Parallelize profiling by Parquet shard

README.md

@@ -103,7 +103,7 @@ swe-qc profile \
   --output qc_outputs/deterministic_v3/metrics.jsonl \
   --errors qc_outputs/deterministic_v3/profile.errors.jsonl \
   --tokenizer-json "$TOKENIZER_JSON" \
-  --workers 8 \
+  --workers 24 \
   --resume
 
 swe-qc summarize \
@@ -119,12 +119,15 @@ swe-qc sample \
   --seed 20260818
 ```
 
-`profile --resume` skips sample IDs already present in the metrics file. Writes
-are append-only and flushed per record. `summarize` and `sample` atomically
-replace their outputs so a partial file is never mistaken for a complete run.
-Any dataset subdirectory containing Parquet shards is also a valid `--input`,
-so the four trajectory families can be profiled in separate CPU processes and
-their metrics concatenated before `summarize`.
+For a Parquet directory and `--workers > 1`, each worker is a separate process
+that reads one shard at a time and writes an atomic shard part. This bypasses
+the Python GIL, bounds live trajectory memory to roughly one small Parquet batch
+per process, and lets `profile --resume` reuse completed shard parts after an
+interruption. The final metrics and error manifests are concatenated atomically
+in stable shard order. For JSONL or a limited pilot, `--resume` instead skips
+sample IDs already present in the append-only output. `summarize` and `sample`
+also replace their outputs atomically.
+
 Profile rows and decisions record the absolute source Parquet shard. `sample`
 uses that provenance to read only shards containing selected IDs instead of
 scanning the complete dataset.