yi_lu/OpenSWETraces_cleanup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Parallelize profiling by Parquet shard

Browse Source
This commit is contained in:
jiachun
2026-08-18 18:22:43 +08:00
parent 4a062134d3
commit fea85047f1
5 changed files with 234 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
changelog.md
+13
View File
@@ -4,6 +4,19 @@ This file records strategy changes that materially affect dataset decisions or
training-data semantics. Generated audit manifests are not treated as stable API
contracts.
## 3.0.1 - 2026-08-18
### CPU profiling
- Replaced directory-level Python thread concurrency with Parquet shard-level
processes so CPU-bound tokenization and metric extraction bypass the GIL.
- Each process streams batches of 16 records to an atomic shard part; the parent
merges completed parts in source order without loading them into memory.
- `profile --resume` now reuses completed shard parts, while interrupted
temporary parts are safely recomputed.
- Review sampling uses stored Parquet provenance and reads only shards that
contain selected trajectories.
## 3.0.0 - 2026-08-18
### Strategy replacement