Add ModelScope dataset download support

This commit is contained in:
Codex
2026-06-24 23:03:00 +08:00
parent 4063246ac2
commit 65153eb4f6
4 changed files with 188 additions and 43 deletions
+17 -2
View File
@@ -1,6 +1,6 @@
# TI Coding Agent Probe Skill
Use this skill when the user wants to run or modify the TI coding-agent SFT probe experiments based on the Hugging Face dataset `ti_coding_agent_training_probe_20260624`.
Use this skill when the user wants to run or modify the TI coding-agent SFT probe experiments based on the ModelScope dataset `eigentom/ti_coding_agent_training_probe_20260624`. Hugging Face remains available as a fallback backend.
## Repository Contract
@@ -25,9 +25,24 @@ git submodule update --init --recursive
./scripts/setup_env.sh
```
Download the training probe dataset:
Download the training/eval probe dataset from ModelScope:
```bash
export MODELSCOPE_API_TOKEN=<modelscope-api-key>
./scripts/download_dataset.py
```
The downloader writes:
- `data/raw/training_probe/{train,validation}.jsonl.gz`
- `data/raw/training_probe/{train,validation}.parquet`
- `data/processed/training_probe/{train,validation}.jsonl`
- `data/processed/training_probe/{train,validation}.parquet`
Use Hugging Face fallback only when requested:
```bash
export DATASET_BACKEND=huggingface
export HF_ENDPOINT=https://hf-mirror.com
export HF_DATASET_REPO_ID=<owner>/ti_coding_agent_training_probe_20260624
./scripts/download_dataset.py