Add ModelScope dataset download support
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# TI Coding Agent Probe Skill
|
||||
|
||||
Use this skill when the user wants to run or modify the TI coding-agent SFT probe experiments based on the Hugging Face dataset `ti_coding_agent_training_probe_20260624`.
|
||||
Use this skill when the user wants to run or modify the TI coding-agent SFT probe experiments based on the ModelScope dataset `eigentom/ti_coding_agent_training_probe_20260624`. Hugging Face remains available as a fallback backend.
|
||||
|
||||
## Repository Contract
|
||||
|
||||
@@ -25,9 +25,24 @@ git submodule update --init --recursive
|
||||
./scripts/setup_env.sh
|
||||
```
|
||||
|
||||
Download the training probe dataset:
|
||||
Download the training/eval probe dataset from ModelScope:
|
||||
|
||||
```bash
|
||||
export MODELSCOPE_API_TOKEN=<modelscope-api-key>
|
||||
./scripts/download_dataset.py
|
||||
```
|
||||
|
||||
The downloader writes:
|
||||
|
||||
- `data/raw/training_probe/{train,validation}.jsonl.gz`
|
||||
- `data/raw/training_probe/{train,validation}.parquet`
|
||||
- `data/processed/training_probe/{train,validation}.jsonl`
|
||||
- `data/processed/training_probe/{train,validation}.parquet`
|
||||
|
||||
Use Hugging Face fallback only when requested:
|
||||
|
||||
```bash
|
||||
export DATASET_BACKEND=huggingface
|
||||
export HF_ENDPOINT=https://hf-mirror.com
|
||||
export HF_DATASET_REPO_ID=<owner>/ti_coding_agent_training_probe_20260624
|
||||
./scripts/download_dataset.py
|
||||
|
||||
Reference in New Issue
Block a user