Migrate stored activation data from the Supabase database to the wisent-ai/activations HuggingFace dataset. Organized as sub-actions for creating the repo, migrating in bulk or per combo, building the index, verifying and cleanup.
python -m wisent migrate-activations ACTION [OPTIONS]
python -m wisent migrate-activations create-repo
python -m wisent migrate-activations single --model meta-llama/Llama-3.1-8B-Instruct --benchmark truthfulqa_mc1 --strategy chat_last
python -m wisent migrate-activations all --combo-start 0 --combo-end 100
| Argument | Description |
|---|---|
| create-repo | Create the wisent-ai/activations HF dataset repo |
| all | Migrate all activations from Supabase to HF |
| consolidate | Build a unified index.json from marker files |
| single | Migrate a single model/benchmark/strategy combo |
| pair-texts | Migrate pair texts for a benchmark |
| raw | Migrate raw activations (chunked) |
| verify-completeness | Check all Supabase combos exist in the HF index |
| cleanup | Drop Supabase rows for combos already in HF |
| verify | Verify HF data matches Supabase |
| Argument | Default | Description |
|---|---|---|
| --database-url | DATABASE_URL env | Database URL |
| --model / --benchmark / --strategy | required | Combo selectors for single, raw and verify |
Upload sub-actions accept a shared set of HuggingFace retry flags (--hf-max-retries, --hf-base-wait, --hf-backoff-max-exp, --hf-jitter-min, --hf-jitter-max, --hf-retryable-pattern) and a preview flag that reports what would be uploaded without writing; run wisent migrate-activations <action> --help for the per-action arguments.
Stay in the loop. Never miss out.
Subscribe to our newsletter and unlock Wisent insights.