Run the AxBench concept steering and detection benchmark against a subject model. Reproduces the published AxBench protocol (a steering-factor sweep over Alpaca-Eval instructions with judge-scored ratings) and reports either steering quality or concept-detection metrics.
python -m wisent axbench --model MODEL --axbench-action {steer,detect} --output FILE [OPTIONS]python -m wisent axbench --model google/gemma-2-2b --axbench-action steer --concept-id 42 --judge-model openai:gpt-4o-mini --output ./axbench_steer.json
python -m wisent axbench --model google/gemma-2-2b --axbench-action detect --all-concepts --max-concepts 50 --output ./axbench_detect.json
| Argument | Description |
|---|---|
| --model | Subject model to steer/probe (HuggingFace id) |
| --axbench-action | steer: judge-scored steering protocol; detect: AUROC/F1 concept detection |
| --output | Output JSON file for the summary |
| Argument | Default | Description |
|---|---|---|
| --variant | concept500 | AxBench dataset variant (concept500, concept16k, concept16k_v2) |
| --concept-set | 2b/l10 | Concept set by GemmaScope subject model/layer |
| --concept-id | (none) | Run a single concept by id |
| --all-concepts | off | Run every concept in the variant |
| --max-concepts | all | Cap the concept count with --all-concepts |
| --layer | midpoint | 1-indexed layer for extraction/steering |
| Argument | Default | Description |
|---|---|---|
| --method | caa | Wisent steering method (caa = AxBench DiffMean baseline) |
| --factors | 14 reference | Comma-separated steering factors |
| --judge-model | (none) | Judge for steer: openai:gpt-4o-mini or a local HF model id |
| --judge-batch-size | 16 | Concurrent judge requests / local judge batch |
Additional protocol knobs (--n-instructions, --seed, --max-new-tokens, --temperature, --use-hard-negatives, --pair-limit, --judge-max-new-tokens, --judge-temperature, --imbalanced-negatives, --work-dir, --device) default to the AxBench reference values; run wisent axbench --help for the full list.
Stay in the loop. Never miss out.
Subscribe to our newsletter and unlock Wisent insights.