optimize-classification

Consider tuning parameter optimization across different tasks. This involves selecting the best layer, setting an appropriate detection threshold and deciding on the most effective way to aggregate tokens for a specific model. To optimize parameters uniformly across

Basic Usage

python -m wisent optimize-classification MODEL [OPTIONS]

Examples

Basic Optimization

python -m wisent optimize-classification \
  meta-llama/Llama-3.1-8B-Instruct \
  --limit 1000 \
  --optimization-metric f1

With Layer Range

python -m wisent optimize-classification \
  meta-llama/Llama-3.1-8B-Instruct \
  --layer-range 10-20 \
  --limit 500 \
  --save-classifiers

Calibration Only

# First, run calibration to estimate timing
python -m wisent optimize-classification \
  meta-llama/Llama-3.1-8B-Instruct \
  --calibrate-only \
  --calibration-file ./calibration.json

Custom Thresholds

python -m wisent optimize-classification \
  meta-llama/Llama-3.1-8B-Instruct \
  --threshold-range 0.3 0.5 0.7 0.9 \
  --aggregation-methods average final max \
  --results-file ./my_results.json

Arguments

Required

Argument	Description
model	Model name or path to optimize

Optimization Settings

Argument	Default	Description
--limit	1000	Maximum samples per task
--optimization-metric	f1	Metric to optimize (f1, accuracy, precision, recall)
--max-time-per-task	15.0	Maximum time per task in minutes
--layer-range	all	Layer range to test (e.g., '10-20')

Search Space

Argument	Default	Description
--aggregation-methods	average, final, first, max, min	Token aggregation methods to test
--threshold-range	0.3-0.9	Detection thresholds to test

Output

Argument	Description
--results-file	Custom file path for saving results
--no-save	Don't save results to model config
--save-logs-json	Save detailed optimization logs to JSON file
--save-classifiers	Save best classifiers for each task (default: true)
--classifiers-dir	Directory to save classifiers

Timing Calibration

Argument	Description
--skip-timing-estimation	Skip timing estimation
--calibration-file	File to save/load calibration data
--calibrate-only	Only run calibration and exit

What Gets Optimized

Layer - Which model layer to extract activations from
Threshold - Classification decision threshold (0-1)
Token Aggregation - How to aggregate token-level scores (average, final, first, max, min)

Related Commands

optimize - Full Optuna-based optimization
optimize-steering - Optimize steering parameters
optimize-sample-size - Find optimal training sample size

Stay in the loop. Never miss out.

Subscribe to our newsletter and unlock Wisent insights.

Contact Careers Privacy Policy Terms of Service