Model evaluation

Wisent Model evaluates the conversations a model holds with a simulated user. Ster executes both the baseline and adapter runs through ster converse, using the same prompt set, user model, turn limit, generation parameters and seed.

Run the actual conversation surface

Submit the evaluation through Stado on the appropriate registered compute target. Within that job, an adapter run uses:

ster converse --model Qwen/Qwen3-8B --adapter ./rl-adapter.safetensors --user-model Qwen/Qwen3-8B --prompts data/rl/prompts.json --turns 20 --max-new-tokens 256 --max-sequence 8192 --temperature 0.9 --seed 7 --precision bf16 --device cuda --output ./trained-conversations.jsonl

Run the baseline with the same settings and without the adapter, retaining its output separately. Record the exact source and model revisions, actual conversations, command statuses and reward measurements. A missing model, failed dependency or rejected run is not a passing comparison.

Interpret the evidence

Measure how long the simulated user continues the conversation and how the trained reward model scores assistant turns. These observations come from the actual model run. Do not replace them with keyword counts, assumed improvements or canned example scores.

wisent-model score --baseline BASELINE_RESULTS --trained TRAINED_RESULTS compares retained results in its supported result format. A comparison of older recorded results does not execute or qualify a new model.

Ownership and storage

Ster owns training and conversation execution; Stado owns the job, compute selection and retained output. Keep private run records in those artifact stores and Oko, rather than publishing workstation paths, provider credentials or incident logs in this guide.

The current repository contract describes preparation, training and evaluation. This page retains its public route without referring to removed evaluation scripts.