generate-pairs-from-task

Generate contrastive pairs from lm-eval benchmark tasks. This command extracts question-answer pairs where one answer is correct (positive) and one is incorrect (negative), creating training data for steering vectors.

Basic Usage

python -m wisent generate-pairs-from-task TASK_NAME --output FILE [OPTIONS]

Examples

Generate from TruthfulQA

python -m wisent generate-pairs-from-task truthfulqa_mc1 \
  --output ./pairs/truthfulqa.json \
  --limit 100

Generate from HellaSwag

python -m wisent generate-pairs-from-task hellaswag \
  --output ./pairs/hellaswag.json \
  --seed 123 \
  --verbose

Arguments

Supported Tasks

Any task from the lm-evaluation-harness that has multiple choice answers can be used. Common tasks include:

• truthfulqa_mc1 - TruthfulQA multiple choice (single answer)
• truthfulqa_mc2 - TruthfulQA multiple choice (multiple answers)
• hellaswag - Commonsense reasoning
• arc_easy - ARC Easy science questions
• arc_challenge - ARC Challenge science questions
• mmlu - Massive Multitask Language Understanding
• winogrande - Winograd schema challenge

Related Commands

• generate-pairs - Generate pairs from custom data
• get-activations - Extract activations from pairs
• generate-vector-from-task - End-to-end pipeline