Ground Truth Evaluator

A Ground Truth Evaluator assesses classifier and detection system effectiveness by aligning their results with known accurate reference values (ground truth data). To measure classifier performance using known correct outcomes, an evaluator compares classifications to actual ground truth information. To measure classifier performance via known right outcomes, evalu

The ground truth evaluator provides multiple methods for determining whether generated responses are truthful or represent hallucinations, serving as the benchmark against which classifier performance is measured. The system supports exact match evaluation that compares generated responses against expected answers using either the lm-eval harness metrics or simple string comparison, though this approach proves problematic for free-form text generation. Substring matching offers a more flexible alternative by checking whether the expected answer appears anywhere within the generated response, but still faces challenges with natural language variation.

For more nuanced evaluation, the interactive method presents each generated response to a human evaluator alongside the expected answer, prompting for manual classification as truthful or hallucination with real-time feedback. The user-specified method allows pre-labeling of responses with ground truth labels that can be provided as input, enabling batch evaluation scenarios. The manual review option marks responses for later human assessment without immediate classification, while the "good" debug mode labels everything as truthful for testing purposes, and the "none" method skips ground truth evaluation entirely when only classifier training is needed.