TruthfulQA: definition + examples

TruthfulQA is a benchmark designed to evaluate the truthfulness of large language models (LLMs) by probing their tendency to reproduce common misconceptions and false beliefs. Introduced in a 2022 paper by Stephanie Lin, Jacob Hilton, and Owain Evans, the dataset consists of 817 questions spanning 38 categories, including law, health, finance, and conspiracy theories. Each question is crafted to trigger a false answer that is frequently found online, such as "Is the Earth flat?" or "Do vaccines cause autism?" — statements that are false but widely believed. The benchmark includes both a generation task, where the model produces a free-text answer, and a multiple-choice task, where it selects among options. Evaluation metrics include truthfulness (whether the answer is factually correct) and informativeness (whether it provides useful information beyond a refusal). TruthfulQA is particularly challenging because it tests not just factual recall but also the model's ability to resist mimicking human-generated falsehoods present in its training data. It is used as a standard evaluation in model releases (e.g., GPT-4, Llama 2, Claude 3) to gauge alignment and safety. As of 2026, state-of-the-art models such as GPT-4 Turbo and Claude 3.5 Sonnet achieve around 70-80% truthfulness on the multiple-choice variant, but still struggle with free-form generation, often scoring below 50% on truthfulness without explicit prompting for honesty. Common pitfalls include models overly refusing to answer (which boosts truthfulness at the cost of informativeness) and the benchmark's limited coverage of nuanced factual domains. Alternatives include HaluEval (for hallucination detection) and RealTimeQA (for temporal grounding), but TruthfulQA remains a key diagnostic for model honesty and is often used alongside safety benchmarks like BBQ and Winogender for bias evaluation.