Generating Zero-Shot Hard-Case Hallucinations: A Synthetic and Open Data Approach

Abstract

We present a novel framework for designing and inducing controlled hallucinations in long-form content generation by LLMs across diverse domains. The purpose is to create fully-synthetic benchmarks and mine hard cases for iterative refinement of zero-shot hallucination detectors. Using Gretel Data Designer to create long-context datasets, we generate question-answer pairs using chain-of-thought approach, apply consensus labeling to filter synthetic examples, and create an automated system for generating hallucinations. The talk operates under open data licenses like Apache-2.0.