We present a novel framework for designing and inducing controlled hallucinations in long-form content generation by LLMs across diverse domains. The purpose is to create fully-synthetic benchmarks and mine hard cases for iterative refinement of zero-shot hallucination detectors. Using Gretel Data Designer to create long-context datasets, we generate question-answer pairs using chain-of-thought approach, apply consensus labeling to filter synthetic examples, and create an automated system for generating hallucinations. The talk operates under open data licenses like Apache-2.0.
Type: Lightning Talk (20 minutes)
Track: Artificial Intelligence
Skill Level: Intermediate
Date: Tuesday, June 10, 2025
Time: 5:20 pm - 5:40 pm
Location: In Person, San Francisco
Technologies: AI/BI, Llama, PyTorch
This lightning talk presents a novel framework for designing and inducing controlled hallucinations in long-form content generation by Large Language Models (LLMs) across diverse domains. The primary purpose is to create fully-synthetic benchmarks and mine hard cases for iterative refinement of zero-shot hallucination detectors.
The Databricks Data + AI Summit is the world’s largest data and AI conference, bringing together thousands of data practitioners, engineers, and leaders to share the latest innovations in data and AI.