Quality assurance · Production

Patronus AI trains Lynx hallucination detection model on Databricks Mosaic AI, outperforming GPT-4o

The problem

LLMs used in RAG applications produce hallucinations that expose users to misinformation; existing LLM-as-a-judge evaluators — including top-performing closed-source models like GPT-4 — frequently fail on complex reasoning tasks; and a significant performance gap exists between open-source and closed-source evaluation models due to lack of challenging domain-specific datasets.

First attempt

Even top-performing closed-source models like GPT-4 used as LLM-as-a-judge evaluators frequently fail to accurately evaluate complex reasoning tasks, with additional concerns about quality, transparency, and cost of closed-source LLMs.

Workflow diagram · grounded in source

Dataset construction via perturbation

trigger

“we first constructed our training and evaluation datasets for a hallucination identification task using a perturbation process”

Fine-tuning job configuration

integration

“To create a fine-tuning job on the Databricks Mosaic AI training infrastructure, we create a config similar to the following”

Distributed GPU training run

ai_action

“For supervised finetuning on 70B models, we trained on 32 NVIDIA H100 GPUs, for an effective batch size of 256. To enhance performance, we used native optimizations in Composer, including FSDP and flash attention.”

Real-time training monitoring

validation

“we used the WandB integration with LLM Foundry to log training results to the WandB dashboard. The Mosaic AI Training console makes it easy to monitor run status, including completion status and job history from teammates”

HaluBench evaluation

validation

“Our results on HaluBench show that our finetuned model outperforms closed-source LLMs and open source LLMs when used as judge evaluator LMs across different tasks”

Open-source model release

output

“We are excited to open source Lynx and HaluBench to advance research in RAG evaluations”

Reported outcome

Lynx outperformed all existing LLM-as-a-judge evaluators on HaluBench, surpassing GPT-4o by almost 1% in accuracy across all tasks and showing a 7.5% difference in medical question-answering, and is the best-performing open-source model by a wide margin.

Reported metrics

Lynx accuracy advantage over GPT-4o (all tasks)almost 1%

Lynx accuracy difference in medical question-answering7.5%

Reported stack

LLM FoundryComposerDatabricks Model TrainingDatabricks Mosaic AILlama-3-70B-InstructHuggingFaceWandB

Source

https://www.databricks.com/blog/patronus-ai-lynx

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Lynx outperformed all existing LLM-as-a-judge evaluators on HaluBench, surpassing GPT-4o by almost 1% in accuracy across all tasks and showing a 7.5% difference in medical question-answering, and is the best-performin…

What tools did this team use?

LLM Foundry, Composer, Databricks Model Training, Databricks Mosaic AI, Llama-3-70B-Instruct, HuggingFace, WandB.

What results were reported?

Lynx accuracy advantage over GPT-4o (all tasks): almost 1%; Lynx accuracy difference in medical question-answering: 7.5% (source-reported, not independently verified).

What failed first in this deployment?

How is this quality assurance AI workflow structured?

Dataset construction via perturbation → Fine-tuning job configuration → Distributed GPU training run → Real-time training monitoring → HaluBench evaluation → Open-source model release.