Snorkel AI builds expert-verified benchmark dataset for evaluating AI agents in insurance underwriting
Enterprise AI agents applied to specialized domains like insurance underwriting are often inaccurate and inefficient because AI R&D has focused on easily verifiable settings with plentiful data, leaving specialized domains without quality benchmarks or expert-validated evaluation.
Frontier AI models demonstrated significant failure modes on the insurance underwriting benchmark: tool call errors occurred in 36% of conversations even for top-performing models, and OpenAI models hallucinated insurance products not present in the provided guidelines in 15-45% of product recommendation conversations.
The benchmark revealed a wide range of model accuracies from single digits to approximately 80%, with actionable granular insights into error modes including tool use failures and domain-knowledge hallucinations, enabling targeted model development.
Show all 10 reported metrics
Frequently asked questions
What did this team achieve with this AI workflow?
The benchmark revealed a wide range of model accuracies from single digits to approximately 80%, with actionable granular insights into error modes including tool use failures and domain-knowledge hallucinations, enab…
What tools did this team use?
LangGraph, Model Context Protocol (MCP), ReAct, SQL, o4-mini.
What results were reported?
Top model task accuracy: ~80%; Tool call error rate across all conversations: 36%; Tool call error rate for top 3 accurate models: 30-50%; OpenAI model hallucination rate on product recommendations: 15-45% (source-reported, not independently verified).
What failed first in this deployment?
Frontier AI models demonstrated significant failure modes on the insurance underwriting benchmark: tool call errors occurred in 36% of conversations even for top-performing models, and OpenAI models hallucinated insur…
How is this back office ops AI workflow structured?
Underwriter submits applicant info → AI copilot probes underwriter → Database and guideline access → Chained SQL queries for qualification → Task solution delivery → Evaluation against benchmark criteria.