Compliance monitoring · Production

Snorkel AI benchmark evaluates frontier model AI agents for insurance underwriting across task types and error modes

The problem

AI agents in enterprise settings are often inaccurate and inefficient because they are not tuned to the critical details of enterprise problems, while AI research has focused on generic use cases that do not translate to enterprise settings.

First attempt

Frontier models made tool call errors in 36% of conversations despite having the metadata needed to use tools correctly, and top OpenAI models hallucinated insurance products not in the provided guidelines 15-45% of the time, with hallucinations also producing misleading questions to the underwriter.

Workflow diagram · grounded in source
1
Underwriter submits application info
Trigger
A junior underwriter provides occasionally incomplete information about an insurance applicant to the AI copilot.
▾ source quote
“The data capture interactions in which a junior underwriter has information about the applicant that is occasionally incomplete, with tasks that they need help with from our AI copilot.”
2
AI copilot probes underwriter
Ai action
The AI assistant asks the underwriter the right questions to gather the information needed to solve the task.
▾ source quote
“We gave underwriters limited information about the applicants, challenging the AI assistant to ask the right questions to solve the task.”
3
Chain of SQL queries
Ai action
The AI copilot enters a chain of SQL queries to determine the correct criteria and thresholds for the underwriting decision.
▾ source quote
“they had to enter a chain of SQL queries to determine the correct criteria and thresholds, interacting with underwriters to obtain the information they needed in the process”
4
Free-text guideline consultation
Ai action
The AI copilot reads free-text underwriting guidelines to inform classification and coverage decisions.
▾ source quote
“Read the free-text underwriting guidelines.”
5
Underwriting recommendation output
Output
The AI copilot produces an underwriting recommendation covering appetite, classification, limits, deductibles, or product offerings.
▾ source quote
“We developed six basic types of tasks: Whether the type of insurance, or "line of business," being applied for was "in appetite" ... What types of deductibles the underwriter should offer the applicant if they are in appetite.”
Reported outcome

The benchmark revealed a wide accuracy range from the single digits up to approximately 80% across frontier models, with even the three most accurate models making tool call errors in 30-50% of conversations, illustrating that top frontier models struggle in surprising ways with proprietary enterprise knowledge.

Reported metrics
Task accuracy range across frontier modelsfrom the single digits up to ~80%
Tool call error rate across all models36%
Tool call error rate for top three accurate models30-50% of the conversations
product recommendation hallucination rate (top OpenAI models)15-45% of the time
Show all 11 reported metrics
task accuracy range across frontier modelsfrom the single digits up to ~80%
tool call error rate across all models36%
tool call error rate for top three accurate models30-50% of the conversations
product recommendation hallucination rate (top OpenAI models)15-45% of the time
deductibles task accuracy (average across models)0.784
business classification task accuracy (average across models)0.772
policy limits task accuracy (average across models)0.762
appetite check task accuracy (average across models)0.615
product recommendations task accuracy (average across models)0.377
accuracy reduction on auto policy deductibles25 points less accurate on average
model turn count vs task accuracy (one model example)7 turns with the underwriter, task accuracy score of about 55%
Reported stack
LangGraphModel Context Protocol (MCP)ReActSnorkel's evaluation suite
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://snorkel.ai/blog/evaluating-ai-agents-for-insurance-underwriting/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The benchmark revealed a wide accuracy range from the single digits up to approximately 80% across frontier models, with even the three most accurate models making tool call errors in 30-50% of conversations, illustra…

What tools did this team use?

LangGraph, Model Context Protocol (MCP), ReAct, Snorkel's evaluation suite.

What results were reported?

Task accuracy range across frontier models: from the single digits up to ~80%; Tool call error rate across all models: 36%; Tool call error rate for top three accurate models: 30-50% of the conversations; product recommendation hallucination rate (top OpenAI models): 15-45% of the time (source-reported, not independently verified).

What failed first in this deployment?

Frontier models made tool call errors in 36% of conversations despite having the metadata needed to use tools correctly, and top OpenAI models hallucinated insurance products not in the provided guidelines 15-45% of t…

How is this compliance monitoring AI workflow structured?

Underwriter submits application info → AI copilot probes underwriter → Chain of SQL queries → Free-text guideline consultation → Underwriting recommendation output.

Related compliance monitoring cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Compliance monitoring cases →