Compliance monitoring · Production

Snorkel AI benchmark evaluates frontier model AI agents for insurance underwriting across task types and error modes

The problem

AI agents in enterprise settings are often inaccurate and inefficient because they are not tuned to the critical details of enterprise problems, while AI research has focused on generic use cases that do not translate to enterprise settings.

First attempt

Frontier models made tool call errors in 36% of conversations despite having the metadata needed to use tools correctly, and top OpenAI models hallucinated insurance products not in the provided guidelines 15-45% of the time, with hallucinations also producing misleading questions to the underwriter.

Workflow diagram · grounded in source
1
Underwriter submits application info
trigger
“The data capture interactions in which a junior underwriter has information about the applicant that is occasionally incomplete, with tasks that they need help with from our AI copilot.”
2
AI copilot probes underwriter
ai_action
“We gave underwriters limited information about the applicants, challenging the AI assistant to ask the right questions to solve the task.”
3
Chain of SQL queries
ai_action
“they had to enter a chain of SQL queries to determine the correct criteria and thresholds, interacting with underwriters to obtain the information they needed in the process”
4
Free-text guideline consultation
ai_action
“Read the free-text underwriting guidelines.”
5
Underwriting recommendation output
output
“We developed six basic types of tasks: Whether the type of insurance, or "line of business," being applied for was "in appetite" ... What types of deductibles the underwriter should offer the applicant if they are in appetite.”
Reported outcome

The benchmark revealed a wide accuracy range from the single digits up to approximately 80% across frontier models, with even the three most accurate models making tool call errors in 30-50% of conversations, illustrating that top frontier models struggle in surprising ways with proprietary enterprise knowledge.

Reported metrics
Task accuracy range across frontier modelsfrom the single digits up to ~80%
Tool call error rate across all models36%
Tool call error rate for top three accurate models30-50% of the conversations
product recommendation hallucination rate (top OpenAI models)15-45% of the time
Show all 11 reported metrics
task accuracy range across frontier modelsfrom the single digits up to ~80%
tool call error rate across all models36%
tool call error rate for top three accurate models30-50% of the conversations
product recommendation hallucination rate (top OpenAI models)15-45% of the time
deductibles task accuracy (average across models)0.784
business classification task accuracy (average across models)0.772
policy limits task accuracy (average across models)0.762
appetite check task accuracy (average across models)0.615
product recommendations task accuracy (average across models)0.377
accuracy reduction on auto policy deductibles25 points less accurate on average
model turn count vs task accuracy (one model example)7 turns with the underwriter, task accuracy score of about 55%
Reported stack
LangGraphModel Context Protocol (MCP)ReActSnorkel's evaluation suite
Source
https://snorkel.ai/blog/evaluating-ai-agents-for-insurance-underwriting/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The benchmark revealed a wide accuracy range from the single digits up to approximately 80% across frontier models, with even the three most accurate models making tool call errors in 30-50% of conversations, illustra…

What tools did this team use?

LangGraph, Model Context Protocol (MCP), ReAct, Snorkel's evaluation suite.

What results were reported?

Task accuracy range across frontier models: from the single digits up to ~80%; Tool call error rate across all models: 36%; Tool call error rate for top three accurate models: 30-50% of the conversations; product recommendation hallucination rate (top OpenAI models): 15-45% of the time (source-reported, not independently verified).

What failed first in this deployment?

Frontier models made tool call errors in 36% of conversations despite having the metadata needed to use tools correctly, and top OpenAI models hallucinated insurance products not in the provided guidelines 15-45% of t…

How is this compliance monitoring AI workflow structured?

Underwriter submits application info → AI copilot probes underwriter → Chain of SQL queries → Free-text guideline consultation → Underwriting recommendation output.

Related compliance monitoring cases