Customer support · Production

Instacart builds LACE, an LLM-based automated evaluation framework for its customer support chatbot

The problem

Instacart needed a reliable, scalable way to evaluate whether its AI-powered customer support chatbot was actually helping customers in real conversations, because human evaluation alone could not scale and the chatbot's quality was difficult to measure objectively.

Workflow diagram · grounded in source

Chat session submitted to LACE

trigger

“We evaluate each chat session — defined as a full, multi-turn conversation between a customer and the support chatbot — against the set of binary (True/False) criteria across five key dimensions”

LLM scores chat against criteria

ai_action

“Direct Prompting — The LLM scores the chat based on a single-pass prompt using predefined criteria”

Agentic debate evaluation

ai_action

“Agentic Evaluation via Debate creates a virtual courtroom where three distinct agents — each simulated by an LLM — evaluate the chatbot's performance from different perspectives. The evaluation is conducted in three sequential steps: A C…”

Human-LLM alignment validation

human_review

“human evaluators rated a carefully selected set of customer conversations using the same criteria as our LACE system. We then compared their ratings to those generated by LACE”

Criteria and prompt refinement

feedback_loop

“When we identified misalignments, we used this feedback to refine our evaluation framework in two ways: Refining existing criteria by improving the definitions and prompts — this was our primary mechanism for improving alignment and was …”

Dashboard monitoring and improvement

output

“LACE feeds into dashboards that let us: Monitor performance trends over time, Analyze specific interaction details to pinpoint issues, Integrate feedback directly into experimentation platforms for real-time improvements”

Reported outcome

LACE provides automated evaluation closely aligned with human judgment, achieving over 90% accuracy on context-dependent criteria and enabling continuous chatbot improvement through dashboard-driven feedback loops that reduced inefficient interactions.

Reported metrics

Accuracy on context-dependent evaluation criteriaover 90%

Accuracy on simple evaluation criterianear-perfect accuracy

Inefficient chatbot interactionsreduced inefficient interactions

Reported stack

o1-preview

Source

https://tech.instacart.com/turbocharging-customer-support-chatbot-development-with-llm-based-automated-evaluation-6a269aae56b2

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LACE provides automated evaluation closely aligned with human judgment, achieving over 90% accuracy on context-dependent criteria and enabling continuous chatbot improvement through dashboard-driven feedback loops tha…

What tools did this team use?

o1-preview.

What results were reported?

Accuracy on context-dependent evaluation criteria: over 90%; Accuracy on simple evaluation criteria: near-perfect accuracy; Inefficient chatbot interactions: reduced inefficient interactions (source-reported, not independently verified).

How is this customer support AI workflow structured?

Chat session submitted to LACE → LLM scores chat against criteria → Agentic debate evaluation → Human-LLM alignment validation → Criteria and prompt refinement → Dashboard monitoring and improvement.