Back office ops · Production

Hermes V3: Building Swiggy's Conversational AI Analyst

The problem

The original Hermes text-to-SQL tool struggled with niche metrics, required users to repeat context in every prompt, generated inconsistent SQL output, and provided no way to debug or trust the results.

First attempt

The previous pipeline achieved only 54% accuracy with 20% of queries fully incorrect, and relied on generic embeddings that frequently failed to resolve inconsistent or obscure column names.

Workflow diagram · grounded in source
1
Employee asks question in Slack
trigger
“Designed as a lightweight interface embedded in Slack, it eliminated the need for analysts to write repetitive queries and empowered non-technical teams to pull data themselves.”
2
Vector similarity few-shot retrieval
ai_action
“Searches for similar prompts using vector similarity. Injects the top results as few-shot examples. Uses these to guide the LLM to generate more accurate SQL.”
3
Orchestrator agent reasoning
ai_action
“Each of these tools is accessible to the orchestrator, which decides what to call and when, using a ReAct-style reasoning loop.”
4
SQL generation
ai_action
“Generate intermediate logic before producing final SQL”
5
Explanation layer output
output
“Breaks down assumptions Hermes made (e.g., interpreting "LD orders" as lm_distance > 4) Details the logical steps used to build the query Assigns a confidence score from 1 (low) to 3 (high)”
6
Weekly feedback and quality control
feedback_loop
“We collect weekly feedback via Slack (Was this correct? Yes/No) and run root cause analyses for every "No". Fixes are rolled out proactively across all similar metrics in that charter.”
Reported outcome

Hermes V3 improved SQL query accuracy from 54% to 93% on a benchmark of approximately 100 manually tagged queries, reduced fully incorrect queries from 20% to 7%, achieved near-zero table-not-found errors, and became the backbone for all internal AI co-pilots at Swiggy.

Reported metrics
SQL query accuracy (new pipeline)93%
SQL query accuracy (old pipeline)54%
Fully incorrect queries (old pipeline)20%
Fully incorrect queries (new pipeline)7%
Show all 7 reported metrics
SQL query accuracy (new pipeline)93%
SQL query accuracy (old pipeline)54%
Fully incorrect queries (old pipeline)20%
Fully incorrect queries (new pipeline)7%
Query accuracy increase on ambiguous prompts20–25%
Table not found errorsNear-zero
Trust ratingsHigher trust ratings
Reported stack
ClaudeSnowflakeSlackvector DBAWS
Source
https://bytes.swiggy.com/hermes-v3-building-swiggys-conversational-ai-analyst-a41057a2279d
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Hermes V3 improved SQL query accuracy from 54% to 93% on a benchmark of approximately 100 manually tagged queries, reduced fully incorrect queries from 20% to 7%, achieved near-zero table-not-found errors, and became…

What tools did this team use?

Claude, Snowflake, Slack, vector DB, AWS.

What results were reported?

SQL query accuracy (new pipeline): 93%; SQL query accuracy (old pipeline): 54%; Fully incorrect queries (old pipeline): 20%; Fully incorrect queries (new pipeline): 7% (source-reported, not independently verified).

What failed first in this deployment?

The previous pipeline achieved only 54% accuracy with 20% of queries fully incorrect, and relied on generic embeddings that frequently failed to resolve inconsistent or obscure column names.

How is this back office ops AI workflow structured?

Employee asks question in Slack → Vector similarity few-shot retrieval → Orchestrator agent reasoning → SQL generation → Explanation layer output → Weekly feedback and quality control.

Related back office ops cases