Finance ops · Production

NYSE/ICE builds a structured RAG text-to-SQL system on Databricks Mosaic AI with 96% execution accuracy in five weeks

The problem

ICE business users had no way to query structured financial data without understanding data models, SQL schemas, or writing SQL queries, creating a barrier to self-service analytics.

Workflow diagram · grounded in source
1
User submits natural language question
trigger
“When a question is submitted, an embedding vector is created and matched against the vector indexes of the table_definitions and sample_queries tables”
2
Vector search retrieves schema and query context
ai_action
“The retrieved context is used to augment the input question, creating a prompt that provides the LLM with a rich understanding of the relevant tables, data, and queries”
3
LLM generates SQL from augmented prompt
ai_action
“The augmented prompt is sent to an LLM of choice, e.g., Llama3.1-70B, via the Foundation Model APIs. The LLM generates a response based on the context provided, from which we utilized regex to extract the SQL statement”
4
SQL evaluated against ground truth
validation
“this dual-pronged evaluation strategy of involving both syntactic and execution matches allowed us to robustly and deterministically assess our text-to-SQL system's performance”
5
SMEs label incorrect queries in Label Studio
human_review
“imported them into an open-source labeling tool called Label Studio. Subject matter experts can add ground-truth SQL statements on the Studio”
6
Incorrect queries used for few-shot learning
feedback_loop
“Incorrect queries can be detected and logged into the sample_queries table. This process allows for a continuous loop that identifies the incorrect SQL queries and then uses those queries for few-shot learning, enabling the model to lear…”
Reported outcome

In five weeks, the team built a text-to-SQL system achieving 77% syntactic accuracy and 96% execution matches, enabling non-technical business users to query structured financial data in natural language.

Reported metrics
Syntactic accuracy77%
Execution match accuracy96%
System development timefive weeks
Reported stack
Unity CatalogVector SearchFoundation Model APIsModel ServingInference TablesAgent Bricks AI GatewayLlama3.1-70BMLflowGitHubSpiderLabel Studio
Source
https://www.databricks.com/blog/unlocking-financial-insights-nyse-ice
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

In five weeks, the team built a text-to-SQL system achieving 77% syntactic accuracy and 96% execution matches, enabling non-technical business users to query structured financial data in natural language.

What tools did this team use?

Unity Catalog, Vector Search, Foundation Model APIs, Model Serving, Inference Tables, Agent Bricks AI Gateway, Llama3.1-70B, MLflow, GitHub, Spider.

What results were reported?

Syntactic accuracy: 77%; Execution match accuracy: 96%; System development time: five weeks (source-reported, not independently verified).

How is this finance ops AI workflow structured?

User submits natural language question → Vector search retrieves schema and query context → LLM generates SQL from augmented prompt → SQL evaluated against ground truth → SMEs label incorrect queries in Label Studio → Incorrect queries used for few-shot learning.

Related finance ops cases