Back office ops · Production

Woowa Brothers builds QueryAnswerBird, an LLM-based AI data analyst using RAG and Text-to-SQL

The problem

More than half of Woowa Brothers employees who wanted to use SQL lacked time to learn it or struggled to generate queries reflecting business logic, and had concerns about the reliability of data extraction.

First attempt

The initial hackathon version using GPT-3.5 had architectural limitations, and GPT-4 alone generated queries lacking company-specific domain knowledge, while LLM hallucination caused inconsistent query quality.

Workflow diagram · grounded in source
1
Employee asks via Slack
trigger
“Employees can easily ask questions and receive answers anytime on the Slack application”
2
Router Supervisor categorizes question
routing
“the Router Supervisor chain identifies the purpose of the question and categorizes it into the appropriate question type in real time”
3
Multi-chain RAG maps to answer chain
ai_action
“The questions are then mapped to multi-chains (e.g. query generation, query interpretation, query syntax validation, table interpretation, log table utilization guide, and column and table utilization guide) that can provide the best pos…”
4
LLM generates SQL via COT reasoning
ai_action
“This prompt goes through a step-by-step reasoning process (COT) to generate the appropriate query for the user's question. Also, it dynamically searches and selects the appropriate data for the question. The response becomes increasingly…”
5
Query syntax validation
validation
“For query generation, the answer also provides an additional description of whether the generated query executes correctly or contains errors by validating the query”
6
Response delivered to user
output
“This feature provides users with responses within 30 seconds to 1 minute, offering high-quality queries that can be referenced for work”
7
User feedback updates GPT cache
feedback_loop
“Answers can be evaluated (satisfied/unsatisfied), which will be reflected in the GPT cache, expanding standardized data knowledge to other users”
Reported outcome

QAB provides employees with high-quality SQL queries and interpretations within 30 seconds to 1 minute, and employees report it helped them greatly in better understanding their work.

Reported metrics
Employees using data for work95%
Query response time30 seconds to 1 minute
A/B tests conductedmore than 500
Employee data understanding improvementhelped them greatly in better understanding their work
Reported stack
LLMRAGLangchainLLMOpsGPT-4oGPT-3.5Microsoft Azure OpenAIVectorDBLangserve PlaygroundSlack
Source
https://deliveryhero.jobs/blog/introducing-the-ai-data-analyst-queryanswerbird-part-1-utilization-of-rag-and-text-to-sql/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

QAB provides employees with high-quality SQL queries and interpretations within 30 seconds to 1 minute, and employees report it helped them greatly in better understanding their work.

What tools did this team use?

LLM, RAG, Langchain, LLMOps, GPT-4o, GPT-3.5, Microsoft Azure OpenAI, VectorDB, Langserve Playground, Slack.

What results were reported?

Employees using data for work: 95%; Query response time: 30 seconds to 1 minute; A/B tests conducted: more than 500; Employee data understanding improvement: helped them greatly in better understanding their work (source-reported, not independently verified).

What failed first in this deployment?

The initial hackathon version using GPT-3.5 had architectural limitations, and GPT-4 alone generated queries lacking company-specific domain knowledge, while LLM hallucination caused inconsistent query quality.

How is this back office ops AI workflow structured?

Employee asks via Slack → Router Supervisor categorizes question → Multi-chain RAG maps to answer chain → LLM generates SQL via COT reasoning → Query syntax validation → Response delivered to user → User feedback updates GPT cache.

Related back office ops cases