Back office ops · Production

Woowa Brothers' BADA team builds LLM-based AI data analyst QueryAnswerBird (QAB) with RAG and Text-to-SQL

The problem

More than half of Woowa Brothers employees wanted to use SQL for data extraction but lacked time to learn it or had difficulty generating queries reflecting business logic, and they were concerned about the reliability of data extraction.

First attempt

The initial hackathon prototype using Microsoft Azure OpenAI's GPT-3.5 API was simple and had limitations in achieving systemization, efficiency, accessibility, and automation; generated queries lacked domain knowledge and suffered from LLM hallucination.

Workflow diagram · grounded in source
1
Employee asks question in Slack
trigger
“Employees can easily ask questions and receive answers anytime on the Slack application”
2
Router Supervisor categorizes question
routing
“the Router Supervisor chain identifies the purpose of the question and categorizes it into the appropriate question type in real time”
3
Multi-chain search algorithm retrieval
ai_action
“When executing multi-chains, utilize search algorithms for each chain to enable the retriever to selectively extract the necessary data”
4
Query generation with ReAct and COT
ai_action
“This prompt goes through a step-by-step reasoning process (COT) to generate the appropriate query for the user's question. Also, it dynamically searches and selects the appropriate data for the question. The response becomes increasingly…”
5
Query syntax validation
validation
“the answer also provides an additional description of whether the generated query executes correctly or contains errors by validating the query”
6
Response delivered in Slack
output
“This feature provides users with responses within 30 seconds to 1 minute, offering high-quality queries that can be referenced for work”
7
User feedback and GPT cache update
feedback_loop
“Answers can be evaluated (satisfied/unsatisfied), which will be reflected in the GPT cache, expanding standardized data knowledge to other users”
Reported outcome

QAB provides high-quality query responses within 30 seconds to 1 minute, won first place at Woowa Hackathon 2023, and generated enough employee demand to warrant full-scale development starting January 2024.

Reported metrics
Employees using data for work (survey)about 95%
employees with SQL difficulty (survey)more than half
Query response timewithin 30 seconds to 1 minute
A/B tests conductedmore than 500
Reported stack
LLMRAGLangchainLLMOpsVectorDBLangserve PlaygroundSlack
Source
https://tech.deliveryhero.com/blog/introducing-the-ai-data-analyst-queryanswerbird-part-1-utilization-of-rag-and-text-to-sql/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

QAB provides high-quality query responses within 30 seconds to 1 minute, won first place at Woowa Hackathon 2023, and generated enough employee demand to warrant full-scale development starting January 2024.

What tools did this team use?

LLM, RAG, Langchain, LLMOps, VectorDB, Langserve Playground, Slack.

What results were reported?

Employees using data for work (survey): about 95%; employees with SQL difficulty (survey): more than half; Query response time: within 30 seconds to 1 minute; A/B tests conducted: more than 500 (source-reported, not independently verified).

What failed first in this deployment?

The initial hackathon prototype using Microsoft Azure OpenAI's GPT-3.5 API was simple and had limitations in achieving systemization, efficiency, accessibility, and automation; generated queries lacked domain knowledg…

How is this back office ops AI workflow structured?

Employee asks question in Slack → Router Supervisor categorizes question → Multi-chain search algorithm retrieval → Query generation with ReAct and COT → Query syntax validation → Response delivered in Slack → User feedback and GPT cache update.

Related back office ops cases