Workflow · Production

Query multiple documents using LlamaIndex, LangChain, and Milvus

The problem

LLMs are useful for personal projects but the challenge is deploying them in production for multi-document querying — querying a single document is insufficient, and without query decomposition, LLMs cannot answer questions that require information from multiple sources.

First attempt

Without decomposable queries, the LLM application returns a response indicating insufficient context to answer a question that spans multiple documents.

Workflow diagram · grounded in source
1
Multi-document query submitted
trigger
“Now we are ready to query over all of these documents. In this example, we ask our LLM app to compare and contrast the Seattle, Houston, and Toronto airports.”
2
Query decomposition
ai_action
“First, the transformer breaks the question into simple queries that a single data source can answer.”
3
Keyword routing to data source
routing
“Then, it uses the keyword index to route the simple queries to the right data source”
4
Vector store semantic retrieval
ai_action
“the vector store index to answer the question”
5
Response synthesis
output
“Finally, the question transformer combines the information and answers our original, complex query.”
Reported outcome

Decomposable queries successfully answer complex multi-document questions by breaking them into simpler targeted sub-queries, routing each to the right data source via a keyword index, and combining results with a vector store index.

Reported stack
LlamaIndexLangChainMilvusOpenAI
Source
https://mlops.community/blog/combine-and-query-multiple-documents-with-llm
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Decomposable queries successfully answer complex multi-document questions by breaking them into simpler targeted sub-queries, routing each to the right data source via a keyword index, and combining results with a vec…

What tools did this team use?

LlamaIndex, LangChain, Milvus, OpenAI.

What failed first in this deployment?

Without decomposable queries, the LLM application returns a response indicating insufficient context to answer a question that spans multiple documents.

How is this workflow AI workflow structured?

Multi-document query submitted → Query decomposition → Keyword routing to data source → Vector store semantic retrieval → Response synthesis.