Customer support · Production

From RAG to Fabric: Lessons Learned from Building Real-World RAGs at GenAIIC — Part 1

The problem

AWS customers have high demand for RAG chatbots that can extract information from massive, heterogeneous knowledge bases, but building effective RAG solutions is difficult because retrieval quality is almost always the primary failure point and evaluation requires significant human effort.

First attempt

Two main RAG failure modes are identified from real-world experience: relevant information is not retrieved (causing the FM to hallucinate or use its own knowledge), or relevant information is buried in excessive irrelevant data (causing the FM to mix up sources and produce wrong answers).

Workflow diagram · grounded in source
1
User query submitted
trigger
“Based on a user's question (1), relevant information is retrieved from a knowledge base (2)”
2
Retrieve from knowledge base
ai_action
“relevant information is retrieved from a knowledge base (2) (for example, an OpenSearch index)”
3
Augment FM prompt
ai_action
“The retrieved information is added to the FM prompt (3.a) to augment its knowledge, along with the user query (3.b)”
4
FM generates answer
output
“The FM generates an answer (4) by using the information provided in the prompt”
Reported outcome

(not stated)

Reported metrics
Running sinceMay 2023
Reported stack
OpenSearchOpenSearch ServerlessAmazon Bedrock Knowledge BasesAmazon S3Amazon TitanAmazon BedrockAnthropic's ClaudeAmazon TextractAWS LambdaTextractorCohere EmbedAmazon Titan Text Embeddings
Source
https://aws.amazon.com/blogs/machine-learning/from-rag-to-fabric-lessons-learned-from-building-real-world-rags-at-genaiic-part-1?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

(not stated)

What tools did this team use?

OpenSearch, OpenSearch Serverless, Amazon Bedrock Knowledge Bases, Amazon S3, Amazon Titan, Amazon Bedrock, Anthropic's Claude, Amazon Textract, AWS Lambda, Textractor.

What results were reported?

Running since: May 2023 (source-reported, not independently verified).

What failed first in this deployment?

Two main RAG failure modes are identified from real-world experience: relevant information is not retrieved (causing the FM to hallucinate or use its own knowledge), or relevant information is buried in excessive irre…

How is this customer support AI workflow structured?

User query submitted → Retrieve from knowledge base → Augment FM prompt → FM generates answer.

Related customer support cases