Accounts payable · Production

Amazon Finance Automation builds a generative AI Q&A chat assistant using Amazon Bedrock

The problem

AP and AR analysts in Amazon Finance Operations spent hours to days answering customer queries, requiring time-consuming back-and-forth with subject matter experts and review of multiple policy documents, with new hires especially lacking immediate access to necessary information.

First attempt

The initial RAG-based chat assistant achieved only 49% response accuracy—far below expectations—due to incomplete contexts from fixed-chunk segmentation, LLM hallucinations when no relevant context was retrieved, and responses that were too brief to be useful.

Workflow diagram · grounded in source
1
Customer query submitted
trigger
“Accounts Payable (AP) and Accounts Receivable (AR) analysts in Amazon Finance operations receive queries from customers through email, cases, internal tools, or phone”
2
RAG document retrieval
ai_action
“When a user submits a query, RAG works by first retrieving relevant documents from a knowledge base”
3
LLM response generation
ai_action
“then generating a response with the LLM from the retrieved documents”
4
Validation and PII removal
validation
“Removes PII from the response and checks whether the generated answer aligns with the retrieved context. If not, it returns a hardcoded "I don't know" response to prevent hallucinations”
5
Answer with citations delivered
output
“In the UI, the citations are listed with hyperlinks following the LLM response, and users can use these citations to validate the LLM performance”
Reported outcome

Through iterative improvements in document segmentation, prompt engineering, and embedding models, the team improved RAG accuracy from 49% to 86%, drastically reducing the time required to address customer queries.

Reported metrics
initial RAG response accuracy49%
RAG accuracy after semantic chunking64%
RAG accuracy after prompt engineering76%
final RAG accuracy86%
Show all 14 reported metrics
initial RAG response accuracy49%
RAG accuracy after semantic chunking64%
RAG accuracy after prompt engineering76%
final RAG accuracy86%
initial context relevance score range55–65%
improved context relevance score range75–80%
retrieved contexts with higher ranks after improvement80%
incorrect contexts in top ranks before improvementmore than 50%
NLP score variance vs human evaluationsapproximately 30%
LLM-based evaluation variance vs human analysisapproximately 5%
inaccuracy share from incomplete contexts14%
cases with too-brief LLM responsesapproximately 13%
time to address customer queriesdrastically reduces
original query resolution timehours to days
Reported stack
Amazon BedrockQUILL EditorAmazon Titan Text Embeddings G1bge-base-en-v1.5all-mpnet-base-v2
Source
https://aws.amazon.com/blogs/machine-learning/how-amazon-finance-automation-built-a-generative-ai-qa-chat-assistant-using-amazon-bedrock?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Through iterative improvements in document segmentation, prompt engineering, and embedding models, the team improved RAG accuracy from 49% to 86%, drastically reducing the time required to address customer queries.

What tools did this team use?

Amazon Bedrock, QUILL Editor, Amazon Titan Text Embeddings G1, bge-base-en-v1.5, all-mpnet-base-v2.

What results were reported?

initial RAG response accuracy: 49%; RAG accuracy after semantic chunking: 64%; RAG accuracy after prompt engineering: 76%; final RAG accuracy: 86% (source-reported, not independently verified).

What failed first in this deployment?

The initial RAG-based chat assistant achieved only 49% response accuracy—far below expectations—due to incomplete contexts from fixed-chunk segmentation, LLM hallucinations when no relevant context was retrieved, and…

How is this accounts payable AI workflow structured?

Customer query submitted → RAG document retrieval → LLM response generation → Validation and PII removal → Answer with citations delivered.

Related accounts payable cases