Accounts payable · Production

Amazon Finance Automation builds a generative AI Q&A chat assistant using Amazon Bedrock

The problem

AP and AR analysts in Amazon Finance Operations spent hours to days answering customer queries, requiring time-consuming back-and-forth with subject matter experts and review of multiple policy documents, with new hires especially lacking immediate access to necessary information.

First attempt

The initial RAG-based chat assistant achieved only 49% response accuracy—far below expectations—due to incomplete contexts from fixed-chunk segmentation, LLM hallucinations when no relevant context was retrieved, and responses that were too brief to be useful.

Workflow diagram · grounded in source
1
Customer query submitted
Trigger
AP and AR analysts receive queries from customers through email, cases, internal tools, or phone.
▾ source quote
“Accounts Payable (AP) and Accounts Receivable (AR) analysts in Amazon Finance operations receive queries from customers through email, cases, internal tools, or phone”
2
RAG document retrieval
Ai action
RAG retrieves relevant documents from the knowledge base when a user submits a query.
▾ source quote
“When a user submits a query, RAG works by first retrieving relevant documents from a knowledge base”
3
LLM response generation
Ai action
An LLM generates a response from the retrieved documents.
▾ source quote
“then generating a response with the LLM from the retrieved documents”
4
Validation and PII removal
Validation
The validation engine removes PII and checks whether the generated answer aligns with the retrieved context.
▾ source quote
“Removes PII from the response and checks whether the generated answer aligns with the retrieved context. If not, it returns a hardcoded "I don't know" response to prevent hallucinations”
5
Answer with citations delivered
Output
The chat assistant UI delivers the answer with citations listed as hyperlinks following the LLM response.
▾ source quote
“In the UI, the citations are listed with hyperlinks following the LLM response, and users can use these citations to validate the LLM performance”
Reported outcome

Through iterative improvements in document segmentation, prompt engineering, and embedding models, the team improved RAG accuracy from 49% to 86%, drastically reducing the time required to address customer queries.

Reported metrics
initial RAG response accuracy49%
RAG accuracy after semantic chunking64%
RAG accuracy after prompt engineering76%
final RAG accuracy86%
Show all 14 reported metrics
initial RAG response accuracy49%
RAG accuracy after semantic chunking64%
RAG accuracy after prompt engineering76%
final RAG accuracy86%
initial context relevance score range55–65%
improved context relevance score range75–80%
retrieved contexts with higher ranks after improvement80%
incorrect contexts in top ranks before improvementmore than 50%
NLP score variance vs human evaluationsapproximately 30%
LLM-based evaluation variance vs human analysisapproximately 5%
inaccuracy share from incomplete contexts14%
cases with too-brief LLM responsesapproximately 13%
time to address customer queriesdrastically reduces
original query resolution timehours to days
Reported stack
Amazon BedrockQUILL EditorAmazon Titan Text Embeddings G1bge-base-en-v1.5all-mpnet-base-v2
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://aws.amazon.com/blogs/machine-learning/how-amazon-finance-automation-built-a-generative-ai-qa-chat-assistant-using-amazon-bedrock?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Through iterative improvements in document segmentation, prompt engineering, and embedding models, the team improved RAG accuracy from 49% to 86%, drastically reducing the time required to address customer queries.

What tools did this team use?

Amazon Bedrock, QUILL Editor, Amazon Titan Text Embeddings G1, bge-base-en-v1.5, all-mpnet-base-v2.

What results were reported?

initial RAG response accuracy: 49%; RAG accuracy after semantic chunking: 64%; RAG accuracy after prompt engineering: 76%; final RAG accuracy: 86% (source-reported, not independently verified).

What failed first in this deployment?

The initial RAG-based chat assistant achieved only 49% response accuracy—far below expectations—due to incomplete contexts from fixed-chunk segmentation, LLM hallucinations when no relevant context was retrieved, and…

How is this accounts payable AI workflow structured?

Customer query submitted → RAG document retrieval → LLM response generation → Validation and PII removal → Answer with citations delivered.

Related accounts payable cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Accounts payable cases →