Customer support · Production

NewDay builds a generative AI-based customer service agent assist with over 90% accuracy

The problem

NewDay's contact center handles 2.5 million calls annually, and with nearly 200 knowledge articles agents had to manually search during live customer calls, slowing resolution times for both agents and customers.

First attempt

An initial full voice-assistant concept was abandoned as too ambitious, and an early text-parsing approach using PyPDF performed at only around 60% accuracy because it did not account for the widget-based structure of NewDay's knowledge articles.

Workflow diagram · grounded in source
1
Agent submits question via UI
trigger
“Users can log in, ask questions, give feedback to answers in the form of thumbs up and thumbs down”
2
Knowledge base ingestion and embedding
integration
“articles are retrieved by using APIs from the third-party knowledge base and chunked with a defined chunking strategy. The chunks are processed to convert to vector embeddings and finally stored in the vector database implemented using O…”
3
Relevant chunk retrieval
ai_action
“retrieves the most relevant chunks and passess these chunks to the large language model (LLM) for generating suggestions based on the context”
4
LLM generates answer suggestion
ai_action
“Anthropic's Claude 3 Haiku was the preferred LLM and was accessed through Amazon Bedrock”
5
Agent feedback capture
feedback_loop
“give feedback to answers in the form of thumbs up and thumbs down, and optionally provide a comment to explain the reason for the bad feedback”
6
Weekly expert review and improvement
feedback_loop
“Every week, business experts review the answers with bad feedback, and AI engineers translate them into experiments that, if successful, increase the solution's performance”
7
Pre-production accuracy validation
validation
“When a new version of NewAssist is created during the experimentation cycles, it is first evaluated in pre-production against an evaluation dataset. If the version's accuracy surpasses a specified threshold, then it can be deployed in pr…”
Reported outcome

NewAssist achieved over 90% accuracy for agent question answering and reduced answer retrieval time from an average of 90 seconds to 4 seconds; the solution is currently rolling out to over 150 agents across Customer Operations.

Reported metrics
Answer accuracyover 90%
Previous answer retrieval time90 seconds
New answer retrieval time4 seconds
Monthly running costunder $400 per month
Show all 11 reported metrics
answer accuracyover 90%
previous answer retrieval time90 seconds
new answer retrieval time4 seconds
monthly running costunder $400 per month
accuracy improvement in initial experimentation phase33%
accuracy increase attributed to knowledge base processing40%
accuracy after custom data parser implementationfrom 60% to 73%
initial accuracy with PyPDF parsingaround 60%
agents in current rolloutover 150 agents
annual contact center call volume2.5 million calls annually
knowledge articles in Customer Servicesnearly 200
Reported stack
Snowflake
Source
https://aws.amazon.com/blogs/machine-learning/newday-builds-a-generative-ai-based-customer-service-agent-assist-with-over-90-accuracy?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

NewAssist achieved over 90% accuracy for agent question answering and reduced answer retrieval time from an average of 90 seconds to 4 seconds; the solution is currently rolling out to over 150 agents across Customer…

What tools did this team use?

Snowflake.

What results were reported?

Answer accuracy: over 90%; Previous answer retrieval time: 90 seconds; New answer retrieval time: 4 seconds; Monthly running cost: under $400 per month (source-reported, not independently verified).

What failed first in this deployment?

An initial full voice-assistant concept was abandoned as too ambitious, and an early text-parsing approach using PyPDF performed at only around 60% accuracy because it did not account for the widget-based structure of…

How is this customer support AI workflow structured?

Agent submits question via UI → Knowledge base ingestion and embedding → Relevant chunk retrieval → LLM generates answer suggestion → Agent feedback capture → Weekly expert review and improvement → Pre-production accuracy validation.

Related customer support cases