Back office ops · Production

Amazon Bedrock Prompt Optimization lifts character dialogue attribution accuracy from 70% to 90% for Yuewen Group

The problem

Yuewen Group's proprietary NLP models had prolonged development cycles and slow updates. After transitioning to LLMs on Amazon Bedrock, the team lacked prompt engineering experience, causing LLMs with unoptimized prompts to underperform traditional NLP models — reaching only around 70% accuracy on character dialogue attribution versus around 80% for the NLP baseline.

First attempt

LLMs with unoptimized prompts underperformed traditional NLP models on specific tasks such as character dialogue attribution, reaching only around 70% accuracy compared to around 80% for the traditional NLP approach.

Workflow diagram · grounded in source
1
User submits original prompt
trigger
“users input their original prompt. The prompt can be a template with the required variables represented by placeholders”
2
Prompt Analyzer decomposes prompt
ai_action
“Prompt Analyzer is a fine-tuned LLM which decomposes the prompt structure by extracting its key constituent elements, such as the task instruction, input context, and few-shot demonstrations”
3
Prompt Rewriter generates optimized prompt
ai_action
“Prompt Rewriter module, which employs a general LLM-based meta-prompting strategy to further improve the prompt signatures and restructure the prompt layout. As the result, Prompt Rewriter produces a refined and enhanced version of the i…”
4
Side-by-side comparison display
output
“console then displays the Compare Variants tab, presenting the original and optimized prompts side-by-side for quick comparison”
Reported outcome

Bedrock Prompt Optimization raised Yuewen Group's character dialogue attribution accuracy to 90%, surpassing traditional NLP models by 10%, and enabled the team to complete prompt engineering processes in a fraction of the time, greatly improving development efficiency.

Reported metrics
character dialogue attribution accuracy — traditional NLP baselinearound 80%
character dialogue attribution accuracy — LLM with unoptimized promptsaround 70%
Character dialogue attribution accuracy — with optimized prompts90%
accuracy improvement over traditional NLP10%
Show all 6 reported metrics
character dialogue attribution accuracy — traditional NLP baselinearound 80%
character dialogue attribution accuracy — LLM with unoptimized promptsaround 70%
character dialogue attribution accuracy — with optimized prompts90%
accuracy improvement over traditional NLP10%
prompt engineering process timefraction of the time
development efficiencygreatly improving development efficiency
Reported stack
Amazon BedrockBedrock Prompt OptimizationClaude 3.5 Sonnet
Source
https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-prompt-optimization-drives-llm-applications-innovation-for-yuewen-group?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Bedrock Prompt Optimization raised Yuewen Group's character dialogue attribution accuracy to 90%, surpassing traditional NLP models by 10%, and enabled the team to complete prompt engineering processes in a fraction o…

What tools did this team use?

Amazon Bedrock, Bedrock Prompt Optimization, Claude 3.5 Sonnet.

What results were reported?

character dialogue attribution accuracy — traditional NLP baseline: around 80%; character dialogue attribution accuracy — LLM with unoptimized prompts: around 70%; Character dialogue attribution accuracy — with optimized prompts: 90%; accuracy improvement over traditional NLP: 10% (source-reported, not independently verified).

What failed first in this deployment?

LLMs with unoptimized prompts underperformed traditional NLP models on specific tasks such as character dialogue attribution, reaching only around 70% accuracy compared to around 80% for the traditional NLP approach.

How is this back office ops AI workflow structured?

User submits original prompt → Prompt Analyzer decomposes prompt → Prompt Rewriter generates optimized prompt → Side-by-side comparison display.

Related back office ops cases