Back office ops · Production

Beekeeper optimizes LLM selection and user personalization with an Amazon Bedrock-powered dynamic evaluation system

The problem

Organizations face a moving target when selecting and maintaining LLMs: the best model and prompt combination shifts as models, prices, and requirements change, and most mid-sized companies lack the resources to continuously evaluate and improve them.

Workflow diagram · grounded in source

Scheduler triggers coordinator

trigger

“A scheduler triggers the coordinator, which fetches test data and sends it to evaluators.”

Automated model/prompt evaluation

ai_action

“These evaluators test each model/prompt pair and return results”

Manual validation sample

human_review

“a portion sent for manual validation”

Prompt mutation

ai_action

“The system mutates promising prompts to create variations, evaluates these again, and saves the best performers.”

User feedback incorporation

feedback_loop

“When user feedback arrives, the system incorporates it through a second phase. The coordinator fetches ranked model/prompt pairs and sends them with user feedback to a mutator, which returns personalized prompts.”

Drift detection

validation

“A drift detector makes sure these personalized versions don't stray too far from quality standards”

Production routing

routing

“Newly selected model/prompt pairs are used in production with ratios 1st: 50%, 2nd: 30%, and 3rd: 20%.”

Reported outcome

Beekeeper's system delivers 13–24% better ratings on responses aggregated per tenant, reduces manual labor in LLM and prompt selection, shortens the feedback cycle, and enables user- and tenant-specific prompt improvements.

Reported metrics

Response rating improvement per tenant13–24% better ratings on response when aggregated per tenant

Baseline evaluation pipeline cost per cyclearound $48

Manual evaluation sample rate7%

Reported stack

Amazon BedrockAmazon EventBridgeAmazon Elastic Kubernetes Service (EKS)AWS LambdaAmazon Relational Database Service (RDS)Amazon Mechanical TurkConverse APIAmazon NovaAnthropic Claude 4 SonnetMeta Llama 3Mistral 8x7BMistral LargeQwen3

Source

https://aws.amazon.com/blogs/machine-learning/how-beekeeper-optimized-user-personalization-with-amazon-bedrock?tag=soumet-20

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Beekeeper's system delivers 13–24% better ratings on responses aggregated per tenant, reduces manual labor in LLM and prompt selection, shortens the feedback cycle, and enables user- and tenant-specific prompt improve…

What tools did this team use?

Amazon Bedrock, Amazon EventBridge, Amazon Elastic Kubernetes Service (EKS), AWS Lambda, Amazon Relational Database Service (RDS), Amazon Mechanical Turk, Converse API, Amazon Nova, Anthropic Claude 4 Sonnet, Meta Llama 3.

What results were reported?

Response rating improvement per tenant: 13–24% better ratings on response when aggregated per tenant; Baseline evaluation pipeline cost per cycle: around $48; Manual evaluation sample rate: 7% (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Scheduler triggers coordinator → Automated model/prompt evaluation → Manual validation sample → Prompt mutation → User feedback incorporation → Drift detection → Production routing.