Marketing ops · Production

Twilio Segment deploys LLM-as-Judge multi-agent evaluation pipeline achieving 90%+ alignment with human assessment for CustomerAI audience generation

The problem

Marketers needed to navigate a complex UI to build customer audiences, and evaluating AI-generated ASTs was difficult because there can be an unbounded number of valid representations of the same audience logic.

Workflow diagram · grounded in source

User enters audience prompt

trigger

“A user provides a prompt, such as "Customers who have purchased at least 1 time."”

Synthetic eval prompt generation

ai_action

“LLM Question Generator Agent: This agent generates potential user input prompts based on the Real World AST Input.”

AST generation from prompts

ai_action

“LLM AST Generator Agent: This agent takes the generated prompts and produces ASTs using LLMs. This LLM Agent is responsible for generating the ASTs given customer's input”

LLM Judge scoring with CoT

validation

“The LLM Judge also needs a Chain of Thought (CoT) to provide reasoning for the scores, which can improve the model's capability and facilitate human evaluation. Implementing CoT allows the model to explain its decisions, making it easier…”

Score-driven iteration

feedback_loop

“these baseline scores enable us to compare future iterations & optimizations. For example, as we explore adding persistent memory via RAG. adopting a new model, or changing our prompting, we can compare the scores to determine the impact…”

Audience output to downstream tools

output

“the output (e.g. a list of users that meet the criteria) get federated into downstream tools like advertising and email marketing tools”

Reported outcome

CustomerAI audiences achieved a 3x improvement in median time-to-audience creation and a 95% feature retention rate when generation succeeds on first attempt; the LLM Judge evaluation system achieved over 90% alignment with human evaluation.

Reported metrics

Median time-to-audience creation3x improvement

Feature retention rate (first-attempt success)95%

LLM Judge alignment with human evaluationover 90%

GPT-4-32k model eval score4.55

Show all 6 reported metrics

median time-to-audience creation3x improvement

feature retention rate (first-attempt success)95%

LLM Judge alignment with human evaluationover 90%

GPT-4-32k model eval score4.55

Claude model eval score4.02

LLM Judge alignment improvement with CoTroughly 89% to 92%

Reported stack

CustomerAI audiencesClaudeGPT-4

Source

https://segment.com/blog/llm-as-judge/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

CustomerAI audiences achieved a 3x improvement in median time-to-audience creation and a 95% feature retention rate when generation succeeds on first attempt; the LLM Judge evaluation system achieved over 90% alignmen…

What tools did this team use?

CustomerAI audiences, Claude, GPT-4.

What results were reported?

Median time-to-audience creation: 3x improvement; Feature retention rate (first-attempt success): 95%; LLM Judge alignment with human evaluation: over 90%; GPT-4-32k model eval score: 4.55 (source-reported, not independently verified).

How is this marketing ops AI workflow structured?

User enters audience prompt → Synthetic eval prompt generation → AST generation from prompts → LLM Judge scoring with CoT → Score-driven iteration → Audience output to downstream tools.