Incident management · Production

Datadog builds a replayable evaluation platform for Bits AI SRE to catch agent regressions

The problem

As Datadog built Bits AI SRE, improvements in one area could quietly introduce regressions in another with no reliable way to detect them, and the team had no way to replay real production context, measure behavior consistently across diverse incidents, or track whether the agent was improving over time.

First attempt

Testing individual tools in isolation failed because agent failures emerged from interactions between steps rather than single tool calls. Live replay of Bits investigations also did not scale because results were not aggregated, environments changed, and signals expired.

Workflow diagram · grounded in source

Customer feedback triggers label

trigger

“When customers provide feedback on a Bits AI investigation, we use that signal, along with the information from the investigation itself, to construct a ground truth root cause analysis and the queries that make up the world snapshot”

Bits derives ground truth RCA

ai_action

“Bits aggregates related signals, derives relevant relationships, and resolves ambiguous references in feedback. For example, it can turn "it was slow" into a more precise statement about the elevated latency in a specific service. Since …”

Confidence scoring and flagging

validation

“each generated label is assigned confidence scores, and anything below a defined threshold is flagged for human review. These scores evaluate the generated RCAs across several dimensions, including thoroughness, specificity, and accuracy.”

Human validation of labels

human_review

“Instead of manually assembling root cause analyses from raw signals, reviewers now validate and refine Bits' outputs.”

Noisy signal reconstruction

integration

“we capture more than the minimal signal needed to explain the issue. We expand the snapshot by discovering related components based on the root cause chain, even if those components are not directly involved in the failure itself.”

Bits runs against label set

ai_action

“The evaluation platform is the system that runs Bits against our label set, scores the results, and tracks performance over time.”

Regression detection and alerting

feedback_loop

“Results from these runs flow into dashboards and Slack notifications, and we alert on significant deviations in overall performance.”

Reported outcome

The evaluation platform scaled label creation by an order of magnitude, reduced label validation time by more than 95%, improved root cause quality by roughly 30%, and now runs Bits against tens of thousands of scenarios drawn from real incidents every week.

Reported metrics

Label validation time per labelmore than 95%

Label creation rateincreased our label creation rate by an order of magnitude

Root cause quality improvementroughly 30%

Evaluation pass rate change from noise calibrationroughly 11%

Show all 6 reported metrics

label validation time per labelmore than 95%

label creation rateincreased our label creation rate by an order of magnitude

root cause quality improvementroughly 30%

evaluation pass rate change from noise calibrationroughly 11%

label count reduction from noise calibration35%

weekly evaluation scaletens of thousands of scenarios

Reported stack

Bits AI SREDatadog LLM ObservabilityClaude Opus 4.5Slack

Source

https://www.datadoghq.com/blog/engineering/bits-ai-eval-platform/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The evaluation platform scaled label creation by an order of magnitude, reduced label validation time by more than 95%, improved root cause quality by roughly 30%, and now runs Bits against tens of thousands of scenar…

What tools did this team use?

Bits AI SRE, Datadog LLM Observability, Claude Opus 4.5, Slack.

What results were reported?

Label validation time per label: more than 95%; Label creation rate: increased our label creation rate by an order of magnitude; Root cause quality improvement: roughly 30%; Evaluation pass rate change from noise calibration: roughly 11% (source-reported, not independently verified).

What failed first in this deployment?

Testing individual tools in isolation failed because agent failures emerged from interactions between steps rather than single tool calls.

How is this incident management AI workflow structured?

Customer feedback triggers label → Bits derives ground truth RCA → Confidence scoring and flagging → Human validation of labels → Noisy signal reconstruction → Bits runs against label set → Regression detection and alerting.