Incident management · Production

Databricks builds AI agent for database debugging, reducing investigation time by up to 90%

The problem

During MySQL incident investigations, Databricks engineers had to jump between multiple disconnected tools, dashboards, CLIs, and SOPs with no cohesive end-to-end workflow. Junior engineers didn't know where to start; senior engineers found the tooling fragmented and cumbersome.

First attempt

A v1 static agentic workflow that followed a debugging SOP was not effective — engineers wanted a diagnostic report with immediate insights, not a manual checklist. A subsequent anomaly detection approach surfaced relevant anomalies but still failed to provide clear next steps.

Workflow diagram · grounded in source
1
Engineer asks in natural language
trigger
“This new agentic capability has enabled engineers to routinely answer questions in natural language about their service health and performance without needing to reach out to on-call engineers in the storage teams”
2
Agent retrieves metrics and logs
ai_action
“Our AI agent interprets, executes, and debugs by retrieving key metrics and logs”
3
Cross-layer symptom correlation
ai_action
“It connects symptoms across layers, such as identifying the workspace driving unexpected load and correlating IOPS spikes with recent schema migrations”
4
Specialized agent routing
routing
“we can easily spin up specialized agents for different domains: one focused on system and database issues, another on client-side traffic patterns, and so on. This decomposition enables each agent to build deep expertise in its area whil…”
5
Judge LLM continuous improvement
feedback_loop
“a separate "judge" LLM to score the responses for accuracy and helpfulness as we modify the prompts and tools”
6
Insights and next steps output
output
“our agent can extract meaningful insights and actively guide engineers through investigations. Within minutes, it surfaces relevant logs and metrics that engineers might not have considered examining”
Reported outcome

The AI-assisted platform reduces time spent debugging by up to 90%, and new hires with zero context can jump-start a database investigation in under 5 minutes.

Reported metrics
Debugging time reductionup to 90%
Time to start database investigation for new hiresunder 5 minutes
Reported stack
DsPyMLflowScala
Source
https://www.databricks.com/blog/how-we-debug-1000s-databases-ai-databricks
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The AI-assisted platform reduces time spent debugging by up to 90%, and new hires with zero context can jump-start a database investigation in under 5 minutes.

What tools did this team use?

DsPy, MLflow, Scala.

What results were reported?

Debugging time reduction: up to 90%; Time to start database investigation for new hires: under 5 minutes (source-reported, not independently verified).

What failed first in this deployment?

A v1 static agentic workflow that followed a debugging SOP was not effective — engineers wanted a diagnostic report with immediate insights, not a manual checklist.

How is this incident management AI workflow structured?

Engineer asks in natural language → Agent retrieves metrics and logs → Cross-layer symptom correlation → Specialized agent routing → Judge LLM continuous improvement → Insights and next steps output.

Related incident management cases