Incident management · Production

Autonomous observability at Pinterest: AI agents bridge fragmented logs, metrics, and traces via MCP

The problem

Pinterest's observability infrastructure predated OpenTelemetry, leaving logs, metrics, and traces in disconnected silos with no shared context or correlation. On-call engineers had to jump across multiple interfaces to root-cause incidents, and a steep per-tool learning curve compounded the time loss, especially for newer engineers.

First attempt

When first building the MCP server agent, Pinterest discovered that allowing the agent to query data organically caused it to exceed its context window and crash, requiring new strategies to constrain query scope.

Workflow diagram · grounded in source
1
Engineer submits alert link
trigger
“The engineer can provide the Tricorder with their alert link/number and sit back while it gathers the relevant information for their investigation”
2
MCP tools fetch observability data
integration
“empower an agent to simultaneously interact with time-series metrics, logs, traces, changefeed events (deployments, experiments, etc.), alerts, and more”
3
Agent explores dependency graph
ai_action
“the agents use tools on multiple parts of the graph, exploring all the incoming and outgoing dependencies to check for the overall health of connections with no specific prompting to do so”
4
Generate filtered dashboard links
output
“have the agent generate links to the dashboards containing that information (already filtered to the correct time periods and relevant services)”
5
Provide root-cause suggestions
output
“providing suggestions and next steps as it gains more information”
Reported outcome

The Tricorder Agent, built on Pinterest's centralized MCP server, lets engineers submit an alert link and receive filtered dashboard links plus root-cause hypotheses and next steps without switching between interfaces, targeting MTTR reduction and freeing engineers to focus on resolving incidents.

Reported metrics
mean time to resolution (MTTR)reducing mean time to resolution (MTTR)
Engineer time switching between interfacessaving an on-call engineer the time spent jumping around all our interfaces
Observability data points processed per minute3 billion data points per minute
Tag key/value combinations per minute12 billion keys (tag key/value combinations) per minute
Show all 6 reported metrics
mean time to resolution (MTTR)reducing mean time to resolution (MTTR)
engineer time switching between interfacessaving an on-call engineer the time spent jumping around all our interfaces
observability data points processed per minute3 billion data points per minute
tag key/value combinations per minute12 billion keys (tag key/value combinations) per minute
logs volume per day7 TB of logs per day
traces volume per day7 TB of traces per day
Reported stack
MCPAgent2AgentLLMsTricorder Agent
Source
https://medium.com/pinterest-engineering/autonomous-observability-at-pinterest-part-1-of-2-eb0adae830ba
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The Tricorder Agent, built on Pinterest's centralized MCP server, lets engineers submit an alert link and receive filtered dashboard links plus root-cause hypotheses and next steps without switching between interfaces…

What tools did this team use?

MCP, Agent2Agent, LLMs, Tricorder Agent.

What results were reported?

mean time to resolution (MTTR): reducing mean time to resolution (MTTR); Engineer time switching between interfaces: saving an on-call engineer the time spent jumping around all our interfaces; Observability data points processed per minute: 3 billion data points per minute; Tag key/value combinations per minute: 12 billion keys (tag key/value combinations) per minute (source-reported, not independently verified).

What failed first in this deployment?

When first building the MCP server agent, Pinterest discovered that allowing the agent to query data organically caused it to exceed its context window and crash, requiring new strategies to constrain query scope.

How is this incident management AI workflow structured?

Engineer submits alert link → MCP tools fetch observability data → Agent explores dependency graph → Generate filtered dashboard links → Provide root-cause suggestions.

Related incident management cases