It support · Production

incident.io controls AI spending with per-prompt cost attribution and OpenAI project billing limits

The problem

incident.io hit OpenAI billing limits and temporarily broke all early-access AI features. As they expanded adoption — particularly Investigations, which costs 100x more than their other AI features — they lacked visibility into which features and code paths were driving spend.

First attempt

Billing limits were breached and all early-access AI features broke. Without per-prompt tracking, the team had no way to attribute spend to features or code paths, and discovered weeks later they had been burning hundreds of dollars on bugs.

Workflow diagram · grounded in source

Per-prompt token attribution

integration

“With this attribution of token usage to a given prompt name, done from the single API that we use to run prompts across our codebase without any change to the function signature, we're now able to track our token usage per execution of a…”

OpenAI project billing limits

integration

“We decided to create individual projects in OpenAI, each with their own billing limit, so we could limit the damage of any run-away code”

Daily Slack cost report

output

“every working morning we receive a screenshot in a Slack channel (#ai-cost-pulse) breaking down the previous 7 days of spend across different environments”

In-product cost and latency tagging

feedback_loop

“For developer environments, and for our organisation in production, we'll tag every message from our bot in Slack with the cost and latency of an interaction”

Backfill cost prediction

ai_action

“Given we know the historical cost of each processor run, we can extrapolate that to give a grounded estimation in what a run of these backfills will cost in LLM spend - giving us a heads up before we spend $50,000”

Eval suite cost reporting

feedback_loop

“our eval suite, the "tests" of the LLM prompting world, report the exact cost of their run”

Reported outcome

incident.io now has control over production, development, and training AI costs, with per-prompt attribution, feature-level billing limits, daily Slack cost reports, and predictive backfill cost estimation, enabling teams to ship AI features confidently.

Reported metrics

investigations cost vs existing AI features100x

Uncontrolled bug spend (historical)hundreds of dollars on bugs

potential backfill LLM cost (high example)$50,000

Example backfill run cost (modest)$112.45

Show all 6 reported metrics

investigations cost vs existing AI features100x

uncontrolled bug spend (historical)hundreds of dollars on bugs

potential backfill LLM cost (high example)$50,000

example backfill run cost (modest)$112.45

actual backfill run cost (specific example)$7.53

hypothetical unchecked backfill cost$128,000

Reported stack

OpenAISlackGitHubGrafanaGoogle Secret ManagerTerraform

Source

https://incident.io/building-with-ai/controlling-costs

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

incident.io now has control over production, development, and training AI costs, with per-prompt attribution, feature-level billing limits, daily Slack cost reports, and predictive backfill cost estimation, enabling t…

What tools did this team use?

OpenAI, Slack, GitHub, Grafana, Google Secret Manager, Terraform.

What results were reported?

investigations cost vs existing AI features: 100x; Uncontrolled bug spend (historical): hundreds of dollars on bugs; potential backfill LLM cost (high example): $50,000; Example backfill run cost (modest): $112.45 (source-reported, not independently verified).

What failed first in this deployment?

Billing limits were breached and all early-access AI features broke.

How is this it support AI workflow structured?

Per-prompt token attribution → OpenAI project billing limits → Daily Slack cost report → In-product cost and latency tagging → Backfill cost prediction → Eval suite cost reporting.