Back office ops · Production

Karrot builds a centralized GenAI platform with LLM Router, Prompt Studio, and KarrotChat

The problem

Teams at Karrot were independently provisioning AI API accounts and keys, creating management overhead, uneven rate-limit availability, and fragmented cost visibility. Every AI feature iteration also required engineering support, preventing rapid experimentation.

Workflow diagram · grounded in source

Route all AI calls via gateway

routing

“We built LLM Router to solve these challenges. The core concept: funnel all AI API calls through a single gateway. API keys and accounts are managed centrally by LLM Router”

Provider-specific translation

integration

“LLM Router handles provider-specific translation internally. When new models launch, we add them to LLM Router once. Teams can immediately use them by specifying the new model name — zero code changes required.”

No-code prompt experimentation

ai_action

“Anyone can create and test AI features without writing code. Enter a prompt, select a model, click run.”

Evaluate prompt with test sets

validation

“Prompt Studio's Evaluation feature lets you upload test sets — hundreds to thousands of examples — generate results in batch, and measure performance quantitatively.”

Deploy prompt without code

output

“Engineers integrate the Prompt Studio API only once. After that, teams ship prompt improvements directly from the UI — no code changes needed.”

User queries DANA in KarrotChat

trigger

“Now anyone can ask questions in KarrotChat and perform sophisticated analysis.”

DANA queries BigQuery via MCP

integration

“It combines the BigQuery MCP, detailed documentation of the data model, and carefully crafted prompts.”

DANA returns analysis results

output

“DANA finds the right table, runs the query, and returns the answer.”

Reported outcome

Karrot eliminated provisioning overhead, unified cost visibility, and enabled non-engineers to build AI features independently.
Internal Agents like DANA now let any team member perform sophisticated data analysis without SQL expertise, measurably boosting productivity across teams.

Reported metrics

GenAI use cases scaledhundreds of GenAI use cases

Platform request volumehundreds of millions of requests

Team productivitymeasurably boosting productivity across teams

Reported stack

LLM RouterPrompt StudioKarrotChatOpenAIAnthropicGooglevLLMOpenAI SDKBigQuery

Source

https://medium.com/daangn/karrots-genai-platform-5cf6e813838e

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Karrot eliminated provisioning overhead, unified cost visibility, and enabled non-engineers to build AI features independently.

What tools did this team use?

LLM Router, Prompt Studio, KarrotChat, OpenAI, Anthropic, Google, vLLM, OpenAI SDK, BigQuery.

What results were reported?

GenAI use cases scaled: hundreds of GenAI use cases; Platform request volume: hundreds of millions of requests; Team productivity: measurably boosting productivity across teams (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Route all AI calls via gateway → Provider-specific translation → No-code prompt experimentation → Evaluate prompt with test sets → Deploy prompt without code → User queries DANA in KarrotChat → DANA queries BigQuery via MCP → DANA returns analysis results.

Karrot eliminated provisioning overhead, unified cost visibility, and enabled non-engineers to build AI features independently.Internal Agents like DANA now let any team member perform sophisticated data analysis without SQL expertise, measurably boosting productivity across teams.