Back office ops · Production

10 Battle Scars from Building Agentic AI Analytics

The problem

Building production-grade agentic AI analytics requires far more than connecting an LLM to SQL; teams encounter pitfalls including framework over-reliance, non-deterministic query generation, unresolved business-term ambiguity, multi-step consistency failures, trust-eroding black-box answers, cold-start onboarding gaps, latency and cost blowouts, and observability gaps that make debugging impossible.

First attempt

Using generic LLM chain frameworks as the runtime hides retries, timeouts, and mutable state, making behaviour hard to trace. Letting generative prompts drive execution produces non-deterministic queries. Black-box feedback mechanisms erode user trust because changes persist with no visibility or review.

Workflow diagram · grounded in source

User submits NL query

trigger

“answers business questions over your company's data”

Pre-parser extracts intent

ai_action

“A pre-parser extracts intent/entities/time into a constrained schema”

LLM proposes typed plan

ai_action

“A planner may use an LLM to propose a plan (temp=0)”

Validator normalizes and policy-checks plan

validation

“a validator normalizes dates, joins, and metric–dimension pairs before execution”

Ambiguity detection and clarification

routing

“if candidates tie or confidence falls below a floor, stop execution and ask”

Policy-aware SQL compilation

output

“The compiler then emits SQL/MDX/GraphQL from the plan”

Answer with inline transparency

output

“Every run must emit a preamble (business view, metric definitions, time intervals), lineage (tables, joins, policy filters), cache status, and fallbacks”

Feedback to governed policies

feedback_loop

“Convert feedback into governed policies with target, scope, TTL, and attribution. Org-wide changes require approval through an admin console.”

Reported outcome

Tellius has implemented deterministic planning, a governed semantic layer, clarification policies, inline transparency, and observability in production; governed feedback and drift checks let the team correct issues quickly without asking users to prompt harder.

Reported metrics

Memory overhead from abstraction layers73%

Latency increase from abstraction layers20%

Cost increase from abstraction layers15%

context consumed by 150-column table (GPT-4)~25%

Show all 16 reported metrics

memory overhead from abstraction layers73%

latency increase from abstraction layers20%

cost increase from abstraction layers15%

context consumed by 150-column table (GPT-4)~25%

compositional task consistency rate (GPT-4)below 50%

AI pilot failure ratenear 95%

LLM query generation time (best models)3.2 seconds

AI-generated query data scan overhead vs human SQL1.5-2x more data than human-written ones

total query processing time6-7 seconds

per-completion cost$0.40+

worst-case per-answer cost$4.00

GPT-4o agent realistic task success rate<50%

GPT-4o consistent cross-attempt success rate~25%

businesses struggling with cross-functional coordination for AI74%

NL2SQL accuracy with domain scoping (AWS/Cisco)>95% accuracy

enterprise schema width vs Spider benchmark300+ columns versus Spider's 28

Reported stack

LangChain

Source

https://www.tellius.com/resources/blog/10-battle-scars-from-building-agentic-ai-analytics

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Tellius has implemented deterministic planning, a governed semantic layer, clarification policies, inline transparency, and observability in production; governed feedback and drift checks let the team correct issues q…

What tools did this team use?

LangChain.

What results were reported?

Memory overhead from abstraction layers: 73%; Latency increase from abstraction layers: 20%; Cost increase from abstraction layers: 15%; context consumed by 150-column table (GPT-4): ~25% (source-reported, not independently verified).

What failed first in this deployment?

Using generic LLM chain frameworks as the runtime hides retries, timeouts, and mutable state, making behaviour hard to trace.

How is this back office ops AI workflow structured?

User submits NL query → Pre-parser extracts intent → LLM proposes typed plan → Validator normalizes and policy-checks plan → Ambiguity detection and clarification → Policy-aware SQL compilation → Answer with inline transparency → Feedback to governed policies.