Back office ops · Production

Credal.ai: RAG on enterprise documents — metadata tagging, data restructuring, and LLM-based routing

The problem

LLMs have limited attention and struggle to provide high-quality answers on complex corporate documents — naive RAG fails to retrieve the most relevant sections, poorly structured data (footnotes detached from citations, opaquely formatted tables) makes questions unanswerable, and LLMs cannot consistently reason about dates.

First attempt

LangChain's naive RAG failed to surface the most relevant document sections; its document parser placed footnotes at the end rather than inline, making citation lookups impossible; its table formatting was token-inefficient; and its structured output formatter distracted GPT-3.5 from the routing task, leaving it accurate only 50% of the time.

Workflow diagram · grounded in source

LLM metadata tag generation

ai_action

“we used LLMs to calculate metadata tags for each document or even each section of each document. Those can then be used in the search.”

User question arrives in Slack

trigger

“When a user asks a question in Slack, the question is routed to the most relevant AI expert”

LLM triager routes to AI expert

routing

“an initial step that could triage the user question to the right AI expert with 95%+ accuracy - an "LLM triager"”

Tag-based section pre-filtering

routing

“when a user question comes in, we can use those tags to pre-filter the text to the sections actually relevant to the user”

RAG retrieval on filtered sections

ai_action

“first retrieve the part of the document relevant to answer this question, and then have the LLM use that part to answer the question. This approach is called "RAG", or Retrieval Augmented Generation”

AI expert generates answer

output

“the question is routed to the most relevant AI expert, which then answers using only relevant data”

Reported outcome

By restructuring data before LLM ingestion — using LLM-generated metadata tags for pre-filtering, inserting footnotes inline, reformatting tables as CSV, and using focused few-shot prompting for routing — Credal turned previously unanswerable questions into straightforward ones and improved routing beyond the 50% baseline.

Reported metrics

LLM triager accuracy — naive approach50%

LLM triager target accuracy95%+

Token cost: LangChain table format vs CSV36% more expensive

Per-question cost using full-context Claudemore than $1

Show all 5 reported metrics

LLM triager accuracy — naive approach50%

LLM triager target accuracy95%+

Token cost: LangChain table format vs CSV36% more expensive

Per-question cost using full-context Claudemore than $1

Per-chunk cost for summary approachup to a $2 per chunk

Reported stack

LangChainPineconeGPT-3.5GPT-4GPT-4-32kClaudeGoogle DocsSlack

Source

https://www.credal.ai/blog/takeaways-from-using-llms-on-corporate-documents

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By restructuring data before LLM ingestion — using LLM-generated metadata tags for pre-filtering, inserting footnotes inline, reformatting tables as CSV, and using focused few-shot prompting for routing — Credal turne…

What tools did this team use?

LangChain, Pinecone, GPT-3.5, GPT-4, GPT-4-32k, Claude, Google Docs, Slack.

What results were reported?

LLM triager accuracy — naive approach: 50%; LLM triager target accuracy: 95%+; Token cost: LangChain table format vs CSV: 36% more expensive; Per-question cost using full-context Claude: more than $1 (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

LLM metadata tag generation → User question arrives in Slack → LLM triager routes to AI expert → Tag-based section pre-filtering → RAG retrieval on filtered sections → AI expert generates answer.