Back office ops · Production

Ramp builds in-house RAG model to migrate industry classification to NAICS codes

The problem

Ramp's industry classification depended on a homegrown taxonomy stitched together from third-party data, sales-entered data, and customer self-reporting, yielding multiple non-auditable sources of truth and preventing teams from sharing a consistent customer view.

First attempt

The homegrown system produced overly broad labels — classifying a hiring platform alongside law firms and dating apps under 'Professional Services' — and required complex many-to-many mappings across 100+ internal levels to translate to standard codes, making cross-team alignment and auditing impossible.

Workflow diagram · grounded in source
1
New business enters pipeline
trigger
“We have internal services that handle embeddings for new businesses and LLM prompt evaluations”
2
Embedding-based retrieval
ai_action
“Generating recommendations involves identifying the most relevant items from the knowledge base (NAICS codes) given a query (business)”
3
Clickhouse similarity search
integration
“Knowledge base embeddings are pre-computed and stored in Clickhouse for fast retrieval of recommendations using similarity scores”
4
LLM first pass — narrow candidates
ai_action
“In the first prompt we include many recommendations but don't include the most specific descriptions, asking the LLM to return a small list of the most relevant codes”
5
LLM second pass — final selection
ai_action
“In the second prompt, we then ask the LLM to choose the best one and provide more context for each code”
6
NAICS code validation
validation
“we validate that the output NAICS codes from each LLM prompt are valid”
7
Kafka logging and iteration
feedback_loop
“We log intermediate results using Kafka so that we can diagnose pathological cases and iterate on prompts”
Reported outcome

Ramp's in-house RAG model increased classification accuracy, gave the team full control over updates and tuning, helped internal teams work more cohesively, and enabled more precise communication with external partners.

Reported metrics
Retrieval stage accuracy improvement (acc@k)up to 60%
fuzzy accuracy improvement after LLM optimization5%-15%
Reported stack
RAGLLMClickhouseKafka
Source
https://engineering.ramp.com/industry_classification
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Ramp's in-house RAG model increased classification accuracy, gave the team full control over updates and tuning, helped internal teams work more cohesively, and enabled more precise communication with external partners.

What tools did this team use?

RAG, LLM, Clickhouse, Kafka.

What results were reported?

Retrieval stage accuracy improvement (acc@k): up to 60%; fuzzy accuracy improvement after LLM optimization: 5%-15% (source-reported, not independently verified).

What failed first in this deployment?

The homegrown system produced overly broad labels — classifying a hiring platform alongside law firms and dating apps under 'Professional Services' — and required complex many-to-many mappings across 100+ internal lev…

How is this back office ops AI workflow structured?

New business enters pipeline → Embedding-based retrieval → Clickhouse similarity search → LLM first pass — narrow candidates → LLM second pass — final selection → NAICS code validation → Kafka logging and iteration.

Related back office ops cases