Back office ops · Production

Ramp migrates customer industry classification to a RAG-powered NAICS system

The problem

Ramp's industry classification relied on a homegrown taxonomy stitched together from third-party data, sales-entered data, and customer self-reporting, producing multiple non-auditable sources of truth that forced many-to-many translation layers and made cross-team alignment and external partner communication difficult.

First attempt

The Homegrown system had documented issues including overly broad and inconsistent categorizations, mandatory many-to-many mappings to external standards, and classifications insufficiently nuanced to satisfy compliance requirements.

Workflow diagram · grounded in source
1
Business embedding generation
ai_action
“We have internal services that handle embeddings for new businesses and LLM prompt evaluations”
2
Similarity search in Clickhouse
integration
“Knowledge base embeddings are pre-computed and stored in Clickhouse for fast retrieval of recommendations using similarity scores”
3
First LLM prompt – initial shortlist
ai_action
“In the first prompt we include many recommendations but don't include the most specific descriptions, asking the LLM to return a small list of the most relevant codes”
4
Second LLM prompt – final selection
ai_action
“In the second prompt, we then ask the LLM to choose the best one and provide more context for each code”
5
NAICS code validation
validation
“we validate that the output NAICS codes from each LLM prompt are valid”
6
Intermediate result logging
feedback_loop
“We log intermediate results using Kafka so that we can diagnose pathological cases and iterate on prompts”
Reported outcome

Migrating to the RAG-based NAICS system increased classification accuracy and gave Ramp's teams a consistent, auditable system with full control over updates and costs, described by stakeholders as significantly upgrading data quality and understanding of customers.

Reported metrics
Acc@k performance improvementup to 60%
Fuzzy accuracy improvement after optimization5%-15%
Data qualitygreatly improved our data quality
Classification accuracyincreased accuracy in industry classification
Reported stack
RAGClickhouseKafkaLLM
Source
https://builders.ramp.com/post/industry_classification
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Migrating to the RAG-based NAICS system increased classification accuracy and gave Ramp's teams a consistent, auditable system with full control over updates and costs, described by stakeholders as significantly upgra…

What tools did this team use?

RAG, Clickhouse, Kafka, LLM.

What results were reported?

Acc@k performance improvement: up to 60%; Fuzzy accuracy improvement after optimization: 5%-15%; Data quality: greatly improved our data quality; Classification accuracy: increased accuracy in industry classification (source-reported, not independently verified).

What failed first in this deployment?

The Homegrown system had documented issues including overly broad and inconsistent categorizations, mandatory many-to-many mappings to external standards, and classifications insufficiently nuanced to satisfy complian…

How is this back office ops AI workflow structured?

Business embedding generation → Similarity search in Clickhouse → First LLM prompt – initial shortlist → Second LLM prompt – final selection → NAICS code validation → Intermediate result logging.

Related back office ops cases