Compliance monitoring · Production

Ramp builds in-house RAG model to standardize industry classification on NAICS codes

The problem

Ramp's industry classification relied on a homegrown system stitched together from third-party data, Sales-entered data, and customer self-reporting, producing multiple non-auditable sources of truth and overly broad labels that made compliance, credit risk profiling, and sales targeting unreliable. Different teams used incompatible taxonomies, requiring many-to-many mappings between over 100 internal levels and thousands of NAICS or SIC codes.

Workflow diagram · grounded in source
1
New business enters pipeline
trigger
“We have internal services that handle embeddings for new businesses and LLM prompt evaluations”
2
Similarity search retrieves NAICS candidates
ai_action
“Knowledge base embeddings are pre-computed and stored in Clickhouse for fast retrieval of recommendations using similarity scores”
3
First LLM prompt narrows candidates
ai_action
“In the first prompt we include many recommendations but don't include the most specific descriptions, asking the LLM to return a small list of the most relevant codes”
4
Second LLM prompt selects final code
ai_action
“In the second prompt, we then ask the LLM to choose the best one and provide more context for each code”
5
Guardrail validation of output codes
validation
“While hallucinations are generally negative, we've also found cases where the LLM predicts the correct code despite it not being present in the recommendations. To filter out just "bad" hallucinations, we validate that the output NAICS c…”
6
Kafka logs intermediate results
feedback_loop
“We log intermediate results using Kafka so that we can diagnose pathological cases and iterate on prompts”
7
NAICS code classification output
output
“a classification model that could predict six-digit NAICS codes for all Ramp businesses”
Reported outcome

Migrating to NAICS codes via an in-house RAG model improved classification accuracy, simplified cross-team workflows, and made the system fully auditable and tunable.
Hyperparameter optimization yielded performance boosts of up to 60% in acc@k and 5%-15% improvement in fuzzy accuracy.

Reported metrics
Acc@k performance boostup to 60%
Fuzzy accuracy improvement5%-15%
Data qualitysignificantly upgrade our data quality
Workflow complexitysimplified workflows
Reported stack
RAGLLMClickhouseKafka
Source
https://engineering.ramp.com/post/industry_classification
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Migrating to NAICS codes via an in-house RAG model improved classification accuracy, simplified cross-team workflows, and made the system fully auditable and tunable.

What tools did this team use?

RAG, LLM, Clickhouse, Kafka.

What results were reported?

Acc@k performance boost: up to 60%; Fuzzy accuracy improvement: 5%-15%; Data quality: significantly upgrade our data quality; Workflow complexity: simplified workflows (source-reported, not independently verified).

How is this compliance monitoring AI workflow structured?

New business enters pipeline → Similarity search retrieves NAICS candidates → First LLM prompt narrows candidates → Second LLM prompt selects final code → Guardrail validation of output codes → Kafka logs intermediate results → NAICS code classification output.

Related compliance monitoring cases