Back office ops · Production

Semantic multi-component embeddings with Redis reduce LLM token consumption by 91% in enterprise tool selection

The problem

An enterprise AI platform with 70+ automated tools was sending all tool definitions to the LLM on every query, consuming over 8,000 tokens per request, causing spiralling costs, slower responses, and irrelevant tool suggestions.

First attempt

Keyword matching and category filtering failed to bridge the vocabulary gap between natural language queries and technical tool names, and could not disambiguate context-dependent queries.

Workflow diagram · grounded in source
1
User query arrives
Trigger
A user submits a natural language query to the enterprise AI platform.
▾ source quote
“Every user query was sending all 70 tool definitions to the model, consuming over 8,000 tokens per request—even when users only needed 2-3 relevant tools”
2
Query embedding generated
Ai action
The system generates a vector embedding of the user query.
▾ source quote
“# Generate query embedding”
3
Multi-component vector search
Ai action
The search is performed across all tool component types.
▾ source quote
“Search across all component types”
4
Weighted relevance scoring
Ai action
Relevance scoring combines semantic similarity with practical factors.
▾ source quote
“The relevance scoring combines semantic similarity with practical factors”
5
Adaptive tool selection
Ai action
The system uses adaptive selection based on score distribution rather than a fixed tool count.
▾ source quote
“Rather than returning a fixed number of tools, the system uses adaptive selection based on score distribution”
6
Minimal tool set sent to LLM
Output
The system sends only the most relevant tools to the LLM.
▾ source quote
“Sent only 3 most relevant tools (638 tokens)”
Reported outcome

Semantic tool selection with multi-component embeddings and Redis vector search reduced token consumption by 91.5% and cost per query by 49%, while improving Precision@3 to 95% and delivering 31% faster response times.

Reported metrics
Token usage reduction91.5%
Cost reduction per query49%
Average input tokens per query (before)7,244
Average input tokens per query (after)198
Show all 16 reported metrics
token usage reduction91.5%
cost reduction per query49%
average input tokens per query (before)7,244
average input tokens per query (after)198
cost per query (before)$0.0118
cost per query (after)$0.0060
Precision@3 (after)95%
Recall@5 (after)90%
Mean Reciprocal Rank (after)0.88
response time improvement31% faster
cache hit rate60%
annual cost savings$20,880
tool library size70+
tokens per request before (stated)over 8,000
tokens in example query before7,433
tokens in example query after638
Reported stack
Redis Stacktext-embedding-3-small
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://mlops.community/blog/how-i-reduced-ai-token-costs-by-91percent-with-semantic-tool-selection-and-redis
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Semantic tool selection with multi-component embeddings and Redis vector search reduced token consumption by 91.5% and cost per query by 49%, while improving Precision@3 to 95% and delivering 31% faster response times.

What tools did this team use?

Redis Stack, text-embedding-3-small.

What results were reported?

Token usage reduction: 91.5%; Cost reduction per query: 49%; Average input tokens per query (before): 7,244; Average input tokens per query (after): 198 (source-reported, not independently verified).

What failed first in this deployment?

Keyword matching and category filtering failed to bridge the vocabulary gap between natural language queries and technical tool names, and could not disambiguate context-dependent queries.

How is this back office ops AI workflow structured?

User query arrives → Query embedding generated → Multi-component vector search → Weighted relevance scoring → Adaptive tool selection → Minimal tool set sent to LLM.

Related back office ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Back office ops cases →