Workflow · Production

Cursor builds local sparse n-gram indexes to accelerate regex search for AI agents in large monorepos

The problem

Regex search via ripgrep is too slow for large Enterprise monorepos, with individual agent search invocations routinely taking more than 15 seconds and stalling agentic coding workflows.

First attempt

ripgrep must scan every file without index-based narrowing; classic trigram inverted indexes produce posting lists too large to query efficiently at scale; bloom-filter-augmented trigram indexes become saturated and lose their filtering power over time.

Workflow diagram · grounded in source
1
Agent issues regex search
trigger
“Matching regular expressions is now a critical part of Agentic development, and we believe it's crucial to target it explicitly”
2
Query decomposed into sparse n-grams
ai_action
“we can pick up a couple terabytes of Open-Source code from the internet and build a frequency table for all the character pairs we find in it. That frequency table is our hash function”
3
Local index lookup narrows candidates
integration
“By decomposing any regular expression into a set of trigrams and loading all the relevant posting lists from the inverted index, we end up with a list of documents that can potentially match our regular expression”
4
Regex matched on candidate files
validation
“The final result set will only be obtained by actually loading all the potential documents and matching the regular expression "the old fashioned way"”
5
Results delivered to agent
output
“removing altogether the time spent searching the codebase provides meaningful time savings —particularly when the Agent investigates bugs— and allows for much more effective iteration”
Reported outcome

Providing text search indexes to fast models creates a qualitative difference for agentic workflows; removing search time from the workflow provides meaningful time savings, particularly when the agent investigates bugs, and allows for more effective iteration.

Reported metrics
Ripgrep invocation latency on large reposmore than 15 seconds
Impact on agentic workflowsqualitative difference for Agentic workflows
Search time savingsmeaningful time savings
Reported stack
ripgrepComposer 2
Source
https://cursor.com/blog/fast-regex-search
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Providing text search indexes to fast models creates a qualitative difference for agentic workflows; removing search time from the workflow provides meaningful time savings, particularly when the agent investigates bu…

What tools did this team use?

ripgrep, Composer 2.

What results were reported?

Ripgrep invocation latency on large repos: more than 15 seconds; Impact on agentic workflows: qualitative difference for Agentic workflows; Search time savings: meaningful time savings (source-reported, not independently verified).

What failed first in this deployment?

ripgrep must scan every file without index-based narrowing; classic trigram inverted indexes produce posting lists too large to query efficiently at scale; bloom-filter-augmented trigram indexes become saturated and l…

How is this workflow AI workflow structured?

Agent issues regex search → Query decomposed into sparse n-grams → Local index lookup narrows candidates → Regex matched on candidate files → Results delivered to agent.