How Ellipsis built a multi-agent AI code review and code search system
Building a reliable AI code review agent is hard: a single mega-agent with a large prompt produces high false positive rates — developers' most common complaint — and LLM performance degrades under large context. Traditional RAG cosine-similarity thresholds work poorly for code search because they fail to capture whether retrieved code is actually useful.
The conventional RAG approach — search, rerank by cosine similarity, drop below a threshold — does not work well for code search because relative ranking matters less than whether the retrieved code is actually useful, and cosine similarity cannot reliably capture that distinction.
The multistage filtering pipeline significantly reduces the false positive rate, incremental vector indexing syncs code changes in a couple of seconds, and user feedback is reflected almost immediately in agent behavior without requiring per-customer fine-tuning.
Frequently asked questions
What did this team achieve with this AI workflow?
The multistage filtering pipeline significantly reduces the false positive rate, incremental vector indexing syncs code changes in a couple of seconds, and user feedback is reflected almost immediately in agent behavi…
What tools did this team use?
GitHub App, Hookdeck, FastAPI, Hatchet, tree-sitter, Turbopuffer, Lsproxy, Modal, DynamoDB, GPT-4o.
What results were reported?
False positive rate: significantly reduces the false positive rate; Index sync time on code changes: a couple seconds; Feedback reflection speed: reflected almost immediately in the agent behavior; Installation time: <30 seconds (source-reported, not independently verified).
What failed first in this deployment?
The conventional RAG approach — search, rerank by cosine similarity, drop below a threshold — does not work well for code search because relative ranking matters less than whether the retrieved code is actually useful…
How is this quality assurance AI workflow structured?
PR opened or tagged → Webhook routed to queue → Parallel comment generators → Code search subagent → Multistage filtering pipeline → User feedback incorporated → Filtered comments posted.