Back office ops · Production

Dropbox Dash: knowledge graphs, MCP, and DSPy prompt optimization for enterprise AI search

The problem

Work content is scattered across dozens of SaaS apps and unavailable to LLMs, making it nearly impossible for employees to find what they need or get AI assistance with their work.

Workflow diagram · grounded in source
1
Connect third-party apps
integration
“we're building custom crawlers and connecting to all these different third-party apps. It's not easy. Everything has its own rate limit, each has its own unique API quirks, each has its own ACL and permission system”
2
Content understanding
ai_action
“we're doing a lot of content understanding—and in certain cases, enriching the content itself. So, first we normalize a lot of the different files that come in and get it into a format like markdown. Then, we're looking at extracting key…”
3
Knowledge graph modeling
ai_action
“we take it a step further to model all these pieces of information together as a graph. Meetings may have associated documents, associated people, transcripts, or prior notes. Building that cross-app intelligence is essential to providin…”
4
Hybrid indexing
integration
“we use both a lexical index—using BM25—and then store everything as dense vectors in a vector store. While this allows us to do hybrid retrieval, we found BM25 was very effective on its own with some relevant signals. It's an amazing wor…”
5
Personalized retrieval and ranking
ai_action
“we apply multiple ranking passes on any retrieved results so they are personalized and ACL'd to you”
6
LLM-as-judge evaluation
validation
“you want to use LLMs as a judge. Broadly speaking, what you're trying to do is judge how relevant a piece of information is between, say, one and five, and then use that to improve over time”
7
DSPy prompt optimization
feedback_loop
“we noticed we could create bullet points with the different disagreements and then have DSPy try to optimize the bullets themselves. So if there were multiple disagreements, it would try to reduce those disagreements overall, and we star…”
Reported outcome

Dropbox Dash combines custom connectors, knowledge graphs, hybrid BM25/vector retrieval, and DSPy-optimized LLM judges to enable cross-app AI search and agentic queries, with LLM judge disagreement reduced from an initial 8% through iterative prompt refinement, model upgrades, and DSPy optimization.

Reported metrics
LLM judge human disagreement rate (initial prompt)8%
MCP simple query latencyup to 45 seconds
Dash context window capabout 100,000 tokens
Total prompts in Dash stackover 30
Reported stack
Dropbox DashBM25MCPDSPyo3CLIP-based modelsNDCGRAGJira
Source
https://dropbox.tech/machine-learning/vp-josh-clemm-knowledge-graphs-mcp-and-dspy-dash
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Dropbox Dash combines custom connectors, knowledge graphs, hybrid BM25/vector retrieval, and DSPy-optimized LLM judges to enable cross-app AI search and agentic queries, with LLM judge disagreement reduced from an ini…

What tools did this team use?

Dropbox Dash, BM25, MCP, DSPy, o3, CLIP-based models, NDCG, RAG, Jira.

What results were reported?

LLM judge human disagreement rate (initial prompt): 8%; MCP simple query latency: up to 45 seconds; Dash context window cap: about 100,000 tokens; Total prompts in Dash stack: over 30 (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Connect third-party apps → Content understanding → Knowledge graph modeling → Hybrid indexing → Personalized retrieval and ranking → LLM-as-judge evaluation → DSPy prompt optimization.

Related back office ops cases