Quality assurance · Production

New Computer improves Dot's memory retrieval by 50% recall and 40% precision with LangSmith

The problem

New Computer needed to rapidly iterate on diverse memory retrieval methods for Dot's agentic memory system while preserving user privacy, facing a combinatorial explosion of experiments as they tested multiple retrieval techniques in parallel.

First attempt

The initial baseline used simple semantic search retrieving a fixed number of memories per query, which proved insufficient across diverse query types where BM25 or meta-field pre-filtering performed better.

Workflow diagram · grounded in source
1
Synthetic user cohort generation
ai_action
“they generated synthetic data by creating a cohort of synthetic users with LLM-generated backstories”
2
Memory seeding and dataset creation
trigger
“After an initial conversation to seed the memory database for each synthetic user, the team began storing queries (messages by synthetic users) along with the full set of available memories in a LangSmith dataset”
3
Memory labeling and metric definition
validation
“the New Computer team labeled relevant memories for each query and defined evaluation metrics like precision, recall and F1”
4
Multi-method retrieval experiments
ai_action
“In some cases, similarity search or keyword methods like BM25 worked better; in others, these methods required some pre-filtering by meta-fields in order to perform effectively”
5
Prompt iteration and comparison
feedback_loop
“they were then able to easily inspect the global effects of prompt changes in LangSmith's experiment comparison view. This let them identify regressed runs derived from prompt changes in a highly-visual manner. In addition, in failure ca…”
Reported outcome

New Computer achieved 50% higher recall and 40% higher precision compared to their baseline dynamic memory retrieval, and greatly improved team iteration speed for evaluating and adjusting conversation prompts.

Reported metrics
Memory recall improvement50% higher recall
Memory precision improvement40% higher precision
Paid tier conversionmore than 45%
Team iteration speedgreatly improved iteration speed
Reported stack
LangSmithBM25
Source
https://blog.langchain.dev/customers-new-computer/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

New Computer achieved 50% higher recall and 40% higher precision compared to their baseline dynamic memory retrieval, and greatly improved team iteration speed for evaluating and adjusting conversation prompts.

What tools did this team use?

LangSmith, BM25.

What results were reported?

Memory recall improvement: 50% higher recall; Memory precision improvement: 40% higher precision; Paid tier conversion: more than 45%; Team iteration speed: greatly improved iteration speed (source-reported, not independently verified).

What failed first in this deployment?

The initial baseline used simple semantic search retrieving a fixed number of memories per query, which proved insufficient across diverse query types where BM25 or meta-field pre-filtering performed better.

How is this quality assurance AI workflow structured?

Synthetic user cohort generation → Memory seeding and dataset creation → Memory labeling and metric definition → Multi-method retrieval experiments → Prompt iteration and comparison.

Related quality assurance cases