Legal ops · Production

How Harvey Scaled Agent Development with Tool Bundles, No Custom Orchestration, and Eval Gates

The problem

Harvey's Assistant product hit a UX, engineering, and collaboration wall as new features accumulated: users weren't discovering Draft mode, multiple retrieval calls were complex to maintain behind a single interface, and new integrations had no clear path to launch. As more engineers touched the monolithic system prompt, their goals conflicted with no safe contribution model.

First attempt

Pre-agent bespoke orchestration meant engineers' goals could directly collide in the shared system prompt—one developer pushing for more tool recall and another for lower latency would overwrite each other's instructions, with no unit-testable boundary between them.

Workflow diagram · grounded in source
1
Developer creates Tool Bundle
trigger
“A Tool Bundle is an interface we designed that allows developers to package together new capabilities, which may be composed of multiple tools or sub-agents, into a single entity”
2
Tool Bundle injects system prompt
integration
“Tool Bundles give feature and integration developers the freedom to inject instructions into the main agent system prompt to achieve their capabilities without needing to make a request of the Assistant team. They also enable capabilitie…”
3
Agent calls tools in loop
ai_action
“We decided to adopt an external library, the OpenAI Agent SDK, instead of writing our own agent library. Compared to other frameworks, it explicitly left out an option to orchestrate code in a more "workflow" type format. Rather than bei…”
4
Leave-one-out eval gate
validation
“Tool Bundles and system prompt upgrades must pass leave-one-out gates to be deployed”
Reported outcome

Adopting a single agent framework with Tool Bundles and eval gates scaled Harvey's in-thread feature development from one team to four, led to emergent feature combinations, and enabled centralized eval.

Reported metrics
Engineering teams contributing to in-thread featuresfrom one team to four
Emergent feature combinationsemergent feature combinations
Reported stack
OpenAI Agent SDKAsk LexisNexisiManage
Source
https://www.harvey.ai/blog/principles-that-helped-us-scale-agent-development
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Adopting a single agent framework with Tool Bundles and eval gates scaled Harvey's in-thread feature development from one team to four, led to emergent feature combinations, and enabled centralized eval.

What tools did this team use?

OpenAI Agent SDK, Ask LexisNexis, iManage.

What results were reported?

Engineering teams contributing to in-thread features: from one team to four; Emergent feature combinations: emergent feature combinations (source-reported, not independently verified).

What failed first in this deployment?

Pre-agent bespoke orchestration meant engineers' goals could directly collide in the shared system prompt—one developer pushing for more tool recall and another for lower latency would overwrite each other's instructi…

How is this legal ops AI workflow structured?

Developer creates Tool Bundle → Tool Bundle injects system prompt → Agent calls tools in loop → Leave-one-out eval gate.

Related legal ops cases