Back office ops · Production

Harvey: Resilient AI Infrastructure for Scaling and Managing Model Performance Across Millions of Daily Requests

The problem

Harvey needed to reliably manage bursty computational load across multiple AI model deployments serving millions of daily requests, while enabling fast onboarding of new model versions and providing granular real-time attribution of every model call.

Workflow diagram · grounded in source
1
Request enters centralized client library
trigger
“At Harvey we have written a centralized Python library that abstracts all model interactions for both the product and our developers”
2
Model endpoint routing and selection
routing
“it picks the right model to forward the request to based on the request's model family. The system verifies which deployments within a model family are healthy and meet our service reliability thresholds ("SRT"), then selects a model dep…”
3
SLI health check and weight adjustment
validation
“we measure latency and success rate Service Level Indicators (SLI) periodically. If a model endpoint is not healthy enough to satisfy our SLA, the system reduces its weight during the weighted selection process”
4
Quota and rate limiting check
validation
“we implemented a distributed, feature-aware rate limiting system utilizing a Redis-backed approximate sliding-window token bucket algorithm”
5
Model proxy forwarding
integration
“a thin proxy that forwards all model requests made outside of our Kubernetes cluster back through the cluster to the model servers”
6
Observability and burn rate alerting
feedback_loop
“we have very strict burn rate alerts around our SLAs to ensure a rapid response by the team. Another critical piece of information that we track is detailed accounting of every prompt and output token consumed by our system. This data is…”
Reported outcome

Harvey achieved high availability across all model deployments through layered fallbacks and retries, a distributed rate limiter that handles bursty traffic without significant impact on throughput or latency, and runtime reconfiguration of limits across all geographically deployed clusters without restart and in just seconds.

Reported metrics
Daily request volumemillions of daily requests
Prompt tokens consumedbillions of prompt tokens
Output tokens producedhundreds of millions of output tokens
Bursty traffic impact on throughput and latencywithout a significant impact on request throughput or latency
Show all 5 reported metrics
daily request volumemillions of daily requests
prompt tokens consumedbillions of prompt tokens
output tokens producedhundreds of millions of output tokens
bursty traffic impact on throughput and latencywithout a significant impact on request throughput or latency
runtime reconfiguration speedwithout any restart and in just seconds
Reported stack
PythonRedisKubernetesSnowflakeOpenAI API
Source
https://www.harvey.ai/blog/resilient-ai-infrastructure
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Harvey achieved high availability across all model deployments through layered fallbacks and retries, a distributed rate limiter that handles bursty traffic without significant impact on throughput or latency, and run…

What tools did this team use?

Python, Redis, Kubernetes, Snowflake, OpenAI API.

What results were reported?

Daily request volume: millions of daily requests; Prompt tokens consumed: billions of prompt tokens; Output tokens produced: hundreds of millions of output tokens; Bursty traffic impact on throughput and latency: without a significant impact on request throughput or latency (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Request enters centralized client library → Model endpoint routing and selection → SLI health check and weight adjustment → Quota and rate limiting check → Model proxy forwarding → Observability and burn rate alerting.

Related back office ops cases