Nextdoor CoreML team achieves 4x latency reduction and 3x throughput increase by tuning ML inference in shared hosting environments
Running ML inference in shared hosting environments (ECS, K8s) introduces unobvious pitfalls that significantly impact latency and throughput.
Nextdoor's ML team experienced latency and throughput degradation traced to poor request queue management and suboptimal OpenMP parameter configuration before discovering and resolving the root causes.
After addressing request queue management and OpenMP parameter tuning, Nextdoor achieved a factor of 4 latency reduction, 3x throughput increase, and CPU utilization improvement from 10% to 50% while maintaining model performance.
Show all 5 reported metrics
Frequently asked questions
What did this team achieve with this AI workflow?
After addressing request queue management and OpenMP parameter tuning, Nextdoor achieved a factor of 4 latency reduction, 3x throughput increase, and CPU utilization improvement from 10% to 50% while maintaining model…
What tools did this team use?
ECS, K8s, OpenMP.
What results were reported?
Latency reduction: factor of 4; Throughput increase: 3x; CPU utilization improvement: CPU 10% -> 50%; real-time ML microservices in production: 30+ (source-reported, not independently verified).
What failed first in this deployment?
Nextdoor's ML team experienced latency and throughput degradation traced to poor request queue management and suboptimal OpenMP parameter configuration before discovering and resolving the root causes.
How is this back office ops AI workflow structured?
ML inference request arrives → Load balancer routes request → Request queue timeout management → OpenMP parameter tuning → ML inference result delivered.