Back office ops · Production

Perplexity AI serves 435 million search queries a month using NVIDIA H100 GPUs, Triton Inference Server, and TensorRT-LLM

The problem

Perplexity AI's inference team faced increasing pressure to provision the hardware and software needed to serve hundreds of millions of AI-powered search queries each month while simultaneously balancing cost efficiency with optimal user experience.

Workflow diagram · grounded in source

User search query arrives

trigger

“handles more than 435 million queries each month. Each query represents multiple AI inference requests”

Classifier detects user intent

ai_action

“the company relies on smaller classifier models that help determine user intent. User tasks detected by the classifiers, like text completion, are then routed to specific models deployed on GPU pods”

Front-end scheduler routes to pod

routing

“a front-end scheduler built in-house that routes traffic to the appropriate pod based on their load and usage, ensuring that the SLAs are consistently met”

Triton serves model inference on H100

ai_action

“Triton Inference Server is a critical component of Perplexity's deployment architecture. It serves optimized models across various backends, batches incoming user requests, and provides GPU utilization metrics to the scheduler. This supp…”

Search results and answers output

output

“serve a wide range of requests—spanning search, summarization, and question answering, among others”

Reported outcome

Perplexity AI serves more than 435 million queries per month across over 20 simultaneous AI models under strict SLAs, and saved approximately $1 million annually by self-hosting models for the Related-Questions feature on cloud-hosted NVIDIA GPUs rather than using third-party LLM provider APIs.

Reported metrics

Monthly queries handledmore than 435 million queries each month

simultaneous AI models servedover 20 AI models simultaneously

annual cost savings on Related-Questions featureapproximately $1 million annually

Reported stack

NVIDIA H100 Tensor Core GPUsNVIDIA Triton Inference ServerNVIDIA TensorRT-LLMKubernetesLlama 3.1CUDA kernels

Source

https://developer.nvidia.com/blog/spotlight-perplexity-ai-serves-400-million-search-queries-a-month-using-nvidia-inference-stack

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Perplexity AI serves more than 435 million queries per month across over 20 simultaneous AI models under strict SLAs, and saved approximately $1 million annually by self-hosting models for the Related-Questions featur…

What tools did this team use?

NVIDIA H100 Tensor Core GPUs, NVIDIA Triton Inference Server, NVIDIA TensorRT-LLM, Kubernetes, Llama 3.1, CUDA kernels.

What results were reported?

Monthly queries handled: more than 435 million queries each month; simultaneous AI models served: over 20 AI models simultaneously; annual cost savings on Related-Questions feature: approximately $1 million annually (source-reported, not independently verified).

How is this back office ops AI workflow structured?

User search query arrives → Classifier detects user intent → Front-end scheduler routes to pod → Triton serves model inference on H100 → Search results and answers output.