It support · Production

Moveworks Copilot achieves 2.3x+ throughput and 2.35x latency improvement with NVIDIA TensorRT-LLM

The problem

LLM processing delays created frustrating lags in the Moveworks Copilot's conversational flow, disrupting employee productivity and limiting the system's ability to scale efficiently on existing infrastructure.

Workflow diagram · grounded in source
1
Employee submits question
trigger
“The lag between user questions and AI responses, even a whisper of a delay, can shatter the magic of conversational AI”
2
TensorRT-LLM optimized inference
ai_action
“Custom-built CUDA kernels make LLM operations exceptionally efficient, outperforming native Hugging Face models”
3
Streamed token output
output
“many LLM-driven applications stream output from the LLM to users one token at a time as the model generates the response. With a lower first token latency, users don't have to wait for the entire response to be fully generated before see…”
Reported outcome

With NVIDIA TensorRT-LLM, the Moveworks Copilot achieved 44 tokens per second (up from 19), average request latency of 1.5 seconds (down from 3.4 seconds), and first token latency of 0.3 seconds (down from 0.8 seconds), enabling smoother conversational flow and more efficient infrastructure utilization.

Reported metrics
tokens per second (with TensorRT-LLM)44 tokens per second
Tokens per second throughput improvementover 2x tokens per second
Tokens per second improvement multiplier2.32x
Average request latency (before and after)from 3.4 seconds to 1.5 seconds
Show all 7 reported metrics
tokens per second (with TensorRT-LLM)44 tokens per second
tokens per second throughput improvementover 2x tokens per second
tokens per second improvement multiplier2.32x
average request latency (before and after)from 3.4 seconds to 1.5 seconds
average request latency improvement multiplier2.35x
first token latency (before and after)from 0.8 seconds to 0.3 seconds
first token latency improvement multiplier2.7x
Reported stack
Flash-DecodingSmoothQuant
Source
https://www.moveworks.com/us/en/resources/blog/moveworks-achieves-low-latency-with-nvidia-tensorrt-llm
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

With NVIDIA TensorRT-LLM, the Moveworks Copilot achieved 44 tokens per second (up from 19), average request latency of 1.5 seconds (down from 3.4 seconds), and first token latency of 0.3 seconds (down from 0.8 seconds…

What tools did this team use?

Flash-Decoding, SmoothQuant.

What results were reported?

tokens per second (with TensorRT-LLM): 44 tokens per second; Tokens per second throughput improvement: over 2x tokens per second; Tokens per second improvement multiplier: 2.32x; Average request latency (before and after): from 3.4 seconds to 1.5 seconds (source-reported, not independently verified).

How is this it support AI workflow structured?

Employee submits question → TensorRT-LLM optimized inference → Streamed token output.

Related it support cases