Recruiting · Production

Upwork builds Uma with in-house custom-trained AI to power agentic hiring workflows

The problem

Off-the-shelf LLMs perform inconsistently in production for domain-specific hiring workflows, failing on edge cases and lacking the platform-specific context needed to support business-critical tasks at Upwork.

First attempt

Generic off-the-shelf LLMs proved insufficient for Upwork's use case—they generalized too broadly, produced inconsistent outputs on minor input variations, and could not encode the hiring domain knowledge Upwork required.

Workflow diagram · grounded in source
1
Client initiates hiring need
trigger
“Uma can now take meaningful action on behalf of clients and talent to move hiring and collaboration forward”
2
AI crafts job post
ai_action
“crafting effective job posts using platform insights”
3
AI surfaces talent matches
ai_action
“surfacing top matches”
4
AI coordinates interviews
ai_action
“Uma actively interprets the client's project needs, and based on that analysis, will call tools to identify talent that is a strong fit, reach out to qualified freelancers, and run real-time interviews”
5
AI evaluates candidates via chain-of-thought
ai_action
“instead of instantly ranking a set of freelancer profiles, Uma works through a structured reasoning process—considering factors like past job success, relevant skills, responsiveness, and client preferences—before generating a shortlist”
6
Tailored recommendations delivered
output
“delivering tailored recommendations”
7
RAG-based Q&A for user questions
ai_action
“Uma's Q&A model works hand-in-hand with the custom RAG tools and our in-house intent recognition systems to ensure the user is getting the most relevant information they need”
8
Continuous model retraining loop
feedback_loop
“Analytics workflows provide fast feedback on how models are performing in public with the help of autonomous AI agents that provide customizable evaluation”
Reported outcome

Uma now handles the full hiring flow agenically—crafting job posts, surfacing matches, coordinating interviews, evaluating candidates, and delivering recommendations—with custom fine-tuned models that substantially outperform off-the-shelf alternatives in production.

Reported metrics
Inference cost reduction vs off-the-shelf models10x or more
Time to deploy new model versionwithin a few days
Synthetic dataset size increase vs human-only databy orders of magnitude
Content and style accuracy improvement from custom trainingdramatically better results in both content and style accuracy
Reported stack
UmaRAGchain-of-thoughtLlamaMistralQwen
Source
https://www.upwork.com/blog/why-upwork-relies-on-in-house-ai-research-to-power-uma-upworks-mindful-ai
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Uma now handles the full hiring flow agenically—crafting job posts, surfacing matches, coordinating interviews, evaluating candidates, and delivering recommendations—with custom fine-tuned models that substantially ou…

What tools did this team use?

Uma, RAG, chain-of-thought, Llama, Mistral, Qwen.

What results were reported?

Inference cost reduction vs off-the-shelf models: 10x or more; Time to deploy new model version: within a few days; Synthetic dataset size increase vs human-only data: by orders of magnitude; Content and style accuracy improvement from custom training: dramatically better results in both content and style accuracy (source-reported, not independently verified).

What failed first in this deployment?

Generic off-the-shelf LLMs proved insufficient for Upwork's use case—they generalized too broadly, produced inconsistent outputs on minor input variations, and could not encode the hiring domain knowledge Upwork requi…

How is this recruiting AI workflow structured?

Client initiates hiring need → AI crafts job post → AI surfaces talent matches → AI coordinates interviews → AI evaluates candidates via chain-of-thought → Tailored recommendations delivered → RAG-based Q&A for user questions → Continuous model retraining loop.

Related recruiting cases