Compliance monitoring · Production

Thumbtack uses a fine-tuned LLM to improve message review precision by 3.7x

The problem

Thumbtack's message review system struggled to detect subtle policy violations — nuanced language, sarcasm, and implied threats — that keyword rules and a CNN-based model could not reliably catch, limiting the platform's ability to protect service professionals at scale.

First attempt

An off-the-shelf LLM with prompt engineering achieved only an AUC of 0.56, far below production requirements, and the legacy CNN model also struggled with nuanced language, sarcasm, and implied threats.

Workflow diagram · grounded in source

Message sent on platform

trigger

“For each message sent on the platform, Thumbtack's message review pipeline runs two primary checks”

Rule-based engine check

validation

“Rule-based engine: This detects obvious policy violations by checking for flagged words or phrases”

CNN model pre-filter

ai_action

“By adjusting its threshold, we repurposed it as a pre-filter to identify and pass through most of the good messages without needing further review. As a result, the LLM only processes around 20% of the total messages”

Fine-tuned LLM reviews

ai_action

“the LLM only processes around 20% of the total messages, focusing its resources on the more challenging or suspicious cases”

Results resolver routes messages

routing

“The results from both systems are then aggregated in the results resolver. If the message is flagged as suspicious, it is forwarded to a manual review team for further investigation. If no issues are detected, the message is immediately …”

Manual review of flagged messages

human_review

“it is forwarded to a manual review team for further investigation”

Message delivered to professional

output

“If no issues are detected, the message is immediately passed along to the professional”

Reported outcome

The fine-tuned LLM reached an AUC of 0.93, with precision improving by a factor of 3.7 and recall improving 1.5 times over the old system.
Using the CNN as a pre-filter reduced LLM processing to around 20% of messages. The system has since processed tens of millions of messages.

Reported metrics

AUC — prompt engineering baseline0.56

AUC — fine-tuned LLM0.93

Precision improvement over old system3.7x

Recall improvement over old system1.5x

Show all 7 reported metrics

AUC — prompt engineering baseline0.56

AUC — fine-tuned LLM0.93

precision improvement over old system3.7x

recall improvement over old system1.5x

share of messages processed by LLMaround 20%

total messages processedtens of millions of messages

overall performance improvementnearly threefold

Reported stack

LLMCNNLangChain

Source

https://medium.com/thumbtack-engineering/using-genai-to-enhance-trust-and-safety-at-thumbtack-2b8355556f1f

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The fine-tuned LLM reached an AUC of 0.93, with precision improving by a factor of 3.7 and recall improving 1.5 times over the old system.

What tools did this team use?

LLM, CNN, LangChain.

What results were reported?

AUC — prompt engineering baseline: 0.56; AUC — fine-tuned LLM: 0.93; Precision improvement over old system: 3.7x; Recall improvement over old system: 1.5x (source-reported, not independently verified).

What failed first in this deployment?

How is this compliance monitoring AI workflow structured?

Message sent on platform → Rule-based engine check → CNN model pre-filter → Fine-tuned LLM reviews → Results resolver routes messages → Manual review of flagged messages → Message delivered to professional.

The fine-tuned LLM reached an AUC of 0.93, with precision improving by a factor of 3.7 and recall improving 1.5 times over the old system.Using the CNN as a pre-filter reduced LLM processing to around 20% of messages. The system has since processed tens of millions of messages.