Customer support · Production

How Thumbtack created a generative AI strategy across search, policy review, and developer productivity

The problem

As LLMs became transformative in 2023, Thumbtack needed a comprehensive gen AI strategy beyond their early LLM search work, including resolving open questions about infrastructure, hallucinations, and democratizing internal access without duplicating efforts.

Workflow diagram · grounded in source
1
Customer natural language search
trigger
“our search experience which lets you describe your project or problem in conversational, everyday language”
2
LLM query understanding and matching
ai_action
“Large language models (LLMs) could help us better understand customer searches and match customers with a pro that fits their needs”
3
ML-based policy violation detection
ai_action
“rules/machine learning models that could detect such violations and flag it for manual human review (for verification purposes)”
4
Gen AI augments violation review
ai_action
“using gen AI to detect Thumbtack policy violations resulted in significant efficiency gains”
5
Human review of flagged violations
human_review
“processes that review potential policy violations had humans in the loop, false positives and hallucinations were less of a concern, and we could more quickly productize the process”
6
Unstructured data insight extraction
ai_action
“Having gen AI as a tool for exploratory analysis of unstructured data has been immensely valuable to data scientists working on problems like mining product reviews for customer-pro interaction insights”
7
GitHub Copilot developer pilot
ai_action
“We decided to launch a pilot program for GitHub Copilot usage. We wanted to measure both adoption and get signals around productivity improvements.”
Reported outcome

Gen AI for policy violation detection delivered significant efficiency gains, gen AI for exploratory analysis of unstructured data proved immensely valuable to data scientists, and a GitHub Copilot pilot was launched to measure developer productivity improvements.

Reported metrics
Policy violation review efficiencysignificant efficiency gains
data scientist productivity from gen AI analysisimmensely valuable
Reported stack
GPT-4LLama 2GitHub Copilot
Source
https://medium.com/thumbtack-engineering/how-we-created-a-generative-ai-strategy-at-thumbtack-e7ab95f8006f
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Gen AI for policy violation detection delivered significant efficiency gains, gen AI for exploratory analysis of unstructured data proved immensely valuable to data scientists, and a GitHub Copilot pilot was launched…

What tools did this team use?

GPT-4, LLama 2, GitHub Copilot.

What results were reported?

Policy violation review efficiency: significant efficiency gains; data scientist productivity from gen AI analysis: immensely valuable (source-reported, not independently verified).

How is this customer support AI workflow structured?

Customer natural language search → LLM query understanding and matching → ML-based policy violation detection → Gen AI augments violation review → Human review of flagged violations → Unstructured data insight extraction → GitHub Copilot developer pilot.

Related customer support cases