Customer support · Production

How AI text generation models are reshaping customer support at Airbnb

The problem

Scaling AI for Airbnb's customer support was difficult due to long-tail corner cases, the high cost of human data labeling, and the limitations of traditional classifiers that could not scale to exhaustive intent taxonomy design.

First attempt

The content recommendation ranker previously used XLMRoBERTa, which was replaced by a generative model. The initial paraphrase model generated bland, generic replies regardless of the specific user input.

Workflow diagram · grounded in source
1
User issue submitted to ranker
trigger
“This pointwise ranker takes the textual representation of two pieces of input — the current user's issue description and the candidate document, in the form of its title, summary, and keywords.”
2
MT5 generative model scores relevance
ai_action
“To enable personalization, we expanded the issue description input with textual representations of the user and their reservation information. We fine-tuned the MT5 model on the task described above.”
3
Ranked documents delivered to Help Center
output
“integrating the model into Airbnb's Help Center, which has millions of active users”
4
Agent conversation captured
trigger
“We concatenate multiple rounds of user-agent conversations to leverage chat history as input text and then ask the prompt we care about at the point in time of serving.”
5
QA model detects user intent
ai_action
“we developed a mastermind Question-Answering (QA) model, aiming to help answer all related questions”
6
Templates recommended to agent
output
“Based on the model's answer, relevant templates are then recommended to agents”
7
Chatbot paraphrase model
ai_action
“Building a paraphrase model, which first rephrases the problem a user describes, can give users some confidence and confirm that the bot's understanding is correct. This has significantly improved our bot's engagement rate.”
8
Training data filtered for quality
feedback_loop
“We labeled all clusters that are too generic and used Sentence-Transformers to filter them out from the training data. This approach worked significantly better and gave us a high-quality model to put into production.”
Reported outcome

Airbnb's generative AI models significantly improved content recommendation ranking performance, drove large engagement rate improvements for CS ambassadors using the agent assistant, and significantly improved chatbot engagement rates through paraphrasing.

Reported metrics
Support document ranking performancesignificant improvements in the key performance metric for support document ranking
Document relevance vs. classification baselinesignificantly higher relevance
CS ambassador engagement ratelarge engagement rate improvement
Chatbot engagement ratesignificantly improved
Show all 5 reported metrics
support document ranking performancesignificant improvements in the key performance metric for support document ranking
document relevance vs. classification baselinesignificantly higher relevance
CS ambassador engagement ratelarge engagement rate improvement
chatbot engagement ratesignificantly improved
business metrics (A/B test)significant business metric improvement
Reported stack
MT5XLMRoBERTaDeepSpeedt5-baseNarrativaT5BARTPEGASUSGPT2Sentence-Transformers
Source
https://medium.com/airbnb-engineering/how-ai-text-generation-models-are-reshaping-customer-support-at-airbnb-a851db0b4fa3
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Airbnb's generative AI models significantly improved content recommendation ranking performance, drove large engagement rate improvements for CS ambassadors using the agent assistant, and significantly improved chatbo…

What tools did this team use?

MT5, XLMRoBERTa, DeepSpeed, t5-base, Narrativa, T5, BART, PEGASUS, GPT2, Sentence-Transformers.

What results were reported?

Support document ranking performance: significant improvements in the key performance metric for support document ranking; Document relevance vs. classification baseline: significantly higher relevance; CS ambassador engagement rate: large engagement rate improvement; Chatbot engagement rate: significantly improved (source-reported, not independently verified).

What failed first in this deployment?

The content recommendation ranker previously used XLMRoBERTa, which was replaced by a generative model.

How is this customer support AI workflow structured?

User issue submitted to ranker → MT5 generative model scores relevance → Ranked documents delivered to Help Center → Agent conversation captured → QA model detects user intent → Templates recommended to agent → Chatbot paraphrase model → Training data filtered for quality.

Related customer support cases