Marketing ops · Production

Nextdoor increases email engagement with generative AI and rejection sampling

The problem

Nextdoor's notification email subject lines defaulted to the first few words of posts, which were often uninformative greetings. Off-the-shelf ChatGPT API produced subject lines that were less engaging, inauthentic, and prone to hallucination.

First attempt

Initial attempts using ChatGPT API with prompt engineering produced subject lines that underperformed user-generated ones in A/B tests; even after multiple iterations, results remained inferior to the control, and the model hallucinated irrelevant content.

Workflow diagram · grounded in source
1
Post selected for notification email
trigger
“we need to determine a subject line of the email for the email audiences”
2
OpenAI API extracts candidate subject
ai_action
“we require that OpenAI API extracts the most interesting part of the post without making any change. This way of extracting user content provides multiple benefits: First, it removes hallucinations. Second, it keeps the subject line auth…”
3
Reward model scores candidate
ai_action
“Given a subject line and the post content, this model predicts if the given subject line would be the better subject line than the user-generated subject line”
4
Rejection sampling selects subject
routing
“accept it only if its reward model score is higher than the user-written subject line's score”
5
Notification email sent
output
“we generate a single subject line for each post”
6
Daily accuracy monitoring and retraining
feedback_loop
“We monitor the reward model's predictive performance daily, using the next day's user clicks after the training phase as the ground truth to compare with the model's output”
Reported outcome

The final system increased Sessions by 3% over the prompt-only approach, raised Weekly Active Users by 0.4%, and grew Ads revenue by 1%, while cutting serving cost to 1/600 via caching.

Reported metrics
Sessions improvement (extraction vs rewrite prompt)3%
Weekly Active Users increase0.4%
Ads revenue increase1%
Reward model accuracyabout 65%
Show all 5 reported metrics
Sessions improvement (extraction vs rewrite prompt)3%
Weekly Active Users increase0.4%
Ads revenue increase1%
Reward model accuracyabout 65%
Serving cost reduction via caching1/600 compared to brute-force
Reported stack
OpenAI APIChatGPT APITenacity
Source
https://engblog.nextdoor.com/let-ai-entertain-you-increasing-user-engagement-with-generative-ai-and-rejection-sampling-50a402264f56
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The final system increased Sessions by 3% over the prompt-only approach, raised Weekly Active Users by 0.4%, and grew Ads revenue by 1%, while cutting serving cost to 1/600 via caching.

What tools did this team use?

OpenAI API, ChatGPT API, Tenacity.

What results were reported?

Sessions improvement (extraction vs rewrite prompt): 3%; Weekly Active Users increase: 0.4%; Ads revenue increase: 1%; Reward model accuracy: about 65% (source-reported, not independently verified).

What failed first in this deployment?

Initial attempts using ChatGPT API with prompt engineering produced subject lines that underperformed user-generated ones in A/B tests; even after multiple iterations, results remained inferior to the control, and the…

How is this marketing ops AI workflow structured?

Post selected for notification email → OpenAI API extracts candidate subject → Reward model scores candidate → Rejection sampling selects subject → Notification email sent → Daily accuracy monitoring and retraining.

Related marketing ops cases