Back office ops · Production

Nextdoor's path from pre-trained to fine-tuned embedding models for notifications, feed, and search ranking

The problem

Nextdoor needed richer content representations to capture nuanced user signals and improve personalization across products, while managing the high storage and serving costs of large fixed-dimensionality embeddings updated daily at scale.

First attempt

Pre-trained off-the-shelf models were trained on public benchmark datasets with semantics different from the Nextdoor domain, and their high fixed dimensionality caused significant storage and serving costs. Earlier word embedding models produced higher rates of null search queries.

Workflow diagram · grounded in source
1
Content text extraction
integration
“The text from content entities, viz. Nextdoor posts & comments, is extracted from post's subject and body and comment text respectively”
2
Multilingual entity embedding generation
ai_action
“fed into a multilingual text embedding model to derive respective entity embeddings for all countries Nextdoor operates in”
3
User embedding aggregation
ai_action
“historical interacted posts' embeddings are weighted aggregated based on interaction type to inform user (interaction) embedding. Ex: Active interaction such as post creation/comment/click would have higher weight compared to a more pass…”
4
Domain fine-tuning with user feedback
ai_action
“we built a two-tower framework to fine-tune embeddings with user feedback collected across Nextdoor surfaces while reducing dimensionality, customizing to our domain, and being cost effective”
5
Feature ingestion to FeatureStore
integration
“content based embedding features are ingested into our Featurestore in near real-time using task worker jobs as they get created/updated”
6
Similarity feature computation and logging
ai_action
“we then write logging code to compute and log the derived features such as cosine similarity and dot product between user & post, user & user”
7
Downstream ranking model promotion
output
“the model with best offline performance lift with new features is promoted for online AB test evaluation and ramping further towards majority member experience”
Reported outcome

Fine-tuned embedding models delivered significant performance lifts in OKR metrics for notifications and feed, reduced null query rates significantly, improved query expansion latencies by more than 10x, and improved user-post cosine similarity by up to 16% while reducing embedding dimensionality by more than 10x.

Reported metrics
Query expansion latency improvementmore than 10x
User-post cosine similarity improvementup to 16%
Embedding dimensionality reductionmore than 10x
product OKR metric performancesignificant performance lifts
Show all 6 reported metrics
query expansion latency improvementmore than 10x
user-post cosine similarity improvementup to 16%
embedding dimensionality reductionmore than 10x
product OKR metric performancesignificant performance lifts
null search query ratereduced significantly
engagement metrics (CTR/sessions/contributions/DAU/WAU)promising lifts
Reported stack
Sentence-BERTSBERTpytorchSageMakerFeatureStoreAirflowHSNWlibBERTopicCLIP
Source
https://engblog.nextdoor.com/from-pre-trained-to-fine-tuned-nextdoors-path-to-effective-embedding-applications-3a13b56d91aa
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Fine-tuned embedding models delivered significant performance lifts in OKR metrics for notifications and feed, reduced null query rates significantly, improved query expansion latencies by more than 10x, and improved…

What tools did this team use?

Sentence-BERT, SBERT, pytorch, SageMaker, FeatureStore, Airflow, HSNWlib, BERTopic, CLIP.

What results were reported?

Query expansion latency improvement: more than 10x; User-post cosine similarity improvement: up to 16%; Embedding dimensionality reduction: more than 10x; product OKR metric performance: significant performance lifts (source-reported, not independently verified).

What failed first in this deployment?

Pre-trained off-the-shelf models were trained on public benchmark datasets with semantics different from the Nextdoor domain, and their high fixed dimensionality caused significant storage and serving costs.

How is this back office ops AI workflow structured?

Content text extraction → Multilingual entity embedding generation → User embedding aggregation → Domain fine-tuning with user feedback → Feature ingestion to FeatureStore → Similarity feature computation and logging → Downstream ranking model promotion.

Related back office ops cases