Marketing ops · Production

Identify User Journeys at Pinterest Using Dynamic Keyword Extraction and ML Ranking

The problem

Pinterest needed to move beyond understanding users' immediate interests to comprehend their underlying, long-term goals, so that recommendations could assist users in achieving those goals rather than only surfacing transient interests.

Workflow diagram · grounded in source
1
User activity data ingestion
trigger
“We leverage a rich set of user data, including: — User search history: Aggregated queries and timestamps. — User activity history: Interactions like Pin closeups, repins, and clickthroughs, extract the annotations and interests from the …”
2
Keyword hierarchical clustering
ai_action
“we adopt the pretrained text embedding for the keywords to perform hierarchical clustering to form journey clusters”
3
Journey naming via ranking model
ai_action
“The current production model is to apply a ranking model to pick the top keyword extracted from each cluster as the journey name. It balances personalization and simplicity by choosing the most relevant keywords from the cluster. We are …”
4
Journey point-wise ranking
ai_action
“our initial approach is to build a point-wise ranking model. We get labels from user email feedback and human annotation. The model takes user features, engagement features (how frequently the user engaged on this journey through search,…”
5
Journey diversification
ai_action
“To prevent the top ranked journeys from always being similar, we implement a diversifier after the journey ranking stage. The most straightforward approach is to apply a penalty if the journey is similar to the journeys that ranked higher”
6
Journey stage prediction
ai_action
“Journeys are categorized based on user engagement patterns and activity duration. If users engage with a journey consistently over an extended period, we classify it as "Evergreen" — these journeys remain perpetually active. In contrast,…”
7
LLM relevance evaluation
validation
“we leverage LLMs to assess the relevance of predicted user journeys. By providing user features and engagement history, we ask the LLM to generate a 5-level score with explanations. We have validated that LLM judgments closely correlate …”
8
Journey-aware notification delivery
output
“We applied user journeys inference to deliver notifications related to the user's ongoing journeys.”
Reported outcome

Journey-aware notifications delivered statistically significant gains in user engagement, including an 88% higher email click rate and a 32% higher push open rate compared to interest-based notifications, and a 23% increase in positive feedback from user surveys.

Reported metrics
Email click rate vs interest-based notifications88% higher
Push open rate vs interest-based notifications32% higher
Positive feedback rate from user surveys23% increase
User engagement gainsstatistically significant gains in user engagements
Reported stack
SearchSageLLMsQwenRay
Source
https://medium.com/pinterest-engineering/identify-user-journeys-at-pinterest-b517f6275b42
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Journey-aware notifications delivered statistically significant gains in user engagement, including an 88% higher email click rate and a 32% higher push open rate compared to interest-based notifications, and a 23% in…

What tools did this team use?

SearchSage, LLMs, Qwen, Ray.

What results were reported?

Email click rate vs interest-based notifications: 88% higher; Push open rate vs interest-based notifications: 32% higher; Positive feedback rate from user surveys: 23% increase; User engagement gains: statistically significant gains in user engagements (source-reported, not independently verified).

How is this marketing ops AI workflow structured?

User activity data ingestion → Keyword hierarchical clustering → Journey naming via ranking model → Journey point-wise ranking → Journey diversification → Journey stage prediction → LLM relevance evaluation → Journey-aware notification delivery.

Related marketing ops cases