Quality assurance · Production

Factory Signals uses LLMs to detect user friction and enable recursive self-improvement in Droid

The problem

Traditional product analytics captured what happened in sessions (duration, tool calls, completion rates) but missed user experience quality — a session could score as a success by metrics while containing significant user frustration from repeated rephrasing loops. Scaling human review of thousands of daily sessions to close this gap was impractical, and reading raw sessions raised privacy concerns.

First attempt

Traditional metrics gave misleading quality signals by scoring frustrated sessions as successes, and human review of raw sessions was both unscalable and privacy-invasive.

Workflow diagram · grounded in source
1
Daily batch session fetch
trigger
“Signals runs as a daily batch process designed for scale and cost efficiency. Sessions from the past twenty-four hours get fetched from BigQuery”
2
Filter for meaningful sessions
validation
“filtered to those with at least thirty agentic steps to ensure meaningful interactions”
3
Facet extraction
ai_action
“Every session gets decomposed into structured metadata. We call these facets: the programming languages involved, the primary intent, how many tool calls were confirmed, whether the session ended in success or abandonment, what framework…”
4
Friction pattern analysis
ai_action
“The friction analyzer scans for patterns that indicate user struggle: error events, repeated rephrasing, escalation in tone, tool calls rejected by the user, and more. Each friction moment gets a severity rating and abstracted citations …”
5
Delight moment identification
ai_action
“Signals doesn't just find problems. It finds moments where Factory genuinely impressed users. Positive exclamations, first-attempt successes on complex tasks, explicit mentions of time saved, rapid approval flows followed by appreciation.”
6
Category evolution via clustering
ai_action
“The facet schema itself evolves over time through semantic clustering. As Signals processes batches of sessions, it generates embeddings for each session's abstracted summary and clusters similar sessions together. The LLM then analyzes …”
7
Results output to BigQuery and Slack
output
“Results flow to BigQuery for historical analysis and to Slack for daily reports.”
8
Threshold-triggered ticket filing
integration
“When patterns cross a threshold, it files Linear tickets automatically.”
9
Droid self-assigns and fixes
ai_action
“Droid picks up those tickets, implements fixes, and reviews its own PRs.”
10
Human PR approval
human_review
“A human still approves the PR before merge.”
Reported outcome

Signals processes thousands of sessions daily and powers a self-improving loop where 73% of issues are auto-resolved with an average fix time of <4h.
The repeated rephrasing friction rate dropped by thirty percent within forty-eight hours of a targeted fix, surfaced without anyone reading individual user sessions.

Reported metrics
Sessions with friction58%
Sessions with delight83%
Average friction events per session1.3
Average delight events per session1.4
Show all 10 reported metrics
sessions with friction58%
sessions with delight83%
average friction events per session1.3
average delight events per session1.4
issues auto-resolved73%
average time to fix<4h
repeated rephrasing friction rate reductiondropped by thirty percent within forty-eight hours
daily sessions analyzed1,946
rephrasing-again probability after three rephrasesroughly a forty percent chance
human approval steps in self-improvement loop1
Reported stack
SignalsLLMsDroidBigQuerySlackLinearOpenAI's batch API
Source
https://factory.ai/news/factory-signals
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Signals processes thousands of sessions daily and powers a self-improving loop where 73% of issues are auto-resolved with an average fix time of <4h.

What tools did this team use?

Signals, LLMs, Droid, BigQuery, Slack, Linear, OpenAI's batch API.

What results were reported?

Sessions with friction: 58%; Sessions with delight: 83%; Average friction events per session: 1.3; Average delight events per session: 1.4 (source-reported, not independently verified).

What failed first in this deployment?

Traditional metrics gave misleading quality signals by scoring frustrated sessions as successes, and human review of raw sessions was both unscalable and privacy-invasive.

How is this quality assurance AI workflow structured?

Daily batch session fetch → Filter for meaningful sessions → Facet extraction → Friction pattern analysis → Delight moment identification → Category evolution via clustering → Results output to BigQuery and Slack → Threshold-triggered ticket filing → Droid self-assigns and fixes → Human PR approval.

Related quality assurance cases