Factory Signals uses LLMs to detect user friction and enable recursive self-improvement in Droid
Traditional product analytics captured what happened in sessions (duration, tool calls, completion rates) but missed user experience quality — a session could score as a success by metrics while containing significant user frustration from repeated rephrasing loops. Scaling human review of thousands of daily sessions to close this gap was impractical, and reading raw sessions raised privacy concerns.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Daily batch session fetch
Signals runs as a daily batch process, fetching sessions from the past twenty-four hours from BigQuery.
Tools used
SignalsLLMsDroidBigQuerySlackLinearOpenAI's batch API
Outcome
Signals processes thousands of sessions daily and powers a self-improving loop where 73% of issues are auto-resolved with an average fix time of <4h. The repeated rephrasing friction rate dropped by thirty percent within forty-eight hours of a targeted fix, surfaced without anyone reading individual user sessions.
What failed first
Traditional metrics gave misleading quality signals by scoring frustrated sessions as successes, and human review of raw sessions was both unscalable and privacy-invasive.