Prior authorization · Production

The Sour Lesson: Building AI Products That Compound with Model Progress at Anterior

The problem

Building AI products on rapidly evolving LLMs creates a tension: serving customers today requires working around current model limitations, but those workarounds become technical debt when capabilities improve.

First attempt

Hierarchical query reasoning—breaking medical guidelines into tree-structured sub-questions answerable within a 16K token window—became unnecessary when GPT-4 Turbo launched with a 128K context window. Finetuning for clinical reasoning was similarly superseded within 12-18 months as frontier general models became strong enough to eliminate the advantage of specialized models; the hierarchical approach itself was replaced within 6 months.

Workflow diagram · grounded in source
1
Medical review case arrives
trigger
“A typical medical review requires processing: Clinical guidelines (often 20-50 pages, and with complex nested logic) - Patient medical records (can be 100+ pages)”
2
Guidelines decomposed into question tree
ai_action
“Parse guidelines into a tree of dependent questions, from high-level requirements down to specific clinical criteria / Answer each question separately against the medical record, starting with leaf nodes and working up the tree”
3
Final determination propagated
output
“Propagate answers up through parent nodes to make a final determination”
4
Clinical knowledge dynamically injected
integration
“building a scalable library of clinical knowledge which is dynamically injected at inference time based on the context the model is operating within”
5
Clinicians review outputs via dashboard
human_review
“I built a dashboard to enable domain experts (clinicians) to perform reviews”
6
Review data feeds product improvement
feedback_loop
“designed a system downstream that used the data we get from reviews in various ways: Provide performance metrics that we can use to prioritize improvement work / Identify and classify 'failure modes' (the ways in which our AI was making …”
Reported outcome

Domain knowledge injection and the expert review system both remained in production 2+ years after being built.
The earlier hierarchical approach, while short-lived, helped acquire the first 2 enterprise customers within 1-2 months.

Reported metrics
Original model context window16K tokens
GPT-4 Turbo context window128K context window
Time to build hierarchical query reasoning approach1-2 months
Enterprise customers acquired via first approach2
Show all 8 reported metrics
original model context window16K tokens
GPT-4 Turbo context window128K context window
time to build hierarchical query reasoning approach1-2 months
enterprise customers acquired via first approach2
time before hierarchical approach was replacedwithin 6 months
time for general models to surpass specialized finetuned models12-18 months
domain knowledge injection production lifespan2+ years
recommended effort split optimization vs experimentationrough 70%:30% split
Reported stack
GPT-3.5-Turbo-16kGPT-4 TurboGPT-4Claude 3
Source
https://chrislovejoy.me/sour-lesson
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Domain knowledge injection and the expert review system both remained in production 2+ years after being built.

What tools did this team use?

GPT-3.5-Turbo-16k, GPT-4 Turbo, GPT-4, Claude 3.

What results were reported?

Original model context window: 16K tokens; GPT-4 Turbo context window: 128K context window; Time to build hierarchical query reasoning approach: 1-2 months; Enterprise customers acquired via first approach: 2 (source-reported, not independently verified).

What failed first in this deployment?

Hierarchical query reasoning—breaking medical guidelines into tree-structured sub-questions answerable within a 16K token window—became unnecessary when GPT-4 Turbo launched with a 128K context window.

How is this prior authorization AI workflow structured?

Medical review case arrives → Guidelines decomposed into question tree → Final determination propagated → Clinical knowledge dynamically injected → Clinicians review outputs via dashboard → Review data feeds product improvement.

Related prior authorization cases