Prior authorization · Production

The Sour Lesson: Building AI Products That Compound with Model Progress at Anterior

The problem

Building AI products on rapidly evolving LLMs creates a tension: serving customers today requires working around current model limitations, but those workarounds become technical debt when capabilities improve.

First attempt

Hierarchical query reasoning—breaking medical guidelines into tree-structured sub-questions answerable within a 16K token window—became unnecessary when GPT-4 Turbo launched with a 128K context window. Finetuning for clinical reasoning was similarly superseded within 12-18 months as frontier general models became strong enough to eliminate the advantage of specialized models; the hierarchical approach itself was replaced within 6 months.

Workflow diagram · grounded in source
1
Medical review case arrives
Trigger
A typical medical review requires processing clinical guidelines and patient medical records.
▾ source quote
“A typical medical review requires processing: Clinical guidelines (often 20-50 pages, and with complex nested logic) - Patient medical records (can be 100+ pages)”
2
Guidelines decomposed into question tree
Ai action
Guidelines are parsed into a tree of dependent questions, from high-level requirements down to specific clinical criteria, and each question is answered separately against the medical record starting with leaf nodes.
▾ source quote
“Parse guidelines into a tree of dependent questions, from high-level requirements down to specific clinical criteria / Answer each question separately against the medical record, starting with leaf nodes and working up the tree”
3
Final determination propagated
Output
Answers are propagated up through parent nodes to make a final determination.
▾ source quote
“Propagate answers up through parent nodes to make a final determination”
4
Clinical knowledge dynamically injected
Integration
A scalable library of clinical knowledge is dynamically injected at inference time based on the context the model is operating within.
▾ source quote
“building a scalable library of clinical knowledge which is dynamically injected at inference time based on the context the model is operating within”
5
Clinicians review outputs via dashboard
Human review
Domain experts (clinicians) perform reviews using a dedicated dashboard.
▾ source quote
“I built a dashboard to enable domain experts (clinicians) to perform reviews”
6
Review data feeds product improvement
Feedback loop
Data from clinician reviews is used to provide performance metrics, identify failure modes, generate domain knowledge injections, and create in-context learning examples.
▾ source quote
“designed a system downstream that used the data we get from reviews in various ways: Provide performance metrics that we can use to prioritize improvement work / Identify and classify 'failure modes' (the ways in which our AI was making …”
Reported outcome

Domain knowledge injection and the expert review system both remained in production 2+ years after being built.
The earlier hierarchical approach, while short-lived, helped acquire the first 2 enterprise customers within 1-2 months.

Reported metrics
Original model context window16K tokens
GPT-4 Turbo context window128K context window
Time to build hierarchical query reasoning approach1-2 months
Enterprise customers acquired via first approach2
Show all 8 reported metrics
original model context window16K tokens
GPT-4 Turbo context window128K context window
time to build hierarchical query reasoning approach1-2 months
enterprise customers acquired via first approach2
time before hierarchical approach was replacedwithin 6 months
time for general models to surpass specialized finetuned models12-18 months
domain knowledge injection production lifespan2+ years
recommended effort split optimization vs experimentationrough 70%:30% split
Reported stack
GPT-3.5-Turbo-16kGPT-4 TurboGPT-4Claude 3
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://chrislovejoy.me/sour-lesson
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Domain knowledge injection and the expert review system both remained in production 2+ years after being built.

What tools did this team use?

GPT-3.5-Turbo-16k, GPT-4 Turbo, GPT-4, Claude 3.

What results were reported?

Original model context window: 16K tokens; GPT-4 Turbo context window: 128K context window; Time to build hierarchical query reasoning approach: 1-2 months; Enterprise customers acquired via first approach: 2 (source-reported, not independently verified).

What failed first in this deployment?

Hierarchical query reasoning—breaking medical guidelines into tree-structured sub-questions answerable within a 16K token window—became unnecessary when GPT-4 Turbo launched with a 128K context window.

How is this prior authorization AI workflow structured?

Medical review case arrives → Guidelines decomposed into question tree → Final determination propagated → Clinical knowledge dynamically injected → Clinicians review outputs via dashboard → Review data feeds product improvement.

Related prior authorization cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Prior authorization cases →