Prior authorization · Production

How to build a custom AI review dashboard for LLM products — lessons from Anterior's Scalpel

The problem

Without metrics on AI performance, teams cannot know how their app is doing or how to improve it; performance could drop unnoticed until customers have already left.

First attempt

Spreadsheets and off-the-shelf tooling providers hit limits quickly — they restrict data views, struggle to expose intermediate LLM steps, and make it hard to translate review outputs directly into application improvements.

Workflow diagram · grounded in source
1
AI makes treatment decision
trigger
“clinical reasoning workflows check medical guidelines against medical evidence to decide whether a treatment should be approved”
2
Context surfaced in dashboard
integration
“we put the two main pieces of context (guidelines and evidence) on one side of the screen, and all AI output information on the left hand side”
3
Clinician reviews AI output
human_review
“enable a small team of clinicians to review >100,000 medical decisions”
4
Failure modes identified
validation
“Identifying failure modes tells you in what way your LLM pipeline is making mistakes. This lets you both focus efforts on improving it and quantify the impact of those improvements”
5
Improvements fed back to AI
feedback_loop
“directly ask for suggestions for improvement and potentially even make those improvements directly (e.g., by tweaking prompts or adding content to a knowledge base)”
Reported outcome

Anterior's Scalpel dashboard enabled a small team of clinicians to review more than 100,000 medical decisions, providing a high-leverage bridge between production AI outputs and continuous product improvement.

Reported metrics
Medical decisions reviewed>100,000
Reported stack
Scalpel
Source
https://chrislovejoy.me/review-dashboard
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Anterior's Scalpel dashboard enabled a small team of clinicians to review more than 100,000 medical decisions, providing a high-leverage bridge between production AI outputs and continuous product improvement.

What tools did this team use?

Scalpel.

What results were reported?

Medical decisions reviewed: >100,000 (source-reported, not independently verified).

What failed first in this deployment?

Spreadsheets and off-the-shelf tooling providers hit limits quickly — they restrict data views, struggle to expose intermediate LLM steps, and make it hard to translate review outputs directly into application improve…

How is this prior authorization AI workflow structured?

AI makes treatment decision → Context surfaced in dashboard → Clinician reviews AI output → Failure modes identified → Improvements fed back to AI.

Related prior authorization cases