Customer support · Production

Shipping a Clinical-Grade Patient Education Agent: Why Observability is Non-Negotiable in Healthcare AI

The problem

A digital health platform needed an AI agent to help patients understand camera-based health scan results conversationally, while maintaining clinical accuracy, HIPAA-compliant audit trails, safe uncertainty handling, and a strict education-versus-diagnosis boundary enforced in code.

First attempt

Early builds suffered invisible hallucinations where the model bridged low-quality retrieval gaps with training data, and lacked a full reasoning-chain audit trail, making it impossible to reconstruct what the AI told a patient during a compliance audit.

Workflow diagram · grounded in source
1
Patient requests scan explanation
trigger
“patients needed conversational help understanding those results”
2
Extract health scan findings
ai_action
“Interpret dermatology test results from camera-based health scans”
3
Retrieve from medical knowledge base
ai_action
“We used RAG over an approved, version-controlled knowledge base of clinical explanations. Every retrieval returned not just content, but a confidence score based on evidence quality.”
4
Confidence-based routing decision
routing
“This is NOT an LLM decision - it's explicit threshold-based routing to ensure safety boundaries are enforced in code, not prompts.”
5
Deliver patient education
output
“Provide plain-language summaries patients could understand”
6
Human clinician review
human_review
“Human-in-the-loop as a first-class node—not a fallback, but a designed path”
7
Audit trail logging
integration
“LangSmith tracing gave us an end-to-end timeline for every conversation: LLM calls, retrieval results, confidence scores, routing decisions, and metadata”
Reported outcome

After threshold tuning, approximately 15% of conversations triggered human review, about 80% of those were appropriately routed per clinician feedback, the false positive review rate decreased 40%, and a very low rate of inappropriate clinical advice was observed in production.

Reported metrics
Conversations triggering human review~15%
Appropriately routed human reviews~80%
False positive review rate reduction40%
Inappropriate clinical advice in productionVery low rate
Reported stack
LangGraphLangSmithPostgreSQLRAG
Source
https://www.lubulabs.com/ai-blog/healthcare-ai-patient-education
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After threshold tuning, approximately 15% of conversations triggered human review, about 80% of those were appropriately routed per clinician feedback, the false positive review rate decreased 40%, and a very low rate…

What tools did this team use?

LangGraph, LangSmith, PostgreSQL, RAG.

What results were reported?

Conversations triggering human review: ~15%; Appropriately routed human reviews: ~80%; False positive review rate reduction: 40%; Inappropriate clinical advice in production: Very low rate (source-reported, not independently verified).

What failed first in this deployment?

Early builds suffered invisible hallucinations where the model bridged low-quality retrieval gaps with training data, and lacked a full reasoning-chain audit trail, making it impossible to reconstruct what the AI told…

How is this customer support AI workflow structured?

Patient requests scan explanation → Extract health scan findings → Retrieve from medical knowledge base → Confidence-based routing decision → Deliver patient education → Human clinician review → Audit trail logging.

Related customer support cases