Clinical documentation · Production

Key Learnings to Elevate the Quality of Doctolib's AI-powered consultation assistant

The problem

Doctolib faced a 'chicken and egg' dilemma building their AI consultation assistant: effective AI requires user feedback, but meaningful feedback requires a reasonably functional AI that earns user engagement.

First attempt

Commonly used NLP evaluation metrics — ROUGE, BLEU, and BERT scores — proved less effective than expected at correlating with actual summary quality and were discarded.

Workflow diagram · grounded in source
1
Speech-to-text transcription
ai_action
“speech recognition to convert audio into text”
2
LLM consultation summarization
ai_action
“summarization to transform text into a medical consultation summary”
3
Medical concept codification
ai_action
“medical content codification to map summaries into standard medical ontologies like ICD-10”
4
LLM quality validation
validation
“we employed a large language model (LLM) as an automated judge. This allowed us to prioritize critical aspects of summary quality like the hallucination rate to assess the factuality or the recall for the summary completeness”
5
Expert safety and usability review
human_review
“our internal experts conducted a thorough safety and usability review to determine whether the Consultation Assistant was ready for real-world deployment with volunteer beta testers”
6
Practitioner feedback collection
feedback_loop
“We track implicit feedback signals such as deletions, edits, validations, and section interactions. Analyzing these patterns reveals how practitioners interact with the product, highlighting areas of potential confusion, frustration, or …”
7
A/B testing of product changes
feedback_loop
“A/B testing is our gold standard for evaluating product changes. This method involves randomly dividing users into test and control groups.”
Reported outcome

The Consultation Assistant launched to widespread adoption with overwhelmingly positive feedback exceeding all expectations, following over a year of continuous improvement with beta testers and medical experts.

Reported metrics
Product adoption at launchwidespread adoption
User feedback qualityoverwhelmingly positive feedback, exceeding all expectations
Reported stack
large language model (LLM)speech recognitionICD-10
Source
https://medium.com/doctolib/key-learnings-to-elevate-the-quality-of-doctolibs-ai-powered-consultation-assistant-3656eb2b9bc7
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The Consultation Assistant launched to widespread adoption with overwhelmingly positive feedback exceeding all expectations, following over a year of continuous improvement with beta testers and medical experts.

What tools did this team use?

large language model (LLM), speech recognition, ICD-10.

What results were reported?

Product adoption at launch: widespread adoption; User feedback quality: overwhelmingly positive feedback, exceeding all expectations (source-reported, not independently verified).

What failed first in this deployment?

Commonly used NLP evaluation metrics — ROUGE, BLEU, and BERT scores — proved less effective than expected at correlating with actual summary quality and were discarded.

How is this clinical documentation AI workflow structured?

Speech-to-text transcription → LLM consultation summarization → Medical concept codification → LLM quality validation → Expert safety and usability review → Practitioner feedback collection → A/B testing of product changes.

Related clinical documentation cases