Clinical documentation · Production

Heidi Health increases prompt testing speed 3x using PromptHub

The problem

Heidi Health's Medical AI Residents stored prompts in Google Sheets and tested them in Jupyter notebooks, quickly outgrowing this setup. A subsequent third-party prompt testing tool proved equally unreliable and clunky, leaving the team without a stable solution that non-technical medical doctors could also use.

First attempt

The Google Sheets and Jupyter notebooks workflow was clunky and inaccessible to non-technical team members. A dedicated prompt testing tool tried before PromptHub was also found to be unreliable and clunky.

Workflow diagram · grounded in source
1
Store prompts and datasets
integration
“the ability to store and reuse data in PromptHub via variables and datasets made it easy to ensure prompts worked at scale”
2
Batch test across models
validation
“Being able to seamlessly batch test prompts against different models helped Kieran understand the trade-offs between output quality, price, and latency”
3
Evaluate parameter effects
validation
“testing how changes to parameters affected output quality enabled the team to get extremely granular with their evaluations”
4
Test new models on release
validation
“Models are added to the platform as soon as they are released, which makes it easy for the Heidi Health team to quickly test new models, making the model selection process extremely fast”
Reported outcome

Heidi Health saw their testing speed increase by 3x, with prompts becoming better and more robust, leading to improved product experiences for users and major efficiency gains.

Reported metrics
Testing speed3x
Efficiency gainsmajor efficiency gains
LLM output qualitybetter outputs from LLMs
Prompt robustnessbetter and more robust
Reported stack
PromptHubJupyter notebooksLLMs
Source
https://www.prompthub.us/customers/heidi-health
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Heidi Health saw their testing speed increase by 3x, with prompts becoming better and more robust, leading to improved product experiences for users and major efficiency gains.

What tools did this team use?

PromptHub, Jupyter notebooks, LLMs.

What results were reported?

Testing speed: 3x; Efficiency gains: major efficiency gains; LLM output quality: better outputs from LLMs; Prompt robustness: better and more robust (source-reported, not independently verified).

What failed first in this deployment?

The Google Sheets and Jupyter notebooks workflow was clunky and inaccessible to non-technical team members.

How is this clinical documentation AI workflow structured?

Store prompts and datasets → Batch test across models → Evaluate parameter effects → Test new models on release.

Related clinical documentation cases