Compliance monitoring · Production

Deploying Agentic AI for Clinical Trial Protocol Deviation Monitoring at Bayezian Limited

The problem

Detecting protocol deviations in clinical trials has traditionally required study teams to manually cross-reference spreadsheets and timelines against lengthy protocol documents — a process described as time-consuming, easy to miss context, and prone to delay.

First attempt

The multi-agent system suffered repeated handover failures — deviations correctly identified by one agent were lost or misclassified by the next — and fragile time-window reasoning, illustrated by a false positive on a Day 14 lab test that the agent raised despite a Day 13 test (within the allowed two-day window) already being on record.

Workflow diagram · grounded in source
1
Document type classification
ai_action
“an early classifier was introduced to determine what type of document had arrived. Was it a screening form or a post-randomisation visit report? That initial decision shaped the downstream path.”
2
Agent routing by document type
routing
“If it was a screening file, the system activated the inclusion and exclusion criteria checker. If it was a visit document, it was handed off to agents responsible for tracking timing, treatment exposure, scheduled procedures, and adverse…”
3
Semantic protocol rule retrieval
ai_action
“protocol content was indexed using FAISS, a vector store optimised for fast similarity-based retrieval. This allowed each agent to fetch relevant rules dynamically and reason through them with appropriate context.”
4
Specialist agent compliance checks
ai_action
“One agent reviewed visit timing. Another checked medication use. Others handled inclusion criteria, missed procedures, or serious adverse events.”
5
Deviation escalation and report
output
“If any agent spotted something unusual, it could escalate the case to a second agent responsible for suggesting possible actions. That might mean logging the issue, generating a report, or prompting a review from the study team.”
6
Human oversight of edge cases
human_review
“Throughout, a human remained involved to validate decisions and interpret edge cases that needed nuance.”
Reported outcome

After targeted improvements including structured memory snapshots, stronger handoff signals, and clearer prompts, the system proved genuinely useful: it spotted patterns early, added pace to routine checks and consistency to decisions, and shifted reviewers from manual line-by-line scanning to focused triage of edge cases.

Reported metrics
Routine check paceadded pace to routine checks
Reviewer focus shiftreviewers began focusing on the edge cases
Reported stack
FAISSPythonCTMS
Source
https://aihub.org/2025/09/15/deploying-agentic-ai-what-worked-what-broke-and-what-we-learned/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After targeted improvements including structured memory snapshots, stronger handoff signals, and clearer prompts, the system proved genuinely useful: it spotted patterns early, added pace to routine checks and consist…

What tools did this team use?

FAISS, Python, CTMS.

What results were reported?

Routine check pace: added pace to routine checks; Reviewer focus shift: reviewers began focusing on the edge cases (source-reported, not independently verified).

What failed first in this deployment?

The multi-agent system suffered repeated handover failures — deviations correctly identified by one agent were lost or misclassified by the next — and fragile time-window reasoning, illustrated by a false positive on…

How is this compliance monitoring AI workflow structured?

Document type classification → Agent routing by document type → Semantic protocol rule retrieval → Specialist agent compliance checks → Deviation escalation and report → Human oversight of edge cases.

Related compliance monitoring cases