Quality assurance · Production

Intuitive Surgical doubles labeled dataset delivery speed for surgical ML models with Labelbox

The problem

Intuitive Surgical's data science team faced a bottleneck in obtaining sufficient labeled video data to train ML models for surgical instrument detection, with frame-by-frame bounding-box annotation across tens of thousands of videos being tedious and time-consuming, and a requirement to maintain a consistent ontology across clinical and data science teams.

Workflow diagram · grounded in source
1
Surgical video annotation need
trigger
“Annotating bounding boxes frame-by-frame in tens of thousands of videos is a tedious and time consuming process, because a large variety of surgical tools and surgeries must be captured for robust model training”
2
Model-assisted labeling
ai_action
“speed up annotation workflows between domain experts and labelers using model-assisted labeling”
3
Expert-labeler collaboration
human_review
“We rely on collaborative software to help align our different teams such as our clinical teams and data science teams to ensure that we have a clearly defined ontology. This ensures that all labeling activities are consistent and provide…”
4
Labeling velocity measurement
validation
“Labelbox's Annotate product, using the native video editor to label their unstructured data, measure labeling velocity & efficiency with detailed metrics”
5
Labeled dataset delivery
output
“double the speed at which they can deliver labeled datasets for their multiple ML models”
Reported outcome

Intuitive Surgical doubled the speed at which they deliver labeled datasets for their ML models, scaled labeling efficiency and throughput, and lowered overhead in gathering performance and quality metrics.

Reported metrics
Labeled dataset delivery speeddoubling the speed at which they can deliver labeled datasets
Labeling efficiency and throughputscale their labeling efficiency and throughput
Overhead in gathering performance and quality metricslower the overhead needed to gather metrics on performance and quality
Reported stack
LabelboxAnnotatenative video editormodel-assisted labeling
Source
https://labelbox.com/customers/intuitive-surgical-customer-story/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Intuitive Surgical doubled the speed at which they deliver labeled datasets for their ML models, scaled labeling efficiency and throughput, and lowered overhead in gathering performance and quality metrics.

What tools did this team use?

Labelbox, Annotate, native video editor, model-assisted labeling.

What results were reported?

Labeled dataset delivery speed: doubling the speed at which they can deliver labeled datasets; Labeling efficiency and throughput: scale their labeling efficiency and throughput; Overhead in gathering performance and quality metrics: lower the overhead needed to gather metrics on performance and quality (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Surgical video annotation need → Model-assisted labeling → Expert-labeler collaboration → Labeling velocity measurement → Labeled dataset delivery.

Related quality assurance cases