Quality assurance · Production

Netflix builds SMAD system to detect speech and music in audio content at production scale

The problem

Netflix needs to systematically classify speech, music, and effects regions across its large audio catalog to enable production and delivery tasks, but collecting fine-resolution frame-level labels is costly, labor-intensive, and restricted by copyright limitations.

Workflow diagram · grounded in source

Audio file received

trigger

“All audio files were originally delivered from the post-production studios in the standard 5.1 surround format at 48 kHz sampling rate”

Audio normalization and downmix

ai_action

“We first normalize all files to an average loudness of −27 LKFS ± 2 LU dialog-gated, then downsample to 16 kHz before creating an ITU downmix”

CRNN speech/music classification

ai_action

“The best model was a CRNN with three convolutional layers, followed by two bi-directional recurrent layers and one fully connected layer. The model has 832k trainable parameters and emits frame-level predictions for both speech and music…”

Temporal metadata output

output

“The detailed temporal metadata SMAD provides about speech and music regions in a polyphonic audio mixture are a first step for structural audio segmentation, indexing and pre-processing audio”

Catalog-wide loudness management

integration

“speech-gated loudness is computed for every audio master track and used for loudness normalization. Speech-activity metadata is thus a central part of accurate catalog-wide loudness management”

Translation and dubbing pipeline

integration

“there are post-production tasks, which take advantage of accurate speech segmentation at the the spoken utterance or sentence level, ahead of translation and dub-script generation”

Reported outcome

Netflix deployed SMAD using a large noisy-labeled catalog dataset and a CRNN architecture, enabling hundreds of audio production and delivery tasks daily across global teams with substantial productivity returns at scale.

Reported metrics

TVSM training dataset size1608 hours

Productivity returnssubstantial productivity returns at scale

Daily task frequencyhundreds of times a day

Reported stack

CRNNPython

Source

https://netflixtechblog.com/detecting-speech-and-music-in-audio-content-afd64e6a5bf8

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Netflix deployed SMAD using a large noisy-labeled catalog dataset and a CRNN architecture, enabling hundreds of audio production and delivery tasks daily across global teams with substantial productivity returns at sc…

What tools did this team use?

CRNN, Python.

What results were reported?

TVSM training dataset size: 1608 hours; Productivity returns: substantial productivity returns at scale; Daily task frequency: hundreds of times a day (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Audio file received → Audio normalization and downmix → CRNN speech/music classification → Temporal metadata output → Catalog-wide loudness management → Translation and dubbing pipeline.