Workflow · Production

Vimeo builds a RAG-powered video Q&A system for knowledge-sharing content

The problem

Video viewers who lack time to watch full recordings need a way to extract information from video content in natural language; knowledge-sharing videos like meetings and lectures are particularly hard to query without watching them entirely.

Workflow diagram · grounded in source
1
Transcript ingestion
trigger
“each video uploaded to Vimeo is already transcribed to generate automatic closed captions, so there's no need to implement anything new”
2
Multi-level chunking and summarization
ai_action
“We start by a standard chunking of the transcript to groups of sentences containing 100 to 200 words (corresponding to 1–2 minutes of playback time). These represent the bottom level of processing. We then take larger chunks of 500 words…”
3
LLM-based speaker name detection
ai_action
“The name detection mechanism is based on votes generated using an LLM. For every transition, we request the LLM to identify speaker names by analyzing each side of the transition individually. This is achieved by masking the first or sec…”
4
Vector database registration
integration
“we take all the textual representations we get from all levels and register them into one large vector database”
5
Question embedding and context retrieval
ai_action
“the question is first converted into a vector embedding and then queried against the vector database to get the best matches by a nearest neighbors search”
6
LLM answer generation
ai_action
“the matches form a context for the LLM, which uses it to generate an answer to the question”
7
Video reference timestamp identification
ai_action
“one prompt that answers the question, followed by a second prompt that finds the relevant quotes in the transcript given the question and answer”
8
Answer and suggestions delivered
output
“In return you get a brief description of the video and some relevant playable moments that you can click to watch. You also get some related questions that you can select to dig deeper into any given topic”
Reported outcome

Vimeo built a video Q&A system using RAG that answers questions in natural language, surfaces playable video moments, and suggests related questions; experiments show the transcript alone is sufficient to answer most important questions for knowledge-sharing videos.

Reported metrics
Question-answering coverage for knowledge-sharing videosable to answer most of the important questions about the video
Two-prompt vs single-prompt answer and reference performancemuch better performance
Reported stack
RAGLLMvector databaseChatGPT 3.5
Source
https://medium.com/vimeo-engineering-blog/unlocking-knowledge-sharing-for-videos-with-rag-810ab496ae59
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Vimeo built a video Q&A system using RAG that answers questions in natural language, surfaces playable video moments, and suggests related questions; experiments show the transcript alone is sufficient to answer most…

What tools did this team use?

RAG, LLM, vector database, ChatGPT 3.5.

What results were reported?

Question-answering coverage for knowledge-sharing videos: able to answer most of the important questions about the video; Two-prompt vs single-prompt answer and reference performance: much better performance (source-reported, not independently verified).

How is this workflow AI workflow structured?

Transcript ingestion → Multi-level chunking and summarization → LLM-based speaker name detection → Vector database registration → Question embedding and context retrieval → LLM answer generation → Video reference timestamp identification → Answer and suggestions delivered.