Back office ops · Production

How Dropbox built AI-powered image content search using EfficientNet and word vectors

The problem

Photos in Dropbox are difficult to find by filename because camera-generated filenames contain no content information, forcing users to browse thumbnails manually to locate specific images.

Workflow diagram · grounded in source
1
Image classified on upload
ai_action
“Our image classifier is an EfficientNet network trained on the OpenImages dataset. It produces scores for about 8500 categories.”
2
Sparse vectors indexed
integration
“In the inverted index, each image is added to 50 posting lists instead of 10,000, at a cost of about 200 bytes. So the total index storage per image is 500 bytes instead of 80 kilobytes.”
3
User initiates image search
trigger
“when a user does a search we run it on all their images and return those that produce a score above a threshold”
4
Query word vector lookup
ai_action
“We use the pre-trained ConceptNet Numberbatch word vectors. These give good results, and importantly to us they support multiple languages, returning similar vectors for words in different languages with similar meanings.”
5
Query projected to category space
ai_action
“multiply by the category space projection matrix C to get qc as described above. C is a fixed matrix that's the same for all users, so we can hold it in memory.”
6
Inverted index lookup
integration
“For each category with a nonzero entry in qc, fetch the posting list from the inverted index. The union of these lists is the search result set of matching images”
7
Relevance scoring and ranking
ai_action
“fetch the category space vector jc from the forward index and multiply by qc to get the relevance score s. Return results with score above a threshold, ranked by the score.”
8
Parallel text and image results
output
“when a user initiates a search we can run both text and image searches in parallel, and show the full set of results together, without making the user wait any longer than they would for a text-only search”
Reported outcome

Image content search is now enabled for all Professional and Business Dropbox users, combining image classification with OCR and full-text search; index storage was reduced to 500 bytes per image from 80 kilobytes, and query latency is on par with text search.

Reported metrics
Index storage per image500 bytes instead of 80 kilobytes
Posting lists scanned at query time10 posting lists — roughly the same amount of work we do for text queries
Query latencyon par with those for text search
Image categories classified8500
Reported stack
EfficientNetOpenImagesTensorFlowConceptNet Numberbatchword2vecNautilusOCR
Source
https://dropbox.tech/machine-learning/how-image-search-works-at-dropbox
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Image content search is now enabled for all Professional and Business Dropbox users, combining image classification with OCR and full-text search; index storage was reduced to 500 bytes per image from 80 kilobytes, an…

What tools did this team use?

EfficientNet, OpenImages, TensorFlow, ConceptNet Numberbatch, word2vec, Nautilus, OCR.

What results were reported?

Index storage per image: 500 bytes instead of 80 kilobytes; Posting lists scanned at query time: 10 posting lists — roughly the same amount of work we do for text queries; Query latency: on par with those for text search; Image categories classified: 8500 (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Image classified on upload → Sparse vectors indexed → User initiates image search → Query word vector lookup → Query projected to category space → Inverted index lookup → Relevance scoring and ranking → Parallel text and image results.

Related back office ops cases