Back office ops · Production

GitHub research: deep learning approach to natural language semantic code search

The problem

Code search on GitHub was limited to keyword matching, requiring users to know exact syntax or anticipate keywords in surrounding comments, with no ability to search by natural language intent.

First attempt

An initial attempt using the Universal Sentence Encoder produced embeddings that worked reasonably but lacked specificity to software development vocabulary and semantics.

Workflow diagram · grounded in source
1
Code summarization model training
ai_action
“we train a sequence-to-sequence model that learns to summarize code. A way to accomplish this for Python is to supply (code, docstring) pairs where the docstring is the target variable the model is trying to predict”
2
Encoder extraction for code representation
ai_action
“we can utilize the encoder from this model as a general purpose feature extractor for code”
3
Text phrase representation learning
ai_action
“we trained a neural language model by leveraging the fast.ai library”
4
Code-to-text vector space mapping
ai_action
“we map the code representations we learned from the code summarization model (part 1) to the vector space of text. We accomplish this by fine-tuning the encoder of this model”
5
Semantic nearest-neighbor search
output
“we can store the vectorized version of all code in a database, and perform nearest neighbor lookups to a vectorized search query”
Reported outcome

The research system achieves a BLEU score of 13.5 on a holdout set of Python code and demonstrates semantic search returning relevant results even when no keywords are shared between the query and the code.

Reported metrics
BLEU score on Python holdout set13.5
Reported stack
fairseq-pyUniversal Sentence EncoderTensorFlow Hubfast.aikubeflow
Source
https://github.blog/ai-and-ml/machine-learning/towards-natural-language-semantic-code-search/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The research system achieves a BLEU score of 13.5 on a holdout set of Python code and demonstrates semantic search returning relevant results even when no keywords are shared between the query and the code.

What tools did this team use?

fairseq-py, Universal Sentence Encoder, TensorFlow Hub, fast.ai, kubeflow.

What results were reported?

BLEU score on Python holdout set: 13.5 (source-reported, not independently verified).

What failed first in this deployment?

An initial attempt using the Universal Sentence Encoder produced embeddings that worked reasonably but lacked specificity to software development vocabulary and semantics.

How is this back office ops AI workflow structured?

Code summarization model training → Encoder extraction for code representation → Text phrase representation learning → Code-to-text vector space mapping → Semantic nearest-neighbor search.

Related back office ops cases