Quality assurance · Production

GitHub builds OctoLingua: an ANN-based machine learning classifier for programming language detection

The problem

Language detection at GitHub is non-trivial because file extensions are ambiguous, shared across languages, or absent entirely. The existing tool Linguist achieved 84% file-level accuracy but its performance declined considerably when extensions were missing or incorrect, making it unsuitable for Gists and inline code snippets in READMEs, issues, and pull requests.

First attempt

Linguist relies on heuristics and a Naive Bayes classifier trained on a small sample of data; it fails as soon as file extension information is altered or removed, revealing that it does not robustly learn from code vocabulary.

Workflow diagram · grounded in source
1
Code pushed to repository
trigger
“When some code is pushed to a repository, it's important to recognize the type of code that was added”
2
Feature extraction
ai_action
“extracting some relevant features in tabular form to be fed to our classifier. The features currently extracted are as follows: Top five special characters per file, Top 20 tokens per file, File extension, Presence of certain special cha…”
3
ANN language classification
ai_action
“a two-layer Artificial Neural Network built using Keras with Tensorflow backend... ultimately produces a 51-dimensional output which represents the predicted probability that the given code is written in each of the top 50 GitHub languag…”
4
Language classification output
output
“recognize the type of code that was added for the purposes of search, security vulnerability alerting, and syntax highlighting—and to show the repository's content distribution to users”
Reported outcome

OctoLingua, built on an ANN architecture using Python, Keras, and TensorFlow, surpasses Linguist in accuracy and performance and maintains good performance under various conditions, learning primarily from code vocabulary rather than file extension metadata.

Reported metrics
Linguist file-level accuracy (baseline)84%
OctoLingua accuracy vs Linguistsurpasses Linguist in accuracy and performance
OctoLingua robustness under varied conditionsmaintains a good performance under various conditions
Reported stack
OctoLinguaLinguistPythonKerasTensorFlowANN
Source
https://github.blog/ai-and-ml/machine-learning/c-or-java-typescript-or-javascript-machine-learning-based-classification-of-programming-languages/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

OctoLingua, built on an ANN architecture using Python, Keras, and TensorFlow, surpasses Linguist in accuracy and performance and maintains good performance under various conditions, learning primarily from code vocabu…

What tools did this team use?

OctoLingua, Linguist, Python, Keras, TensorFlow, ANN.

What results were reported?

Linguist file-level accuracy (baseline): 84%; OctoLingua accuracy vs Linguist: surpasses Linguist in accuracy and performance; OctoLingua robustness under varied conditions: maintains a good performance under various conditions (source-reported, not independently verified).

What failed first in this deployment?

Linguist relies on heuristics and a Naive Bayes classifier trained on a small sample of data; it fails as soon as file extension information is altered or removed, revealing that it does not robustly learn from code v…

How is this quality assurance AI workflow structured?

Code pushed to repository → Feature extraction → ANN language classification → Language classification output.

Related quality assurance cases