Workflow · Production

Meta releases Code Llama: an open large language model for code generation, completion, and debugging

The problem

Developer workflows needed tools to become faster and more efficient, the barrier to entry for people learning to code needed to be lowered, and open-source code-specific LLMs lagged behind proprietary alternatives.

Workflow diagram · grounded in source
1
Developer submits text prompt
trigger
“can use text prompts to generate code”
2
Model generates code output
ai_action
“It can generate code, and natural language about code, from both code and natural language prompts (e.g., "Write me a function that outputs the fibonacci sequence.")”
3
Fill-in-the-middle completion
ai_action
“The 7B and 13B base and instruct models have also been trained with fill-in-the-middle (FIM) capability, allowing them to insert code into existing code”
4
Code completion and debugging
ai_action
“It can also be used for code completion and debugging”
Reported outcome

Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP, outperforming other open-source code-specific LLMs and performing on par with ChatGPT, while supporting contexts up to 100,000 tokens.

Reported metrics
HumanEval score (34B model)53.7%
MBPP score (34B model)56.2%
Maximum context window100,000 tokens
training data (7B/13B/34B models)500B tokens
Show all 7 reported metrics
HumanEval score (34B model)53.7%
MBPP score (34B model)56.2%
maximum context window100,000 tokens
training data (7B/13B/34B models)500B tokens
training data (70B model)1T tokens
Python fine-tuning data100B tokens
developer workflow efficiencymake workflows faster and more efficient
Reported stack
Code LlamaLlama 2PyTorchHumanEvalMBPP
Source
https://ai.meta.com/blog/code-llama-large-language-model-coding/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Code Llama 34B scored 53.7% on HumanEval and 56.2% on MBPP, outperforming other open-source code-specific LLMs and performing on par with ChatGPT, while supporting contexts up to 100,000 tokens.

What tools did this team use?

Code Llama, Llama 2, PyTorch, HumanEval, MBPP.

What results were reported?

HumanEval score (34B model): 53.7%; MBPP score (34B model): 56.2%; Maximum context window: 100,000 tokens; training data (7B/13B/34B models): 500B tokens (source-reported, not independently verified).

How is this workflow AI workflow structured?

Developer submits text prompt → Model generates code output → Fill-in-the-middle completion → Code completion and debugging.