Back office ops · Production

eBay adapts Meta Llama 3.1 to the e-commerce domain via continued pretraining, achieving approximately 25% benchmark improvement

The problem

General-purpose LLMs such as GPT-4 and Claude were too costly for eBay's at-scale needs and introduced data security risks by relying on third-party providers, while training a domain-specific LLM entirely from scratch was too time- and resource-intensive.

First attempt

Third-party LLMs (GPT-4 and Claude) were prohibitively expensive and introduced data security constraints, while off-the-shelf open models such as Llama 3.1 lacked the e-commerce domain knowledge required for eBay's use cases.

Workflow diagram · grounded in source
1
E-commerce data gathering and filtering
integration
“we gather data from public listings and product reviews from the eBay website. This data is then thoroughly filtered and serialized to fit the task of autoregressive language modeling”
2
E-commerce classifier extraction
ai_action
“we train an e-commerce classifier and use it to extract e-commerce specific examples from a larger open-source dataset”
3
Continued pretraining on e-commerce data
ai_action
“we continue training the Llama base models on a large amount of e-commerce data in order to infuse domain specific knowledge into the model. This technique is known as "continued pretraining"”
4
Instruction tuning with human feedback
human_review
“we further instruction-tuned the models, aligning them with human feedback to ensure they generated safe and contextually appropriate content. This tuning also helped the models learn guardrails and follow explicit instructions”
5
Benchmark validation
validation
“The final e-Llama models demonstrate approximately 25% improvement in e-commerce-specific benchmarks for English and about 30% improvement for non-English when compared to the corresponding Llama 3.1 base models. At the same time, we obs…”
Reported outcome

The final e-Llama models demonstrate approximately 25% improvement in English e-commerce benchmarks and about 30% for non-English, with only 1% degradation on general domain NLU benchmarks, enabling eBay to drive new AI initiatives across the company.

Reported metrics
e-commerce benchmark improvement (English)approximately 25%
e-commerce benchmark improvement (non-English)about 30%
general domain NLU benchmark degradation (e-Llama 70B)1%
training compute (70B model on 1 trillion tokens)about 340k GPU-hours
Show all 5 reported metrics
e-commerce benchmark improvement (English)approximately 25%
e-commerce benchmark improvement (non-English)about 30%
general domain NLU benchmark degradation (e-Llama 70B)1%
training compute (70B model on 1 trillion tokens)about 340k GPU-hours
training duration (70B model)around one month
Reported stack
Llama 3.1Megatron-LMNVIDIA H100 80GB GPUNVIDIA NVLinkInfiniBandflash-attention-2
Source
https://innovation.ebayinc.com/tech/features/scaling-large-language-models-for-e-commerce-the-development-of-a-llama-based-customized-llm-for-e-commerce/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The final e-Llama models demonstrate approximately 25% improvement in English e-commerce benchmarks and about 30% for non-English, with only 1% degradation on general domain NLU benchmarks, enabling eBay to drive new…

What tools did this team use?

Llama 3.1, Megatron-LM, NVIDIA H100 80GB GPU, NVIDIA NVLink, InfiniBand, flash-attention-2.

What results were reported?

e-commerce benchmark improvement (English): approximately 25%; e-commerce benchmark improvement (non-English): about 30%; general domain NLU benchmark degradation (e-Llama 70B): 1%; training compute (70B model on 1 trillion tokens): about 340k GPU-hours (source-reported, not independently verified).

What failed first in this deployment?

Third-party LLMs (GPT-4 and Claude) were prohibitively expensive and introduced data security constraints, while off-the-shelf open models such as Llama 3.1 lacked the e-commerce domain knowledge required for eBay's u…

How is this back office ops AI workflow structured?

E-commerce data gathering and filtering → E-commerce classifier extraction → Continued pretraining on e-commerce data → Instruction tuning with human feedback → Benchmark validation.

Related back office ops cases