Ecommerce ops · Production

Faire fine-tunes Llama3-8b to scale semantic search relevance measurement to 70M predictions per day

The problem

Evaluating search relevance at Faire was a manual, expensive, and slow process limited to monthly human-labeled snapshots, making it hard to scale and act on relevance signals as the search ecosystem grew more complex with personalization.

First attempt

Prompt engineering alone could not capture Faire's definition of semantic search relevance, and the fine-tuned GPT solution was increasingly constrained by external API costs, limiting throughput for labeling.

Workflow diagram · grounded in source
1
Human-labeled ground truth
trigger
“we worked with a data annotation vendor to label a sample of query-product pairs every month for measuring the relevance of our search system. We built out several iterations of decision trees to reach > 90% agreement amongst labelers an…”
2
Llama3-8b fine-tuning with LoRA
ai_action
“employed Parameter Efficient Fine-Tuning using the LoRA adapter. This significantly reduced the amount of trainable parameters, resulting in lower memory usage and far faster training speed. In the fine-tuning, the training only optimize…”
3
ESCI relevance classification
ai_action
“The model was tasked to complete the text with one of the four ESCI labels to measure the relevance of the search query and the product.”
4
Self-hosted batch inference
output
“Quantize the model to 8 bit Run batches on a single A100 GPU Improve inference speed using DeepSpeed Horizontally scale the number of GPU instances By doing so, we were able to reach 70 million predictions per day using 16 GPUs when we b…”
5
Downstream search optimization
feedback_loop
“These scaled out predictions unlocked many downstream use cases including offline retrieval analysis, measurement of personalization, measurable contribution of experiments towards relevance, Pareto frontier exploration between engagemen…”
Reported outcome

The fine-tuned Llama3-8b model achieves a 28% improvement in Krippendorff's Alpha over the existing GPT production model and enables 70 million relevance predictions per day using 16 GPUs, making relevance a measurable and actionable dimension across all retailer search sessions.

Reported metrics
Daily relevance predictions70 million predictions per day
Llama3-8b vs Llama2-7b performance gains1.4% to 8%
Krippendorff's Alpha (GPT production model)0.56
query-product pairs labeled per hour (GPT model)~300k query product pairs per hour
Show all 7 reported metrics
daily relevance predictions70 million predictions per day
Llama3-8b vs Llama2-7b performance gains1.4% to 8%
Krippendorff's Alpha (GPT production model)0.56
query-product pairs labeled per hour (GPT model)~300k query product pairs per hour
labeler agreement rate> 90%
fine-tuning trainable parametersabout 4%
GPT accuracy vs prompt engineeringalmost 2x accuracy
Reported stack
Llama2-7bLlama2-13bGPTDeepSpeedA100 GPUFireworks.ai
Source
https://craft.faire.com/fine-tuning-llama3-to-measure-semantic-relevance-in-search-86a7b13c24ea
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The fine-tuned Llama3-8b model achieves a 28% improvement in Krippendorff's Alpha over the existing GPT production model and enables 70 million relevance predictions per day using 16 GPUs, making relevance a measurabl…

What tools did this team use?

Llama2-7b, Llama2-13b, GPT, DeepSpeed, A100 GPU, Fireworks.ai.

What results were reported?

Daily relevance predictions: 70 million predictions per day; Llama3-8b vs Llama2-7b performance gains: 1.4% to 8%; Krippendorff's Alpha (GPT production model): 0.56; query-product pairs labeled per hour (GPT model): ~300k query product pairs per hour (source-reported, not independently verified).

What failed first in this deployment?

Prompt engineering alone could not capture Faire's definition of semantic search relevance, and the fine-tuned GPT solution was increasingly constrained by external API costs, limiting throughput for labeling.

How is this ecommerce ops AI workflow structured?

Human-labeled ground truth → Llama3-8b fine-tuning with LoRA → ESCI relevance classification → Self-hosted batch inference → Downstream search optimization.

Related ecommerce ops cases