Back office ops · Production

Wix builds a domain-adapted custom LLM that outperforms GPT-3.5 on Wix-specific tasks

The problem

Standard LLM customization techniques—prompt engineering, RAG, and task-specific fine-tuning—suffered from high cost, high latency, model hallucination, and an inability to handle multiple domain tasks simultaneously.

First attempt

Prompt engineering and RAG required overly complex prompts prone to overfitting, vendor-provided fine-tuning services overfitted to specific prompts without yielding cross-domain capabilities, and existing LLM and RAG solutions did not perform well enough on Wix domain tasks.

Workflow diagram · grounded in source
1
Build domain evaluation dataset
integration
“To estimate the knowledge of the LLM we built a custom Wix Question and Answers (Q&A) dataset. The question and answer data was taken from existing customer service live chats and FAQs.”
2
LLM-as-judge evaluation
ai_action
“we applied the LLM-as-a-judge technique as presented in Figure 3. The "judge" is a prompt that compares answers suggested by an LLM to the ground-truth one. After assessing the performance of several open-source LLMs as judges, we decide…”
3
Synthesize training data
ai_action
“we synthetically generated Q&As using organizational data such as knowledge base articles, customer support chats, technical documentation, and internal reports”
4
Fine-tune LLaMa2 on AWS P5
ai_action
“We decided to limit our work to one high-power GPU running on AWS P5 that allowed us to experiment with full-scale fine-tuning and LoRA with high ranks on LLaMa2 7b”
5
Custom LLM delivers Wix domain tasks
output
“Our smaller, customized Wix LLM showed better results than GPT 3.5 models on a variety of Wix tasks and opened the door for more impact in the organization”
Reported outcome

Wix's smaller customized LLM showed better results than GPT 3.5 models on a variety of Wix tasks and opened the door for more impact in the organization.

Reported metrics
custom LLM performance vs GPT-3.5better results than GPT 3.5 models on a variety of Wix tasks
Domain-specific training data ratio2%
Reported stack
RAGLLaMa2LoRAAWS P5
Source
https://www.wix.engineering/post/customizing-llms-for-enterprise-data-using-domain-adaptation-the-wix-journey
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Wix's smaller customized LLM showed better results than GPT 3.5 models on a variety of Wix tasks and opened the door for more impact in the organization.

What tools did this team use?

RAG, LLaMa2, LoRA, AWS P5.

What results were reported?

custom LLM performance vs GPT-3.5: better results than GPT 3.5 models on a variety of Wix tasks; Domain-specific training data ratio: 2% (source-reported, not independently verified).

What failed first in this deployment?

Prompt engineering and RAG required overly complex prompts prone to overfitting, vendor-provided fine-tuning services overfitted to specific prompts without yielding cross-domain capabilities, and existing LLM and RAG…

How is this back office ops AI workflow structured?

Build domain evaluation dataset → LLM-as-judge evaluation → Synthesize training data → Fine-tune LLaMa2 on AWS P5 → Custom LLM delivers Wix domain tasks.

Related back office ops cases