Data entry ops · Production

Databricks builds a bespoke fine-tuned LLM for AI-generated data catalog documentation in 1 month for under $1,000

The problem

In virtually every organization, the vast majority of database tables are undocumented, making it difficult for humans to discover data and for AI agents to automatically find datasets. An initial prototype using off-the-shelf SaaS LLMs ran into challenges with quality, performance, and cost that blocked production launch.

First attempt

All tested versions of SaaS LLMs exhibited the same challenges: as general-purpose models they were too slow and costly at scale, and risked regressions on the narrow documentation use case as they evolved for other use cases.

Workflow diagram · grounded in source
1
Schema-based doc generation trigger
trigger
“automatically generate documentation for tables and their columns based on their schema”
2
LLM suggests descriptions
ai_action
“automatically suggest descriptions for the tables and columns”
3
User accepts or modifies
human_review
“allow users to either individually accept, bulk accept, or modify the suggestions for higher fidelity”
4
Synthetic training data generation
ai_action
“synthesized CREATE TABLE statements using the above use cases to yield a diverse set of tables and generated sample responses including table descriptions and column comments using another LLM. In total, we generated ~3600 training examples”
5
Double-blind model evaluation
validation
“we set up a simple double-blind evaluation framework in which we asked 4 employees to rate table descriptions generated from the two models we wanted to compare using a set of 62 unseen tables”
6
Production acceptance rate feedback
feedback_loop
“we could measure a model's quality through production metrics such as the rate of users accepting the suggestions”
Reported outcome

Databricks built and deployed a bespoke fine-tuned LLM that delivered better quality, higher throughput, and more than a 10-fold reduction in cost, with more than 80% of table metadata updates now AI-assisted in production on Amazon Web Services and Google Cloud.

Reported metrics
table metadata updates AI-assistedmore than 80%
cost reduction vs SaaS LLMmore than 10-fold reduction in cost
Fine-tuning compute costless than $1,000
Fine-tuning durationaround 15 minutes
Show all 6 reported metrics
table metadata updates AI-assistedmore than 80%
cost reduction vs SaaS LLMmore than 10-fold reduction in cost
fine-tuning compute costless than $1,000
fine-tuning durationaround 15 minutes
development time1 month
training examples generated~3600
Reported stack
Unity CatalogMPT-7BDatabricks Data Intelligence PlatformAmazon Web ServicesGoogle Cloud
Source
https://www.databricks.com/blog/creating-bespoke-llm-ai-generated-documentation
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Databricks built and deployed a bespoke fine-tuned LLM that delivered better quality, higher throughput, and more than a 10-fold reduction in cost, with more than 80% of table metadata updates now AI-assisted in produ…

What tools did this team use?

Unity Catalog, MPT-7B, Databricks Data Intelligence Platform, Amazon Web Services, Google Cloud.

What results were reported?

table metadata updates AI-assisted: more than 80%; cost reduction vs SaaS LLM: more than 10-fold reduction in cost; Fine-tuning compute cost: less than $1,000; Fine-tuning duration: around 15 minutes (source-reported, not independently verified).

What failed first in this deployment?

All tested versions of SaaS LLMs exhibited the same challenges: as general-purpose models they were too slow and costly at scale, and risked regressions on the narrow documentation use case as they evolved for other u…

How is this data entry ops AI workflow structured?

Schema-based doc generation trigger → LLM suggests descriptions → User accepts or modifies → Synthetic training data generation → Double-blind model evaluation → Production acceptance rate feedback.

Related data entry ops cases