Quality assurance · Production

GoDaddy builds a hybrid LLM and Spark synthetic data generator to eliminate test data bottlenecks

The problem

GoDaddy's data teams lacked sufficient, realistic test data for validating pipelines before production. Using real production data in lower environments posed privacy and security risks, while manually crafting test data was slow, costly, and impossible to maintain across hundreds of schemas.

First attempt

Prior approaches all proved unworkable: engineers spent days on manual JSON and SQL scripts that could not scale; off-the-shelf generators failed on complex schemas; and pure LLM generation was prohibitively expensive and slow at row-scale.

Workflow diagram · grounded in source
1
Schema submitted via API
trigger
“Producer or consumer submits schema: via Data Lake API. Payload includes schema definition, dataset name, target bucket/path, and generation parameters (row count, partitions, date range).”
2
Persist schema in DynamoDB
integration
“Data Lake API writes a new record to DynamoDB: schema, owner, target S3, desired state, and an initial status (e.g., SUBMITTED).”
3
DLMS orchestrates generation workflow
routing
“DLMS API is invoked to start a generation workflow for that schema/version. It acts as an orchestrator (auth, validation, idempotency, retries).”
4
GoCode generates Datagen template
ai_action
“a GoCode template engine renders a strongly typed data model and generation logic from the schema (e.g., field types, constraints, distributions, PK/FK rules). Output is compiled or packaged for Spark execution.”
5
Template validation layer
validation
“We implemented a validation layer that checks generated templates against schema constraints and a library of known-good patterns. Invalid templates are regenerated with more specific prompts.”
6
EMR Spark synthesizes records
output
“Spark job (EMR) uses the generated code to synthesize realistic, schema-conformant records at scale (partitioning, skew handling, nullability, referential integrity). Data is serialized to Parquet with proper compression and column types.”
7
Write Parquet files to S3
output
“Output is written to the producer's designated S3 prefix (e.g., s3://producer-bucket/datasets/<name>/dt=YYYY-MM-DD/). Cross-account write is permitted via bucket policy or an assumed role (depending on org policy).”
8
Status updated in DynamoDB
integration
“On success or failure, the workflow updates DynamoDB (RUNNING → SUCCEEDED/FAILED) and exposes status via DLMS API for UI/automation polling.”
Reported outcome

Since launching, GoDaddy achieved a 90% reduction in time spent creating test data, 100% elimination of production data in test environments, and 5x faster pipeline development cycles.

Reported metrics
Time spent creating test data90%
Production data in test environments100%
Pipeline development cycle speed5x faster
cost vs always-on EMR clusters80%
Show all 5 reported metrics
time spent creating test data90%
production data in test environments100%
pipeline development cycle speed5x faster
cost vs always-on EMR clusters80%
cost vs pure LLM generation99.9%
Reported stack
GoCodeDatabricks Labs DatagenEMR ServerlessLambdaDynamoDBS3SparkDLMSDeX
Source
https://www.godaddy.com/resources/news/building-a-synthetic-data-generator
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Since launching, GoDaddy achieved a 90% reduction in time spent creating test data, 100% elimination of production data in test environments, and 5x faster pipeline development cycles.

What tools did this team use?

GoCode, Databricks Labs Datagen, EMR Serverless, Lambda, DynamoDB, S3, Spark, DLMS, DeX.

What results were reported?

Time spent creating test data: 90%; Production data in test environments: 100%; Pipeline development cycle speed: 5x faster; cost vs always-on EMR clusters: 80% (source-reported, not independently verified).

What failed first in this deployment?

Prior approaches all proved unworkable: engineers spent days on manual JSON and SQL scripts that could not scale; off-the-shelf generators failed on complex schemas; and pure LLM generation was prohibitively expensive…

How is this quality assurance AI workflow structured?

Schema submitted via API → Persist schema in DynamoDB → DLMS orchestrates generation workflow → GoCode generates Datagen template → Template validation layer → EMR Spark synthesizes records → Write Parquet files to S3 → Status updated in DynamoDB.

Related quality assurance cases