Quality assurance · Production

Edmunds automates dealer review moderation with GPT-4 via Databricks Model Serving

The problem

Edmunds received over 300 dealer quality-of-service reviews daily that were moderated manually, sometimes taking up to 72 hours before vetted reviews could be published. IAM-based data governance also created coarse access control and little visibility into pipeline dependencies.

First attempt

Training an off-the-shelf model produced poor results, and fine-tuning still failed to handle the complex moderation rules because all rules had to be captured in the prompt with no flexibility for edge cases. Prompt engineering experiments were also difficult because there was no easy way to compare outputs across different models.

Workflow diagram · grounded in source
1
Daily review submissions
trigger
“With over 300 daily reviews submitted on both the quality of new and used cars and the dealers selling them”
2
GPT-4 auto-moderation
ai_action
“What we are using now is GPT-4 that is called through Databricks Model Serving endpoints with a lot of custom prompts on how to moderate it, and that has worked the best for us”
3
Accept or reject decision
validation
“Their custom prompt instructions direct the model whether to accept or reject a review — in a matter of seconds, not hours”
4
Moderator publishes reviews
human_review
“Edmunds moderators can now analyze and publish new reviews in minutes”
5
Unity Catalog governance
integration
“they decided to migrate to Databricks Unity Catalog in place of using their existing workspaces. They used external tables for the majority of their important pipelines, so they created metadata sync scripts to keep these tables in sync …”
Reported outcome

Edmunds now auto-moderates dealer reviews in minutes instead of up to 72 hours, saving three to five hours of staff time per week and requiring only two moderators freed for higher-value tasks.
Unity Catalog migration delivered improved auditing, compliance, security, and data discovery.

Reported metrics
Staff time saved per weekthree to five hours per week
Previous review turnaround timeup to 72 hours
New review publication timein minutes
Moderators requiredtwo
Show all 5 reported metrics
staff time saved per weekthree to five hours per week
previous review turnaround timeup to 72 hours
new review publication timein minutes
moderators requiredtwo
daily reviews processedover 300
Reported stack
DatabricksDatabricks Model ServingGPT-4Amazon S3
Source
https://www.databricks.com/customers/edmunds/gen-ai
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Edmunds now auto-moderates dealer reviews in minutes instead of up to 72 hours, saving three to five hours of staff time per week and requiring only two moderators freed for higher-value tasks.

What tools did this team use?

Databricks, Databricks Model Serving, GPT-4, Amazon S3.

What results were reported?

Staff time saved per week: three to five hours per week; Previous review turnaround time: up to 72 hours; New review publication time: in minutes; Moderators required: two (source-reported, not independently verified).

What failed first in this deployment?

Training an off-the-shelf model produced poor results, and fine-tuning still failed to handle the complex moderation rules because all rules had to be captured in the prompt with no flexibility for edge cases.

How is this quality assurance AI workflow structured?

Daily review submissions → GPT-4 auto-moderation → Accept or reject decision → Moderator publishes reviews → Unity Catalog governance.

Related quality assurance cases