Hr ops · Production

DDI reduces behavioral simulation scoring from 48 hours to 10 seconds with Databricks

The problem

DDI's behavioral simulation assessments relied on trained human assessors taking 24 to 48 hours to evaluate and score candidate responses, limiting scalability and driving operational costs. Infrastructure challenges — hardware orchestration, scaling, data privacy, and multi-vendor coordination — had also blocked effective ML automation.

Workflow diagram · grounded in source
1
Candidate submits assessment
trigger
“Candidates complete an assessment and submit their responses.”
2
LLM classifies and scores responses
ai_action
“The instruction fine-tuned Llama3-8b achieved an F1 score of 0.86, compared to the baseline score of 0.76”
3
Model endpoints serve scores
integration
“Unity Catalog, combined with the Model Serving feature, has been particularly beneficial for deploying models with auto-scaling and serverless computing capabilities.”
4
Simulation report delivered
output
“drastically reduced the simulation report delivery time from 48 hours to just 10 seconds”
Reported outcome

ML models on Databricks reduced simulation report delivery time from 48 hours to 10 seconds.
Prompt optimization with DSPy improved recall from 0.43 to 0.98, and the instruction fine-tuned Llama3-8b reached an F1 score of 0.86 versus a baseline of 0.76. The LLMs demonstrated high reliability and precision in scoring.

Reported metrics
Simulation report delivery time48 hours to just 10 seconds
recall score improvement via DSPy0.43 to 0.98
F1 score (Llama3-8b instruction fine-tuned vs baseline)0.86 vs baseline 0.76
Reported stack
Databricks Data Intelligence PlatformDatabricks NotebooksMLflowUnity CatalogDSPyLlama3-8bChat GPT-4Azure Active Directory
Source
https://www.databricks.com/customers/ddi
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

ML models on Databricks reduced simulation report delivery time from 48 hours to 10 seconds.

What tools did this team use?

Databricks Data Intelligence Platform, Databricks Notebooks, MLflow, Unity Catalog, DSPy, Llama3-8b, Chat GPT-4, Azure Active Directory.

What results were reported?

Simulation report delivery time: 48 hours to just 10 seconds; recall score improvement via DSPy: 0.43 to 0.98; F1 score (Llama3-8b instruction fine-tuned vs baseline): 0.86 vs baseline 0.76 (source-reported, not independently verified).

How is this hr ops AI workflow structured?

Candidate submits assessment → LLM classifies and scores responses → Model endpoints serve scores → Simulation report delivered.

Related hr ops cases