Back office ops · Production

Vannevar Labs fine-tunes multilingual sentiment analysis model in 2 weeks with Databricks Mosaic AI, reducing latency by 75%

The problem

Vannevar Labs needed accurate multilingual sentiment analysis for defense intelligence work, but GPT-4 with prompt engineering achieved only 65% accuracy, was too expensive, and struggled with lower-resourced languages like Tagalog. GPU resource shortages and infrastructure management complexity also blocked the team from fine-tuning their own model.

First attempt

GPT-4 with prompt engineering failed to meet accuracy requirements for multilingual sentiment classification, achieving only 65% accuracy while being cost-prohibitive and inadequate for lower-resourced languages.

Workflow diagram · grounded in source
1
Ingest data from public sources
integration
“MCLI's robust capabilities for data ingestion allowed seamless, secure connection to Vannevar's datasets”
2
Fine-tune Mistral 7B on domain data
ai_action
“Punma's team leveraged Databricks Model Training to fine-tune their models. Specifically, they fine-tuned Mistral's 7B parameter model using domain-specific data. This model was chosen for its open source nature and its ability to effici…”
3
Orchestrate multi-GPU training
integration
“Databricks also facilitated efficient training across multiple GPUs by managing the configurations through YAML files”
4
Export model to production storage
output
“convert their trained models to a standard Hugging Face format and export them to their Amazon S3 or Hugging Face Model Repository for production use”
5
Classify narrative sentiment
ai_action
“improve the accuracy of classifying the sentiment of news articles, blogs and social media related to specific narratives”
Reported outcome

The fine-tuned model achieved an F1 score of 76%, reduced latency by 75%, and was deployed within 2 weeks, enabling Vannevar Labs to process significantly more data more efficiently at lower cost.

Reported metrics
sentiment model F1 score76%
previous GPT-4 accuracy65%
Latency reduction75%
Time to deploy fine-tuned model2 weeks
Show all 6 reported metrics
sentiment model F1 score76%
previous GPT-4 accuracy65%
latency reduction75%
time to deploy fine-tuned model2 weeks
cost savingssignificant cost savings
data processing volumeprocess significantly more data more efficiently
Reported stack
Databricks Mosaic AIDatabricks Model TrainingMCLIPython SDKMistralNVIDIA A10 Tensor Core GPUWeights & BiasesGPT-4Amazon S3Hugging Face
Source
https://www.databricks.com/customers/vannevar-labs
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The fine-tuned model achieved an F1 score of 76%, reduced latency by 75%, and was deployed within 2 weeks, enabling Vannevar Labs to process significantly more data more efficiently at lower cost.

What tools did this team use?

Databricks Mosaic AI, Databricks Model Training, MCLI, Python SDK, Mistral, NVIDIA A10 Tensor Core GPU, Weights & Biases, GPT-4, Amazon S3, Hugging Face.

What results were reported?

sentiment model F1 score: 76%; previous GPT-4 accuracy: 65%; Latency reduction: 75%; Time to deploy fine-tuned model: 2 weeks (source-reported, not independently verified).

What failed first in this deployment?

GPT-4 with prompt engineering failed to meet accuracy requirements for multilingual sentiment classification, achieving only 65% accuracy while being cost-prohibitive and inadequate for lower-resourced languages.

How is this back office ops AI workflow structured?

Ingest data from public sources → Fine-tune Mistral 7B on domain data → Orchestrate multi-GPU training → Export model to production storage → Classify narrative sentiment.

Related back office ops cases