Back office ops · Production

Vivian Health connects healthcare professionals to their dream job with dbt Cloud and Metaplane

The problem

Vivian Health's data team faced recurring data incidents with downstream impacts that were hard to identify and fix. Manual pipeline testing was slow (one hour to a full day per pipeline), test thresholds required constant updates as data evolved, and new objects from product growth went unmonitored.

First attempt

Using Persistent Derived Tables in Looker for data transformation required specialized LookML knowledge and produced fragile, hard-to-maintain code. Manual unit tests were too slow to set up and could not keep pace with scale.

Workflow diagram · grounded in source
1
Multi-source data ingestion
integration
“Vivian combines Stitch, Airflow, Kinesis Firehose, and in-house builds, to funnel data into Snowflake. Upstream sources include multiple PostgreSQL databases that serve different workloads, various business applications, and events captu…”
2
Dbt transformation and modeling
output
“Everything from analytics tables to machine learning models is transformed and modeled in dbt. For example, the training data for ML models used to show the most accurate recommendations to job seekers in-product.”
3
Dbt pre-deployment testing
validation
“The ability to load sample data into dbt allows us to verify our models work as intended before deployment”
4
Metaplane data quality monitoring
validation
“Data quality is monitored from the upstream transactional PostgreSQL database to dbt-modeled tables in Snowflake, with the ability to see how incidents impact dashboards in Looker.”
5
Alert on data incident
output
“If I get a Metaplane alert, it's always something that's gone wrong, which is what we want. We want to ensure that we're catching everything without over-alerting”
6
Downstream data distribution
integration
“the team funnels their dbt models further downstream using Census to send modeled data feeds back into business applications themselves, making it even more convenient for internal stakeholders to draw insights in the tools they're famil…”
Reported outcome

Vivian Health tripled the number of data pipeline contributors, moved 100% of data models into dbt, achieved their data quality OKR, and saved weeks of effort on creating and maintaining custom tests.

Reported metrics
Data pipeline contributors3x increase
data quality OKR achieved1
Data models created in dbt100%
Effort on custom test creation and maintenancesaving weeks of effort
Reported stack
dbt CloudMetaplaneSnowflakeStitchAirflowKinesis FirehoseSegmentLookerCensusSnowparkSalesforceAmplitude
Source
https://www.getdbt.com/case-studies/vivian-health
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Vivian Health tripled the number of data pipeline contributors, moved 100% of data models into dbt, achieved their data quality OKR, and saved weeks of effort on creating and maintaining custom tests.

What tools did this team use?

dbt Cloud, Metaplane, Snowflake, Stitch, Airflow, Kinesis Firehose, Segment, Looker, Census, Snowpark.

What results were reported?

Data pipeline contributors: 3x increase; data quality OKR achieved: 1; Data models created in dbt: 100%; Effort on custom test creation and maintenance: saving weeks of effort (source-reported, not independently verified).

What failed first in this deployment?

Using Persistent Derived Tables in Looker for data transformation required specialized LookML knowledge and produced fragile, hard-to-maintain code.

How is this back office ops AI workflow structured?

Multi-source data ingestion → Dbt transformation and modeling → Dbt pre-deployment testing → Metaplane data quality monitoring → Alert on data incident → Downstream data distribution.

Related back office ops cases