Back office ops · Production

Symend migrates from Azure Data Factory to Airbyte, cutting data latency 75% and projecting $900K annual savings

The problem

Symend's legacy data infrastructure built on Microsoft Azure Data Factory could not scale with its growing customer base and AI ambitions, suffering from performance, scalability, cost, and reliability problems that limited its ability to leverage data effectively.

First attempt

Azure Data Factory lacked sufficient parallelization, causing cascading failures where a single pipeline failure would halt all subsequent client jobs.

Workflow diagram · grounded in source
1
Enterprise source data ingestion
trigger
“Easy-to-manage connectors for Microsoft SQL, unstructured data in S3/Blob, and other enterprise sources”
2
Parallel pipeline loading to Snowflake
integration
“With Airbyte, things are run in parallel because of the platform's distributed nature, which means that we can process multiple clients at the same time without impacting performance”
3
Customer payment behavior scoring
ai_action
“Scoring models in Databricks to predict customer payment behavior”
4
Sentiment analysis and ML feature engineering
ai_action
“Sentiment analysis and feature engineering for ML applications”
5
Natural language data chatbot
ai_action
“This chatbot is a true self-service experience, removing bottlenecks from technical teams and enabling clients, business users, project managers, CEOs, CFOs, to just ask any questions to the data through natural language and gain insights”
6
Self-serve analytics dashboards
output
“Self Serve analytical dashboards to allow analysis of customer engagement performance”
Reported outcome

Replacing Azure Data Factory with Airbyte eliminated cascading pipeline failures, reduced data refresh latency from 2 hours to as low as 30 minutes — a potential 75% performance improvement — and is projected to save approximately $900,000 annually, while enabling AI-powered scoring models, sentiment analysis, and a self-service chatbot for business users.

Reported metrics
Projected annual cost savingsapproximately $900,000 annually
Data refresh latency reductionfrom 2 hours to an hour, and in some cases as low as 30 minutes
Performance improvement (potential)75%
Cascading pipeline failureseliminated
Reported stack
AirbyteMicrosoft Azure Data FactoryCortex AISnowflakeDatabricks
Source
https://airbyte.com/success-stories/symend
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Replacing Azure Data Factory with Airbyte eliminated cascading pipeline failures, reduced data refresh latency from 2 hours to as low as 30 minutes — a potential 75% performance improvement — and is projected to save…

What tools did this team use?

Airbyte, Microsoft Azure Data Factory, Cortex AI, Snowflake, Databricks.

What results were reported?

Projected annual cost savings: approximately $900,000 annually; Data refresh latency reduction: from 2 hours to an hour, and in some cases as low as 30 minutes; Performance improvement (potential): 75%; Cascading pipeline failures: eliminated (source-reported, not independently verified).

What failed first in this deployment?

Azure Data Factory lacked sufficient parallelization, causing cascading failures where a single pipeline failure would halt all subsequent client jobs.

How is this back office ops AI workflow structured?

Enterprise source data ingestion → Parallel pipeline loading to Snowflake → Customer payment behavior scoring → Sentiment analysis and ML feature engineering → Natural language data chatbot → Self-serve analytics dashboards.

Related back office ops cases