Compliance monitoring · Production

ESGpedia unifies ESG data with Databricks lakehouse and RAG, achieving 4x cost savings in pipeline management

The problem

ESGpedia managed approximately 300 fragmented data pipelines across multiple platforms, each requiring extensive precleaning, processing, and relationship mapping, leading to slower response times and hampering AI-driven initiatives.

Workflow diagram · grounded in source

Continuous data ingestion

trigger

“The Databricks Platform unlocked streaming data capabilities, enabling continuous data ingestion from various sources”

Unified governance via Unity Catalog

integration

“Unity Catalog played a critical role in data management and governance, supporting compliance requirements with stringent access controls and detailed data lineage”

RAG solution for internal teams

ai_action

“The company used the Agent Bricks Custom Agents to develop a RAG solution specifically tailored to improve the efficiency and effectiveness of their internal teams”

Few-shot prompting for classification

ai_action

“we currently are using few-shot prompting to help with the classification of our datasets”

Context-aware insights delivered

output

“The implementation of Agent Bricks Custom Agents RAG techniques has enhanced ESGpedia's ability to provide nuanced, context-aware insights to their corporate and bank clients”

Reported outcome

ESGpedia achieved 4x cost savings in data pipeline management and migrated approximately 300 pipelines in six months, while significantly improving time to insight and enabling nuanced, context-aware ESG insights via RAG for corporate and bank clients.

Reported metrics

Data pipeline management cost savings4x

Pipeline migration durationsix months

Time to insightgreatly improved

Employee productivityenhances productivity and empowers employees to perform their roles with greater efficacy

Reported stack

DatabricksDatabricks Data Intelligence PlatformUnity CatalogDatabricks Mosaic AIAgent Bricks Custom AgentsRAGLLMs

Source

https://www.databricks.com/customers/esgpedia

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

ESGpedia achieved 4x cost savings in data pipeline management and migrated approximately 300 pipelines in six months, while significantly improving time to insight and enabling nuanced, context-aware ESG insights via…

What tools did this team use?

Databricks, Databricks Data Intelligence Platform, Unity Catalog, Databricks Mosaic AI, Agent Bricks Custom Agents, RAG, LLMs.

What results were reported?

Data pipeline management cost savings: 4x; Pipeline migration duration: six months; Time to insight: greatly improved; Employee productivity: enhances productivity and empowers employees to perform their roles with greater efficacy (source-reported, not independently verified).

How is this compliance monitoring AI workflow structured?

Continuous data ingestion → Unified governance via Unity Catalog → RAG solution for internal teams → Few-shot prompting for classification → Context-aware insights delivered.