Compliance monitoring · Production

Grab automates PetaByte-scale data classification with LLM, scanning 20,000+ entities per month

The problem

Grab managed PetaByte-level data across countless schemas, but manual PII campaigns were inconsistent — developers interpreted classification policies differently — and the data volume made table-level manual tagging infeasible. The existing third-party classification tool could not be customised and its regex classifiers generated too many false positives.

First attempt

The initial Gemini service built around a third-party classification tool had two blocking gaps: its ML classifiers could not be customised to Grab's internal taxonomy, and regex patterns produced excessive false positives. Building a bespoke in-house model was equally impractical due to the labelling and data-science investment required.

Workflow diagram · grounded in source
1
Data platform scan request
trigger
“Data platforms trigger scan requests to the Gemini service to initiate the tag classification process”
2
Request batching and rate limiting
routing
“it helps aggregate the requests into reasonable mini-batches. Aggregation is achievable through the message queue at fixed intervals. In addition, a rate limiter is attached at the workflow level”
3
LLM column-level tag classification
ai_action
“We ask the language model to be a column tag generator and to assign the most appropriate tag to each column”
4
Predictions published to Kafka
output
“The predictions are published to the Kafka queue to downstream data platforms”
5
Data owner tag verification
human_review
“The platforms inform respective users weekly to verify the classified tags to improve the model's correctness”
6
Iterative prompt improvement
feedback_loop
“to improve the model's correctness and to enable iterative prompt improvement”
Reported outcome

Within a month of rollout the LLM-powered system scanned over 20,000 data entities at 300–400 per day, saving an estimated 360 man-days per year.
Eighty percent of data owners reported the new tagging process helped them, and acknowledged tables required fewer than one tag change on average.

Reported metrics
Data entities scanned within one month of rolloutmore than 20,000
Daily classification throughput300-400 entities per day
Annual time savedapproximately 360 man-days per year
Data owner satisfaction with new tagging process80%
Show all 5 reported metrics
data entities scanned within one month of rolloutmore than 20,000
daily classification throughput300-400 entities per day
annual time savedapproximately 360 man-days per year
data owner satisfaction with new tagging process80%
average tags changed per acknowledged tableless than one tag
Reported stack
GeminiGPT3.5Azure OpenAIKafka
Source
https://engineering.grab.com/llm-powered-data-classification
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Within a month of rollout the LLM-powered system scanned over 20,000 data entities at 300–400 per day, saving an estimated 360 man-days per year.

What tools did this team use?

Gemini, GPT3.5, Azure OpenAI, Kafka.

What results were reported?

Data entities scanned within one month of rollout: more than 20,000; Daily classification throughput: 300-400 entities per day; Annual time saved: approximately 360 man-days per year; Data owner satisfaction with new tagging process: 80% (source-reported, not independently verified).

What failed first in this deployment?

The initial Gemini service built around a third-party classification tool had two blocking gaps: its ML classifiers could not be customised to Grab's internal taxonomy, and regex patterns produced excessive false posi…

How is this compliance monitoring AI workflow structured?

Data platform scan request → Request batching and rate limiting → LLM column-level tag classification → Predictions published to Kafka → Data owner tag verification → Iterative prompt improvement.

Related compliance monitoring cases