Back office ops · Production

Grab enables conversational data discovery with LLMs via HubbleIQ

The problem

With over 200,000 tables in their data lake, Grab's data consumers struggled to find the right datasets: Elasticsearch-based search could not handle semantic queries, documentation coverage was only 20% for critical P80 tables, and 51% of users spent multiple days hunting for data via tribal knowledge and Slack.

First attempt

Elasticsearch in its vanilla form could not perform semantic search, causing users to abandon Hubble searches and fall back to asking colleagues on Slack for dataset help.

Workflow diagram · grounded in source
1
Data consumer submits query
trigger
“where data consumers come to ask contextual search queries for their domains”
2
Query classification and routing
routing
“the Hubble team identified four categories of data search queries that were seen either on the Hubble UI or in Slack channels”
3
Enhanced Elasticsearch serves reference queries
integration
“Through user interviews, the team discovered how Elasticsearch should be tuned to better fit the Grab context. They implemented the following optimisations: Tagging and boosting P80 tables”
4
GPT-4 generates table documentation
ai_action
“we built a documentation generation engine that used GPT-4 to generate documentation based on table schemas and sample data. We refined the prompt through multiple iterations of feedback from data producers. Such docs were visible to dat…”
5
Data producer reviews generated docs
human_review
“notifying data producers to review the generated documentation”
6
HubbleIQ answers semantic discovery queries
ai_action
“we used Glean Apps to create the HubbleIQ bot, which was essentially an LLM with a custom system prompt that could access all Hubble datasets that were catalogued on Glean. Finally, we integrated this bot into Hubble search, such that fo…”
7
Results delivered in Hubble or Slack
output
“we integrated HubbleIQ with Slack, allowing data consumers to discover datasets without breaking their flow”
Reported outcome

The click-through rate rose from 82% to 94%, P80 table documentation coverage increased by 70 percentage points to ~90%, and 73% of survey respondents found it easy to discover datasets—a 17 percentage point improvement over the previous survey.

Reported metrics
Data lake tables (scale)over 200,000
Click-through rate baseline82%
Click-through rate (post-enhancement)94%
Click-through rate improvement12 percentage point increase
Show all 11 reported metrics
data lake tables (scale)over 200,000
click-through rate baseline82%
click-through rate (post-enhancement)94%
click-through rate improvement12 percentage point increase
P80 table documentation coverage (baseline)20%
P80 table documentation coverage increase70 percentage points to ~90%
users finding generated docs useful~95%
respondents finding dataset discovery easy73%
dataset discovery ease improvement17 percentage point increase
data consumers taking multiple days to find datasets51%
exact and partial searches share of Hubble traffic75%
Reported stack
HubbleDatahubElasticsearchGPT-4GleanGlean AppsHubbleIQSlack
Source
https://engineering.grab.com/hubble-data-discovery
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The click-through rate rose from 82% to 94%, P80 table documentation coverage increased by 70 percentage points to ~90%, and 73% of survey respondents found it easy to discover datasets—a 17 percentage point improveme…

What tools did this team use?

Hubble, Datahub, Elasticsearch, GPT-4, Glean, Glean Apps, HubbleIQ, Slack.

What results were reported?

Data lake tables (scale): over 200,000; Click-through rate baseline: 82%; Click-through rate (post-enhancement): 94%; Click-through rate improvement: 12 percentage point increase (source-reported, not independently verified).

What failed first in this deployment?

Elasticsearch in its vanilla form could not perform semantic search, causing users to abandon Hubble searches and fall back to asking colleagues on Slack for dataset help.

How is this back office ops AI workflow structured?

Data consumer submits query → Query classification and routing → Enhanced Elasticsearch serves reference queries → GPT-4 generates table documentation → Data producer reviews generated docs → HubbleIQ answers semantic discovery queries → Results delivered in Hubble or Slack.

Related back office ops cases