Back office ops · Production

Building a Resilient Embedding System for Semantic Search at Airtable

The problem

Building a robust embedding system for semantic search over customer data involves non-trivial challenges across lifecycle management, eventual consistency, migrations, and failure recovery at scale.

Workflow diagram · grounded in source
1
Data change detection
trigger
“When data changes, we update the embedding state's lastUpdatedTransaction to reflect the current transaction.”
2
Embedding task creation
routing
“Tasks are created to generate the embeddings for each config that saw data in its data subscription updated.”
3
Embedding generation
ai_action
“Our embedding service processes the task(s), and generates the embeddings. We have ample retry logic in case this fails”
4
Conditional out-of-order guard
validation
“We prevent out-of-order writes by making the insertion conditional on the transaction number for the write being greater than the transaction number stored in the vector DB.”
5
Vector DB persistence
output
“Once embeddings are generated we store them in our vector database.”
6
MemApp state confirmation
integration
“We then confirm with MemApp that the write has happened, updating the lastPersistedTransaction.”
7
Config reset for invalid embeddings
feedback_loop
“any time the system detects that embeddings are, or about to become, invalid, we reset the embedding config”
Reported outcome

Airtable built a resilient embedding system that accepts eventual consistency and uses a reset-based strategy to handle all migrations and failures, with p99.9 re-embedding time under 2 minutes.

Reported metrics
P99.9 re-embedding latencyp99.9 under 2 minutes
Embedding storage size relative to source data10x the size of the underlying data
Manual labor for insight discoveryhours of manual labor
Reported stack
MemAppMySqlOpenAIBedrock
Source
https://medium.com/airtable-eng/building-a-resilient-embedding-system-for-semantic-search-at-airtable-d5fdf27807e2
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Airtable built a resilient embedding system that accepts eventual consistency and uses a reset-based strategy to handle all migrations and failures, with p99.9 re-embedding time under 2 minutes.

What tools did this team use?

MemApp, MySql, OpenAI, Bedrock.

What results were reported?

P99.9 re-embedding latency: p99.9 under 2 minutes; Embedding storage size relative to source data: 10x the size of the underlying data; Manual labor for insight discovery: hours of manual labor (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Data change detection → Embedding task creation → Embedding generation → Conditional out-of-order guard → Vector DB persistence → MemApp state confirmation → Config reset for invalid embeddings.

Related back office ops cases