Field service · Production

Multimodal RAG with Vision: Microsoft ISE experiments on image-enriched document retrieval for enterprise Q&A

The problem

Enterprise documents contain both textual and image content such as photographs, diagrams, and screenshots; standard text-only RAG pipelines cannot surface image information, limiting the relevance of LLM responses to image-related queries.

First attempt

Multi-modal embeddings like CLIP were initially considered but rejected due to word-count limits and inability to capture detailed visual information; the inference model also did not reliably return parsable JSON output.

Workflow diagram · grounded in source
1
Document ingestion via custom loader
trigger
“The ingestion process extracts both text and image data from source documents using a custom loader”
2
Image classification for relevance
ai_action
“The image tagging functionality in Azure Computer Vision Image Analysis is used for this purpose – we don't generate descriptions for images that are classified as a logo (based on a configurable confidence score threshold) or that don't…”
3
MLLM image description generation
ai_action
“images in the source document are replaced with MLLM-generated textual descriptions if they are deemed to be potentially relevant to the document subject matter”
4
Indexing enriched data in Azure AI Search
integration
“The extracted data is then ingested into Azure AI Search, making it available for user queries”
5
User query and chunk retrieval
trigger
“End users submit queries to receive meaningful responses from the AI system. Upon receiving a query, the system utilizes Azure AI Search to retrieve initial chunks of relevant data”
6
Optional reranking for precision
ai_action
“These data chunks can optionally undergo a reranking process to enhance search result precision”
7
LLM answer and citation generation
output
“The final chunks are then passed as context to an Azure OpenAI LLM to generate an answer to the user query grounded in the relevant source information, resulting in the final LLM-generated response to the user's query and related citations”
Reported outcome

Including document metadata produced a statistically significant improvement in source recall; storing image annotations as separate chunks yielded notable statistical improvements in both source document and image retrieval metrics; and introducing an image classifier substantially reduced ingestion time while maintaining statistically similar recall.

Reported metrics
Source recall improvement from metadatastatistically significant improvement in source recall performance
Retrieval metrics improvement from separate image chunksnotable statistical improvements in both source document and image retrieval metrics
Ingestion time reduction from classifiersubstantially reducing ingestion time
Search latency change from separate chunksno change in terms of search latency
Reported stack
GPT-4VGPT-4oAzure AI SearchAzure Computer Vision Image AnalysisAzure OpenAI ServiceAzure AI Services
Source
https://devblogs.microsoft.com/ise/multimodal-rag-with-vision/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Including document metadata produced a statistically significant improvement in source recall; storing image annotations as separate chunks yielded notable statistical improvements in both source document and image re…

What tools did this team use?

GPT-4V, GPT-4o, Azure AI Search, Azure Computer Vision Image Analysis, Azure OpenAI Service, Azure AI Services.

What results were reported?

Source recall improvement from metadata: statistically significant improvement in source recall performance; Retrieval metrics improvement from separate image chunks: notable statistical improvements in both source document and image retrieval metrics; Ingestion time reduction from classifier: substantially reducing ingestion time; Search latency change from separate chunks: no change in terms of search latency (source-reported, not independently verified).

What failed first in this deployment?

Multi-modal embeddings like CLIP were initially considered but rejected due to word-count limits and inability to capture detailed visual information; the inference model also did not reliably return parsable JSON out…

How is this field service AI workflow structured?

Document ingestion via custom loader → Image classification for relevance → MLLM image description generation → Indexing enriched data in Azure AI Search → User query and chunk retrieval → Optional reranking for precision → LLM answer and citation generation.

Related field service cases