Field service · Production

Multimodal RAG with Vision: Microsoft ISE experiments on image-enriched document retrieval for enterprise Q&A

The problem

Enterprise documents contain both textual and image content such as photographs, diagrams, and screenshots; standard text-only RAG pipelines cannot surface image information, limiting the relevance of LLM responses to image-related queries.

First attempt

Multi-modal embeddings like CLIP were initially considered but rejected due to word-count limits and inability to capture detailed visual information; the inference model also did not reliably return parsable JSON output.

Workflow diagram · grounded in source
1
Document ingestion via custom loader
Trigger
The ingestion process extracts both text and image data from source documents using a custom loader.
▾ source quote
“The ingestion process extracts both text and image data from source documents using a custom loader”
2
Image classification for relevance
Ai action
The image tagging functionality in Azure Computer Vision Image Analysis is used to classify images, excluding logos and images without text from enrichment.
▾ source quote
“The image tagging functionality in Azure Computer Vision Image Analysis is used for this purpose – we don't generate descriptions for images that are classified as a logo (based on a configurable confidence score threshold) or that don't contain any …”
3
MLLM image description generation
Ai action
Images in the source document are replaced with MLLM-generated textual descriptions if they are deemed potentially relevant to the document subject matter.
▾ source quote
“images in the source document are replaced with MLLM-generated textual descriptions if they are deemed to be potentially relevant to the document subject matter”
4
Indexing enriched data in Azure AI Search
Integration
The extracted and enriched data is ingested into Azure AI Search, making it available for user queries.
▾ source quote
“The extracted data is then ingested into Azure AI Search, making it available for user queries”
5
User query and chunk retrieval
Trigger
End users submit queries and the system utilizes Azure AI Search to retrieve initial chunks of relevant data.
▾ source quote
“End users submit queries to receive meaningful responses from the AI system. Upon receiving a query, the system utilizes Azure AI Search to retrieve initial chunks of relevant data”
6
Optional reranking for precision
Ai action
Retrieved data chunks can optionally undergo a reranking process to enhance search result precision.
▾ source quote
“These data chunks can optionally undergo a reranking process to enhance search result precision”
7
LLM answer and citation generation
Output
The final chunks are passed as context to an Azure OpenAI LLM to generate a grounded answer and related citations.
▾ source quote
“The final chunks are then passed as context to an Azure OpenAI LLM to generate an answer to the user query grounded in the relevant source information, resulting in the final LLM-generated response to the user's query and related citations”
Reported outcome

Including document metadata produced a statistically significant improvement in source recall; storing image annotations as separate chunks yielded notable statistical improvements in both source document and image retrieval metrics; and introducing an image classifier substantially reduced ingestion time while maintaining statistically similar recall.

Reported metrics
Source recall improvement from metadatastatistically significant improvement in source recall performance
Retrieval metrics improvement from separate image chunksnotable statistical improvements in both source document and image retrieval metrics
Ingestion time reduction from classifiersubstantially reducing ingestion time
Search latency change from separate chunksno change in terms of search latency
Reported stack
GPT-4VGPT-4oAzure AI SearchAzure Computer Vision Image AnalysisAzure OpenAI ServiceAzure AI Services
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://devblogs.microsoft.com/ise/multimodal-rag-with-vision/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Including document metadata produced a statistically significant improvement in source recall; storing image annotations as separate chunks yielded notable statistical improvements in both source document and image re…

What tools did this team use?

GPT-4V, GPT-4o, Azure AI Search, Azure Computer Vision Image Analysis, Azure OpenAI Service, Azure AI Services.

What results were reported?

Source recall improvement from metadata: statistically significant improvement in source recall performance; Retrieval metrics improvement from separate image chunks: notable statistical improvements in both source document and image retrieval metrics; Ingestion time reduction from classifier: substantially reducing ingestion time; Search latency change from separate chunks: no change in terms of search latency (source-reported, not independently verified).

What failed first in this deployment?

Multi-modal embeddings like CLIP were initially considered but rejected due to word-count limits and inability to capture detailed visual information; the inference model also did not reliably return parsable JSON out…

How is this field service AI workflow structured?

Document ingestion via custom loader → Image classification for relevance → MLLM image description generation → Indexing enriched data in Azure AI Search → User query and chunk retrieval → Optional reranking for precision → LLM answer and citation generation.

Related field service cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Field service cases →