Customer support · Production

Building a context-sensitive AI assistant with RAG: Lessons from building Vectorize's in-product documentation assistant

The problem

Vectorize users had to leave the product UI to visit the docs site, creating disruptive context-switching, while the in-product Intercom chat was not always staffed and some users were reluctant to ask for help.

Workflow diagram · grounded in source

User clicks Ask AI in UI

trigger

“When the user clicks on "Ask AI" a chat window appears at the top right of screen”

Multi-source RAG pipeline ingestion

integration

“Originally, I was just going to scrape the docs site using the web crawler connector. But because Vectorize supports multiple source connectors in a pipeline and has integrations with Discord and Intercom, I was also able to easily pull …”

Topic-prefixed retrieval query

ai_action

“// Prefix the question with the topic to improve retrieval const contextualizedQuestion = `(${topic}) ${question}`;”

Reranking and relevance filtering

validation

“I found by experimenting that a relevance threshold of 0.5 works well. Any data retrieved below that threshold is not sent to the LLM”

LLM generates answer

ai_action

“we have been using the Llama 3.1 70B model hosted on Groq”

Response delivered in chat window

output

“instructing the LLM to answer the question in Markdown format (for better presentation)”

User thumbs feedback to analytics

feedback_loop

“Clicking on these buttons sends messages to our analytics system so we can track how users are responding to our AI assistant”

Reported outcome

Vectorize built a context-sensitive AI assistant embedded directly in the product UI, powered by a self-improving multi-source RAG pipeline.
Topic-aware retrieval and reranking improved response quality, and anti-hallucination prompting prevented the LLM from fabricating answers.

Reported metrics

Retrieval response qualityreally improved the quality of the responses

LLM inference speedlightning fast

Inference costlower cost

Reported stack

VectorizeGroqLlama 3.1 70BReactreranking modelDiscordIntercom

Source

https://vectorize.io/creating-a-context-sensitive-ai-assistant-lessons-from-building-a-rag-application/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Vectorize built a context-sensitive AI assistant embedded directly in the product UI, powered by a self-improving multi-source RAG pipeline.

What tools did this team use?

Vectorize, Groq, Llama 3.1 70B, React, reranking model, Discord, Intercom.

What results were reported?

Retrieval response quality: really improved the quality of the responses; LLM inference speed: lightning fast; Inference cost: lower cost (source-reported, not independently verified).

How is this customer support AI workflow structured?

User clicks Ask AI in UI → Multi-source RAG pipeline ingestion → Topic-prefixed retrieval query → Reranking and relevance filtering → LLM generates answer → Response delivered in chat window → User thumbs feedback to analytics.

Vectorize built a context-sensitive AI assistant embedded directly in the product UI, powered by a self-improving multi-source RAG pipeline.Topic-aware retrieval and reranking improved response quality, and anti-hallucination prompting prevented the LLM from fabricating answers.