Sales ops · Production

MSX Sales Copilot: real-time content recommendation for Microsoft sellers using bi-encoder and cross-encoder LLM architecture

The problem

Microsoft sellers needed to find and share relevant technical documentation with customers in real time during calls, but were limited to an external filter-based search on the Seismic website rather than a tool embedded directly in their CRM interface.

First attempt

The previous external Seismic filter-based search was sub-optimal and not embedded in the MSX CRM interface sellers used daily.

Workflow diagram · grounded in source
1
Seller query entered
trigger
“a seller should be able to ask questions to the Copilot chatbot in natural language about any of the topics covered by the skills and get relevant answers to their questions”
2
Semantic Kernel planner routing
routing
“the planner in Semantic Kernel is invoked. Planners are used to generate plans and invoke the respective skills based on the context of the query”
3
Bi-encoder document retrieval
ai_action
“We apply the DistillBERT (pre-trained on MSMarco dataset) to all prompts in and cache the resulting set of embeddings”
4
Cross-encoder re-ranking
ai_action
“cross-encoder MSMarco MiniLM model which returns a similarity score for each one of the pairs”
5
Top-5 documents returned to seller
output
“Each of these has a link to the document which the seller can use to open it and share with the end customer using livesend links”
Reported outcome

The new real-time content recommendation system was deployed into the production MSX Copilot and described as a tremendous improvement by the seller community, receiving a daily task relevance score of 4 out of 5 and a document relevancy rating of 3.7 out of 5 in seller satisfaction surveys.

Reported metrics
Content recommendation skill daily task relevance (seller survey)4 out of 5
Recommended document relevancy rating (seller survey)3.7 out of 5
System response latencyfew seconds
Seller reception of new system vs predecessortremendous improvement
Reported stack
Semantic KernelAzure Machine LearningSeismicDynamics CRM
Source
https://arxiv.org/html/2401.04732v1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The new real-time content recommendation system was deployed into the production MSX Copilot and described as a tremendous improvement by the seller community, receiving a daily task relevance score of 4 out of 5 and…

What tools did this team use?

Semantic Kernel, Azure Machine Learning, Seismic, Dynamics CRM.

What results were reported?

Content recommendation skill daily task relevance (seller survey): 4 out of 5; Recommended document relevancy rating (seller survey): 3.7 out of 5; System response latency: few seconds; Seller reception of new system vs predecessor: tremendous improvement (source-reported, not independently verified).

What failed first in this deployment?

The previous external Seismic filter-based search was sub-optimal and not embedded in the MSX CRM interface sellers used daily.

How is this sales ops AI workflow structured?

Seller query entered → Semantic Kernel planner routing → Bi-encoder document retrieval → Cross-encoder re-ranking → Top-5 documents returned to seller.

Related sales ops cases