Back office ops · Production

Toqan.ai: Prosus deploys a generative AI assistant across 24 group companies for collective discovery at scale

The problem

Prosus needed a scalable way to give all colleagues across 24 group companies first-hand access to generative AI, enable bottom-up discovery of viable use cases, and maintain safety guardrails at scale across a highly varied user base.

Workflow diagram · grounded in source
1
Employee submits task via Slack
trigger
“Initially accessible through Slack”
2
Multi-model AI processing
ai_action
“integrates several Large Language Models — commercial, OpenSource and fine-tuned in house — but also image interpretation and generation, voice encoding and generation, large document processing, data analysis and code creation, for a to…”
3
Internal knowledge base grounding
integration
“It also accesses the internal knowledge bases of the companies to provide grounded responses”
4
Privacy and security guardrails
validation
“We implemented several guardrails, including privacy and security measures like no-learning and no-retention policies, to protect data from being used to train future models”
5
User feedback capture
feedback_loop
“we introduced a feedback mechanism, including options for positive (thumbs up, heart) and negative (thumbs down, Pinocchio for unreliable or fabricated answers) feedback”
6
Model quality improvement loop
feedback_loop
“this rate dropped to below 3% by June 2023 and stabilized around 1.5%, thanks to improvements in the underlying models, enhanced prompting techniques, and better user proficiency in crafting prompts”
7
Use case graduation to production
output
“First they stress test the use cases with the AI Assistant until convinced that they work, and then graduate them into the regular engineering practices and into production”
Reported outcome

Over 81% of users report productivity increases of more than 5–10%, A/B testing shows time reductions of 50% or more for certain tasks, and the hallucination rate dropped from almost 10% to around 1.5%.
The assistant now serves approximately 13,000 colleagues across 24 companies.

Reported metrics
Users reporting productivity increase of more than 5–10%over 81%
time reduction for certain tasks (A/B testing)50% or more
Users turning to assistant as first help resourceabout 60%
Pinocchio (hallucination) feedback rate — fall 2022almost 10%
Show all 7 reported metrics
users reporting productivity increase of more than 5–10%over 81%
time reduction for certain tasks (A/B testing)50% or more
users turning to assistant as first help resourceabout 60%
Pinocchio (hallucination) feedback rate — fall 2022almost 10%
Pinocchio (hallucination) feedback rate — stabilizedaround 1.5%
colleagues using the assistantapproximately 13,000
engineering-related task share of interactionsapproximately 50%
Reported stack
Toqan.aiSlack
Source
https://medium.com/prosus-ai-tech-blog/harnessing-generative-ai-for-collective-discovery-lessons-from-two-years-of-deployment-at-scale-5792d6e46cac
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Over 81% of users report productivity increases of more than 5–10%, A/B testing shows time reductions of 50% or more for certain tasks, and the hallucination rate dropped from almost 10% to around 1.5%.

What tools did this team use?

Toqan.ai, Slack.

What results were reported?

Users reporting productivity increase of more than 5–10%: over 81%; time reduction for certain tasks (A/B testing): 50% or more; Users turning to assistant as first help resource: about 60%; Pinocchio (hallucination) feedback rate — fall 2022: almost 10% (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Employee submits task via Slack → Multi-model AI processing → Internal knowledge base grounding → Privacy and security guardrails → User feedback capture → Model quality improvement loop → Use case graduation to production.

Related back office ops cases