Back office ops · Production

IDIADA optimizes AIDA chatbot interaction routing using Amazon Bedrock classifiers

The problem

As AIDA's user interactions grew more complex over time—from simple queries to document translations, service requests, and specialized inquiries—a coherent system was needed to categorize those interactions and route each one to a more specialized pipeline.

First attempt

LLM-based classification with examples improved accuracy but faced critical scalability limits: the volume of examples required caused infrastructure overflow, quota issues with Amazon Bedrock, and runtimes up to 18 seconds—unacceptable for user experience.

Workflow diagram · grounded in source

User submits request to AIDA

trigger

“Initially, users were making simple queries to AIDA, but over time, they started to request more specific and complex tasks. These included document translations, inquiries about IDIADA's internal services, file uploads, and other specia…”

Classifier identifies interaction type

ai_action

“We present three different classification approaches: two based on LLMs and one using a classic machine learning (ML) algorithm. The aim is to understand which approach is most suitable for addressing the presented challenge.”

Route to specialized pipeline

routing

“By categorizing the interactions into three main groups—conversation, services, and document translation—the system can better understand the user's intent and respond accordingly”

Specialized pipeline delivers response

output

“This approach allows for tailored responses and processes for different types of user needs, whether it's a simple question, a document translation, or a complex inquiry about IDIADA's services”

Reported outcome

SVM and ANN models using Cohere's multilingual embeddings achieved the best balance of performance and speed, with the SVM reaching F1 scores of 0.99, 0.80, and 0.93 across the three classes at a runtime of approximately 0.3 seconds—far faster than the LLM's 1.2 seconds and LLM with examples' 18 seconds.

Reported metrics

ANN Cohere Document Translation F10.96

LLM Services F1 (baseline)0.22

LLM Document Translation F1 (baseline)0.46

LLM inference runtimeapproximately 1.2 seconds

Show all 9 reported metrics

ANN Cohere Document Translation F10.96

LLM Services F1 (baseline)0.22

LLM Document Translation F1 (baseline)0.46

LLM inference runtimeapproximately 1.2 seconds

LLM with examples inference runtimearound 18 seconds

ANN inference runtimeapproximately 0.15 seconds

SVM inference runtime0.3 seconds

training and test dataset size1,668

AIDA efficiency and accuracysignificant increase in efficiency and accuracy

Reported stack

Amazon BedrockClaude 3 SonnetLangChainBoto3Amazon Titan Text Embeddings G1Amazon S3Coherescikit-learnKerasTensorFlowPython

Source

https://aws.amazon.com/blogs/machine-learning/how-idiada-optimized-its-intelligent-chatbot-with-amazon-bedrock?tag=soumet-20

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

SVM and ANN models using Cohere's multilingual embeddings achieved the best balance of performance and speed, with the SVM reaching F1 scores of 0.99, 0.80, and 0.93 across the three classes at a runtime of approximat…

What tools did this team use?

Amazon Bedrock, Claude 3 Sonnet, LangChain, Boto3, Amazon Titan Text Embeddings G1, Amazon S3, Cohere, scikit-learn, Keras, TensorFlow.

What results were reported?

ANN Cohere Document Translation F1: 0.96; LLM Services F1 (baseline): 0.22; LLM Document Translation F1 (baseline): 0.46; LLM inference runtime: approximately 1.2 seconds (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

User submits request to AIDA → Classifier identifies interaction type → Route to specialized pipeline → Specialized pipeline delivers response.