Procurement · Production

Scoutbee's 16-18 month journey building LLM-powered supplier discovery through four production stages

The problem

Scoutbee, a supply chain supplier discovery platform serving enterprises like Unilever and Walmart, wanted to bring LLMs into a new generation of their product, but foundational models lacked domain knowledge about supplier discovery, produced hallucinated results, and raised enterprise data privacy concerns.

First attempt

The initial ChatGPT API integration failed because foundational models lacked domain knowledge and hallucinated fake suppliers. A subsequent agent-based approach remained unpredictable and nearly impossible to debug, with agents randomly fabricating supplier answers even after domain adaptation and guardrails were introduced.

Workflow diagram · grounded in source

User submits supplier query

trigger

“Once the user starts to have a conversation with the system”

Chain of Thoughts query processing

ai_action

“Our planner and execution layer for LLMs now went from having an agent and a guardrail, to having Chain of Thoughts prompting, query rewrite, splitting into multiple query generation, custom guardrails based on Graphs of Thought, query b…”

Graphs of Thought guardrails

validation

“we thought of our entire business process as a graph. At any given point in time, we knew where the user was, and we invoked different sorts of guardrails to make sure that LLM is not misleading the user.”

RAG retrieval from knowledge graphs

ai_action

“We're using knowledge graphs as our data source, and not vector database”

Cited supplier results delivered

output

“we force it to provide citations with data provenance, it actually tells the user that this calmcoffee supplier who produces coffee beans, they're based out of Brazil, and we found this information from this place”

Reported outcome

After introducing RAG with Chain of Thoughts prompting, hallucinations drastically reduced and testing became much easier.
Results now include citations with data provenance, allowing users to trust and verify the source of each supplier answer.

Reported metrics

Hallucinationsdrastically reduced

Testing easewhole lot easier

User trust in resultsnow they can start to trust

Reported stack

ChatGPTLangChainLLaMA-13BFastChat APIHugging FaceRagasSparkknowledge graphs

Source

https://www.infoq.com/presentations/architecture-llm/?topicPageSponsorship=88befbbd-30f0-4d18-9d43-0bf2cb3e751d

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After introducing RAG with Chain of Thoughts prompting, hallucinations drastically reduced and testing became much easier.

What tools did this team use?

ChatGPT, LangChain, LLaMA-13B, FastChat API, Hugging Face, Ragas, Spark, knowledge graphs.

What results were reported?

Hallucinations: drastically reduced; Testing ease: whole lot easier; User trust in results: now they can start to trust (source-reported, not independently verified).

What failed first in this deployment?

The initial ChatGPT API integration failed because foundational models lacked domain knowledge and hallucinated fake suppliers.

How is this procurement AI workflow structured?

User submits supplier query → Chain of Thoughts query processing → Graphs of Thought guardrails → RAG retrieval from knowledge graphs → Cited supplier results delivered.

After introducing RAG with Chain of Thoughts prompting, hallucinations drastically reduced and testing became much easier.Results now include citations with data provenance, allowing users to trust and verify the source of each supplier answer.