Back office ops · Production

Hard-Earned Lessons from a Year of Building AI Agents at IBM Research

The problem

Non-experts could not translate generative AI capabilities into solving high-value problems, while only teams with deep LLM and systems engineering expertise could unlock AI's potential—leaving a broad set of everyday builders behind.

Workflow diagram · grounded in source
1
Non-expert AI gap identified
trigger
“Non-experts struggled to capture the promised productivity gains of generative AI. While generative AI offers vast potential, many users— struggled to translate its capabilities into solving high-value problems.”
2
AI agent prototyped
ai_action
“we rapidly prototyped an AI agent that — at the time — outperformed commercially available search solutions. This agent could break down complex queries into sub-steps, navigate multiple web pages to gather supporting information, execut…”
3
Trajectory explorer added
output
“A key addition we made was the trajectory explorer, a visual tool that allowed users to investigate the steps taken by the agent to generate its response. This feature significantly improved transparency and user trust.”
4
Early adopter testing
validation
“We tested this prototype with early adopters and uncovered several key insights”
5
BeeAI Framework built
output
“We evaluated existing tools for building agents and found a clear gap in addressing the needs of full-stack developers. This led to the development of the BeeAI Framework, a TypeScript-based library built specifically to fill this gap.”
6
Open-source stack launched
output
“we chose to open-source the entire stack to see what resonated with the community. This allowed us to reach a diverse range of users and measure real-world traction.”
Reported outcome

IBM Research open-sourced BeeAI Framework and quickly found an audience with TypeScript developers, with standout community implementations including Bee Canvas and UI Builder emerging.

Reported metrics
Prototype performance vs. commercial searchoutperformed commercially available search solutions
Trajectory explorer impact on trustsignificantly improved transparency and user trust
Reported stack
BeeAI FrameworkLlama 3-70B-ChatRAGReActAgent
Source
https://medium.com/@mayamurad/hard-earned-lessons-from-a-year-of-building-ai-agents-945d90c78707
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

IBM Research open-sourced BeeAI Framework and quickly found an audience with TypeScript developers, with standout community implementations including Bee Canvas and UI Builder emerging.

What tools did this team use?

BeeAI Framework, Llama 3-70B-Chat, RAG, ReActAgent.

What results were reported?

Prototype performance vs. commercial search: outperformed commercially available search solutions; Trajectory explorer impact on trust: significantly improved transparency and user trust (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Non-expert AI gap identified → AI agent prototyped → Trajectory explorer added → Early adopter testing → BeeAI Framework built → Open-source stack launched.

Related back office ops cases