Back office ops · Production

Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agent Software

The problem

There is growing imprecision about what AI agents are, what they can do, and how effectively they can be used. A systematic understanding of how the tech industry conceives AI agents and how end-users actually experience them in practice is lacking.

First attempt

Historical agent designs such as Microsoft Clippy frustrated users by having interfaces that promised more than the underlying AI could deliver, and current commercial AI agents risk repeating these same mistakes.

Workflow diagram · grounded in source
1
User objective provided
trigger
“generate commands with a given user objective via the underlying large language model (LLM)”
2
VLM reads GUI state
ai_action
“they read the state of a computer GUI using vision language models”
3
LLM generates commands
ai_action
“generate commands with a given user objective via the underlying large language model (LLM)”
4
Controller executes actions
output
“execute them via a controller that simulates human input. This allows agents to produce actions that interact with real-world state”
5
Agent loops until done
feedback_loop
“They operate in a loop where they read the state of a computer GUI using vision language models, generate commands with a given user objective via the underlying large language model (LLM), and execute them via a controller that simulate…”
Reported outcome

A systematic review of 102 commercial AI agents yielded a taxonomy of three umbrella categories (Orchestration, Creation, Insight).
A think-aloud usability study on Operator and Manus found users were generally impressed but faced five critical usability barriers.

Reported metrics
commercial AI agents reviewed102
Taxonomy umbrella categoriesthree
Critical usability barriers identifiedfive
Automation orchestration agents in taxonomy36
Show all 7 reported metrics
commercial AI agents reviewed102
taxonomy umbrella categoriesthree
critical usability barriers identifiedfive
automation orchestration agents in taxonomy36
information retrieval insight agents in taxonomy98
writing creation agents in taxonomy25
recommendations insight agents in taxonomy44
Reported stack
OperatorManus
Source
https://arxiv.org/html/2509.14528v1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

A systematic review of 102 commercial AI agents yielded a taxonomy of three umbrella categories (Orchestration, Creation, Insight).

What tools did this team use?

Operator, Manus.

What results were reported?

commercial AI agents reviewed: 102; Taxonomy umbrella categories: three; Critical usability barriers identified: five; Automation orchestration agents in taxonomy: 36 (source-reported, not independently verified).

What failed first in this deployment?

Historical agent designs such as Microsoft Clippy frustrated users by having interfaces that promised more than the underlying AI could deliver, and current commercial AI agents risk repeating these same mistakes.

How is this back office ops AI workflow structured?

User objective provided → VLM reads GUI state → LLM generates commands → Controller executes actions → Agent loops until done.

Related back office ops cases