Back office ops · Production

Building Boba AI: lessons and patterns for LLM-powered co-pilot applications

The problem

Using an LLM effectively requires considerable prompt engineering skill from users, and LLMs cannot access current information beyond their training cutoff, limiting their usefulness for tasks requiring up-to-date knowledge.

Workflow diagram · grounded in source
1
User enters strategic prompt
trigger
“When a user navigates to the Boba application, they see an initial screen similar to this”
2
Prompt enriched via template
ai_action
“Boba will then ask the LLM to generate scenarios, using Templated Prompt to enrich the user's prompt with additional elements both from general knowledge of the scenario building task and from the user's selections in the UI”
3
LLM returns structured JSON
ai_action
“You will respond with only a valid JSON array of scenario objects. Each scenario object will have the following schema: "title": <string>, //Must be a complete sentence written in the past tense "summary": <string>, //Scenario descriptio…”
4
Results rendered as UI elements
output
“Boba receives a Structured Response from the LLM and displays the result as set of UI elements for each scenario”
5
User selects context to carry
human_review
“Select and Carry Context allows the user to narrow or broaden the scope of interaction to perform subsequent tasks - also known as the task context. This is typically done by selecting one or more elements in the user interface and then …”
6
Web search and article extraction
integration
“Use a Google SERP API to perform the web search based on the user's query and get the top 10 articles (search results) - Read the full content of each article using the Extract API”
7
Vector store and LLM answer
ai_action
“Save the content of each article in short-term memory, specifically an in-memory vector store. The embeddings for the vector store are generated using the OpenAI API, and based on chunks of each article (versus embedding the entire artic…”
Reported outcome

The team built a working LLM-powered co-pilot that mediates between users and the LLM using templated prompts, structured JSON responses, context carrying, and embedded external knowledge, spending approximately 80% of development effort on the user interface.

Reported metrics
development time — UI vs AIabout 80% on UI, about 20% on AI
Reported stack
GPT 3.5LangchainGoogle SERP APIExtract APIOpenAI APIStable DiffusionChatGPT
Source
https://martinfowler.com/articles/building-boba.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The team built a working LLM-powered co-pilot that mediates between users and the LLM using templated prompts, structured JSON responses, context carrying, and embedded external knowledge, spending approximately 80% o…

What tools did this team use?

GPT 3.5, Langchain, Google SERP API, Extract API, OpenAI API, Stable Diffusion, ChatGPT.

What results were reported?

development time — UI vs AI: about 80% on UI, about 20% on AI (source-reported, not independently verified).

How is this back office ops AI workflow structured?

User enters strategic prompt → Prompt enriched via template → LLM returns structured JSON → Results rendered as UI elements → User selects context to carry → Web search and article extraction → Vector store and LLM answer.

Related back office ops cases