Workflow · Production

All the Hard Stuff Nobody Talks About when Building Products with LLMs

The problem

Building a production AI feature backed by LLMs is far harder than demos suggest: context windows may be too small for large customer schemas, LLM latency is high, prompt engineering has few established best practices, correctness and broad input acceptance are in tension, prompt injection has no complete solution, and compliance obligations add significant overhead.

First attempt

Zero-shot prompting produced no usable output; single-shot worked but poorly; chaining LLM calls compounded latency and accuracy problems; Claude 100k with a full schema dump was slower and hallucinated more than a targeted embedding approach; LangChain provided no tangible improvement in query generation.

Workflow diagram · grounded in source
1
User submits NL query
trigger
“Query Assistant allows you to express a desired Honeycomb query in natural language”
2
Schema trimmed to active fields
validation
“when you constrain a schema to only include fields that received data in the past seven days, you can trim the size of a schema and usually fit the whole thing in gpt-3.5-turbo's context window”
3
LLM generates query output
ai_action
“Query Assistant is all about prompting, which assembles a task and data/context as input to an LLM”
4
Parse and validate LLM output
validation
“We parse the output of an LLM into a particular format and run validation against it”
5
Query executed in UI
output
“execute the query against our query engine”
Reported outcome

Honeycomb shipped Query Assistant to all users within a one-month timeline; the feature is live and actively used, a full security and compliance audit of LLM providers was completed, new legal terms were drafted, and prompt injection mitigation measures were put in place.

Reported metrics
LLM response latency rangetwo to 15+ seconds
time to ship Query Assistantunder a month
Reported stack
gpt-3.5-turboClaudegpt-4LangChainOpenAI
Source
https://www.honeycomb.io/blog/hard-stuff-nobody-talks-about-llm
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Honeycomb shipped Query Assistant to all users within a one-month timeline; the feature is live and actively used, a full security and compliance audit of LLM providers was completed, new legal terms were drafted, and…

What tools did this team use?

gpt-3.5-turbo, Claude, gpt-4, LangChain, OpenAI.

What results were reported?

LLM response latency range: two to 15+ seconds; time to ship Query Assistant: under a month (source-reported, not independently verified).

What failed first in this deployment?

Zero-shot prompting produced no usable output; single-shot worked but poorly; chaining LLM calls compounded latency and accuracy problems; Claude 100k with a full schema dump was slower and hallucinated more than a ta…

How is this workflow AI workflow structured?

User submits NL query → Schema trimmed to active fields → LLM generates query output → Parse and validate LLM output → Query executed in UI.