Building Your Own Product Copilot: Challenges, Opportunities, and Needs
Software engineers building product copilots face pain points at every stage: prompt engineering is fragile and time-consuming, orchestrating multi-step AI workflows is hard to steer, and existing software engineering processes have not caught up with the demands of building AI-powered applications.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · User natural-language query
The user provides a query or command, and the copilot provides a response in a single-turn interaction.
Tools used
GPT-4ClaudeLLaMAGitHub CopilotMural
Outcome
The study produced collaborative findings and tool design directions for the software engineering community, with participants converging on prompt asset management, orchestration tooling, and AI-specific benchmarking as the highest-priority unmet needs.
What failed first
Models produce inconsistent, hallucinated, or malformed outputs; get stuck in loops; and mistakenly signal task completion prematurely, while unit-testing frameworks designed for deterministic code cannot cope with generative model variability.