Back office ops · Production

Instacart's prompt engineering techniques for internal LLM productivity tooling

The problem

LLMs used in Instacart's internal productivity tools face challenges including hallucinations, context size limits, difficulty completing tasks, and unreliable output formatting when used programmatically.

Workflow diagram · grounded in source
1
Code diff input received
trigger
“here is a section of the prompt we use to generate pull request (PR) titles and descriptions for internal code reviews”
2
Room for Thought outline
ai_action
“First let's create an outline for the pull request description. Do not generate a title and description, only write the outline.”
3
Monte Carlo option generation
ai_action
“Generate 5 possible radically different titles and then critique them.”
4
Self-correction critique
ai_action
“By letting the model form critiques, you allow the model to improve on its observations”
5
Logit-bias forced classification
validation
“we can nudge that probability by using the logit_bias parameter in our request to OpenAI, and if we set the bias to 100, we can force the model to choose from a specific set of tokens”
6
Puppetry JSON output
output
“when we want the model to output a JSON object for the pull request script, we do this”
Reported outcome

Instacart developed and deployed a suite of prompt engineering techniques for the Ava family of internal productivity products, achieving reliable structured output and improved LLM response quality.

Reported metrics
LLM output reliabilityguarantees valid output
Few-shot model performance with polite phrasingactually performs better than without
Reported stack
GPT-4GPT-3.5OpenAI
Source
https://tech.instacart.com/monte-carlo-puppetry-and-laughter-the-unexpected-joys-of-prompt-engineering-4b9272e0c4eb
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Instacart developed and deployed a suite of prompt engineering techniques for the Ava family of internal productivity products, achieving reliable structured output and improved LLM response quality.

What tools did this team use?

GPT-4, GPT-3.5, OpenAI.

What results were reported?

LLM output reliability: guarantees valid output; Few-shot model performance with polite phrasing: actually performs better than without (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Code diff input received → Room for Thought outline → Monte Carlo option generation → Self-correction critique → Logit-bias forced classification → Puppetry JSON output.

Related back office ops cases