Workflow · Production
JOBifAI: working around unreliable LLM safety filters with a retry mechanism
The problem
LLM safety filtering is so unreliable that technical failures cannot be distinguished from safety rejections, making it impossible to return clean errors or build a predictable user experience without expensive retry workarounds.
Workflow diagram · grounded in source
1
Player submits input
trigger
“Whenever we allow user input, the LLM responses will frequently miss the mark”
2
LLM generates JSON response
ai_action
“Give the result as a json dictionary of the form '{"choice": c, "sentence": s}', where c is the choice that best describes the player action and s is a description of the result of this action”
3
Safety filter evaluation
validation
“If the player crosses the boundary of what is acceptable, the secretary calls security, and it's an instant game over”
4
Retry on failure
routing
“the current mechanism to bypass these problems is to retry asking 3 times”
Reported outcome
The team implemented a 3-retry mechanism achieving roughly 99% success rate and released JOBifAI as a free proof of concept, though its unreliable foundations would deter full-product development.
Reported metrics
Answer success rate after 1 attemptroughly 75%
Answer success rate after 2 attemptsroughly 90%
Answer success rate after 3 attemptsroughly 99%
Reported stack
LLM
Frequently asked questions
What did this team achieve with this AI workflow?
The team implemented a 3-retry mechanism achieving roughly 99% success rate and released JOBifAI as a free proof of concept, though its unreliable foundations would deter full-product development.
What tools did this team use?
LLM.
What results were reported?
Answer success rate after 1 attempt: roughly 75%; Answer success rate after 2 attempts: roughly 90%; Answer success rate after 3 attempts: roughly 99% (source-reported, not independently verified).
How is this workflow AI workflow structured?
Player submits input → LLM generates JSON response → Safety filter evaluation → Retry on failure.