Back office ops · Production

Five LLM development lessons from building an AI Jira analytics tool at Luna

The problem

Luna's team found that approximately 80% of problematic LLM outputs traced to data quality rather than prompt engineering, and that building reliable LLM-based Jira analytics required navigating challenges around temporal reasoning, model sycophancy, and parameter configuration.

First attempt

Initial efforts focused on prompt engineering produced minimal improvement; setting temperature to 0 made models too rigid for nuanced analysis; models lacked inherent date awareness causing errors in time-sensitive sprint and deadline analysis; and models sometimes reinforced user biases rather than providing objective assessments.

Workflow diagram · grounded in source
1
Jira data ingestion
trigger
“extract actionable insights from complex Jira data to help engineering and product teams track progress, identify risks, and predict potential delays more effectively”
2
Data quality preprocessing
validation
“By cleaning and standardizing our Jira data inputs, we achieved far more reliable and accurate outputs”
3
Temporal context computation
validation
“By pre-computing all time-based calculations and providing explicit context (including the "current date" used for calculations), we eliminated an entire category of errors”
4
LLM chain-of-thought analysis
ai_action
“requiring models to explain their reasoning step-by-step before providing the final conclusion”
5
Summaries and insights output
output
“AI Jira Fix Version summaries automate progress updates, risk detection, and trade-off decisions: giving PMs and EMs instant visibility”
Reported outcome

After addressing data quality, temporal context, temperature tuning, chain-of-thought prompting, and output scoping, Luna dramatically improved the reliability, accuracy, and overall effectiveness of their AI Jira analytics products, eliminating entire categories of errors.

Reported metrics
Problematic cases caused by data qualityapproximately 80%
Reliability and accuracy improvementdramatically improved
Error category eliminatedeliminated an entire category of errors
ROI of chain-of-thought promptingsignificant
Reported stack
GPT-4Claude 3.7Jira
Source
https://withluna.ai/blog/llm-development-lessons-ai-jira-analytics
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After addressing data quality, temporal context, temperature tuning, chain-of-thought prompting, and output scoping, Luna dramatically improved the reliability, accuracy, and overall effectiveness of their AI Jira ana…

What tools did this team use?

GPT-4, Claude 3.7, Jira.

What results were reported?

Problematic cases caused by data quality: approximately 80%; Reliability and accuracy improvement: dramatically improved; Error category eliminated: eliminated an entire category of errors; ROI of chain-of-thought prompting: significant (source-reported, not independently verified).

What failed first in this deployment?

Initial efforts focused on prompt engineering produced minimal improvement; setting temperature to 0 made models too rigid for nuanced analysis; models lacked inherent date awareness causing errors in time-sensitive s…

How is this back office ops AI workflow structured?

Jira data ingestion → Data quality preprocessing → Temporal context computation → LLM chain-of-thought analysis → Summaries and insights output.

Related back office ops cases