Five LLM development lessons from building an AI Jira analytics tool at Luna
Luna's team found that approximately 80% of problematic LLM outputs traced to data quality rather than prompt engineering, and that building reliable LLM-based Jira analytics required navigating challenges around temporal reasoning, model sycophancy, and parameter configuration.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Jira data ingestion
Complex Jira data is ingested to extract actionable insights for engineering and product teams tracking progress, risks, and potential delays.
Tools used
GPT-4Claude 3.7
Outcome
After addressing data quality, temporal context, temperature tuning, chain-of-thought prompting, and output scoping, Luna dramatically improved the reliability, accuracy, and overall effectiveness of their AI Jira analytics products, eliminating entire categories of errors.
What failed first
Initial efforts focused on prompt engineering produced minimal improvement; setting temperature to 0 made models too rigid for nuanced analysis; models lacked inherent date awareness causing errors in time-sensitive sprint and deadline analysis; and models sometimes reinforced user biases rather than providing objective assessments.