Behind the scenes of Elastic Security's generative AI features: a quantitative approach to prompt tuning and LLM evaluation
Elastic's Security GenAI team needed a robust, reproducible way to evaluate LLM prompt quality and compare model providers at scale; the initial approach relied on manual spreadsheet-based testing that was effective but time-intensive and did not scale as more features were added.
Elastic transitioned from manual to automated LLM evaluations using LangSmith and LangGraph, building a framework that enables quantitative comparison of prompts and models, with a real-time rubric check in production that regenerates responses if they fall below quality standards.
Frequently asked questions
What did this team achieve with this AI workflow?
Elastic transitioned from manual to automated LLM evaluations using LangSmith and LangGraph, building a framework that enables quantitative comparison of prompts and models, with a real-time rubric check in production…
What tools did this team use?
Elastic AI Assistant, Attack Discovery, Automatic Import, LangSmith, LangGraph, Elasticsearch, ES|QL.
What results were reported?
Evaluation workflow improvement: significantly improving our workflow; Prompt accuracy threshold for acceptance: 85% (source-reported, not independently verified).
How is this incident management AI workflow structured?
User submits request → RAG retrieval via Elasticsearch → LLM response generation → Real-time rubric evaluation → Regeneration on failure → Response displayed.