Compliance monitoring · Production

Capital One refines LLM input guardrails with chain-of-thought prompting and fine-tuned alignment

The problem

LLM-powered applications at Capital One faced adversarial attacks—including jailbreak prompts and prompt injections—that could cause unsafe outputs, while base open-source LLMs lacked sufficient detection capability.

First attempt

Base open-source LLMs achieved F1 scores well below 80% on adversarial input detection, and performance gaps on jailbreaks and prompt injections persisted even in many-shot settings.

Workflow diagram · grounded in source
1
User input intercepted
trigger
“the primary job of these guardrail components is to prevent such harmful interactions from reaching the main conversation-driving LLM (or the "brain" of the assistant)”
2
LLM-as-Judge classifies input
ai_action
“the judge LLM is prompted with instructions to determine whether inputs are safe or unsafe, based on the provided policy violation definitions”
3
Chain-of-thought reasoning generates verdict
ai_action
“CoT prompting involves instructing the LLM to first generate a logical and concise explanation of its thought process and leverage this information to arrive at a final verdict”
4
Verdict routed to conversational agent
routing
“the input guardrail's safety verdict accompanied by an explanation is passed to the conversational agent in charge of deciding how to best synthesize a response to the query”
5
Fine-tuning improves guardrail model
feedback_loop
“all LLMs achieved strong performance gains across all evaluation metrics after fine-tuning and alignment”
Reported outcome

Fine-tuning with SFT, DPO, and KTO yielded over 50% improvements in F1 score and attack detection ratio with only a maximum 1.5% increase in false positive rate, and the best model—DPO-aligned Llama3 8B—outperformed LlamaGuard-2 and other public guardrail models by wide margins.

Reported metrics
F1 score and attack detection ratio improvement50+%
False positive rate increase after fine-tuning1.5%
baseline F1 score ceiling on base LLMswell below 80%
performance vs LlamaGuard-2significant improvements against LlamaGuard-2 across all metrics
Show all 5 reported metrics
F1 score and attack detection ratio improvement50+%
false positive rate increase after fine-tuning1.5%
baseline F1 score ceiling on base LLMswell below 80%
performance vs LlamaGuard-2significant improvements against LlamaGuard-2 across all metrics
performance vs DeBERTaV3 and PromptGuardimprovements by wide margins across all metrics
Reported stack
Mistral 7B Instruct v2Mixtral 8x7B Instruct v1Llama2 13B ChatLlama3 8B InstructLoRACoTSFTDPOKTO
Source
https://medium.com/capital-one-tech/refining-input-guardrails-for-safer-llm-applications-capital-one-715c1c440e6b
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Fine-tuning with SFT, DPO, and KTO yielded over 50% improvements in F1 score and attack detection ratio with only a maximum 1.5% increase in false positive rate, and the best model—DPO-aligned Llama3 8B—outperformed L…

What tools did this team use?

Mistral 7B Instruct v2, Mixtral 8x7B Instruct v1, Llama2 13B Chat, Llama3 8B Instruct, LoRA, CoT, SFT, DPO, KTO.

What results were reported?

F1 score and attack detection ratio improvement: 50+%; False positive rate increase after fine-tuning: 1.5%; baseline F1 score ceiling on base LLMs: well below 80%; performance vs LlamaGuard-2: significant improvements against LlamaGuard-2 across all metrics (source-reported, not independently verified).

What failed first in this deployment?

Base open-source LLMs achieved F1 scores well below 80% on adversarial input detection, and performance gaps on jailbreaks and prompt injections persisted even in many-shot settings.

How is this compliance monitoring AI workflow structured?

User input intercepted → LLM-as-Judge classifies input → Chain-of-thought reasoning generates verdict → Verdict routed to conversational agent → Fine-tuning improves guardrail model.

Related compliance monitoring cases