Finance ops · Production

Fintool uses Braintrust to build a scalable LLM evaluation workflow for financial AI insights

The problem

Institutional investors face a sheer volume of daily regulatory filings that makes it impossible for humans to review every document, and Fintool needed real-time monitoring to maintain quality and user confidence in its AI-generated financial insights.

Workflow diagram · grounded in source
1
Investor configures monitoring
trigger
“Investors select the companies they want to monitor and configure alerts by specifying what type of information they want to be summarized”
2
Document section summarization
ai_action
“Fintool addressed this problem by developing Fintool Feed, a Twitter-like interface where they summarize key sections of documents based on user prompts”
3
Citation and source validation
validation
“Fintool makes sure every insight includes a reliable source, like an SEC document ID, and automatically flags anything that's missing or doesn't look right”
4
LLM-as-judge quality scoring
ai_action
“Each generated insight is evaluated using LLM-as-a-judge scorers on key metrics like accuracy, relevance, and completeness. Braintrust automatically updates whenever Fintool adjusts prompts or ingests new data, preventing surprise regres…”
5
Human expert review on low scores
human_review
“When content gets a low score or is downvoted, a human expert is immediately notified to step in. They can approve, reject, or edit the Markdown to fix issues like poor formatting”
6
Live content update via Braintrust
feedback_loop
“Since the Fintool database is linked directly to Braintrust, the expert can update the live content right from the Braintrust UI”
Reported outcome

Fintool's evaluation workflow now manages millions of LLM-generated insights, processing 1.5 billion tokens across 70 million data chunks daily while improving accuracy, consistency, and efficiency at scale.

Reported metrics
Daily token processing volume1.5 billion tokens
Daily data chunks processed70 million data chunks
LLM-generated insights managedmillions of LLM-generated insights
Accuracy, consistency, and efficiencyimproving accuracy, consistency, and efficiency at scale
Reported stack
BraintrustLLM-as-a-judge scorersLLMClassifier
Source
https://www.braintrust.dev/blog/fintool?utm_source=weeklyupdate0201&utm_medium=newsletter&utm_campaign=signups
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Fintool's evaluation workflow now manages millions of LLM-generated insights, processing 1.5 billion tokens across 70 million data chunks daily while improving accuracy, consistency, and efficiency at scale.

What tools did this team use?

Braintrust, LLM-as-a-judge scorers, LLMClassifier.

What results were reported?

Daily token processing volume: 1.5 billion tokens; Daily data chunks processed: 70 million data chunks; LLM-generated insights managed: millions of LLM-generated insights; Accuracy, consistency, and efficiency: improving accuracy, consistency, and efficiency at scale (source-reported, not independently verified).

How is this finance ops AI workflow structured?

Investor configures monitoring → Document section summarization → Citation and source validation → LLM-as-judge quality scoring → Human expert review on low scores → Live content update via Braintrust.

Related finance ops cases