Back office ops · Production

Digits: Lessons from two years running AI agents in production for accounting

The problem

Moving AI agents from prototype to production in an accounting platform requires solving hard infrastructure problems—observability, guardrails, memory, and safe tool generation—that open-source frameworks do not yet address in production-ready form.

First attempt

Open-source agent frameworks like LangChain and CrewAI were evaluated but found too complex with too many dependencies to be production-ready without significant modification.

Workflow diagram · grounded in source
1
Task objective defined
trigger
“An objective - What needs to be accomplished”
2
Reasoning model plans task
ai_action
“Using reasoning models to plan tasks upfront achieves faster completion times and higher accuracy with lower latency”
3
LLM processes and invokes tools
ai_action
“An LLM - The language model that processes and reasons Tools - The capabilities the system can invoke”
4
Guardrail LLM validates response
validation
“Use a different LLM to evaluate responses”
5
Response delivered to user
output
“A response - The output delivered back”
6
Feedback-driven model fine-tuning
feedback_loop
“We capture user feedback about agent responses, design reward functions, and explore reinforcement learning to fine-tune agent-specific models. Each iteration makes our agents more effective at their specific tasks.”
Reported outcome

Digits has been running AI agents in production for over two years across three use cases—vendor data enrichment, client onboarding, and complex user questions—with qualitative improvements in manual research time and data quality, and faster task completion with higher accuracy using upfront task planning.

Reported metrics
Manual research time for vendor datareducing manual research time
Vendor data qualityimproving data quality
Task completion time with upfront planningfaster completion times
Task accuracy with upfront planninghigher accuracy
Show all 5 reported metrics
manual research time for vendor datareducing manual research time
vendor data qualityimproving data quality
task completion time with upfront planningfaster completion times
task accuracy with upfront planninghigher accuracy
task latency with upfront planninglower latency
Reported stack
OpenTelemetryGo
Source
https://digits.com/blog/mlops-world-2025-slides/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Digits has been running AI agents in production for over two years across three use cases—vendor data enrichment, client onboarding, and complex user questions—with qualitative improvements in manual research time and…

What tools did this team use?

OpenTelemetry, Go.

What results were reported?

Manual research time for vendor data: reducing manual research time; Vendor data quality: improving data quality; Task completion time with upfront planning: faster completion times; Task accuracy with upfront planning: higher accuracy (source-reported, not independently verified).

What failed first in this deployment?

Open-source agent frameworks like LangChain and CrewAI were evaluated but found too complex with too many dependencies to be production-ready without significant modification.

How is this back office ops AI workflow structured?

Task objective defined → Reasoning model plans task → LLM processes and invokes tools → Guardrail LLM validates response → Response delivered to user → Feedback-driven model fine-tuning.

Related back office ops cases