It support · Production

Dosu uses LangSmith to scale evaluation-driven development for their AI GitHub assistant

The problem

As Dosu's installation base grew, their manual approach of reviewing logs with grep and print statements became unscalable, making it nearly impossible to monitor responses and identify failure modes in production—a step critical to their evaluation-driven development workflow. The broader problem Dosu was built to address is that up to 85% of developers' time is spent on non-coding tasks such as answering questions and triaging issues.

First attempt

Manual log review with grep and print statements could not scale with Dosu's growth, and changing LLM prompts frequently caused regressions in areas that had previously been working well.

Workflow diagram · grounded in source

User submits GitHub issue

trigger

“Dosu receives a myriad of requests from users—everything from simple questions about a codebase, to error traces from upgrading to a new library version, to asking about the status of a feature”

Dosu AI generates response

ai_action

“We've built dozens of submodules to intelligently handle different types of scenarios, and we're constantly iterating on our approach to problem solving as models and the research in the field evolve”

LangSmith traces the run

integration

“It only took us minutes to instrument, and immediately upon pushing these changes to production we saw traces flooding into the LangSmith UI. An unexpectedly awesome feature of the @traceable decorator is it can send the function and LLM…”

Search for failure signals

validation

“LangSmith's advanced search functionality makes it easy to identify anomalous behaviors. We can perform searches using a range of criteria, including: explicit user feedback, recent error incidents, response time delays, or negative sent…”

Add failures to eval datasets

feedback_loop

“We search LangSmith for additional examples. Add them to our eval datasets. Iterate against the evals. Push a new version of Dosu, and repeat”

Reported outcome

LangSmith gave Dosu out-of-the-box visibility into all their activity, enabling the team to identify unforeseen failure modes at scale and integrate production monitoring into their EDD workflow.
The team is now building automated evaluation dataset collection from production traffic.

Reported metrics

repositories with Dosu installedthousands of repositories

Developer time on non-coding tasks (industry statistic)85%

Reported stack

LangSmithLangChainGitHubOpenAI

Source

https://blog.dosu.dev/iterating-towards-llm-reliability-with-evaluation-driven-development/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LangSmith gave Dosu out-of-the-box visibility into all their activity, enabling the team to identify unforeseen failure modes at scale and integrate production monitoring into their EDD workflow.

What tools did this team use?

LangSmith, LangChain, GitHub, OpenAI.

What results were reported?

repositories with Dosu installed: thousands of repositories; Developer time on non-coding tasks (industry statistic): 85% (source-reported, not independently verified).

What failed first in this deployment?

Manual log review with grep and print statements could not scale with Dosu's growth, and changing LLM prompts frequently caused regressions in areas that had previously been working well.

How is this it support AI workflow structured?

User submits GitHub issue → Dosu AI generates response → LangSmith traces the run → Search for failure signals → Add failures to eval datasets.

LangSmith gave Dosu out-of-the-box visibility into all their activity, enabling the team to identify unforeseen failure modes at scale and integrate production monitoring into their EDD workflow.The team is now building automated evaluation dataset collection from production traffic.