Quality assurance · Production

Slack engineers achieve 80% automated migration of 15,500 Enzyme tests to React Testing Library using a hybrid AST and LLM pipeline

The problem

Slack had 15,500 Enzyme tests requiring migration to React Testing Library to support a React 18 upgrade, with an estimated total of more than 10,000 engineering hours if done manually and no existing open-source tools or adapters to help automate the work.

First attempt

A pure AST codemod approach achieved only a 45% success rate due to the complexity of Enzyme's 65 methods and JavaScript/TypeScript variability. An initial Claude 2.1 LLM-only trial achieved just 40% to 60% success with high variability and was not released to developers.

Workflow diagram · grounded in source

File submitted to pipeline

trigger

“This is the pipeline, end-to-end, where you provide the file.”

Context collection

integration

“steps 1 to 3 is context collection where we get the file code. We look at the DOM tree for all of those test cases. We get the partially converted code with our codemod, then we package it all”

AST partial conversion and annotation

validation

“AST partial conversion with rules, so all of those AST conversions that we knew would result in 100% accurate conversions, we performed them, but for everything else we just added annotations”

LLM generates converted code

ai_action

“In step number 4, we send the AI API request, we parse the response, we run the linters and auto-fix stuff.”

Test execution check

validation

“We actually run the code and see if it's passing or not passing, check the results.”

Feedback loop on failure

feedback_loop

“If it fails, we have one more feedback step where we also add the logs and dynamically create prompts based on what happened.”

Output results

output

“Then, if it doesn't fail or if it fails, we just output the results.”

Reported outcome

The hybrid AST and LLM pipeline converted 80% of code in each file correctly and saved 22% of developer time, enabling the team to complete all 15,500 test migrations by May 2024.
The tool was open-sourced on npm for other companies to use.

Reported metrics

Initial test cases to migrate15,500

Manual time per test case30 to 45 minutes

Estimated total manual engineering effortmore than 10,000 engineering hours

AST-only conversion success rate45%

Show all 16 reported metrics

initial test cases to migrate15,500

manual time per test case30 to 45 minutes

estimated total manual engineering effortmore than 10,000 engineering hours

AST-only conversion success rate45%

LLM-only conversion success rate40% to 60%

hybrid AST and LLM conversion success rate80%

improvement beyond LLM-only capability20% to 30%

developer time saved by AI codemod22%

time saved per test case45 minutes

fully converted files in evaluation16%

partially converted files in evaluation24%

test cases passing after conversion22%

improvement from single feedback loop10% to 20%

files run in evaluation batch338

test cases in evaluation batchmore than 2,000

files converted using the codemod (adoption)64

Reported stack

Claude 2.1ASTcodemodnpm

Source

https://www.infoq.com/presentations/ai-migration-large-scale/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The hybrid AST and LLM pipeline converted 80% of code in each file correctly and saved 22% of developer time, enabling the team to complete all 15,500 test migrations by May 2024.

What tools did this team use?

Claude 2.1, AST, codemod, npm.

What results were reported?

Initial test cases to migrate: 15,500; Manual time per test case: 30 to 45 minutes; Estimated total manual engineering effort: more than 10,000 engineering hours; AST-only conversion success rate: 45% (source-reported, not independently verified).

What failed first in this deployment?

A pure AST codemod approach achieved only a 45% success rate due to the complexity of Enzyme's 65 methods and JavaScript/TypeScript variability.

How is this quality assurance AI workflow structured?

File submitted to pipeline → Context collection → AST partial conversion and annotation → LLM generates converted code → Test execution check → Feedback loop on failure → Output results.

The hybrid AST and LLM pipeline converted 80% of code in each file correctly and saved 22% of developer time, enabling the team to complete all 15,500 test migrations by May 2024.The tool was open-sourced on npm for other companies to use.