The Pulse: We need to talk about migrations with AI
The Pulse: We need to talk about migrations with AIAsana migrated off testing framework Enzyme in two weeks: without AI, this work would surely have been kicked down the road. Airbnb and Uber share similar stories and AI seems excellent at migrations.
Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from last week’s The Pulse issue. Full subscribers received the article below seven days ago. If you’ve been forwarded this email, you can subscribe here. OpenAI put an impressive-sounding case study about how they helped Asana save $5.9M with a single migration. From OpenAI (emphasis mine.)
For context, Asana migrated from Enzyme to React Testing Library, which indeed would not be a simple migration. However, their estimate of four engineers (each on circa $300K/year, according to OpenAI’s arithmetic), spending five years on the project, had me like:
But thinking about this for longer raised the question: what does a migration from Enzyme to React Testing Library even look like? Enzyme to React Testing Library migrationLet’s take a simple test and see how it looks in various testing libraries. For our test, we want to verify that a button increments a counter. Here’s our button in React:
Now, the test to verify this, in Enzyme:
And let’s rewrite this test in React Testing Library:
The two tests do the same, but they have completely different syntax! Let’s see just how different they are, with a side-by-side comparison:
The main reason for this difference is that the two frameworks use a fundamentally different approach to testing. Whereas Enzyme is oriented towards component testing (notice how the test operates on a component instance), the React Testing Library operates on the rendered Document Object Model (a data structure representing the HTML shipped to the client) so the test sees the whole rendered page, not just the component its written for. That’s why the testing approach will differ radically between the two frameworks, especially when testing complex user journeys. You can learn more here on the tradeoffs between the two approaches from the React Testing Library author. It took Airbnb 6 weeks to migrate 3,500 tests with AILast year, Airbnb revealed how they migrated their Enzyme test suite of 3,500 component test files within six weeks with LLMs. The estimate of doing this by hand was 1.5 engineering years. Airbnb did the LLM-aided migration in a multi-phase process:
Airbnb’s team had to build loops to keep retrying migrations; once they did, 75% of files were migrated in just four hours, and the migrations were straightforward. They then built a more sophisticated refactor pipeline for the remaining 25% of tests; after building the pipeline, the new loop migrated most of the remaining tests (97%) in total, after running over 4 days. The remaining 3% was done with LLM input, with engineers finishing it in a week. This was in March 2025, when the frontier coding model was Claude 3.7 Sonnet. Today, models are a lot more capable, such as the likes of GPT-5.6 Sol and Claude Fable 5. AI makes impractical migrations doableOn the basis that it took Airbnb six weeks, I find it credible that it took Asana two weeks to migrate what is probably a similarly complex test infrastructure from Enzyme to RTL, a year later. The time and $6M cost as quoted by OpenAI feels inflated. I assume the numbers were based on an estimate that a fulltime engineer could do a maximum of X tests migrated per day, where X was between 5 and 10. Then, calculate the number of engineering years this takes (perhaps 20 engineering years), and multiply by the cost of an engineer. You estimate a project like this when it’s an undesirable project you really don’t want to do as an engineer! So, looking at it from this point of view: does it matter if the estimate was 1.5 years (Airbnb) or half a decade (Asana, hypothetically)? Or if the estimated cost was $1M or $5M? It would still be an impractical migration and a foolish endeavor: too long and distracting, at least, in the “traditional” way! Pre-AI, years-long migrations were rare. In 2021, Sentry took 1.5 years to convert their frontend codebase from JavaScript to TypeScript. That was a migration of 1,100 files and 95,000 lines of code! Around 10 engineers worked on the migration, so if we assume a $300K-per-engineer cost, that’s a $2–4M cost for circa 95,000 lines of code. Indeed, Asana’s reported $12,000 migration cost could actually cost even less if there was ruthless focus on cost optimization. What about using a model that’s 10x cheaper than OpenAI, like an open model running on inference providers? After all, why use the most expensive model, especially if a company already owns GPUs, running models, making inference practically free (except for power costs, that is.) For a first run, $12,000 would be affordable for a company paying $300K for engineers. But for subsequent runs, I’d wager it’s worth spending time optimizing the cost, and saving $10K per migration (or more!) A few more details from inside AsanaI managed to catch up with Dan Ubilla at Asana, who leads the Developer Productivity group at the company. He helpfully clarified a few things about the Enzyme migration post on the Asana site:
And finally, one addition from my end: The $6M cost was a back-of-the envelope estimation. I asked Dan how this estimate of the migration costing $6M came together. Dan confirmed that the estimation was done the same way as most of us do these estimations:
This estimation does not take into account that migrating even ten files by hand takes less than ten times more than it takes to migrate one file (you become more efficient), nor does it assume LLM usage. It’s a baseline to get a sense of how the work would have been done, pre-AI, if distributed across all Asan engineers to pick up, and migrate few files, whenever they have time. My take is that this number is probably an overestimation, but the point is less about the number, and more to convey that this is a lot of work! Internally, Asana’s team and leadership are now convinced about the usefulness of LLMs for migrations, and they’ll be using them for other, long-running, otherwise soul-sucking migrations. Expect long-avoided migrations to finally happenA few months ago, Uber shared that they executed a massive JUnit migration in four months with two engineers and AI: moving 600,000 unit tests spanning 15 million lines of code (!!) by moving from the unsupported JUnit 4 to JUnit 5. During the migration, 1.25M lines of code were modified. This type of migration used to be impractical; with AI tools, it took eight engineering months of effort, plus AI costs. And there’s the Bun migration (530,000 lines of code from Zig to Rust in two weeks for a $165K API cost) as another example of drastically faster migrations with AI. The best part about rapid, AI-assisted migrations is not needing to worry about supporting “old” libraries. One of the main reasons library migrations have been such a pain was the need to keep supporting the old library or technology during the migration. Shortening this window could well be worth the additional cost, at least until we figure out how to drastically reduce the time and cost of these migrations, as well. The shared characteristic of all of the above migrations is that engineers needed to plan for it, design verification loops, and be involved throughout. I always dreaded migrations, so I see it as good news that we have a new tool to use for “grunt work” like this. Read the full issue of last week’s The Pulse, or check out this week’s The Pulse. This week’s issue covers:
You’re on the free list for The Pragmatic Engineer. For the full experience, become a paying subscriber. Many readers expense this newsletter within their company’s training/learning/development budget. If you have such a budget, here’s an email you could send to your manager. This post is public, so feel free to share and forward it. If you enjoyed this post, you might enjoy my book, The Software Engineer's Guidebook: navigating senior, tech lead, staff and principal positions at tech companies and startups.
|
Comments
Post a Comment