Laura Tacho, CTO of DX, shares findings from 180+ companies on how AI is really impacting dev productivity, what most teams get wrong, and why measuring dev experience first is critical.

͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­͏     ­

	Forwarded this email? Subscribe here for more   The Pragmatic Engineer Measuring the impact of AI on… 0:00 1:11:29   Listen now   Measuring the impact of AI on software engineering – with Laura Tacho Laura Tacho, CTO of DX, shares findings from 180+ companies on how AI is really impacting dev productivity, what most teams get wrong, and why measuring dev experience first is critical. Gergely Orosz Jul 23   READ IN APP   Stream the Latest Episode Listen and watch now on YouTube, Spotify and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom. Brought to You By Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more. Graphite — The AI developer productivity platform. — In This Episode There’s no shortage of bold claims about AI and developer productivity, but how do you separate signal from noise? In this episode of The Pragmatic Engineer, I’m joined by Laura Tacho, CTO at DX, to cut through the hype and share how well (or not) AI tools are actually working inside engineering orgs. Laura shares insights from DX’s research across 180+ companies, including surprising findings about where developers save the most time, why devs don’t use AI at all, and what kinds of rollouts lead to meaningful impact. We also discuss: The problem with oversimplified AI headlines and how to think more critically about them An overview of the DX AI Measurement framework Learnings from Booking.com’s AI tool rollout Common reasons developers aren’t using AI tools Why using AI tools sometimes decreases developer satisfaction Surprising results from DX’s 180+ company study How AI-generated documentation differs from human-written docs Why measuring developer experience before rolling out AI is essential Why Laura thinks roadmaps are on their way out And much more! Interesting ideas from the conversation Here are three interesting observations that came in this conversation: Idea #1: Code is a liability! Laura (at 16:04): “One of my more controversial opinions is that source code is a liability. It sounds controversial — but then when people think about it, they realize, yeah, it actually is a liability! We're in a world where it is trivially easy to produce a tremendous amount of source code. And so what does that actually mean for productivity and business impact when what could have been written in one line is now written in five lines? Do we really want to measure AI impact in terms of lines of code generated? I certainly don't. We did not include acceptance rate in our framework for good reason. I think it does give insight into whether the tools are fit for purpose. But when we're looking at broadly measuring business impact and the impact on developer experience and the impact on the business, acceptance rate is just such a tiny part of the story.” Idea #2: Roadmaps are on their way out Gergely (at 55:45) “As an end user of a customer of a company that has invested heavily in AI made that be Google, Microsoft, a startup. What should you be seeing? Laura: “As an end user, what I expect is faster time to market. And that's really on the other side, on the building side, what we're trying to emphasize and what a lot of our conversations have focused on with other executives and engineering leaders. We're really trying to reduce the amount of time to market. This has a lot of implications. Another one of my unconventional opinions, I think roadmaps are on their way out in the age of AI. Companies that are going to win with AI are not ones that think about things in roadmap sequential form, but think about it more as experiment portfolios. And so rapid experimentation and trying to figure out what delights your customers is going to help companies win. The companies that will win are the ones that focus or that already have muscles to do experimentation, A/B tests, trying to figure out, you know, how to delight their customers. As an end user, what I don't want is thrashing as an end user. And I could see that happening. It's just like, we're just gonna, cause now there are fewer reasons to say no to things. And there's probably good reason that some of those things weren't built.” Idea #3: AI time savings are smaller than most people assume Laura (at 27:34): “Typing speed has never been the bottleneck in development. Now we have all this code generated faster than we can type. That's great. But it still takes me time to review that code! It's not that we're saving time because we don't have to type. A lot of that time, we're just reallocating to reviewing or other parts of code authoring that's not typing. (…) The industry average [of spending time on coding] is like 25%. There was a study at AWS that an average AWS engineer only spends 20 % of their time coding. And so when we apply AI to the coding tasks, we're only working with 20 % of that time to begin with. And when we save 10 % of that time, that actually doesn't amount to, ‘we can, you know, ship 10 new product lines overnight!’ ” The Pragmatic Engineer deepdives relevant for this episode AI Engineering in the real world Measuring software engineering productivity The AI Engineering stack A new way to measure developer productivity – from the creators of DORA and SPACE Timestamps (00:00) Intro (01:23) Laura’s take on AI overhyped headlines (10:46) Common questions Laura gets about AI implementation (11:49) How to measure AI’s impact (15:12) Why acceptance rate and lines of code are not sufficient measures of productivity (18:03) The Booking.com case study (20:37) Why some employees are not using AI (24:20) What developers are actually saving time on (29:14) What happens with the time savings (31:10) The surprising results from the DORA report on AI in engineering (33:44) A hypothesis around AI and flow state and the importance of talking to developers (35:59) What’s working in AI architecture (42:22) Learnings from WorkHuman’s adoption of Copilot (47:00) Consumption-based pricing, and the difficulty of allocating resources to AI (52:01) What DX Core 4 measures (55:32) The best outcomes of implementing AI (58:56) Why highly regulated industries are having the best results with AI rollout (1:00:30) Indeed’s structured AI rollout (1:04:22) Why migrations might be a good use case for AI (and a tip for doing it!) (1:07:30) Advice for engineering leads looking to get better at AI tooling and implementation (1:08:49) Rapid fire round References Where to find Laura Tacho: • X: https://x.com/rhein_wein • LinkedIn: https://www.linkedin.com/in/lauratacho/ • Website: https://lauratacho.com/ • Laura’s course (Measuring Engineering Performance and AI Impact) https://lauratacho.com/developer-productivity-metrics-course Mentions during the episode: • AI Measurement Framework: https://getdx.com/research/measuring-ai-code-assistants-and-agents/ • Are Coders' Jobs At Risk? AI's Impact On The Future Of Programming: https://www.forbes.com/sites/sylvainduranton/2024/04/15/are-coders-jobs-at-risk-ais-impact-on-the-future-of-programming/ • Guide to AI-Assisted Engineering: https://getdx.com/guide/ai-assisted-e Engineering/ • DORA's Impact of Generative AI in Software Development report: https://dora.dev/publications/#impact-of-generative-ai-in-software-development • Booking Case Study: https://getdx.com/customers/booking-uses-dx-to-measure-impact-of-genai/ • AI coding assistants wave goodbye to junior developers: https://www.cio.com/article/3509174/ai-coding-assistants-wave-goodbye-to-junior-developers.html • OpenAI Just Released a Coding Tool to ‘Help’ Programmers (Replace Their Jobs, Probably): https://gizmodo.com/openai-just-released-a-coding-tool-to-help-programmers-replace-their-jobs-probably-2000603414 • Cursor: https://cursor.com/ • Measuring Software Engineering Productivity: https://newsletter.pragmaticengineer.com/p/engineering-productivity • DORA, SPACE, and DevEx: Which framework should you use?: https://getdx.com/guide/dora-space-devex/ • Abi Noda on LinkedIn: https://www.linkedin.com/in/abinoda/ • How AI is changing software engineering at Shopify with Farhan Thawar: https://newsletter.pragmaticengineer.com/p/how-ai-is-changing-software-engineering • How Linux is built with Greg Kroah-Hartman: https://newsletter.pragmaticengineer.com/p/how-linux-is-built-with-greg-kroah • Amazon says developers spend a surprisingly small amount of time per day coding: https://www.businessinsider.com/amazon-developers-spend-only-one-hour-coding-daily-aws-ai-2024-12 • Impact of Generative AI in Software Development: https://dora.dev/research/ai/gen-ai-report/ • Vercel Clerk: https://vercel.com/marketplace/clerk • WorkHuman: https://www.workhuman.com/ • Workhuman increases ROI from AI assistants 21% with DX: https://getdx.com/customers/workhuman-increases-roi-from-ai-assistants-21-with-dx/ • Visual Studio: https://visualstudio.microsoft.com/ • Jesse Adametz on LinkedIn: linkedin.com/in/jesseadametz • DX Core 4: https://getdx.com/core-4-reporting/? • Indeed: https://www.indeed.com/ • Accelerating code migrations with AI: https://research.google/blog/accelerating-code-migrations-with-ai/ • Granola: https://www.granola.ai/ • Write Useful Books: A modern approach to designing and refining recommendable nonfiction: https://www.amazon.com/Write-Useful-Books-recommendable-nonfiction-ebook/dp/B0983HFQX7 • Unsavory Truth: How Food Companies Skew the Science of What We Eat: https://www.amazon.com/dp/1541697111/? — Production and marketing by Pen Name. You’re on the free list for The Pragmatic Engineer. For the full experience, become a paying subscriber. Many readers expense this newsletter within their company’s training/learning/development budget. If you have such a budget, here’s an email you could send to your manager. Upgrade to paid This post is public, so feel free to share and forward it. Share The Pragmatic Engineer If you enjoyed this post, you might enjoy my book, The Software Engineer's Guidebook. Here is what Tanya Reilly, senior principal engineer and author of The Staff Engineer's Path said about it: "From performance reviews to P95 latency, from team dynamics to testing, Gergely demystifies all aspects of a software career. This book is well named: it really does feel like the missing guidebook for the whole industry." Get The Software Engineer's Guidebook   Like Comment Restack   © 2025 Gergely Orosz 548 Market Street PMB 72296, San Francisco, CA 94104 Unsubscribe