Building resilient systems with Sam Newman
Building resilient systems with Sam NewmanSam Newman joins me to discuss when to use microservices, how to build resilient distributed systems, and how some of the tech world fundamentally misunderstands AI
Stream the latest episodeListen and watch now on YouTube, Apple, and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom. Brought to You by
• turbopuffer – vector and full-text search built on object storage: fast, 10x cheaper, and extremely scalable. Under the hood, the team are rewriting the tpuf storage engine. Follow along until it hits production at turbopuffer.com/v3 • Get a free introduction property-based testing: through a live, online webinar hosted by Hillel Wayne (a recent podcast guest!), author of Logic for Programmers and Developer Educator at Antithesis. What if you only needed a single test to harden your system? RSVP here. • Entire – Git hosting, rebuilt for the agentic era. Entire hosts your code in-region, and is up to 89x faster than any other competitor. Mirror from GitHub with a single click – I’ve already done so. In this episodeSam Newman wrote one of the most-read books about microservices (“Building Microservices”), but he calls them an architecture of “last resort.” In this episode of the Pragmatic Engineer podcast, Sam explains his thinking on this, and he’s certainly well placed to do so; he was in the room when the term “microservices” was coined. We discuss what teams get wrong when adopting microservices, why independent deployment matters, and how microservices can help teams work more autonomously. We also peek into his new book, ‘Building Resilient Distributed Systems’, and Sam explains his three rules for distributed systems, why observability is essential, and also why it’s vital to take business context into account when deciding whether to fail open or fail closed on errors.
We also explore how AI is changing software development, from specs versus code as a source of truth to cognitive debt and cognitive surrender. Sam shares how modular architecture can help teams experiment with AI while maintaining understanding of the systems being built. Takeaways from the conversation with Sam
1. In the room when “microservices” was coined. In the early years of the 2010s, James Lewis – Sam’s colleague at Thoughtworks – kept running into companies building service-oriented architectures where each service could be replaced quickly. At an architecture symposium in the Lake District, England, James pitched a name for this trend: “micro apps,” and in a room of about 10 people (including Sam), someone else suggested “microservices.” Lewis and Martin Fowler then wrote the first article on the topic in March 2014, ‘Microservices: a definition of this new architectural term’, and Sam published the book Building Microservices the following year. 2. Sam spent 18 months teaching Googlers how to write good automated tests. In 2007, Sam was embedded at Yahoo and Google as a ThoughtWorks consultant, alongside people from Pivotal Labs. Their job was to teach engineers how to write automated tests, and code that’s testable. He recalls that some Google devs did not believe it was possible to write automated functional tests for websites, until Sam and his colleagues showed them how Selenium works! 3. Sam spent 10+ years at a large company, three years at startups, and has been independent for more than a decade. After 13 years at Thoughtworks, he quit to avoid becoming “institutionalized.” He recalls thinking he could hang around at Thoughtworks and grow bitter. “So I’ll just go now.” He then worked at three different startups before going independent, which he’s now done for 15 years. The biggest challenge of being your own boss? He says: “The problem of working for yourself is that the boss is frequently an a*****e. So, current Sam frequently is cursing past Sam for the commitments he made!” 4. So, what’s a “microservice?” Sam gives two definitions: a clear one, and a softer one:
5. Three rules of distributed systems: Sam sometimes struggled to recall the eight fallacies of distributed computing, and so he boiled them down to the three that cause the most problems: 1. Information takes time to travel. You cannot send information instantaneously from point A to point B. This is basically the “speed of light” constraint. 2. Sometimes, the thing you want to talk to isn’t there. That is, resources can and do go offline. 3. Resources are not infinite; you will eventually run out of things like CPU, memory, storage, and networking bandwidth. In Sam’s experience, most distributed systems outages are caused by #3; resource pools running out. 6. Dealing with idempotency: keys or fingerprints? Idempotency is performing the same operation multiple times and it having the same effect as if done once only. For example, if you repeatedly hit the “down” button for an elevator, nothing more happens than what happened upon that first press: multiple button smashes don’t bring more elevators your way. Payments are a classic example where idempotency is needed: when you retry or refresh a payments page, you should not be charged twice. There are two common ways to deal with idempotency:
7. Four concepts of resiliency: Sam frequently references the four outcomes for military systems, explained by David Woods:
8. What if code is a side effect of collaboration? When Sam was at Thoughtworks, some clients refused to do pair programming as a practice because they could only see one developer typing and the other doing nothing, which is a waste. Martin Fowler’s dictum that “programming is not typing” helped Sam to explain to clients that typing out code is an output, but not an outcome. Sam conceives programming as building a system collectively, and code is one of the mechanisms delivering the system. Now that with LLMs we no longer spend most time typing out code, we’ll discover the next bottleneck in creating good software. Could it be delayed feedback cycles? 9. Hedging the vendor options is sensible when integrating AI into systems. It’s hard to tell which AI companies have long-term, sustainable business models and which don’t, so it’s the responsible thing when designing systems to hedge the providers. Aim to be multi-vendor, multi-model in your choices, so you can easily change the vendor or model layer. Also, consider if you can swap out LLM-powered functionality for deterministic code that runs faster and cheaper. 10. We must resist “cognitive surrender” to AI. Sam believes that more AI usage is leading to less critical thinking – or could we be using it wrong? He says: “The original pitch of AI was it was going to free us from drudgery. We’d have less toil, and we can focus more on critical thinking and smarts. However, much of our current use of AI is not freeing us up from critical thinking! It’s causing more context switching and leaving less time for thinking. People are working longer hours; we’ve got studies that show this. But [longer hours and more context switching] is not necessarily AI’s fault. It’s just how we’re using it.” Putting in the effort often leads to better results. For example, when writing Building Resilient Distributed Systems, Sam used NotebookLM for research, but he manually clicked through each link it surfaced, and checked whether it was information he wanted to reference. 11. Starting with modules encourages better software architecture. Also: production is truth. I asked Sam for advice on what to do to get better at designing systems:
12. Much of the tech world might still be naive about what an LLM is. As Sam said:
Sam’s comments feel very timely, given Opus 5.5 just formatted the hard drive of a dev who ran the model with the —dangerously-skip-permissions tag, and was surprised to see all of his data gone.
Finally, recommended reading to get better at software architecture. Sam’s recommendations:
The Pragmatic Engineer deepdives relevant for this episode• Scaling Uber with Thuan Pham (Uber’s first CTO) • What is good software architecture at Netflix? • The past and future of modern backend practices • What is reliability engineering, and the history of SRE • How to debug large, distributed systems: Antithesis • Designing Data-intensive Applications with Martin Kleppmann • Building Bluesky: a Distributed Social Network Timestamps00:00 Intro 03:16 Sam’s path into tech 09:32 Thoughtworks 20:55 The rise of microservices 33:37 Are specs becoming more important than code? 45:22 Building Resilient Distributed Systems (Sam’s new book) 52:35 Three rules of distributed systems 56:16 Observability 1:02:20 Resilience tradeoffs 1:07:54 Idempotency 1:15:58 Thundering herds 1:21:03 Business context and resilience decisions 1:25:51 Resilience engineering: four concepts 1:32:42 AI and resilience 1:36:26 AI’s limitations and where to use it 1:40:06 Cognitive debt and cognitive surrender 1:45:02 Modular architecture and AI software factories 1:51:53 Resources for learning software architecture 1:55:30 Where to find Sam ReferencesWhere to find Sam Newman: • LinkedIn: https://www.linkedin.com/in/samnewman • Website: https://samnewman.io Mentions during the episode: • Acorn Electron: https://en.wikipedia.org/wiki/Acorn_Electron • Commodore 64: https://en.wikipedia.org/wiki/Commodore_64 • Alstom: https://www.alstom.com • Fortran: https://en.wikipedia.org/wiki/Fortran • Thoughtworks: https://www.thoughtworks.com • TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent • How Kent Beck shapes the software engineering industry: https://newsletter.pragmaticengineer.com/p/how-kent-beck-shapes-the-software • Extreme programming: https://martinfowler.com/bliki/ExtremeProgramming.html • Growing Object-Oriented Software Guided by Tests: https://growing-object-oriented-software.com/ • Dave Farley’s website: https://www.davefarley.net • Jez Humble on X: https://x.com/jezhumble • Continuous Delivery: https://continuousdelivery.com • Mike Bland’s publications: https://mike-bland.com/publications • How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler • James Lewis on LinkedIn: https://www.linkedin.com/in/james-lewis-microservices • Building Microservices: Designing Fine-Grained Systems: https://www.amazon.com/dp/1492034029 • Microservices Guide: https://martinfowler.com/microservices • Ben Christensen on LinkedIn: https://www.linkedin.com/in/benjchristensen • Scaling Uber with Thuan Pham (Uber’s first CTO): https://newsletter.pragmaticengineer.com/p/scaling-uber-with-thuan-pham-ubers • Stop being skeptical about AI for development with Charity Majors: https://newsletter.pragmaticengineer.com/p/stop-being-skeptical-about-ai-for • Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl: https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html • Chris Ford on LinkedIn: https://www.linkedin.com/in/ctford • Beyond Vibe Coding with Addy Osmani: https://newsletter.pragmaticengineer.com/p/beyond-vibe-coding-with-addy-osmani • Building Resilient Distributed Systems: https://samnewman.io/books/building-resilient-distributed-systems • Honeycomb: https://www.honeycomb.io • Monzo: https://monzo.com • How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt: https://margaretstorey.com/blog/2026/02/09/cognitive-debt • Fundamentals of Software Architecture: A Modern Engineering Approach: https://www.amazon.com/dp/1098175514 • Balancing Coupling in Software Design: Universal Design Principles for Architecting Modular Software Systems: https://www.amazon.com/Balancing-Coupling-Software-Design-Addison-wesley/dp/0137353480 • Learning Domain-Driven Design: Aligning Software Architecture and Business Strategy: https://www.amazon.com/Learning-Domain-Driven-Design-Vlad-Khononov-ebook/dp/B09J2CMJZY • Software Fundamentals: Collected Papers by David L. Parnas: https://www.abebooks.com/9780201703696/Software-Fundamentals-Collected-Papers-David-0201703696/plp • Just Enough Software Architecture: https://www.georgefairbanks.com/book • Modern Software Engineering: https://www.youtube.com/c/ContinuousDelivery — Production and marketing by Pen Name. You’re on the free list for The Pragmatic Engineer. For the full experience, become a paying subscriber. Many readers expense this newsletter within their company’s training/learning/development budget. If you have such a budget, here’s an email you could send to your manager. This post is public, so feel free to share and forward it. If you enjoyed this post, you might enjoy my book, The Software Engineer's Guidebook: navigating senior, tech lead, staff and principal positions at tech companies and startups.
|
%26stroke%3Dnone%26strokeWidth%3D3.6){kind=small}
Comments
Post a Comment