Why is Meta destroying its engineering organization?
Hi – this is Gergely with a free issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of senior engineers and engineering leaders. If you’ve been forwarded this email, you can subscribe here. Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager. Why is Meta destroying its engineering organization?Leadership at the social media giant has been on an AI-fueled rampage through its engineering org. We report what’s happened
For two decades, Meta had a unique, high-performance engineering org; right up until around April of this year. For the first 20 years of the company’s existence, it had a “move-fast-and-break-things” culture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the company were empowered to do good work, focus on impact, and to balance business interests with solid engineering. But in the past few weeks, all that has changed, as if the leadership has been following detailed blueprints on how to demolish a proven, successful engineering culture in the most ruthlessly efficient way possible. For the past few weeks, I’ve been sharing how bad things are inside the social media company for engineers in one of Silicon Valley’s most prestigious workplaces. In this article, we walk through what’s happened, and ask what’s going through the minds of leadership who are reducing software engineering there from the profit center that it was between 2004 until very recently, to the disdained cost center that it has become in just a few weeks. We cover:
1. Meta’s pre-AI engineering cultureI’d split Meta’s engineering culture into two eras: “move fast and break things”, and then “move fast with stable infra.” “Move fast and break things”In the 2010s, Facebook’s unconventional engineering culture had grown somewhat legendary in the tech industry, as the company went against conventional best practices and succeeded massively. In 2012, when Facebook hit the billion-users landmark, the company produced a small physical book about its culture which was placed on employees’ desks. Presented with retro propaganda design, it was dubbed the “little red book”, co-opting the name of a famous volume of the thoughts of Chairman Mao, (1964). At around 70 pages long, Facebook’s version codified its engineering culture: speed, fearlessness, taking ownership, and thinking outside of the box.
Back then, mantras in Facebook’s little red book were also in print across campus, and included:
There was genuine focus on building good products. Also from the book:
“Move fast with stable infra” cultureIn 2022, I did what is one of the longest deepdives we’ve published on the topic of Meta’s engineering culture. By then, things had evolved, and much of any former recklessness was gone, replaced by the principle of moving fast, but with stable infra. Here’s how I described Meta’s engineering culture then:
Also, Facebook, as a product, has one of the most sophisticated auto rollout systems in the industry. Instagram has a battle-tested infrastructure where it was almost trivial to launch a new social network (Threads) with 100 million users served in its first week. Engineers whom I knew inside the company are capable, motivated, and product-minded, and their work was appreciated. CEO, Mark Zuckerberg, was influential: he personally coded the first version of Facebook, had stayed close to engineering, and valued software engineers very much. Engineers there felt they were working inside a profit center. 2. Investing in AI and pressing engineers to always use itMeta has been the only company among the big five of Apple, Microsoft, Amazon, Google, and itself not to own a hardware platform or operating system. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle. Stepping back, it looks as though the Mark Zuckerberg of today has resolved not to miss a platform opportunity, after the company failed to build its own mobile OS or mobile phone during the 2010s. This is one reason for investing so much in virtual reality (VR) with Oculus, and in augmented reality with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR – and the metaverse – could be massive. Billions was spent on ensuring Meta would be the market leader in this space. But once again, VR didn’t go mainstream; since the end of the pandemic, popular interest in the segment has died down considerably. When it became clear that AI would become a mega-trend in 2022, Zuckerberg didn’t miss it: he assembled the internal FAIR group (Fundamental AI Research team), and released a series of open-weight AI models:
In June that year, Meta acquired Scale AI to reboot its AI efforts for a whopping $14.8B, and brought Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strategy. The acquisition of Chinese startup Manus AI for $2B is currently in question after China blocked the deal from being completed. Based on the investment made into Scale AI, it’s pretty clear that Meta – and Zuckerberg – is determined to build a state-of-the-art LLM that can be competitive with the latest versions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to deliver. Scale AI brings in a very specific kind of expertise to Meta, as one of the best in the industry in:
Wang seems to have a very broad reign to do what he has been an expert in: creating training data, doing data labeling and RLHF. This is being pulled off with the labor of Meta’s engineering workforce, and by surveilling it. Problem #1: Tracking keystrokes and mouse clicks, with no option to opt out. In late April, Meta told engineers they were being enrolled into a system that tracks every keystroke and click, to produce training data for Meta’s new AI. There’s no way to opt out. Needless to say, this is invasive and raises privacy questions: If you log into your personal bank account, does the tool track you? What about when you’re writing a personal email, or responding to a personal call? Meta held no consultation and there are no workarounds; just a top-down decision being pushed through. This month, Reuters reported that people’s concerns there are finally being heard:
From talking with current Meta engineers, I understand the logging system has not been rolled out in the UK due to data protection regulation. Problem #2: 30-50% of engineers on core teams have been forcefully reassigned to data labeling and RLHF, upsetting folks even more. Also starting in late April, product engineering teams received a mandate from above, whereby 30-50% of engineers were to leave the team and join the ADO org (Agent Data Optimisation). “Forceful” reassignment is very relevant here because of Meta’s traditional engineering culture. Between its founding in 2004 and until last year, Meta gave engineers autonomy to choose where they work and what they work on. This was structural to how the company worked:
Team selection via bootcamp started to die down in around 2024, but any Meta engineer with at least two years’ tenure knows that previously they chose what to work on, and of course, could pick the most impactful thing to work on. And then, out of the blue, they’re assigned to a division where the impact is not clear, the work is menial, and doing it too long will surely hurt their career prospects. Infrastructure and security teams were hit especially hard by reassignments. I talked with several engineers in infra orgs, who had 30-50% of their teams drafted into the ADO org. And in some cases, it was the best engineers who left. One engineer told me that the whole situation feels like the movie, The Hunger Games, when tributes are randomly selected and then removed from their environment, to something completely different. Except, at Meta, many more folks are being affected, with between three and five from a 10-person team going from building products used by hundreds of millions, to giving human feedback on AI-generated GitHub repos, over and over. So, a wider impact than in the Hunger Games, but with less drastic consequences. Around 6,500 people are in the ADO org, more than at OpenAI and Anthropic. Roughly four to five thousand of these are software engineers. Meta has around 25,000 engineers, meaning that one in every 5-6 software engineers may now find themselves doing data labeling full time. As you can imagine, people are actively open to new positions, and nobody is updating their job title on LinkedIn and elsewhere to “data labeling at Meta.” I’ve spoken with people in this role and they don’t like doing it, and feel upset about the top-down decision making. The silver lining is that they still have a job, have retained their salary, and were not part of layoffs. They still have time to leave Meta for something that pays comparably and is not a data labeling job. 3. Core engineering folks feel treated like trashProblem #3: a month-long waiting game, stoking fear across the company. On 20 April, Reuters reported that Meta planned to lay off 10% of staff in a month’s time, and Meta confirmed the news, meaning there was a period of four weeks when everyone knew that they could be unemployed very soon. Forced reassignments to data labeling started to happen. As I covered at the time:
Problem #4: Performance review is hyper-aggressive at Meta, so devs optimize all metrics. The internal performance review process, PSC (Performance Summary Cycle), is very stringent, compared to Google and Apple, I’ve learned. Managers inside Meta “fight” over the pay packets of their employees, which involves “knocking down” the packet of engineers on other teams, so their direct reports are ranked higher. It’s common to weaponize metrics in this process – be that business impact, the number of code reviews, number of lines of code written, pre-AI (see our Coding Machine archetype podcast on this.) Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets. After a few years, engineers at Meta learn that the best way to not get a bad PSC rating is to have all metrics – impact, code committed, and other numbers – higher than their peers’ are. Learn more about the internal politics of performance calibrations. Problem #5: tokens are measured as part of perf, so devs aggressively optimize for it. When layoffs were confirmed, engineers also learned that managers shall inspect token count during perf reviews. This raised worries that those with low token counts might be marked as underperformers and dismissed. So, what is the natural reaction to this as an engineer at Meta? They started using AI tools for the sake of generating more tokens. This happened while Meta had an internal token leaderboard, encouraging tokenmaxxing. As I wrote on 16 April:
The biggest problem: people stop caring about real work and focus on performative work. Let’s check the four ingredients that Meta’s leadership has decided to introduce to their workplace:
Shake this mix up well, and what do you get? Two things:
To its credit, Meta has given out generous retention equity packages to several engineers considered key on the remaining teams. These packages make it harder to get matching compensation elsewhere. Still, I talked to one engineer who got an equity top-up, and said that this approach helped him decide to leave as soon as possible because he feels bitter about the lack of autonomy and having no control over things. 4. Most embarrassing-ever outageMeta’s core infra and security teams have suddenly found themselves severely understaffed. Most folks are pushing AI-generated code merged with AI-only reviews, without paying much attention to quality. After all, they’re dealing with the possibility of unemployment, while firefighting to operate a team without its best engineers whose headcount has been cut in half, all with the knowledge that AI usage could affect their own job security. Two weeks ago, on 30 May, the most embarrassing outage in Meta’s history happened. Here’s software engineer Siddharth Sundharam’s summary (emphasis mine):
This is a security breach in which Meta left its extra-secure, reinforced front door unlocked, so that anyone could come in, and there was no alarm to notify anybody when it happened! It seemed Meta only noticed when users started reporting it on social media! From talking with folks inside Meta, I’ve learned that AI was at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted were together the cause of this beyond-embarrassing incident. I poked around, and here’s what I gathered:
Meta’s Chief Security Officer resigned the very next day. The outage was resolved on Monday, 1 June, and an investigation started as part of the SEV process. On Tuesday, Meta’s Chief Information and Security Officer (CISO), Guy Rosen, announced his departure. Coincidence? I suspect not: the CISO might have stepped down if they warned against the Security org being gutted but were then ignored, and so no longer trusts leadership. I also imagine the CISO didn’t have the idea to move a good half of Instagram’s security team over to data labeling. As a note, the previously worst outage was all Meta services going down for seven hours in 2021, due to a DNS / BGP configuration issue. It was a bad outage, but Meta handled the follow up well, in my opinion. After that 2021 outage, Meta shared a postmortem and apology. It has not done so for the latest Instagram account takeover outage. 5. Internal messWired shares more details on just how bad the situation is inside Meta, right now:
There’s more: Meta’s Chief Product Officer, Chris Cox, reportedly admitted to staff that Meta’s upper leadership (the folks above him, meaning the C-level at Meta) created the mess. Also from Wired:
6. Self-inflicted woundsSo, is there an ultimate source of the “insanity of this company”, as CPO Chris Cox put it? Engineers whom I talked to point the finger at two individuals: Mark Zuckerberg and Alexandr Wang. Zuckerberg has full control over the business, and has made the decisions to reallocate a good part of engineering folks to data labeling, to roll out tracking software, and to lay off 10% of staff when Meta achieved record revenue and profits. As the CEO, the buck clearly stops with him. But it’s hard to unsee that – outside of layoffs – everything that Meta is doing is taken from the Scale AI playbook, and that surely comes from Wang:
Before all this happened, Meta was on track to overtake Google as the world’s #1 ads business by the end of the year. But for some reason, Mark Zuckerberg decided that building a coding LLM is more important. Meta’s leadership is now trying to undo all the damage they have done. Wired reports that Meta CTO, Andrew Bosworth, admitted to staff that the AI reorg was atrocious and committed to better communication in the future. To me, it looks obvious that Zuckerberg doesn’t care how engineers feel about the massive changes, and that Bosworth likely ignored the chaos, all while engineers know for a fact that the next AI model matters more than they do to the business. Bosworth also said that employees will have access to AI coaching tools. Very considerate, given the situation! Based on all I’ve learned, Meta’s engineering culture is dead because leadership has made it clear that engineering at the company is a cost center.
Needless to say, I hope my assessment is way off, but I’ve seen nothing yet from Mark Zuckerberg and Alexandr Wang – the two executives creating this current mess – to suggest it is. There may be a short time period where, if major changes like data labeling assignments and staff tracking are undone, then things at Meta could return to normal. The longer the current conditions persist, the more tenured engineers will surely leave. 7. Is it just Meta, or are other companies also acting irrationally?It’s tragic to see a technical founder at Meta so focused on AI that he neglects the engineers who built the heart of his company. But is Meta a one-off exception? Mitchell Hashimoto (creator of Ghostty, founder of HashiCorp) says he is seeing similar behavior by other founders (emphasis mine:)
The takeover outage at Instagram was exactly like this: the engineering team dropped the quality bar for AI-generated and AI-reviewed code, probably expecting that they could recover quickly from failures. And they did indeed recover… after the damage was done, high-profile Instagram accounts were hacked, and the system was compromised, all very publicly. Mitchell highlights the specific concern of founders over-estimating the capabilities of AI, and consequently casting aside sensible safeguards when shipping code to production. TakeawaysMost of us probably have something to learn from the disastrous events at Meta caused by hyper-focus on AI to the exclusion of people who are the lifeblood of that company. In some good news, I’m hearing that in the UK, some of the 10% layoffs have suddenly been cancelled: at the end of the mandatory consultation period, several infra and security teams are learning that no engineers on their team will be let go, as originally expected. Meta has a booming business, and is already a beneficiary of AI via increased ads revenue. Meanwhile, my Facebook feed is filled with fake, AI-generated videos, with hundreds of comments from bots and people who seemingly don’t realize it’s AI. It all seems like just more content for Meta to show ads next to. And yet, despite business booming, Meta’s leadership has gone on a crusade to inflict the most damage possible on its engineering org. Apparently, they’re now learning that most of it was pointless. If you’re in a leadership position and feeling the temptation to make drastic org changes for AI-related reasons, take a deep breath and see where it left Meta. Meanwhile, If you’re an engineer at a company whose leadership is over-indexing on AI, consider forwarding this article as additional context. If you’re hiring standout engineers who are extremely hands-on with AI, then it’s never been easier to get talent from Meta, than right now. Every engineer I know at the company is an extremely early adopter of AI, and knows how to build products and AI infra. These folks have soured on the company and its leadership. Meta’s loss of talent will be the gain of other startups and the rest of Big Tech; it’s one benefit of AI that’s probably a bit unexpected – not least of all by Meta! It seems like the old mantra of “move fast and break things” has now extended to Meta’s engineering org itself, with the company’s rush to over-invest in AI, so it will avoid missing the latest mega-trend in the tech industry. You’re on the free list for The Pragmatic Engineer. For the full experience, become a paying subscriber. Many readers expense this newsletter within their company’s training/learning/development budget. If you have such a budget, here’s an email you could send to your manager. This post is public, so feel free to share and forward it. If you enjoyed this post, you might enjoy my book, The Software Engineer's Guidebook: navigating senior, tech lead, staff and principal positions at tech companies and startups.
|
Comments
Post a Comment