Inside OpenAI’s agentic software factory
👋 Hi, this is Gergely with a subscriber-only issue of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. If you’ve been forwarded this email, you can subscribe here. Inside OpenAI’s agentic software factoryA deepdive into how Codex has “taken over” OpenAI, how the frontier lab builds its agentic software factory, and the engineering challenges of one billion users. Details from inside OpenAI
It’s rare to work with an unlimited token budget, but at OpenAI, that’s what all engineers, researchers, finance colleagues, and marketing folks do. Recently, I visited one of the world’s leading frontier labs to find out how OpenAI operates today – and for a glimpse at where software engineering might be headed as a profession. Plenty has changed since I visited OpenAI’s headquarters last year. Within a year, Codex has gone from a “nice-to-have” tool to being the backbone of pretty much everything at the company. To learn more, I talked with seven engineering leaders and engineers there: Venkat Venkataramani (VP of Engineering, Applied Infra), Sulman Choudhry (Head of Engineering, ChatGPT), Andrew Ambrosino (Lead, Desktop), Joe Gershenson (Lead, Core Agent team), Akshay Nathan (Engineering Lead, Productivity), Ahmed Ibrahim (Engineer, Codex) and Steve Coffey (Engineer, Responses API). Thanks to all for taking part! Today, we cover:
Before we start, a scheduling update: I’m in New York for the week, attending the LDX3 conference and visiting a few startups and tech companies in the city, so there will be no edition of The Pulse on Thursday. Normal service resumes next week! The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online. 1. Codex takes over at OpenAIThe takeaway from my visit to the company’s headquarters which really sticks out is that Codex – and lately Codex and ChatGPT Work – have taken over everything there, starting in around January. Desktop lead, Andrew Ambrosino, told me:
The token usage chart below shows this sudden adoption surge:
In a four-month period, non-engineering orgs like finance, recruitment, and legal went from ~0% usage of Codex to 90% usage. Now, almost all OpenAI employees use Codex and ChatGPT Work weekly. So, what happened? OpenAI released the Codex app for Mac in February and for Windows in March, and ChatGPT Work (powered by the Codex harness) in July. Following that, non-engineers there moved all their workflows over to Codex and then to Work. Caveat: OpenAI’s internal version of Codex is a lot more advanced than its external counterpart because it’s plugged into pretty much every OpenAI system – similar to how Ramp’s Inspect AI agent has been wired up. The fascinating part of this is that OpenAI got close to 40% adoption across non-engineering teams at a time when the Codex app was hostile to non-engineering users (hard to use). Between February and April, the Codex app still showed the code on-screen, but even so, non-technical colleagues outside of engineering still used it because it could do complex work like researching and creating a presentation, document, spreadsheet, or tasks that produce rich output. Today, those folks are very heavy users of it. Being able to work for longer on more complex things drove adoption. OpenAI added the /goal setting to Codex, where you can set up a goal for the agent and it keeps working until it is complete. Between April and May, usage surged from 60% to 90%. Andrew believes improvement in the harness’s handling of long-running tasks was one cause of this:
“Awareness overhang” is another cause of the rapid adoption, the Codex team believes. As Akshay Nathan, Engineering Lead, Productivity team, told me:
Role-specific and team-specific plugins are created and distributed. Another thing that sped up adoption is that each group started to distribute useful role-specific workflows as plugins. Andrew explained why it’s important to not just offer a generic coding agent:
Subject matter experts are embedded in ChatGPT Work engineering teams. The models have become “smarter” than developers in some domains, so devs cannot channel “taste” into the harness in those areas. So, people who are domain experts are onboarded onto engineering teams. This is one outcome of ChatGPT Work being used by so many non-engineering domains: experts embedded with engineering advise developers on things like what a good slide deck, spreadsheet, or business report looks like. Of course, domain experts being in engineering teams is a decades-old best practice for building quality products. It seems like this gets rediscovered in different contexts every few years! OpenAI is fully dependent on Codex and Work. This is so much the case that in the event of even a minor outage, internal messages from colleagues alert the Codex and Work teams at the same time as – or before – automated alerts. Basically, work happens through Codex and Work, and pretty much nothing else. From the outside, this dependence on a single shared harness is particularly eye-catching; two years ago, there were no AI agents, only advanced AI autocomplete! 2. Death of the IDE & pull requestsLate last year, the Codex team was torn about whether to release the Codex desktop app. Andrew recalls the hesitation:
Indeed, since January, IDE usage has gone down and OpenAI’s bet looks like a good one. However, the Codex app is becoming a little more akin to an IDE: for example, the ability to edit files inside the app was shipped in June. CI/CD systems are seeing massive load increases. One sign of productivity gains from Codex is the amount of additional code flowing through OpenAI’s dev infra systems. More code being created and pushed leads to new scaling challenges which the team is currently heads-down on solving. Venkat Venkataramani, VP of Engineering, Applied Infra, said:
In this context, PRs and code reviews are being rethought. They have “core primitives” in software engineering, but this level of development acceleration is an opportunity to reimagine them. Again, from Venkat:
An increasingly painful bottleneck is in deploying native mobile apps. When there are ten times more pull requests, it’s challenging to deploy on the backend or the web and more infrastructure is needed to do so. Then, after you rework a CI/CD system and make sure there’s enough capacity to run them, you’ll be deploying that many more PRs to production. This arises in the shipping of updates to native iOS and Android apps because every app update needs to go through Apple’s and Google’s manual approval processes which take hours or days to complete. Talking with Sulman Choudhry, Head of Engineering, ChatGPT, he explained how the app review bottleneck is affecting iteration speed. Sulman used to work at Facebook and remembers how the social media company sped up shipping mobile releases:
There’s some irony in how shipping a native iOS or Android app has the exact same challenges today as in 2008, when the App Store was launched. In 18 years, not much has changed! Apple still does not officially allow apps to bypass the App Store review process to ship meaningful experience changes. 3. OpenAI’s agentic software factoryThe idea of a “software factory” is similar to a physical factory where robots and humans produce autos together. In the software context, it is AI agents and humans producing software. Some manufacturing sites are fully automated “dark factories” where illumination isn’t needed because there are no humans. Could the same fully automated process emerge in software engineering? At OpenAI today, there’s a “software factory” running and it’s all built around Codex. Here’s how the “traditional” software development pipeline used to look, compared to what OpenAI’s agentic infra pipeline looks like today, as described by VP of Engineering, Applied Infra, Venkat Venkataramani:
The pipeline: 1. A human builder defines the desired outcome. A software engineer or product manager specifies the problem and desired outcome. Judgment, prioritization, and taste are becoming more important for this phase. Interestingly, Venkat told me that engineers at OpenAI are becoming more like product managers than traditional systems engineers. 2. Codex gathers context. OpenAI has moved all its documentation inside of the source code, which makes it easier for agents to understand more of the code. Codex also has access to:
Codex is so “plugged” into OpenAI that new engineers are directed to ask Codex any questions they have during onboarding because it has a surprising amount of context. 3. Codex implements code changes. This part is trivial enough: Codex gets to work and makes a series of code changes until it reaches its goal, and then verifies that the software works as it should. 4. Build & test, then CI. The agent builds the code, runs the tests, fixes the code when it breaks the test, and then creates a pull request. This pull request triggers the continuous integration (CI) server to run and execute a more thorough suite of linters and tests. The agent babysits the PR until it’s “green”, fixing any CI failures and automatically updating the PR. New: a “perf harness”: the agent also uses a perf harness to send problematic PRs to the Synthetics A/B framework for evaluating performance implications. As mentioned above, the load upon CI systems has increased greatly in the past six months. 5. Agentic code review. Instead of using one generic AI code reviewer, OpenAI spins off multiple agents, each with a “domain specialist” configuration. Venkat told me they see this as equivalent to having a human domain expert from each relevant infrastructure team review every change. Note from Gergely: I was skeptical about the claim that an agent that’s told to be a cloud infra specialist would produce a different review from a generic agent. However, all Codex agents have full access to OpenAI’s code and docs, so this “cloud infra expert” agent likely has gathered a lot of context about cloud infra setup and best practices, meaning it should provide highly targeted feedback. The important thing is how these “domain specialist” agents are set up, the context they have access to, and how they focus only on their own domain to make best use of their limited context window. Code changes are classified by risk. High-risk changes can be sent through stricter processes; for example, they might invoke more AI code reviews, or mandate that a human reviews it after the AI agents finish. Low-risk changes follow an easier path; areas of the codebase can opt in to an agent that will auto-approve low risk PRs, removing human acceptance as a bottleneck and improving velocity. A neat thing about risk assessment is that OpenAI can automate when additional compliance input is needed: either automated (via another agent) or human review. With the quantity of PRs being produced, it simply wouldn’t be possible for humans to review all code without assistance. Just like with CI, the coding agent babysits the comments and updates the PR to fix issues surfaced. 6. Agentic deploy. After a human approves a change to go to production, it is assigned its own agent with an instruction that could be summarized as: “Handhold this change until it is safely and fully rolled out into production.” Agents handhold both the code changes and the changes behind feature flags. For example, in the case of a change behind the feature flag, the agent will:
OpenAI’s long term goal is to have something like a “per-change autonomous SRE” (site reliability engineer) in the form of an agent that can deploy pretty much autonomously. 7. Observe production. Tools which track the production system:
One big change at OpenAI since my previous visit, pre-Codex, is that back then, engineers created dashboards to monitor services whereas now, agents do this at the granularity of per-change deployment. 8. Production monitoring feeds back into development. OpenAI’s “Perf Factory” uses agents to sift through alerts and dashboards, de-duplicate signals, identify real latency regressions, root-cause them and propose fixes. This helps catch performance issues introduced by ongoing code changes, extending the workflow beyond deployment into continuous improvement. 9. Respond to outages. Sevbot is OpenAI’s internal incident response agent; unsurprisingly, it’s also built on top of Codex. When an incident is detected, the bot “wakes up.” Here’s what it does:
OpenAI’s goal is to get to the point where Sevbot can take autonomous action when mitigating some outages. The dream is that no humans be woken up outside of their working hours during an outage because Sevbot can handle “routine” outages autonomously, with humans reviewing its actions when they return to work. But as of now, oncall duty is not a thing of the past at the company. 4. How engineering tooling & practices are changingUnsurprisingly, Codex is changing how easy it is to build internal tools and having an impact on standard engineering practices like debugging. Here’s what I gathered from talking with folks at OpenAI... Subscribe to The Pragmatic Engineer to unlock the rest.Become a paying subscriber of The Pragmatic Engineer to get access to this post and other subscriber-only content. A subscription gets you:
|
Comments
Post a Comment