Richland Technology
Richland Technology
← Back to Blog
llmagentsproductionopenaiperformancecodex

OpenAI's Codex "Performance Factory" finds and fixes ChatGPT regressions — but its own numbers tell the honest story

A Codex-driven Regression Workflow root-caused 600+ cases in three months — but OpenAI's own 70% accuracy figure and "fixing is the hard part" caveat are the real story.

Richland AI Desk·19 November 2026·4 min read

OpenAI has shipped an autonomous, Codex-driven "Performance Factory" that finds and fixes ChatGPT performance regressions end to end. In just over three months the Regression Workflow tracked more than 1,700 investigations and root-caused over 600 cases (17 automated investigations a day). But OpenAI reports only 70% of root-cause suggestions were accurate in a manual review of 100, and it says fixing a regression is harder than finding it.

What shipped

The Performance Factory, described in a November 19, 2026 engineering post by OpenAI staff Okay Zed and Cyrus Forbes, is a cloud system that runs agentic performance workflows. Engineers define workflows as configuration files plus Markdown prompts. A dispatch loop assigns ready work to Codex agents, and subscriptions wake an agent when something it's waiting on changes — a GitHub review comment or a completed deployment, for example.

The first workflow is the Regression Workflow, which moves a task through four states — needs investigation, root caused, fix proposed, resolved. When a configured performance alert fires, it opens a task. An agent investigates and records findings; a separate agent proposes a fix and opens a pull request; an engineer reviews and merges; and after deployment another agent checks whether performance returned to baseline. Task state lives outside any single agent's context, so work can pass between agents and engineers and resume after a review or deployment.

The system has four parts: a task service with configurable workflows, a dispatch loop that assigns ready work, an internal API that runs Codex turns in the cloud under a service identity (with access to the same repos and observability tools engineers use), and subscriptions that re-queue waiting tasks.

Why it matters

The interesting part is how OpenAI describes the reliability problem and the checks it built around it. The post walks through a six-stage evolution from manual work to the factory: pre-Codex manual investigation (the team accumulated more than 30 performance tools), Codex as a partner, reusable skills, a "human proxy" stage where engineers pasted alerts into Codex, automated investigations, and finally the factory.

The reusable skill is perf-forensics, which tells Codex which tools are available and how to work through a slowdown from three angles: find when the slowdown began, identify the affected requests, and investigate whether deployments or configuration changes explain it. The team refined it against evals built from past regressions.

The key design decision is adversarial review. Early on, agents "sometimes stopped at the first plausible explanation they found, and some of those diagnoses turned out to be wrong." So OpenAI added a separate Codex session whose only job is to disprove the proposed diagnosis — rechecking assumptions and testing counterfactuals. The post gives a concrete example: an agent traced failing ChatGPT conversations to errors in image downloads and marked the cause found, but the reviewer noticed the internal file-locating service had already recovered while image requests were still failing. The diagnosis couldn't explain the continuing failures, so the investigation was reopened.

That adversarial step is why the accuracy number is worth reading carefully. In a manual review of 100 root-cause suggestions, OpenAI graded 70% as accurate and 30% as needing further refinement or not fully establishing causality. The post frames the adversarial reviewer as a mechanism for pushing that number up through "checks and balances."

OpenAI is explicit that fixing is the hard part. Agent-proposed fixes were sometimes more complicated than needed or introduced new issues, which is why it added adversarial review of fixes too, and why an engineer still reviews and merges every PR. Each task carries an auto-generated report — regression summary, impact, root cause, proposed fix, metrics time series, CPU flame graphs, traces — to make that handoff easy.

Caveats

Every headline number here is vendor-reported. OpenAI's own post is the only source for the 1,700 investigations, 600 root-caused cases, 17-per-day rate, and the 70% accuracy figure; no third party has independently measured them. The post also publishes no user-visible latency, error-budget, or uptime numbers — the figures it gives measure the process (investigations, root causes, review accuracy), not how much faster ChatGPT actually got.

The 70% accuracy figure is modest and self-reported, and OpenAI itself uses it to justify keeping a human on the merge button. The Performance Factory is also just one part of OpenAI's broader engineering flow, so it should not be read as a full account of how ChatGPT stays up.

Sources

Start a Conversation

Ready to build something like this?

Tell us about your project and we'll respond with a clear plan within one business day.

Tell Us About Your Project

Takes 3 minutes. We read every submission personally.