ci-cd — jd:/dev/blog

Steve Yegge Deleted the Wrong Thing

Tue, 18 Aug 2026 00:00:00 GMT

Steve Yegge deleted the wrong thing. He's right that merging one pull request at a time is mathematically dead at agent scale, and I'll grant that without hedging, because seven weeks ago I wrote down this exact scenario as the one way my bet loses. I sell merge queues for a living. Discount me accordingly, then keep reading, because the part he got wrong is more interesting than the part he got right.

In The Continuous Thunderdome, Yegge describes running about forty agents around the clock on Wyvern, the MMO he's been writing for twenty-five years. They produce roughly 175 real commits a day, some days 250. His build takes about half an hour. He started with batches of ten and bisection on failure, and watched his merge queue grow past a hundred, then to a hundred and sixty-six.

Then he stopped:

Just slam all the commits onto main, and then just friggin' deal with it. No bisections, no sequencing, no blame, none of that old crap.

He calls it the Land Rush. Let the queue build to a hundred, drop 120 to 150 commits onto main at once, and when main goes red, point agents at the wreckage. "Swarm diagnosis (not bisection)." He found out afterwards that AAA studios have done a version of this for years and call it Game DevOps, where main generally stays red and nobody minds.

The math is not in dispute

Here's the line worth keeping:

Once your commit rate outruns your build slots, one commit per green build becomes mathematically impossible.

That's true, and it's the whole diagnosis. If you can run two builds an hour and you receive ten changes an hour, no amount of clever ordering saves you. The serial queue was sized for humans opening four pull requests a day. I said as much in June: the human-scale queue strains under agent churn, grant it and move on.

So we agree on the problem. We disagree about what he actually removed.

He didn't delete the queue, he deleted the intent

Look at what a batch of 150 commits is before it lands, and what it becomes after.

Before, it's 150 changes, each with an agent that wrote it, each with a goal, a plan, and a reason. Every failure in there, if it fails, belongs to something.

After, it's one merged artifact. Main is red. The mapping from "this test is failing" back to "this agent was trying to do this" is gone, and it's gone permanently, because by then those agents have finished and moved on. David Crawshaw put his finger on why that's fatal in a post that reaches the opposite conclusion from the same premises: by the time CI reports a failure, "the agent context window is dead and gone."

So the swarm arrives at a red main with no author, no intent, and no plan, and has to reconstruct from scratch what 150 agents were collectively attempting. That is the most expensive debugging context you can possibly build. Yegge is right that binary search is too slow. He's wrong that the fix is to stop attributing, because attribution is the thing that makes the repair cheap.

And look at what "swarm diagnosis, not bisection" actually is. It's a parallel search for which change broke the build. That's still attribution. He kept the expensive half of bisection and ran it on the worst possible substrate, then threw away the half that was free.

Bisection was never really about finding the guilty commit. It was about finding the guilty intent, and handing the problem back to whoever had it.

Fix one thing, in one context

The alternative isn't the queue he abandoned. It's the queue nobody has built yet.

You still gate before merging. Main stays green, because a green main is what lets everything downstream stop thinking about it. But the queue stops behaving like a turnstile that admits or rejects, and starts behaving like the place where you sort things out as fast as possible.

A batch goes red. Instead of evicting the offender and making it start again from the back of the line, the queue works out what's actually broken and hands one diff back to the one agent that owns it, with the failure attached. Not "your pull request was dequeued, resubmit." Closer to "this assertion fails because of this thing you changed, here it is, fix it."

That's a different job than any merge queue does today, and I'm not going to pretend we ship it. But it's cleaner in the way that matters: you fix one piece of code, in one context, with the intent still attached, instead of untangling a hundred and fifty changes that have already been stirred together.

Yegge's own vision needs this more than mine does. What he's building toward, across Gas Town and these essays, isn't a framework you download. It's a civilization of colleagues running on datacenter silicon. Civilizations need infrastructure, and the CI and the merge queue that one would run on don't exist yet. That's the interesting problem. Not whether Git survives.

Where I'd lose

Two places, and both are real.

The first is coherence. Sometimes a batch fails and no single change owns it: three agents each wrote something defensible and together they contradict. Attribution doesn't help, because there's nothing to attribute to. Green is not coherent. A merge queue keeps main buildable, it does not keep main good, and that half is a review problem that nobody has answered.

The second is that a red main costs Yegge nothing. He's the only human on that codebase, and he says so, in the same essay where he enters Sam Altman's solo unicorn contest. When you're alone, "main generally stays red" is a workflow. The moment a second team pulls from your main, the same sentence is an outage, and the cost stops landing on the person who chose it. That isn't a refutation of the Land Rush for him. It's the reason it doesn't travel.

The real constraint was never the queue

Strip it back and the thing that forced his hand is a thirty-minute build and a finite number of build slots. That's it. He didn't have an ordering problem, he had a throughput budget, and he balanced it by setting the quality bar to zero.

Which is a legitimate trade. It's just a trade, and it deserves to be stated as one.

Main stays green. Changes land fast. CI spend stays under a number. Most days you get all three, and when they collide, something has to give. Nobody running a merge queue today gets to say which one. What they get is a batch size, which is a mechanism, and then they get to work backwards from it to an outcome they were never invited to state. Your CI pipeline wasn't built for this, and neither was the settings screen bolted to the front of it.

Most of the work ahead is making CI cheap enough that none of this requires a sacrifice. Yegge's real innovation was noticing that when you can't afford the gate, you can simply stop paying for it.

I'd rather make it cheap enough to keep.

GitHub Actions Isn't a CI

Tue, 11 Aug 2026 00:00:00 GMT

GitHub Actions runs your tests when you push. By the 2006 definition of continuous integration, that's all it takes, so technically, yes, it's CI. And yet nobody who has run a real pipeline on it believes that's what they have.

Here's the confusion I want to clear up: a runner is not a CI system. Actions is a superb runner. It is not a CI system, and the industry has spent a decade mistaking one for the other.

Say the awkward part first: Mergify sells into this layer, so I have a horse in this race. I'll disclose it once and let the argument stand on its own.

The word is doing two jobs

"CI" means two different things now, and the whole mess comes from smashing them together.

There's the practice: integrate often, run the build, catch breakage early. Martin Fowler wrote that down in 2006, and Actions does it fine. And there's the system: the machine teams actually picture when they say "our CI," the one that understands their tests, owns the build graph, keeps the pipeline fast, and decides what's safe to merge. Buying a runner gets you the practice, and it feels like it should get you the system. It doesn't. That gap is the entire post.

The 2006 definition predates everything hard about CI today: flaky tests at scale, monorepo build graphs, thousand-job pipelines, merge queues. The bar didn't move. The problems did. Actions clears a bar that was set before the hard part existed.

What Actions actually is

Strip the branding and Actions is a general-purpose, event-driven job runner. Something happens (a push, a label, a schedule, a webhook) and it runs jobs in containers. That's it, and that's a lot. Deploys, releases, bots, cron, search reindexing at 3am, and yes, your tests, which sit in that list right next to "post to Slack."

To the engine, your test suite is a command that exits zero or non-zero. It has no idea it's a test suite. That isn't a flaw in the runner. That's the definition of a runner, and Actions is a good one. Good is generous (look closely and it's a fairly basic runner too), but I'll grant it anyway, because even a great runner still isn't a CI system.

The four places the seam shows

Every daily pain with "Actions as CI" is the same thing underneath: a runner with no model of the work you handed it.

It has no model of your tests. It runs a script and reads an exit code, and that's the whole relationship. It can't flag a flaky test, because it doesn't know a test ran. It can't run only the tests your change affects, because it doesn't know what a test is or what you touched. It can't quarantine anything. A CI system treats tests as objects with history and behavior. Actions treats them as stdout and a number.

Its primitives are runner primitives, not CI primitives. Caching exists, as a step you wire by hand with keys you manage (actions/cache), not as a build graph the system understands, so it can't reuse or invalidate intelligently. Secrets are a model everyone quietly works around. You can re-run a failed job, but not a single step inside it, so retrying one broken thing re-runs the whole job. And yes, there are matrices, reusable workflows, composite actions. They're clever ways to script a runner. None of them make it understand what it's running.

You author it blind. There's no faithful local run. act approximates it, but it won't match the runner images, the secrets, or the hosted context, so real debugging is still edit, commit, push, wait four minutes, read the logs, change one line, push again. The development loop for the thing that guards all your other development loops is the worst one you own. There's no model to test against locally, only a remote runner to execute against.

The logic has nowhere to live but YAML. Expressions, if: conditionals, matrix expansions, bash templated inside a workflow key. Because the engine has no model of your process, every scrap of process logic gets crammed into config, and YAML quietly becomes a programming language. Every gnarly workflow file is a program in exile.

None of these are Actions failing at its job. They're a runner being asked to do a job that needs a model it was never built to have.

Why the mislabel costs you

If this were only semantics I wouldn't bother. It isn't, because names set expectations.

Listen to how teams talk about CI. "Is CI green?" "CI is slow again." "CI is flaky." Nobody says "CI caught the flaky one" or "CI ran only the two hundred tests that mattered," because their CI can't do those things, so the words for them fell out of use. The vocabulary shrank to a status light, because the tool is a status light. A whole generation now believes that's the ceiling of what CI is. It's the floor. Teams mistake the floor for the building.

The real capabilities all exist. I've argued that GitHub stopped tending the primitives everyone builds on; this is the same story from the other side. You stopped asking for a CI system because you were handed a runner and told it was one.

What to do instead

Keep Actions. It's a great runner and you have a hundred legitimate uses for one: deploys, releases, automation, glue. Keep all of it.

Just stop expecting the runner to be the system, and start closing the gap on purpose. If you're building it yourself, that means putting a real test layer in front of the exit code (flaky detection, test selection, quarantine), giving caching an actual dependency graph instead of hand-managed keys, and putting a merge queue between "passed" and "merged" so that green-in-isolation stops being mistaken for safe-to-integrate. If you'd rather buy it, several tools do this, mine included, and now I've said so twice. Either way the move is identical: stop letting a runner set the ceiling for what CI is allowed to be.

A screwdriver is a fine tool. It's a terrible hammer. We've spent ten years hammering with it and calling the bent nails "CI."

The Hidden Corruption Tax of AI Delegation

Tue, 12 May 2026 00:00:00 GMT

Frontier LLMs corrupt 25% of what you delegate to them in long workflows. The new genre of "I went back to writing code by hand" essays says the answer is to fire the LLM. They have the diagnosis wrong. We solved this exact problem for humans twenty years ago, with linters and CI. We just forgot to apply any of it to the new worker.

A new paper from Microsoft Research, LLMs Corrupt Your Documents When You Delegate by Laban, Schnabel and Neville, put a number on something most of us had been seeing for months. Across 52 professional domains, in long delegated workflows, even the frontier (Gemini 3.1 Pro, Claude 4.6 Opus, GPT 5.4) corrupted an average of 25% of document content by the end of a 20-interaction relay. The average across all 19 models tested was 50%. Errors were sparse but severe. They compounded with document size, turn count, and distractor files in context. Agentic tool use made it worse, not better.

The 25% comes from a backtranslation simulation (forward edit, reverse edit, compare to original), so it's a proxy for cumulative drift, not a literal claim about your next delegated diff. The point isn't the exact number. The point is the proxy holds across 19 models, 52 domains, document sizes and turn counts, in the same direction. The shape of the failure is real even when the headline number is conservative.

The honest reaction to that shape isn't "see, AI is useless." It's: of course it drifts, what did you build to catch the wrong slice?

Sparse but severe is the worst kind of bug

If LLM errors were dense and obvious, we would catch them. The whole industry would have rejected vibe coding by now. The paper's finding is more uncomfortable: the model is right most of the time, then quietly wrong about one critical thing, then right again. The result reads correctly. It compiles. It even ships.

This is the worst possible error profile for human reviewers. Rare anomalies in a sea of normal-looking output are exactly what humans miss, every time, in every field that has bothered to study it. The LLM accidentally built an attention trap.

It also gets worse with size. Bigger document, longer interaction, more distractor files in the context: degradation compounds. So the more useful you try to make delegation ("rewrite the whole spec" instead of "fix this one line"), the worse the corruption gets. The delegation interface rewards exactly the workflow that fails hardest.

Going back to writing by hand is firing the junior

Every few weeks now, a senior engineer publishes the same essay. I quit the AI assistants. I am writing code by hand again. I am faster, the bugs are gone, my brain is back. Each one gets three hundred HN comments saying "finally, someone said it."

I read those posts and I see the same mistake a stretched manager makes about a struggling junior: I should just write the code myself, they suck. No. They don't suck. You skipped onboarding. You didn't give them a feedback loop. You didn't pair-review their first ten PRs. You set them up to fail, then concluded that hiring juniors is broken.

LLMs are the same shape of problem. We have been onboarding humans into software projects for decades, and we long ago stopped trusting them blindly. We require their code to compile. We require it to pass a linter. We make them write tests. We force it through type checkers, static analysis, security scanners, dependency auditors. We don't do this because junior engineers are bad people. We do this because anyone producing code at speed quietly breaks things, and the deterministic harness catches it before production does. (As I wrote a while back, AI won't kill juniors, it'll expose stagnant seniors: same problem, same diagnostic mistake.)

Now the industry has hired the most prolific junior in history, and most teams forgot to point any of that harness at its output.

We already built the fix. It's called a linter.

The first thing you do when you start a new repository is not turn on Ruff, ESLint, Biome, cargo clippy, mypy, gosec, semgrep. It should be. It almost never is. Most new repos start out with a default GitHub Action template at best. Rarely a security linter. Almost never a custom rule for the smells that actually bite this codebase. And those are the projects humans are writing.

Then we plug an LLM into the same unguarded repo and act surprised when the same failure mode shows up in the diff.

Linters are the cheapest deterministic check we have. They exist precisely because we already learned the lesson the DELEGATE-52 paper had to re-prove with a small mountain of inference: confident producers of code make sparse but severe errors at scale. That's the entire reason the toolchain exists. Pointing it at an LLM doesn't require a research breakthrough. It requires turning on the thing we built for humans and forgot to wire up.

The corruption tax compounds because we removed the harness when we changed the worker. Same mistake we make when we send a junior into a repo with no CI and then blame the junior.

The obvious objection: linters catch syntactic bugs, not semantic ones. A regex flipped from ^foo$ to ^foo passes every static check. So does a dosage that went from 5mg to 50mg, an off-by-one in a business rule, a swapped argument when both types match. That is correct. It's also exactly why the current agentic coding tools (Cursor, Claude Code, Aider) run linters and unit tests inside their loop, and the paper's degradation numbers are still collected on top of harnesses like that. A single linter pass isn't the harness. That's the whole point. We also built property-based testing, fuzzing, golden-file diffs, contract tests, snapshot review, and the boring discipline of two humans signing off on a change. The harness is layered. The frontier LLMs are sparse-but-severe at every layer of it, which is what the paper actually shows. You don't pick one tool; you stack them, because every layer catches a different class of wrong.

The paper itself makes this point in passing: out of 52 domains tested, Python was the only one where most LLMs reached "ready" status. The paper credits "verifiable rewards", the fact that code can be checked deterministically. That is just another way of saying Python comes with a harness, the one programmers built precisely because any writer, junior or LLM, will quietly drift unless something deterministic catches them. Crystallography, music notation and accounting ledgers have spell-check at best. The domain with the densest harness is the domain where delegation actually works. That is not a coincidence.

In every other domain, DELEGATE-52's 25% becomes a corrupted file you ship and nobody catches, because there's no automated reader behind the human one. If you delegate to an LLM in a domain without deterministic checks, you are the harness. Act like it.

The interface helps too, but it's not the whole game

There's still a real interface problem on top of this. The dominant UX for delegated work is "here is the doc, do your thing": the model returns a new version, you skim it, you ship it. Implicit acceptance, hidden diff. That is the worst possible UI for a tool whose failure mode is silent corruption.

Cursor, Claude Code and Aider all show diffs. Real ones. But "show a diff" is not "force a diff". You can still click accept-all on a thousand-line refactor at three in the morning. The fix is making review the default path, not an option: forced span-level acceptance, drift indicators across long sessions, distractor-file warnings, visible quality-loss curves as the doc grows. As I argued in AI is a human interface nightmare, the pattern is depressingly consistent. Decent models surrounded by terrible UI for catching when they're wrong.

These are product problems, not research problems. They are exactly the same guardrail work we did for code review tooling over the last fifteen years. Same job, new worker.

You have to read what you delegate. You also have to lint it.

Until both the interface and the harness catch up, the defenses are boring. Diff what you delegated. Re-read every line the model touched. Run the linter. Run the tests. Run the security scanner. If you don't have time to do that, you didn't have time to delegate the task in the first place. You were borrowing against your future self, and a slice of the document is already gone.

I have shipped features at Mergify where I wrote maybe five percent of the lines myself and Claude wrote the other ninety-five. It worked. The reason it worked is not that I read every line with a microscope. It's that Biome, mypy, our test suite, and a Copilot review pass read the diff before I did. The harness was the reviewer I didn't run. The corruption tax was paid quietly in the background, and the linters caught what mattered. Take the harness away, and that same PR is a future incident.

The AI free lunch was about who pays for the compute. The corruption tax is about who pays for the silence. Both bills come due. Both are starting to.

The answer to both, weirdly, is the same boring thing: build the safety net you always should have had. The LLM didn't change that. It just punished the teams that never bothered.

Your CI Pipeline Wasn't Built for This

Tue, 17 Mar 2026 00:00:00 GMT

From where I sit at Mergify, the trend is obvious: same teams, same repos, no headcount changes, but more pull requests and more CI jobs than a year ago. The driver isn't a mystery. AI started writing code.

And the number will keep climbing. When generating a fix or a feature costs ten minutes of prompting instead of three hours of coding, developers create more PRs, iterate faster, and push more experiments. Creating code got cheap. Testing it didn't.

The Bill Nobody Budgeted For

More code means more tests. More tests means more CI minutes. More CI minutes means a bill that grows faster than the team.

This catches people off guard because the promise of AI-assisted development was productivity: do more with less. And that's true on the code side. But CI doesn't care who wrote the code. Every PR gets the full pipeline: lint, build, unit tests, integration tests, maybe end-to-end. Human PR or AI PR, same cost.

When you were shipping five PRs a day, running the full suite each time was fine. When you're shipping thirty, you're running the same tests six times as often, and most of those runs produce no new information. You're paying for redundancy that made sense at human speed and makes no sense at AI speed.

The solution isn't to skip tests. It's to stop running tests that can't tell you anything new. The usual advice (test selection, caching, fast checks before expensive suites) isn't new. It's just not how most pipelines work, because most teams configured them when five PRs a day felt busy.

Flaky Tests Are Poisoning Your Agents

Cost is painful but manageable. You can throw money at it. For a while. The real problem is signal.

On our main branch at Mergify, where the code is already merged and most runs are about integration stability, roughly 90% of CI failures are transient: infrastructure hiccups, network timeouts, resource limits, the kind of failures that disappear when you hit "retry." That's high, but our test suite is large and infrastructure-heavy. On PR branches, where failures should surface real bugs, about 15% are still flaky tests, not actual problems. Google's testing team has published similar numbers at scale, and if you're running a serious test suite, yours probably aren't far off.

When a human developer sees a red build, they open the logs, recognize the flaky test, swear under their breath, and hit retry. They carry context. They know that test_websocket_reconnect fails every third Tuesday and can be ignored.

An LLM doesn't know that.

Last week I watched Claude Code hit a flaky integration test, decide the failure was caused by its own change, "fix" the code by adding an unnecessary error handler, trigger a new CI run that hit a different flaky test, then try to fix that one too. Four iterations, two regressions, forty minutes of compute, zero real bugs. I killed the session and hit retry myself. Green on first try.

That's the loop. At human pace, flaky tests are an annoyance. At AI pace, they're a multiplier on wasted compute and wrong decisions. The LLM is making choices based on bad signal, and it's making them at machine speed.

Yes, agents will get smarter about this. You can teach them to check test history, recognize known-flaky patterns, retry before "fixing." But that pushes CI knowledge into every agent, every tool, every workflow. It's the wrong layer. The CI system should know which signals to trust.

From Status to Signal

Today, CI is a gate: green means go, red means stop. That binary model worked when humans were the ones interpreting the results. It breaks when the consumer of CI output is an LLM that takes "red" at face value and starts debugging a ghost.

What CI actually needs is to become aware of its own reliability. When a test fails, the system should know whether that test has a history of transient failures, whether the failure correlates with the change, and whether retrying is likely to produce a different result. That context exists in build history and test failure patterns. Almost no pipeline uses it.

The next generation of CI needs to output signal, not just status. Not "failed" but "failed, likely flaky, recommend retry." Not "13 tests failed" but "2 failures correlate with your change, 11 are known flaky." Give the LLM (or the human) the information to make a good decision instead of a fast one.

This matters beyond individual PRs. Merge queues depend on CI signal to decide what lands in main. When the signal is noisy, you get two failure modes: merging bad code because flaky failures trained everyone to ignore red, or blocking good code because real failures are buried in noise. Both get worse as volume increases.

The Real Waste

Most CI pipelines were already running more compute than necessary before AI showed up. Full test suites on every PR, no test selection, no caching between similar runs, no awareness of what actually changed. The waste was tolerable at human speed because the volume was low.

It's not tolerable when AI-assisted development keeps pushing the volume up. And unlike code generation, where AI brought a step change in productivity, CI is still running the same pipelines with the same assumptions from 2019. AI didn't create the flaky test problem or the redundant pipeline problem. It just made both impossible to ignore.

Your CI pipeline was built for a world where code was expensive to write and cheap to test. That world is gone.