devops — jd:/dev/blog

GitHub Actions Pricing: The Platform Reality Check

Thu, 18 Dec 2025 00:00:00 GMT

GitHub just announced pricing changes for GitHub Actions, and as expected, parts of the CI ecosystem panicked.

Some people are celebrating the price drop on GitHub-hosted runners. Others are furious that GitHub will start charging for self-hosted runners. And a few businesses are suddenly asking existential questions.

Since then, GitHub has paused the self-hosted runner billing change, acknowledging they moved too fast and didn’t involve the ecosystem enough.

That doesn’t change the underlying reality. It just delays the conversation.

GitHub Actions Was Never “Free”

GitHub Actions is not just a binary that runs on your machines.

It’s a platform:

job queuing and scheduling
runner registration and lifecycle management
workflow orchestration
security, isolation, secrets handling
reliability at massive scale

Even when you run your own hardware, GitHub is still doing a lot of work on your behalf. That infrastructure has always existed; hosted runners simply subsidized it.

GitHub explicitly said it:

“We have real costs in running the Actions control plane.”

That’s not new. It’s just now being made explicit.

Charging a small per-minute platform fee for self-hosted runners isn’t conceptually unfair: it’s GitHub aligning pricing with reality.

If you believe this is unacceptable, there has always been a clear alternative: run Jenkins, GitLab CI, or any other system where you fully own the control plane.

But you don’t get GitHub Actions “for free” just because the CPU cycles are yours.

Vendor Lock-In? Yes. And Everyone Chose It.

Some people are suddenly discovering that GitHub Actions can lead to vendor lock-in.

That’s… not new.

GitHub Actions is a GitHub App deeply embedded in the GitHub ecosystem. YAML workflows, permissions, APIs, events. The lock-in was the trade-off for convenience, reliability, and speed of adoption.

And let’s be honest: most teams are perfectly happy with vendor lock-in (right up until pricing becomes visible). You can’t have a deeply integrated platform and complain when the platform prices itself like one.

The Real Problem: CI Cost Arbitrage

The real pain isn’t for users. It’s for companies whose business model is essentially:

“We’ll run GitHub Actions cheaper than GitHub.”

That model was always fragile. If you are:

buying cloud compute from AWS, GCP, or another provider
reselling CI minutes
competing on price against Microsoft + Azure

You are not competing on technology. You are competing on arbitrage.

And arbitrage disappears the moment the platform owner decides to price closer to cost, or decides they don’t want that game played anymore.

This is not new. This is how platforms work.

When Self-Hosting Still Makes Sense

Self-hosted runners absolutely still make sense when:

you are very large
you have predictable workloads
you already operate infra at scale
growth is slow, and margin optimization matters more than velocity

In other words, when infrastructure is your business or a stable internal cost.

But for growing startups, optimizing CI costs too early is usually a mistake. Time spent shaving a few cents off a CI minute is time not spent shipping product.

(And yes: this is precisely why engineers should not be the sole decision-makers on infra strategy.)

What GitHub’s Pause Actually Signals

GitHub’s follow-up message is important:

They acknowledged real platform costs
They admitted poor communication
They paused to listen, not to abandon the direction

This is not GitHub “giving up.” It’s GitHub realizing that CI/CD has become critical infrastructure, and changes must be introduced with more ecosystem buy-in.

Hosted runners still get cheaper. Actions is still being positioned as a core execution layer (including for agentic workloads). The platform direction hasn’t changed.

Only the timeline has.

The Takeaway

GitHub Actions isn’t “turning evil.” It’s finishing its transition from a feature to a platform.

If your CI strategy depends on GitHub never charging for orchestration, scheduling, and reliability, that was never a safe assumption.

And if your business depends on undercutting a hyperscaler on compute, you were always racing the clock.

For everyone else, this remains mostly good news:

clearer economics
cheaper hosted runners
a stronger, more explicit platform contract

And a reminder that CI/CD is not just about cost. It's about leverage — whether that's merge queues that keep your main branch green or cheaper runners that let you ship faster.

Why Engineers Shouldn’t Decide Your Cloud Strategy

Tue, 17 Jun 2025 00:00:00 GMT

Every few months, a new wave of engineers proudly announces their exit from the cloud. “We’re going bare metal. Look at our savings!”.

The thread goes viral. Everyone nods wisely.

But here’s the truth: if you’re making infra decisions without thinking about your growth model, you’re optimizing the wrong thing.

The cloud is not a cost problem. It’s a scaling solution.

Startups don’t pay AWS bills because it’s cheap. They pay because it gives them instant access to global infrastructure they couldn’t build or operate themselves, and arguable, they should not spend time building a team to operate it.

If your business is growing 100% year over year, optimizing gross margin is not your first battle. Surviving growth is.

Datadog has been in the cloud since day one. They’ve scaled revenue 150× over 10 years. The cloud didn’t kill them. It enabled them. They did that while controlling and optimizing their gross margin, but also without spinning up a giant project to double them by leaving the cloud (yet). Why? Because they’re still (a little bit more slowly) growing.

Bare metal works — if you’re not growing much.

Basecamp left AWS. They made noise. But they also “only” grew 6**×** in 12 years — not 150**×**. When growth is slow and predictable, you can (and should) optimize for margin. You have time. You have predictability. Maybe you even have ops engineers with spare cycles. And if you don’t, as you’re not struggling to grow your team, you can expand into infrastructure and internalize it.

When you run out of stamina for growth, you optimize your gross margin; therefore, your cost is what you want to shrink. It’s a different phase.

The same goes for any small or internal project; there might be no need to deal with a cloud provider if you know your infrastructure will not double every year. Just rent or buy a bunch of bare metal servers and deal with them.

Most engineers don’t see the whole picture.

If you follow engineers, they always want to optimize. The problem is that most of them can’t optimize your market, your growth. The only thing they know how to optimize is resource consumption and cost by working more.

Therefore, they’ll look at a line item on the AWS invoice and say, “We could get this cheaper with low-cost bare metal and our team spending time spinning things up.”

Maybe.

Who’s factoring in the cost of talent to manage infra? The time you won’t spend shipping product? The opportunity cost of slowing down? (This is a classic case of solving build vs. buy.)

So… bare metal or cloud?

It depends.

If you’re building a startup and aiming for fast growth: cloud, 100%.
If you’re a slow-growth company with predictable traffic: maybe bare metal.
If you’re a big org running an intranet or legacy app: buy servers, no big deal.

But let’s stop pretending this is just a technical decision.

It’s not.

It’s a strategic one.

Why We Left Heroku

Tue, 28 Jan 2025 00:00:00 GMT

In January 2023, everything was smooth sailing for Mergify. Our infrastructure was humming along on Heroku, a platform we had trusted for over three years. Heroku was once the go-to choice for startups—simple, reliable, and developer-friendly. We were happy customers, growing steadily and paying our invoices month-to-month.

Then things started to change.

The Start of a Rocky Relationship

In early 2023, Heroku reached out with an enticing offer: transition from month-to-month billing to an annual Heroku Enterprise contract. The deal included significant discounts on everything—dynos (containers), databases, and add-ons—in exchange for a one-year commitment to a certain number of resources.

We were told we’d be allowed to overuse our resources up to 30% during the year without being bothered—with the understanding that if we grew beyond that, the contract would be adjusted fairly in the next cycle.

It sounded like a win-win.

We signed the contract and carried on. For the first year, everything was fine. By the end of 2023, we had indeed surpassed the 30% growth threshold, but Heroku didn’t reach out. The contract auto-renewed, and we moved into 2024 with no issues.

That was until the automated emails started arriving.

A Series of Surprises

In May 2024, we received an automated email from Heroku. It informed us that the discounts on our containers were being rescinded, effective immediately. Naturally, we contacted Heroku’s support team to understand how that would affect our current contract and were redirected to a new account executive to clarify.

Their explanation was straightforward: we had doubled our usage, and they wanted us to pay the difference for the current contract term—for the next 9 months.

While this was unexpected, we decided to comply. We signed an amendment to the contract and paid the outstanding amount. We chalked it up to a policy change, and, as Heroku has been fair so far, we decided to move on.

But then, in October 2024, another email arrived. This time, Heroku announced that discounts on add-ons, such as PostgreSQL databases and Redis, would also be removed. Once again, we reached out to their team for clarification.

This conversation, however, was very different.

The Heroku automated email we received

When Contracts Don’t Matter

Our current account executive explained that the discounted add-ons we had purchased as part of our original enterprise agreement were no longer “fair” for Heroku.

Indeed, two years before, our previous account executive offered us a 60% discount on the listed price, which was a power move to make us commit for a whole year to the platform. A practice that worked: we committed to Heroku, and the account executive won a “top deal France SMB” award at Heroku.

But now, Heroku wanted us to pay the full price for these services, even though our contract explicitly stated otherwise.

We reminded them of the terms we agreed to in the contract, but their response was, essentially, “it’s not fair for us anymore.”

I spent a lot of time trying to understand how getting a few thousand euros more from a loyal startup would impact Salesforce P&L, or how bullying us into paying money we didn’t owe would help our account executive gain respect from their boss, with no luck.

Despite their efforts to pressure us to pay more, we held firm. A contract is a contract, and we weren’t going to be oppressed into paying for something that wasn’t part of the original agreement.

However, at this point, it was clear that Heroku was no longer a reliable partner for us. Their lack of stability, constant policy changes, and disregard for contractual terms made it impossible to trust them with our infrastructure.

The Move Away from Heroku

By late 2024, we made the decision to move Mergify’s infrastructure to Google Cloud Platform (GCP). Migrating a live product is never easy, but it was the right choice. Heroku, once the pioneer of developer-friendly hosting, had stagnated. The platform’s lack of innovation, combined with its increasingly unpredictable business practices, made it clear that it was time to leave.

GCP offered the flexibility, scalability, and reliability we needed to grow. The migration was a success, and while it wasn't a move we originally planned for, it's one we're glad we made. Google helped us a lot in moving to their platform, which made the whole process smooth — and it gave us a chance to rethink our entire CI/CD stack, including how we handle merge queue challenges.

Reflections on Heroku

Despite the rocky ending, it’s important to acknowledge Heroku’s role in our journey. The platform played a significant part in our early success, providing the simplicity and ease of use that helped us focus on building our product. For small apps and early-stage startups, Heroku can still be a good choice.

But over time, Heroku failed to evolve. As the tech industry moved forward, Heroku seemed to stand still. Features stagnated, the platform became less relevant, and dealing with them as a customer grew increasingly frustrating. In 2025, it’s hard to recommend Heroku as a reliable choice for scaling companies.

Advice for Other Startups

Our experience with Heroku taught us some valuable lessons:

Beware of Contracts with Large Companies: Big corporations can change their terms, policies, or priorities on a whim. Make sure you fully understand the risks before signing long-term agreements.
Stand Your Ground: If a vendor tries to pressure you into unfair terms, don’t be afraid to push back. Contracts exist for a reason. Be ready to jump and save your ass.
Choose Platforms That Grow with You: Heroku was perfect for us in the beginning, but as our needs grew, it became clear that we needed a more robust and innovative platform.

For startups navigating similar challenges, remember that your infrastructure choices are critical. Hosting platforms should be partners in your growth, not obstacles.

Final Thoughts

Leaving Heroku wasn’t an easy decision, but it was the right one for Mergify. We’ve learned a lot from this experience, and we’re excited about what’s ahead with our new infrastructure.

If you’re a startup considering Heroku—or debating whether to stay or move on—ask yourself this: is your hosting platform helping you scale or holding you back? At the end of the day, it’s all about finding a partner you can trust to grow with you.

There's (almost) no GitLab

Tue, 29 Oct 2024 00:00:00 GMT

Do you guys support GitLab? Is there any way this can work with GitLab? Does this support merge requests?

No, we don’t.

As I spent hours screening software engineer candidates those last weeks, I repeatedly answered the same question: where’s Mergify support for GitLab?

To put things into perspective, we mostly hire in the French market, and I think it deserves some context.

Culture Bias

I’ve known software engineers for more than 20 years in the country of the baguette, and something is pretty clear. We have amazing engineers, but they suck at understanding what ROI means. Most people have no conception of the value of time, and for most average French engineers, it’s OK to spend time on anything as long as it avoids spending money (or requesting a budget).

It’s not even a frugality thing; it really is just the inability to compute a basic return on investment and put a price on an hour of work.

In the context of software forges, that means something: French companies, from startups to scaleups, are heavily biased towards deploying GitLab Community Edition because it’s free. They would do this over a cheap hosting bare metal server. You can find such hosting for around $20/month.

At this price, the average French software engineer will not buy a GitHub license. They’d call that a rip-off.

I’ve heard it.

Fire at OVH datacenter in Strasbourg, March 2021

If you don’t factor in the time it takes to spin up and maintain the GitLab instance or the impact of having your server on fire, then, indeed, a price of $20/month is unbeatable.

I remember asking one young French engineer in a startup about their GitLab instance and how they would maintain it and manage its security compliance. The answer was straightforward:

We just run `apt upgrade` every night so we’re sure we’ve every security deployment installed.

YMMV.

The Market Share

In April 2024, the Mergify team spent a few days at Devoxx France, the country's largest developer conference with nearly 5,000 attendees. We talked to dozens of engineers, and roughly 50% of them were using GitLab at work. Some large teams were moving away from GitLab to GitHub, but for a large majority, we were weirdos for not supporting GitLab. Their view of the market share is biased toward the French market, where GitLab might indeed have a large usage, which might not tend to generate large revenue, though. Remember that the Community Edition of GitLab is free.

Mergify’s team at Devoxx France 2024

If we compare GitHub’s $2 billion revenue to GitLab’s $579 million revenue for 2024, this is a 1:4 ratio, which is already pretty huge. Sure, revenue is not usage, but considering that GitHub has Microsoft behind it and GitLab is reportedly looking for a buyer, the future looks way brighter for GitHub — something I explored in more depth in Is GitHub the Future?. And I'm not even talking about the fact that the vast majority of open-source projects use GitHub.

No Gauthier, I don’t think that this is how it works.

(LinkedIn post)

Innovation

Now that I’ve set the scenery, I feel it’s safe to answer about GitLab support for Mergify. The main reasons why Mergify’s Merge Queue is not looking for GitLab support anytime soon are:

Innovation happens on GitHub nowadays. Ten years ago, GitHub was behind on certain topics (hello CI), but they are now way ahead of the competition. GitHub is the place where most open-source is built and where you need to be if you’re building new products;
Most teams using GitLab CE have no intention to buy any software;
The market share of GitLab is small and probably shrinking.

I’ve nothing against GitLab, and their software might be pretty good. All we know is that they’re outsiders, while a lot of the tech market in Europe seems to think that betting on GitLab is the best go-to-market strategy.

I beg to differ.

What's going on with Dependabot?

Tue, 15 Oct 2024 00:00:00 GMT

I loved Dependabot. I’ve used it since Grey Baker started it in 2017. I’ve seen it grow from a one-person shop to being acquired by GitHub in 2019. It’s been a fantastic tool that created more than 5,000 pull requests on Mergify repositories. I remember the excitement of finally having a tool that would bring all the new fancy features and bug fixes of my dependencies to my project in a snap.

Dependabot allowed us to fix a lot of security updates introduced by dependencies, and to be aware of anything new being released in the libraries we use.

But today, we kicked Dependabot out. Dependabot let us down.

Security You Said?

The move to acquire Dependabot from GitHub was smart. It was one of the key features that started their security roadmap with the acquisition of Semmle and their CodeQL engine the same year. GitHub has become all about security over the last few years, which makes sense considering the cybersecurity segment's hyper-growth over the last and upcoming years.

However, as Dependabot matured under GitHub, cracks started to show in what was once a flawless experience. The tool that was supposed to streamline security and updates became a source of frustration.

It has major design flaws that GitHub does not seem to care about.

First, Dependabot can fail silently. That happened to us multiple times a year when Dependabot would just stop working and create pull requests to update our packages. You’d think that debugging such an issue would be possible by going into the Dependabot tab of your repository, but no. The log for this is actually hidden in Insight → Dependency graph → Dependabot. A strange and unintuitive location for such crucial information.

Once you find your log, you can then read it and debug it yourself.

That’s a major problem because there’s nothing warning you that Dependabot is broken. We are used to updating our packages regularly, so we’d know, but there’s nothing preventing your dependencies and security updates from getting stale for months without you noticing. Terrible experience.

Always Lagging Behind

We’re a Python shop. We leverage poetry to manage our dependencies, and we use the latest Python version in our containers.

As a Python shop, staying on the latest version helps us ensure security, performance, and compatibility. So we update it as soon as we can, usually a few days after it’s released.

And then Dependabot is broken.

And you have to wait weeks for GitHub to fix the problem.

The last few times, we had to update ourselves Dependabot itself, as shown here or even here. We’re basically doing GitHub’s job for free, maintaining the Dependabot database ourselves for all their customers.

We contacted GitHub support about this already, and they did not care at all. Their laconic answer was, “Wait for it to be updated.”

Sure, thank you. We’re the ones doing the updates.

I get it—maybe Fortune 500 companies don’t care about the latest Python micro releases. But for startups like ours? It’s a big deal.

So today, we got rid of Dependabot and replaced it with Renovate. It seems better maintained and supports a larger package ecosystem than Dependabot. So far, it has simplified our workflow and is not broken on a simple Python micro update. 🤞

We're also adding support for Renovate in Mergify Merge Protections, as we have done for Dependabot in the past. That will ensure you can write advanced rules for automating your GitHub workflows, including automatically merging your dependency update. 🦾 Let me know if you’re interested in trying it out!

Is GitHub the Future or Becoming Obsolete?

Tue, 17 Sep 2024 00:00:00 GMT

Over the last few months, I stumbled upon a few articles on GitHub's history and future. I find those quite interesting. Greg Foster gives a quick history of the rise of GitHub over the years, while Scott Chacon, one of GitHub’s co-founder, retraces the history of GitHub from the inside.

The story is great, and I’ll leave it to you to read those posts if you want to learn more. I was alive at that time; I saw and lived it all. I started using Git in 2005, and I’m the 2644th of the 100 million GitHub users—I joined GitHub in 2008 when it was still in beta.

It’s definitely true that Git won, and especially GitHub. I’ll probably have to write about GitLab at some point (I eventually did) — but you won’t read any ranting here today. As Scott Chacon writes, GitHub had a taste and perfect timing and soon gained traction from the open-source community.

Ok, great, so GitHub wins. Where do we go now?

Microsoft

I believe that GitHub's growth was already on track before Microsoft bought it in 2018, but that move still changed everything. At that time, GitHub still lacked major features, such as a CI/CD system, and had a hard time being compared to GitLab. The following year that changed, and GitHub Actions started (thanks, Azure) and shook everything up.

If you read opinions about GitHub vs. GitLab, CodeCommit, Azure DevOps, or any other platform, you’ll mostly see engineers comparing user-visible features. And sure, this has value, and that might be your main criteria to pick one or the other if you’re a small team with full power over their choice.

However, this is not what GitHub and Microsoft are after anymore. Take a look at the roadmap of the last few years, and you’ll see a pattern: enterprise. Pushing code, creating pull requests, and any part of software engineers' day-to-day activities have been covered for the last 15 years.

They designed it, the industry adopted it, and GitHub has nothing in its roadmap to change that paradigm. They’re building on the momentum they raised.

Security, Copilot (AI), and compliance are items that the largest corporation needs to embrace a platform such as GitHub. This is only the beginning: this year, the GitHub sales organization underwent a reorganization to look more like Microsoft's sales organization. I suspect GitHub is now able to leverage even more Microsoft resources to push its platform to large corporations—which definitely makes sense. The link between GitHub and Azure is tightening, both technically and commercially.

For the best and the worst.

Relevance

How is GitHub still relevant if it does not innovate on developer workflow? Is it becoming legacy software?

There is certainly a disjunction between what developers and corporations expect, and at this point, if you look at the ratio between features, compliance, and price, there's no better alternative than GitHub. I don’t think this is going to change anytime soon.

For open-source projects, there might be alternatives, but if you’re pragmatic (and lazy), GitHub is the pick. There have been a number of projects trying to escape GitHub over the years, such as when Microsoft acquired it. The latter is still seen as an enemy of open source by some folks (I suspect this is PTSD caused by the Balmer era). More recently, another exodus was triggered by the announcement of Copilot and the fear that the training was done on free software. However, at this stage, it’s like emptying an ocean with a spoon, and the impact does not compensate for larger projects moving to GitHub (e.g., Python).

Steve Ballmer

On the other hand, GitHub is attracting more competitors. It has grown to a point where many startups want to disrupt it: one can look at Radicle and its decentralized approach, Pierre and its modern design, Diversion with its game-centric approach, or Palmier, who’s building a new kind of repository.

They might succeed, but the road is going to be long to get massive adoption — and migration.

There’s nothing replacing GitHub in the short term. We better deal with it.

Reflecting on the Journey of "Nom d'un Pipeline !"

Tue, 30 Jul 2024 00:00:00 GMT

"Nom d'un Pipeline !" (translation for English readers: "What a Pipeline!") has been an incredible journey, filled with learning, growth, and connecting with some of the brightest minds in the French tech scene. As a French-based startup for over five years, with most of our customers and audience in the US, we noticed a significant gap in the knowledge and maturity around CI/CD among French engineering teams. This realization motivated us to create a platform to elevate the understanding and practices of CI/CD in our home country, leading to the birth of "Nom d'un Pipeline !"

The Birth of an Idea

Starting a podcast was a new adventure for me. With the help of Mergify’s marketing team, we built everything from the ground up and began reaching out to potential guests. We aimed to find the right candidates and teams to feature in our 45-minute episodes, sharing their journeys and insights around continuous integration, continuous deployment, and quality assurance.

Highlights from Season 1

All episodes were amazing, and the content we built was really valuable. I think several episodes stood out for their depth and impact:

Mathieu Leroux-Huet discussed performance, offering valuable insights into optimizing systems for better efficiency.
Cyril Rohr from RunsOn shared how improving GitHub Actions runner can significantly enhance CI workflows speed and cost.
Olivier Pillaud-Tirard from ManoMano detailed how they have improved their CI over the last couple of years, providing a roadmap for other teams to follow.

These episodes were not only informative but also showcased the diverse approaches and innovations happening within the French tech community.

Overcoming Challenges

Producing a podcast comes with its own set of challenges. One of the biggest hurdles we faced was recording remotely. There were times when technical issues disrupted the recordings, but we managed to overcome these obstacles with perseverance and technical adjustments. While I would love to record in a studio, living in Toulouse makes it challenging since none of my guests are local. Despite these difficulties, the remote setup has allowed us to connect with a broader range of guests.

Positive Feedback and Growing Momentum

The feedback we've received has been overwhelmingly positive. Listeners appreciate the insights and real-world experiences shared by our guests. Knowing that the show provides value and helps the French tech scene advance in CI/CD practices is incredibly rewarding. Recently, we passed the 2,000 views/listens mark, indicating growing momentum after just eight months. This milestone is a testament to the show's impact and the increasing interest in CI/CD topics.

Personal and Professional Growth

On a personal level, hosting "Nom d'un Pipeline !" has been a delightful experience. I discovered that I genuinely enjoy talking to people and learning about their teams and tech stacks. It's been an eye-opening journey that has enriched my understanding of CI/CD and connected me with some brilliant minds in the industry.

Acknowledging Our Guests

I want to extend my heartfelt thanks to all our guests for their confidence and for sharing their journeys: Clément, Sofiyan, Romaric, Aurélien, Frédéric, Olivier, Dan, Thomas, Mathieu, Cyril and François.

Your contributions have been invaluable in making this podcast a success.

Looking Ahead to Season 2

As we wrap up the first season, we are already gearing up for season 2, set to launch in September. We are excited to bring new hosts from major French companies like Doctolib and Alan, promising even more insightful content and engaging discussions. While we don't have specific plans for season 2 yet, we are committed to continuing our mission of educating and inspiring the French tech community about CI/CD.

"Nom d'un Pipeline !" has been an extraordinary journey, and I'm proud of what we've accomplished so far. We hope that this podcast continues to serve as a valuable resource for the French tech scene, helping teams to improve their continuous integration, testing, continuous deployment, and overall development workflows — including tackling tough topics like the challenges of merge queues. Stay tuned for more exciting episodes and insights in the upcoming season!

If you understand French, don’t forget to subscribe to the podcast on your favorite podcast platform or on YouTube!

The Challenges of Merge Queues

Tue, 23 Jul 2024 00:00:00 GMT

Merge queues are a tough concept to grasp, and over the last five years at Mergify, we've spent countless hours educating developers about their importance and utility. We've published numerous blog posts, written extensive documentation, and even gone to conferences to teach software engineers what a merge queue is. This process of spreading awareness has been a rewarding yet challenging endeavor.

One of our developers, Charly Laurent, gave an insightful talk on the subject, highlighting how merge queues can revolutionize CI/CD processes. You can check out his talk here:

Understanding Merge Queues

Merge queues are not an obvious choice for most teams, and they often require a shift in the balance between safety and speed of delivery. Deploying a merge queue means prioritizing quality over quantity, which is not an easy decision for many development teams, who might be pressured to ship fast.

For example, without a merge queue, teams often merge untested code. This is due to outdated test runs, meaning that they are deploying code that might not work. Without a merge queue, there is no way to prevent merging pull requests with outdated tests and breaking the CI for everyone — which is exactly why you should stop merging your pull requests manually in the first place. One of our customers faced this exact issue, which meant they needed the equivalent of a full-time engineer dedicated to tracking issues in the main branch that broke the CI.

Most platform engineers find the concept of moving the post-merge tests to pre-merge tests challenging.

The Trade-offs

This blog post from Vercel captures this common misunderstanding and the trade-off around merge queues and their CI costs and latency:

Despite the majority of commits being safe to merge after the local CI checks complete on their pull request, the merge queue will incur running the cost of running the CI again every time.

While this is true, the problem here lies in the word "majority." The definition of "majority" can vary significantly across teams. If a minority of pull requests break the main branch after merging, it can cause considerable downtime and require substantial effort from CI engineers to restore stability. We've seen teams come to Mergify with a 30% failure rate on their main branch. While a merge queue won't magically improve the failure rate, it ensures that it doesn't worsen, even if it means a small decrease in merge speed. That ensures that the effort invested in improving the CI is not wasted the day after.

Another perspective from Vercel states:

With merge queues, changes from developers depend on changes from other developers even if they are unrelated to each other, and this makes it hard to scale monorepo merge times with more developers.

This concern is valid for merge queues that don't support monorepo and queue parallelization. However, most modern merge queues (GitHub's own being an exception) do allow for optimization in these scenarios.

Vercel’s blog post concludes with:

With this workflow in place, the merge queue can be safely removed because checks will still always be run before users ever see the deployment.

This reflects the workflow of many teams that don't use a merge queue: merge, run tests on main, then deploy. However, this approach doesn't solve the issue of merging something that breaks the main branch. During the downtime, teams have to identify the culprit, revert changes, and ensure everything works, causing delays and frustration. Bad developer experience ensues.

Teams like Uber recognized this problem six years ago and started building their merge queues. Similarly, in OpenStack, we had a system supporting multiple repositories with Zuul over ten years ago.

Build New Solutions

Considering the merge queue adoption issues, we've spent the last few months reworking our merge queue system to simplify deployment and enhance user experience. We know for a fact that developers appreciate the reliability it brings to CI processes, but we also observe the difficulty of discovering and integrating the system. By deploying a merge queue, teams can eliminate the need for a "check that main works before deployment" step because this is done before the actual merge.

One notable example is a team that previously needed a full-time engineer to manage CI issues due to frequent breaks in the main branch. After adopting Mergify's merge queue, they drastically reduced these disruptions, allowing their engineers to focus on more productive tasks.

The Road Ahead

Merge queues are not without their challenges, and the trade-offs between safety and speed are not always apparent. However, we believe in their potential to transform development workflows. We're on the verge of redefining the merge queue concept at Mergify, and we think it has far greater potential than what has been realized over the past decade. I’ll be happy to write about that soon and share what we’ve built.