mergify — jd:/dev/blog

GitHub Actions Pricing: The Platform Reality Check

Thu, 18 Dec 2025 00:00:00 GMT

GitHub just announced pricing changes for GitHub Actions, and as expected, parts of the CI ecosystem panicked.

Some people are celebrating the price drop on GitHub-hosted runners. Others are furious that GitHub will start charging for self-hosted runners. And a few businesses are suddenly asking existential questions.

Since then, GitHub has paused the self-hosted runner billing change, acknowledging they moved too fast and didn’t involve the ecosystem enough.

That doesn’t change the underlying reality. It just delays the conversation.

GitHub Actions Was Never “Free”

GitHub Actions is not just a binary that runs on your machines.

It’s a platform:

job queuing and scheduling
runner registration and lifecycle management
workflow orchestration
security, isolation, secrets handling
reliability at massive scale

Even when you run your own hardware, GitHub is still doing a lot of work on your behalf. That infrastructure has always existed; hosted runners simply subsidized it.

GitHub explicitly said it:

“We have real costs in running the Actions control plane.”

That’s not new. It’s just now being made explicit.

Charging a small per-minute platform fee for self-hosted runners isn’t conceptually unfair: it’s GitHub aligning pricing with reality.

If you believe this is unacceptable, there has always been a clear alternative: run Jenkins, GitLab CI, or any other system where you fully own the control plane.

But you don’t get GitHub Actions “for free” just because the CPU cycles are yours.

Vendor Lock-In? Yes. And Everyone Chose It.

Some people are suddenly discovering that GitHub Actions can lead to vendor lock-in.

That’s… not new.

GitHub Actions is a GitHub App deeply embedded in the GitHub ecosystem. YAML workflows, permissions, APIs, events. The lock-in was the trade-off for convenience, reliability, and speed of adoption.

And let’s be honest: most teams are perfectly happy with vendor lock-in (right up until pricing becomes visible). You can’t have a deeply integrated platform and complain when the platform prices itself like one.

The Real Problem: CI Cost Arbitrage

The real pain isn’t for users. It’s for companies whose business model is essentially:

“We’ll run GitHub Actions cheaper than GitHub.”

That model was always fragile. If you are:

buying cloud compute from AWS, GCP, or another provider
reselling CI minutes
competing on price against Microsoft + Azure

You are not competing on technology. You are competing on arbitrage.

And arbitrage disappears the moment the platform owner decides to price closer to cost, or decides they don’t want that game played anymore.

This is not new. This is how platforms work.

When Self-Hosting Still Makes Sense

Self-hosted runners absolutely still make sense when:

you are very large
you have predictable workloads
you already operate infra at scale
growth is slow, and margin optimization matters more than velocity

In other words, when infrastructure is your business or a stable internal cost.

But for growing startups, optimizing CI costs too early is usually a mistake. Time spent shaving a few cents off a CI minute is time not spent shipping product.

(And yes: this is precisely why engineers should not be the sole decision-makers on infra strategy.)

What GitHub’s Pause Actually Signals

GitHub’s follow-up message is important:

They acknowledged real platform costs
They admitted poor communication
They paused to listen, not to abandon the direction

This is not GitHub “giving up.” It’s GitHub realizing that CI/CD has become critical infrastructure, and changes must be introduced with more ecosystem buy-in.

Hosted runners still get cheaper. Actions is still being positioned as a core execution layer (including for agentic workloads). The platform direction hasn’t changed.

Only the timeline has.

The Takeaway

GitHub Actions isn’t “turning evil.” It’s finishing its transition from a feature to a platform.

If your CI strategy depends on GitHub never charging for orchestration, scheduling, and reliability, that was never a safe assumption.

And if your business depends on undercutting a hyperscaler on compute, you were always racing the clock.

For everyone else, this remains mostly good news:

clearer economics
cheaper hosted runners
a stronger, more explicit platform contract

And a reminder that CI/CD is not just about cost. It's about leverage — whether that's merge queues that keep your main branch green or cheaper runners that let you ship faster.

The Day I Got Custom Table Legs

Tue, 22 Jul 2025 00:00:00 GMT

Last week, I was with my team at our Mergify on-site — what we call our MAHOS (Mergify All-Hands On-Site). Yes, we’re a fully remote team, so your regular off-site are called on-site for us. 😉

We talked about the usual: roadmap, strategy, alignment. But then I brought up something a little different. I wanted to explain how we think about customer support at Mergify—not as a checkbox, not as a cost center, but as a way to deliver what I call the wow effect.

To make my point, I told them a story. A true one — about a table.

Two years ago, I had just moved into a new house. My first real garden. I was excited to enjoy it, so I decided to buy a garden table and chairs. After browsing around, I picked Lafuma — a French brand I’ve liked since I was a kid.

Fun fact: my very first school backpack in first grade was a Lafuma. Yes, I still remember it.

Anyway, the table and chairs arrived. Great delivery. Good packaging. Quality seemed top-notch.

But something felt… off.

I sat down, and it wasn’t right. The proportions felt weird. The table was just a little too high, or the chairs too low. So I did what every curious engineer does: I grabbed a tape measure. Compared it to my indoor table. And there it was — the Lafuma table was exactly 2cm too tall. Just enough to make every meal feel slightly awkward.

So I wrote them a message. Not angry, not demanding. Just a note saying:

“Love the brand, love the product, but this feels like a design oversight.”

I didn’t expect a reply.

One week later, I got a call. It was someone from Lafuma’s support team — a QA engineer.

He said:

“I read your message. I’d like to understand exactly what’s wrong.”

I explained. He listened. Then, without hesitation, he said:

“Okay. I’ll send you custom table legs, 2cm shorter. You’ll have them next week.”

I was stunned.

“Wait, really? You can do that?”

“Of course,” he replied. “We have spare legs in the workshop. We’ll just trim and ship them to your size.”

And that’s exactly what happened. A week later, I swapped out the legs. Perfect fit. Perfect height. Perfect support.

They didn’t have to do that. I wasn’t going to return the table. I wasn’t even asking for anything. But they did it anyway — because they cared. Because they listened. Because they understood what great support means.

New leg size approved by my wife.

That’s the kind of service we try to deliver at Mergify.

Even in B2B, even in software, even at scale — you can still surprise people. (Why we still care about quality is about the same mindset.) You can still make them feel heard. You can still wow them. That’s what Amazon did so well for years. And it’s what so many companies forget as they grow.

But it’s not optional. It’s the difference between a satisfied user and a loyal one. Between a customer and a fan.

Build the table. And send the legs.

Why We Still Care About Quality

Tue, 24 Jun 2025 00:00:00 GMT

I recently read Linear’s excellent blog post on why quality is so rare, and it resonated deeply with me. Craft, quality, care — these aren’t buzzwords. They’re a way of working, a way of thinking, and frankly, the only way I’ve ever known how to build things.

Linear: Why is quality so rare?

For me, it started with open source. When you put your code out in the open, you naturally want to make it good. Maybe even beautiful. I started more than 20 years ago, polishing my Debian packages, making sure they were clean, understandable, and useful. Later I poured that same mindset into building awesomewm, striving to write the best C code I could — because that code was me, visible to anyone curious enough to look.

Open source taught me that quality is not an accident. It’s a habit. And a commitment.

Even though Mergify is no longer open source, the ethos never left. We still build like our code is going to be read by thousands, because, well, it is at least read by our folks. Our team ships work we’re proud of. Whether we win a deal or not, it’s common to hear people tell us:

The quality of Mergify stands out.

That never gets old.

I know I’m not alone in this. Mehdi, my cofounder, and I have been building together for over 15 years. It’s in our DNA: we hate mediocrity. We won’t ship something that we wouldn’t use ourselves — joyfully.

That said, I’ve also seen the flip side. Back when I worked on OpenStack, a massive open-source project, there was a lot of code… and not always a lot of care. Many contributors came from companies that didn’t value quality — and it showed. Open source can be beautiful, but when it’s driven by quantity instead of pride, it becomes exhausting. I hated that part.

Quality isn’t just aesthetic. It’s a business strategy. Linear nailed that in their post. When you build something that feels right — fast, polished, thoughtful — users notice. They stay. They tell others. We’ve seen this at Mergify: our growth has been fueled not just by features but by how those features feel to use.

But quality is more than just a great UI or bug-free code.

It’s also:

A fast, reliable, intuitive product.
Clean code that enables long-term agility.
Thoughtful defaults and edge-case handling.
Being able to say “no” when something adds complexity without enough payoff.

Getting there isn’t easy. You need judgment — to know what’s worth doing and what can wait. That comes with experience and the humility to know you’ll never get everything right. We aim for 80/20, not 100/0. Sometimes that means leaving the last 20% for another day — or maybe never. Not because we don’t care, but because we care about the whole system staying healthy and fast.

Quality isn’t free. But it pays back. In speed, trust, and joy.

So yes, it’s a choice. One you make every day.

You can take the shortcut, or you can make something that lasts.

We still choose the latter.

From Failure to Focus

Tue, 22 Apr 2025 00:00:00 GMT

Startups love to talk about iteration. Failing fast. Learning from mistakes. But when you’re six months into a project that doesn’t ship — not once, but twice — that mantra starts to feel a bit too real.

At Mergify, we recently spent almost a year building two separate products around CI/CD. Both had potential. Both looked promising. And both ended up in the graveyard.

I have written about this in three posts already: The $100,000 Mistake, When Nobody Wants Your Product, and When Great Tech Isn't Enough.

But here’s the thing: that journey was the best thing that could’ve happened to us because it led to CI Insights, our newest product, which we’re shipping for real this time.

When R&D Goes Off-Road

If you’ve followed the first parts of this series, you know the backstory. In 2023, after years of growing Mergify’s Merge Queue product, we started exploring new ideas.

The first was CI Optimizer — a tool to help teams reduce CI/CD costs. But we quickly learned something important: engineers aren’t the ones who care about CI spend. That’s a FinOps conversation. And FinOps teams weren’t our users.

Then came CI Issues, a project aimed at tracking flaky tests and infrastructure problems. This time we had interest. Teams did struggle with these problems. But we made the mistake of diving into R&D without doing proper design work. We built a complex system — and it worked — but it was so hard to deploy and operate that we never felt confident letting users in.

No design, you said?

So once again, we shelved it.

Two product attempts, zero releases.

But what survived both efforts was a deeper understanding of the pain engineers feel every day around CI.

The Real Problem: CI Is a Black Box

Across all our conversations, one theme kept showing up: visibility.

Teams weren’t desperate to reduce CI costs. They weren’t obsessed with infrastructure failures. But they were all asking the same questions:

Why is our CI pipeline so slow?
Which PRs are consistently the bottleneck?
Which tests are flaky?
Where are we wasting time?

Nobody had good answers. CI is treated like a utility — flip the switch and hope the light turns on. But when it doesn’t, or when it flickers, most teams don’t have the tools to understand why.

We realized what was missing wasn’t a FinOps tool or a smart test tracker — it was observability.

CI Insights: The Missing Layer

With all the groundwork we’d laid — our CI connectors, our data pipelines, our internal dashboards — we already had most of the pieces. We just needed to reframe the problem.

So we did.

CI Insights is the observability layer for your CI.

It helps teams:

Spot flaky jobs and tests
Identify long-running or unstable jobs
Understand where their pipeline is slowing them down
Track trends over time across teams and repos

It’s not about cost savings. It’s not about blaming the CI tool. It’s about clarity — understanding what’s going on, so teams can ship faster and with less frustration.

This Time, We Built It Right

We learned from our previous mistakes.

This time, we:

Started with real use cases from customers and our own internal needs
Designed first, coded second
Focused on value over complexity
Shipped it to users. Yes. Really.

We’ve been using CI Insights ourselves for months now. It already helped us catch flaky jobs, detect broken test workflows, and reduce merge queue delays.

Now, we’re rolling it out to early users — and so far, the feedback has been 🔥.

The Bigger Picture

CI Insights is more than just a tool. It’s a shift in how we think about CI.

It’s not just a thing that “runs your tests.” It’s a critical part of your development workflow. And it deserves the same kind of visibility, metrics, and tooling that you already have for production systems.

We’re building CI Insights to be the best observability tool for CI — because engineers deserve better tools.

What’s Next?

We’re just getting started. The roadmap is full. We’re onboarding users slowly and shaping the product based on real feedback.

If CI is a black box for your team — if you’re tired of guessing why things are slow — we’d love to hear from you.

👉 Request early access

Not Just a Job, It’s a Ride

Tue, 15 Apr 2025 00:00:00 GMT

Hiring is easily one of the hardest jobs I’ve had to do at Mergify. Not because there aren’t smart people out there — there are. But because we’re not just hiring for skill. We’re hiring for mindset.

And let’s be honest: that’s a lot harder to screen for than technical chops.

When you’re building a startup, every new hire changes the shape of the team. Every person matters. You’re not adding a cog to a big machine — you’re inviting someone on the ride with you, and they’d better be ready for the speed bumps.

What We Actually Look For

At Mergify, we look for people who care. Not just about clean code or nice UI — but about the mission. The thing we’re building. The problems we’re solving. If what we’re doing doesn’t excite you, we’re not going to try to sell it to you. We want you to come in already leaning forward.

We’re looking for people who aren’t title-driven but outcome-driven. People who will get things done even when no ticket has been assigned or clear ownership has been defined yet. That happens a lot.

You need to bring ideas, not just execute someone else’s. We expect initiative, curiosity, and autonomy.

We provide a good summary of our vision and mission on our website.

Autonomy Isn’t a Buzzword, It’s the Filter

Here’s a small story that stuck with me.

We had a candidate once who asked me during the interview, “Who’s in charge of breaking down the project into tickets?”

I said, “You are.”

That was enough to scare them off — and that’s OK. At Mergify, you’ll be expected to do exactly that. Define your work, structure your plan, ask questions when you need to — but no one’s going to hand you a Jira board and a user story spec for every task.

If that makes you nervous, we might not be the right place. If it makes you excited? You should talk to us.

Startups Are Messy. That’s the Point.

You can’t build a startup with neat little boxes around every role.

One week, you might be writing code. The next, you’re giving a talk at a meetup. The week after, you’re helping a customer figure out something weird in their CI pipeline.

We look for people who can jump between lanes without crashing the car. If you need clear job boundaries, startup life will drive you insane. But if you like wearing multiple hats (sometimes in one day), you’ll thrive here.

Wearing multiple hats.

Hiring Mistakes: Yep, We Made Some

One of our early missteps? Underestimating how hard remote work can be — for some people.

We’re fully remote, and we love it. But not everyone is cut out for it. We’ve had brilliant engineers who struggled because they needed more structure, more hand-holding, and more real-time sync. (I wrote about this in Remote Work: Great, But Not Perfect.) And we’ve learned the hard way that great resumes don’t always mean great remote workers.

Now, we screen harder for that. Autonomy, again, is a big part of the puzzle.

Our Hiring Process (And Why We Meet in Person)

Our process is pretty standard on the surface:

👉 A technical test

👉 A technical interview

👉 A CEO chat

👉 An onsite interview

👉 Reference checks

We’ve documented the whole thing on our website, so there are no surprises.

But one thing we insist on: we meet you in person.

Remote or not, hiring is a human process. A real-life conversation can reveal things that a dozen Zoom calls won’t. We’ve avoided a few bad hires because we took the time to meet face-to-face.

Motivation Over Résumé

The biggest signal we look for? Why you want to join.

We try not to overexplain this part publicly, because it’s one of our most effective filters. But let’s just say: if your reason for leaving your current job is “looking for a remote job,” we’ll probably pass.

We want people who are actively choosing this kind of work, people who are ready for the ambiguity, the responsibility, and, yes, the chaos.

Hiring for Roles You Don’t Know? Brutal.

When we hired our first designer, it took us way longer than it should have. Why? Because we didn’t know how to assess the role.

If you’ve never done the job yourself, hiring for it is like ordering dinner in a language you don’t speak. You might get lucky, but you probably won’t.

We’ve learned to spend more time understanding what we actually need before we go looking for someone to do it. Sounds obvious. It’s not.

What’s Hard Now?

Right now, our biggest challenge is finding candidates who combine technical skill with startup DNA.

We’re looking for people who’ve spent a few years in early-stage companies, who know what it’s like to ship fast, wear multiple hats, and stay sane through ambiguity. Not just smart — adaptable.

If That’s You…

Then maybe we should talk. If you’re looking for more than just a job — if you want to build something, shape it, and take pride in it — you might belong here.

We’re picky, yeah. It slows us down. But we’ve learned the cost of the wrong hire is much higher than waiting for the right one.

So if you’re ready for the ride, we’re hiring.

When Great Tech Isn’t Enough

Wed, 26 Mar 2025 00:00:00 GMT

Welcome to the third post in our “we-built-something-and-killed-it” series.

In the first chapter, we shared how we started building CI Optimizer—our ambitious attempt to help teams cut down on CI/CD costs.

In the second, we explained why that effort never made it past the runway: while the problem existed, no one really wanted the solution.

But that wasn’t the end of the story.

Because just as we were winding down CI Optimizer, something else started to take shape—almost accidentally.

From CI Cost to CI Chaos

As we were working on CI Optimizer, we had to dig deeper into CI platforms like GitHub Actions or CircleCI. We needed to understand the structure, failures, and performance of CI pipelines to measure their cost.

And the more we explored, the more something else stood out: teams weren’t just struggling with CI/CD costs—they were struggling with CI reliability.

❗Flaky tests.

❗Unreliable runners.

❗Timeouts.

❗Random infra failures.

And as users of our Merge Queue product kept telling us:

“Our workflow is fine—until CI starts acting up.”

So we asked ourselves a new question:

💡 What if we stopped focusing on how much CI costs, and started looking at how much CI hurts?

That’s how the idea for our next product—CI Issues—was born.

The Pivot: CI Issues

CI Issues was meant to do one thing really well:

Track, identify, and alert on CI problems before they silently torpedo developer productivity.

We wanted to give teams insight and visibility into:

How often their tests flaked
Whether their CI infrastructure was unreliable
Which PRs were impacted
Which workflows deserved attention

The goal wasn’t just dashboards. It was detection and action. You’d be able to see patterns, set alerts, and flag recurring issues before developers noticed them.

And as we started to pitch the concept to engineers, the excitement was real:

💬 “We have this exact pain.”

💬 “We’ve built half of this internally.”

💬 “Please let us know when it’s ready.”

We felt like we were onto something.

The R&D Rabbit Hole

So we jumped in headfirst. We already had code collecting and analyzing CI data, so we started adapting it for CI Issues.

We ran the system internally, refined metrics, tested detection logic, built a first UI. And then we iterated. And iterated. And iterated again.

But something was off.

Every time we looked at what we had, the same thought came back:

“This is good… but it’s not a product.”

It was barely working for us internally. Even we had trouble using it.

It was noisy. It was complex. It was fragile. It wasn’t obvious how to deploy or operate it at scale.

We had built tech.

But we hadn’t designed a product.

The Realization That Stopped Us

After almost a year of work, we paused and took a step back. And it hit us:

We had made the same mistake again—but in a different way.

With CI Optimizer, we had no market.

With CI Issues, we had no design.

This time, it wasn’t the problem that was flawed—it was our approach.

We had focused on research, experimentation, pipelines, metrics, code—but we hadn’t put the same energy into figuring out how the product should be used.

How would teams onboard?

How would they configure it?

How would they act on the data?

What does success look like for them?

The longer we waited to answer those questions, the more we realized:

💣 “If we ship this now, we’ll be building another tool that’s hard to use, hard to maintain, and ultimately, unadopted.”

So we made the call—again.

We stopped.

What We Learned (This Time)

This second failure didn’t sting the same way as the first.

In fact, it felt like a necessary part of the journey.

Here’s what we learned:

Validation isn’t enough—you need design.

Even if users want a solution, they won’t use a product that’s hard to operate or understand.
Great tech doesn’t mean great UX.

CI Issues worked, technically—but without thoughtful design, it was dead in the water.
You need both clarity and empathy.

Clarity on what you’re solving, and empathy for how your users will experience it.

What’s Next?

The story doesn’t end here.

CI Issues gave us a powerful insight into how fragile and painful the CI experience can be—and how underserved engineers still are when things go wrong.

So we took everything we learned from CI Optimizer and CI Issues, and went back to the drawing board—with a new vision, new design principles, and a better understanding of how to build the right thing the right way.

Stay tuned for the final post in the series: what we built next, and how it’s going to change how developers deal with CI failures.

When Nobody Wants Your Product

Tue, 04 Mar 2025 00:00:00 GMT

In the first part of this series, we introduced CI Optimizer, a product we were convinced would help engineering teams reduce their CI/CD costs. Given the economic downturn in 2023, we saw budgets tightening, companies folding, and engineering teams being forced to justify every dollar they spent.

If you missed the first part, read it here.

It seemed like the perfect time to launch a tool that would bring cost visibility and optimization to CI/CD workflows.

We started building immediately, setting up a landing page and a waitlist, and running early customer interviews. Our approach was clear:

Build the product.
Talk to potential users.
Iterate based on feedback.

But as we reached out to customers, one thing became clear:

💡 Nobody really cared about optimizing CI/CD costs.

This was the moment we realized we were building something that might never find an audience.

Trying to Sell the Product Before It Existed

I strongly believe that you should be selling a product before it even exists. If you can’t generate demand when it’s just an idea, chances are you won’t generate demand once it’s built.

So, as we were writing the first lines of code, we also launched:

✅ A marketing campaign to build awareness.

✅ A landing page with a waitlist.

✅ Customer outreach to gauge interest.

Our goal was to validate demand early—before we wasted months building something nobody wanted.

But things didn’t go as expected.

The First Red Flag: Engineers Didn’t Care

As we started talking to users, the first warning sign was that engineers were simply not interested in optimizing their CI/CD costs.

💬 “Sure, spending less money is nice, but it’s not a priority.”

💬 “We’ve never been asked to reduce our CI/CD spend.”

💬 “CI is just a necessary cost of doing business.”

This was surprising. We expected companies to be actively looking for ways to cut costs, but instead, we found:

👉 Engineers weren’t incentivized to optimize costs. Most of them were measured by features delivered and bugs fixed, not by how much they spent on infrastructure.

👉 Budgets were tight, but existing expenses weren’t scrutinized. Many teams were cutting new expenditures, but existing CI/CD costs were just accepted as part of doing business.

👉 It wasn’t an engineering problem—it was a finance problem. Even when engineers acknowledged CI/CD spending was high, they said, “This isn’t my job to fix.”

In short:

🚨 We had built a solution for a problem our audience didn’t care about.

But we weren’t ready to give up yet.

The Second Red Flag: Talking to the Wrong People

Since engineering teams didn’t seem to care, we were often redirected to FinOps teams—the financial teams responsible for tracking cloud spend.

So we thought, “Great! Maybe this is our actual target audience.”

We started talking to FinOps teams, and here’s what we discovered:

💬 “We don’t need another tool—we just need a report in a spreadsheet.”

💬 “Can you just give us an API so we can generate cost breakdowns?”

💬 “We don’t want to ‘optimize’ CI/CD automatically. We just need visibility.”

💬 “If we were to buy your product, we’d need more than reporting. We want automatic cost optimization.”

Here’s where we ran into our second major issue:

🚨 We were not equipped to build a product for FinOps teams.

We understood engineers. We had deep experience with CI/CD workflows.

But we knew nothing about selling to FinOps.

Selling to FinOps teams is a completely different game.

They care about budgets, forecasting, and high-level cost reporting, not about how CI/CD actually works.

Even worse:

❌ The product we had in mind was too technical for FinOps teams.

❌ The version they needed was much more complex and would take a year to build.

❌ We would be competing against massive cloud cost monitoring tools, not other DevOps tools.

At this point, we had two choices:

Keep building a product for engineers who didn’t care.
Completely pivot to a new audience we didn’t understand.

Neither option looked good.

The Hard Decision: Killing the Product

By the six-month mark, we had spent:

🕚 Hundreds of hours building a proof-of-concept.
📞 Countless customer calls trying to validate the idea.
💬 Weeks refining our messaging to see if we could spark interest.

But deep down, we knew the truth:

❌ Engineers wouldn’t pay for cost optimization.
❌ FinOps teams needed something completely different.
❌ There was no clear path forward.

And so, after six months of work, we made the hardest decision a product team can make:

We killed the project.

Instead of pushing forward with a product that had no market, we pivoted to something else—which I’ll reveal in the final part of this series.

Lessons Learned

Even though we ultimately abandoned CI Optimizer, the experience taught us some critical lessons about building new products:

Talk to Customers Before Writing Code

We should have validated demand before starting development. Building first and testing later is a risky approach. Fortunately we mitigated this by talking while building.

Engineers Don’t Always Care About Cost Savings

Developers are focused on shipping code, not cutting costs. If a product doesn’t directly impact their work, they won’t engage with it.

Just Because a Problem Exists Doesn’t Mean It Needs a Product

Companies do spend too much on CI/CD, but that doesn’t mean they’re looking for a tool to fix it. Some problems are simply not painful enough to justify a new product.

Selling to Finance Teams is a Whole Different Game

FinOps teams think differently from engineers. If your product doesn’t fit into their existing finance workflows, they won’t use it.

Know When to Walk Away

One of the hardest skills in startups is knowing when to cut your losses. We could have wasted another 6–12 months building something nobody wanted. Instead, we chose to fail fast and pivot.

What’s Next?

Even though CI Optimizer never launched, it wasn’t a wasted effort.

In fact, the insights we gained from this failure led us to build something even better.

In Part 3, I’ll reveal how we took everything we learned from this failure and pivoted to a product that actually resonated with engineers—and how that decision changed the trajectory of Mergify.

The $100,000 Mistake

Tue, 18 Feb 2025 00:00:00 GMT

A Journey into Startup Reality

Not every startup success story begins with a garage, two co-founders, and an overnight explosion of users. And not every failure is a dramatic, fiery crash. Some of the most valuable lessons happen in the quieter moments—when you’ve built something, spent months refining it, and then realized you were solving the wrong problem all along.

This is the story of CI Optimizer, a product we believed would transform the way companies managed their CI/CD costs. We spent six months designing, building, and testing it—only to ultimately kill it before launch.

Why? Because we made one fundamental mistake.

This three-part series is not just about the technical challenges of optimizing CI/CD or the intricacies of pricing cloud infrastructure. It’s about the hard reality of building a product, talking to customers, and realizing you missed the mark.

If you’re an entrepreneur, a product builder, or just someone fascinated by the messy, unpredictable world of startups, this series is for you.

Let’s start at the beginning.

How It Started

At the end of 2022, Mergify was on a roll.

We had spent the past year growing steadily, tripling our revenue, refining our Merge Queue product, and deepening our place in the DevOps ecosystem. Our customers were engaged, our product-market fit felt strong, and our team fired on all cylinders.

But as the year drew to a close, something in the air felt different.

Conversations with prospects were shifting. Instead of discussing new features and scaling up their usage, they were hesitant. Startups—our core audience—were tightening their budgets. Investors were slowing down. Some of our customers simply disappeared.

First, it was the crypto companies. Then, real estate tech. One by one, they went silent—not because they didn’t love our product, but because their businesses were collapsing. The market was crashing. Funding was drying up.

The message became clear:

“We love Merge Queue, but our budget is frozen.”

We knew we couldn’t just sit back and hope the market would bounce back. We needed to adapt. That’s how startups survive.

And that’s when we had what we thought was a brilliant idea.

The Birth of CI Optimizer

One undeniable truth about software engineering is that CI/CD is expensive.

Every build, every test, every deployment—it all costs money.

At scale, those costs grow exponentially, often without teams fully understanding where their budget is going. Developers push a change, run a full test suite, and move on. But in the background, cloud bills are racking up, and finance teams are left wondering where all that money is going.

So we thought:

“What if we built a tool that gave teams complete visibility into their CI/CD costs? What if we could identify waste, eliminate unnecessary builds, and optimize pipelines automatically?” What if we could help companies save money on their CI without slowing them down?”

It sounded like a no-brainer. We’d build a smart system that could analyze CI/CD usage and recommend cost-saving adjustments—maybe even automate them.

This is something we could even use ourselves to save on our CI/CD bills.

This wasn’t just an idea—we were convinced we had struck gold.

Building the Future of CI/CD Cost Optimization

By early 2023, we had begun prototyping.

The first step? Connecting to GitHub Actions. GitHub, like many CI/CD providers, is well known for not providing a good report on cost analysis (still true as of today). Since GitHub is where all of our customers are, this made sense.

We needed to pull in detailed CI/CD usage data and break down the cost per minute of every build. Our system would scan pipelines, report metrics, identify inefficiencies, and provide actionable insights—like which jobs were wasting the most money.

It felt like a natural extension of what I had worked on years earlier at Datadog, where I had pushed for replacing CPU seconds with dollar values in profiling tools. The goal was simple: make CI/CD costs tangible, trackable, and optimizable.

We saw an opportunity to build something that would fit neatly into engineering workflows. The logic was airtight:

✅ Developers hate waiting for builds.

✅ Developers need to get more budget.

✅ We could solve both problems at once.

Or so we thought.

What We Hoped to Prove

Our plan for early 2023 was straightforward:

1️⃣ Build an MVP that could accurately measure CI/CD costs at a granular level.

2️⃣ Talk to real users and validate whether CI/CD engineers cared about cost optimization.

3️⃣ Launch a first version of CI Optimizer and start onboarding teams.

We expected engineers to tell us:

“This is amazing! We’ve been waiting for something like this!”

But that’s not what happened.

Instead, we hit an unexpected roadblock that completely changed the course of the project. (Read what happened next in When Nobody Wants Your Product.)

SaaS Pricing is Hard

Tue, 04 Feb 2025 00:00:00 GMT

Pricing is one of the hardest things to get right in SaaS. If you’re a startup founder, especially in B2B, you’ve likely wrestled with pricing questions:

💰 How much should I charge?

📊 What pricing model makes sense?

⚖️ How do I ensure fairness while maximizing revenue?

At Mergify, we’ve spent years experimenting, iterating, and learning the hard way. Here’s a breakdown of our journey—and what we’d do differently if we had to start over.

How We Picked Our First Pricing Model

When we first launched Mergify, we had no idea what the right pricing model should be. So, like many startups, we copied GitHub.

We charged per user, based on the size of the entire organization.

This meant that if a company had 200 engineers, they had to pay for all 200 engineers—even if only 20 or 30 of them actually used Mergify.

For small companies (e.g., 20–30 engineers), this wasn’t a big deal. They usually had one team using Mergify across all their repos. But as we grew and larger companies came in, things got tricky.

🛑 Larger companies had multiple teams, and only some teams used Mergify.

💰 They didn’t want to pay for everyone—just for the engineers who actually needed it.

We needed to change.

Counting Users the “Right” Way

To address this, we moved to a new model:

✅ Instead of charging per organization, we charged per collaborator—engineers who had write access to a repository where Mergify was active.

This felt fairer. A company with 100 engineers could now pay only for the repositories where Mergify was used, rather than for the entire org.

At the same time, we doubled our price per user. Why?

Customers were already seeing the value, and price wasn’t their biggest concern.
The new model lowered the user count for most companies, so we had to balance revenue.

The Math: Why This Worked

For a company with 100 engineers:

Old model: $2 per user × 100 users = $200/month
New model: $4 per user × 50 repo contributors = $200/month

For many customers, their bill stayed roughly the same. But they were happier because they felt they were paying for what they actually used.

Fairness is Everything

But the journey didn’t stop there.

Larger organizations often gave write access to all engineers by default, even if only a subset was actually making commits. That meant some companies were being charged for engineers who weren’t actively contributing.

So we introduced another iteration:

✅ Charging per “active user”—engineers who actually made commits.

This approach, inspired by Slack’s active user model, made more sense. Now, companies only paid for users who actively used Mergify.

The Math: Why This Worked (Again)

For a company with 100 engineers, where only 40 engineers actively pushed commits:

Previous model: $4 per user × 100 write-access users = $400/month
New model: $8 per user × 40 active users = $320/month

Again, the price per user increased, but total spending often decreased or stayed the same. More importantly, it felt fairer to customers.

The Real Takeaway: Fairness > Exact Pricing Models

What we’ve learned is that most customers don’t scrutinize how much they pay—but they deeply care about why they are paying it.

💡 Customers want fairness more than they want cheap pricing.

They don’t want to pay for people who never use the product.
They want transparency in billing.

Now, no matter how much we refine our pricing, some customers will always question it. That’s fine. What matters is that we keep the discussion focused on value—not just pricing mechanics.

Advice for SaaS Startups Navigating Pricing

Don’t get stuck in the weeds of perfect pricing.

Focus on maximizing total revenue, not obsessing over per-user logic.
Price increases aren’t scary if you frame them well.

Every time we changed how we counted users, we also raised prices—and it worked fine. You can always grandfather happy customers.
People care more about fairness than numbers.

If customers understand why they’re paying what they’re paying, they’re much less likely to complain.

Final Thoughts

Pricing is a constant work in progress. We’ll probably keep refining it at Mergify as we grow. But the core lesson is this: be transparent, focus on fairness, and anchor pricing to the value your product delivers.

If you want more on how we think about pricing, I wrote about why we're sticking with seat-based pricing over work-based models and the broader build vs buy dilemma from the customer's perspective.

Why We Left Heroku

Tue, 28 Jan 2025 00:00:00 GMT

In January 2023, everything was smooth sailing for Mergify. Our infrastructure was humming along on Heroku, a platform we had trusted for over three years. Heroku was once the go-to choice for startups—simple, reliable, and developer-friendly. We were happy customers, growing steadily and paying our invoices month-to-month.

Then things started to change.

The Start of a Rocky Relationship

In early 2023, Heroku reached out with an enticing offer: transition from month-to-month billing to an annual Heroku Enterprise contract. The deal included significant discounts on everything—dynos (containers), databases, and add-ons—in exchange for a one-year commitment to a certain number of resources.

We were told we’d be allowed to overuse our resources up to 30% during the year without being bothered—with the understanding that if we grew beyond that, the contract would be adjusted fairly in the next cycle.

It sounded like a win-win.

We signed the contract and carried on. For the first year, everything was fine. By the end of 2023, we had indeed surpassed the 30% growth threshold, but Heroku didn’t reach out. The contract auto-renewed, and we moved into 2024 with no issues.

That was until the automated emails started arriving.

A Series of Surprises

In May 2024, we received an automated email from Heroku. It informed us that the discounts on our containers were being rescinded, effective immediately. Naturally, we contacted Heroku’s support team to understand how that would affect our current contract and were redirected to a new account executive to clarify.

Their explanation was straightforward: we had doubled our usage, and they wanted us to pay the difference for the current contract term—for the next 9 months.

While this was unexpected, we decided to comply. We signed an amendment to the contract and paid the outstanding amount. We chalked it up to a policy change, and, as Heroku has been fair so far, we decided to move on.

But then, in October 2024, another email arrived. This time, Heroku announced that discounts on add-ons, such as PostgreSQL databases and Redis, would also be removed. Once again, we reached out to their team for clarification.

This conversation, however, was very different.

The Heroku automated email we received

When Contracts Don’t Matter

Our current account executive explained that the discounted add-ons we had purchased as part of our original enterprise agreement were no longer “fair” for Heroku.

Indeed, two years before, our previous account executive offered us a 60% discount on the listed price, which was a power move to make us commit for a whole year to the platform. A practice that worked: we committed to Heroku, and the account executive won a “top deal France SMB” award at Heroku.

But now, Heroku wanted us to pay the full price for these services, even though our contract explicitly stated otherwise.

We reminded them of the terms we agreed to in the contract, but their response was, essentially, “it’s not fair for us anymore.”

I spent a lot of time trying to understand how getting a few thousand euros more from a loyal startup would impact Salesforce P&L, or how bullying us into paying money we didn’t owe would help our account executive gain respect from their boss, with no luck.

Despite their efforts to pressure us to pay more, we held firm. A contract is a contract, and we weren’t going to be oppressed into paying for something that wasn’t part of the original agreement.

However, at this point, it was clear that Heroku was no longer a reliable partner for us. Their lack of stability, constant policy changes, and disregard for contractual terms made it impossible to trust them with our infrastructure.

The Move Away from Heroku

By late 2024, we made the decision to move Mergify’s infrastructure to Google Cloud Platform (GCP). Migrating a live product is never easy, but it was the right choice. Heroku, once the pioneer of developer-friendly hosting, had stagnated. The platform’s lack of innovation, combined with its increasingly unpredictable business practices, made it clear that it was time to leave.

GCP offered the flexibility, scalability, and reliability we needed to grow. The migration was a success, and while it wasn't a move we originally planned for, it's one we're glad we made. Google helped us a lot in moving to their platform, which made the whole process smooth — and it gave us a chance to rethink our entire CI/CD stack, including how we handle merge queue challenges.

Reflections on Heroku

Despite the rocky ending, it’s important to acknowledge Heroku’s role in our journey. The platform played a significant part in our early success, providing the simplicity and ease of use that helped us focus on building our product. For small apps and early-stage startups, Heroku can still be a good choice.

But over time, Heroku failed to evolve. As the tech industry moved forward, Heroku seemed to stand still. Features stagnated, the platform became less relevant, and dealing with them as a customer grew increasingly frustrating. In 2025, it’s hard to recommend Heroku as a reliable choice for scaling companies.

Advice for Other Startups

Our experience with Heroku taught us some valuable lessons:

Beware of Contracts with Large Companies: Big corporations can change their terms, policies, or priorities on a whim. Make sure you fully understand the risks before signing long-term agreements.
Stand Your Ground: If a vendor tries to pressure you into unfair terms, don’t be afraid to push back. Contracts exist for a reason. Be ready to jump and save your ass.
Choose Platforms That Grow with You: Heroku was perfect for us in the beginning, but as our needs grew, it became clear that we needed a more robust and innovative platform.

For startups navigating similar challenges, remember that your infrastructure choices are critical. Hosting platforms should be partners in your growth, not obstacles.

Final Thoughts

Leaving Heroku wasn’t an easy decision, but it was the right one for Mergify. We’ve learned a lot from this experience, and we’re excited about what’s ahead with our new infrastructure.

If you’re a startup considering Heroku—or debating whether to stay or move on—ask yourself this: is your hosting platform helping you scale or holding you back? At the end of the day, it’s all about finding a partner you can trust to grow with you.

Reflecting on 2024

Tue, 07 Jan 2025 00:00:00 GMT

As 2025 begins, I’m taking a moment to reflect on an eventful 2024—a year of evolution, challenges, and new beginnings for me and Mergify. Building a bootstrapped company comes with its own unique highs and lows, and 2024 was no exception. Here’s what the past year taught me and how it has shaped the future of Mergify.

Keeping Mergify Thriving in a Changing Market

Surviving and thriving as a bootstrap startup is always worth celebrating, especially in a niche like ours. In 2024, we doubled down on what makes Mergify special: our Merge Queue product, which remains the best in the market for helping engineering teams manage pull request workflows.

That said, we also recognized our limitations. While we’ve always been proud of our technology, we realized this year that having great tech isn’t enough. For years, Mergify was more of a tech-driven company than a product-focused one. This year, we worked hard to change that, shifting our mindset to prioritize product design, usability, and scalability.

The advancement of competitors, such as GitHub, forced us to rethink our strategy. We’ve been evolving in a niche for years, and it’s now time for us to expand our vision beyond what we’ve been doing so far.

2024 also brought some tough lessons about the realities of the market. We initially decided to double down on marketing in late 2022, deploying our efforts during all of 2023. However, after the startup market crash of late 2022, we spent much of 2023 navigating a challenging environment where companies were hesitant to adopt new tools. As that trend continued into early 2024, we ultimately decided to scale back our marketing efforts and rethink how we approach growth.

The truth is that marketing alone can’t solve every problem—especially in a niche like ours. Instead, we’re focusing on building the best products possible and letting our work speak for itself. This approach has already started to pay off, and we’re more confident than ever in Mergify’s future.

A Shift Toward Product and Design

Consequently, one of the pivotal moments in 2024 was hiring a designer—a first for Mergify. This move sparked a transformation in how we approached our product. It wasn’t just about solving technical challenges anymore; it was about creating an experience developers love.

New Mergify design

This new focus led to a complete redesign of our branding and dashboard, making it easier than ever for teams to onboard and use Mergify. It also paved the way for new products like Merge Protections, a tool for managing repository freezes and policies. This was the first product we built with a product-driven mindset from the ground up, and it’s already gaining traction with customers.

Back to Founder Mode

In 2024, I found myself returning to what is now called “founder mode.” For the first time in years, I rolled up my sleeves and dove back into coding and product development. Writing Python again, designing architecture, new workflows, and collaborating directly with the team reminded me of the early days of Mergify—and how much I enjoy building things.

This hands-on approach was fueled in part by the rise of AI tools, which have transformed how we work. From speeding up R&D to enhancing productivity, AI has helped us stay agile and efficient as we tackle big challenges in CI/CD.

Doubling Down on R&D

Speaking of challenges, research and development was a major focus for us in 2024. We spent a lot of time exploring how to solve some of the toughest problems in CI/CD, like flaky tests, CI failures, and observability issues. These pain points resonate deeply with our customers, and we’re excited to bring solutions to market in 2025.

Our work so far has been a team effort, but it’s also required stepping outside our comfort zone. We’ve been engaging more directly with developers, learning from their frustrations and workflows, and using those insights to shape our next big product.

On a Personal Note

Outside of Mergify, 2024 was a year of rediscovery for me. I started writing again, leaning on tools like GPT to help me produce regular content and share my thoughts with a wider audience. Writing has always been a way for me to reflect, and having the discipline to do it consistently has been rewarding.

I’m also trying to teach my kids to take photographs

I also continued my work as a business angel, supporting other startups and sharing what I’ve learned from building Mergify. While balancing this with my responsibilities at Mergify can be challenging, it’s incredibly fulfilling to help others navigate the ups and downs of entrepreneurship.

I continued publishing episodes of Nom d’un Pipeline !, my French CI/CD-themed podcast. This has been a tremendous way to meet new people and learn stuff. I really enjoy the exercise of podcasting, and I’m looking for an excuse to start a new one this year.

I also recorded several episodes as a guest, one in French in IFTTD, another one in French and in Toulouse on the future of tech, another one in French on The Cloud Screener, one on The Python Show, and, finally, one with saas.group.

Looking Ahead to 2025

As I look to the year ahead, I’m more excited than ever about what’s next for Mergify. We’re working on new products that tackle critical pain points in CI/CD, and we’re committed to helping developers ship faster and with less friction. At the same time, we’re staying true to our roots as a bootstrap startup—focused, agile, and always learning.

2024 was a year of growth in every sense of the word. It challenged us to think differently, to embrace new ideas, and to keep pushing forward. As we enter 2025, I’m grateful for the lessons we’ve learned, the customers we serve, and the incredible team that makes it all possible.

Here’s to another year of building, learning, and growing.

Aligning Project Management with Company's Values

Tue, 10 Dec 2024 00:00:00 GMT

When Mehdi and I co-founded Mergify, we didn’t just set out to create a great product—we wanted to build a company that reflected our values. Part of that journey has been rethinking project management, a task informed by years of working in organizations where “Agile” had become synonymous with bureaucracy. What started as a flexible, team-oriented methodology often felt bogged down by rituals that added complexity without delivering real results.

Over the past year, we’ve refined our project management approach at Mergify, shaping it to fit not only the needs of our team but also our belief in simplicity, ownership, and autonomy. Here’s the story of how we got there—and why building a workflow that aligns with your values matters as much as the work itself.

My Journey Through Agile Overload

When I look back at my experiences with Agile in various organizations, one story sticks out. Early in my career, I joined a team at Red Hat, one of the most functional, productive groups I’ve ever worked with. We didn’t rely on heavy Scrum processes; instead, we used a lightweight Kanban board and had stand-ups over IRC. It wasn’t fancy, but it worked. We focused on the work, not the process.

Contrast that with some other teams I observed. One team, with over 20 members, struggled to maintain a sense of ownership. The two-pizza rule, famously touted by Jeff Bezos, couldn’t have been more relevant here: a team too big to share two pizzas is often too big to stay effective. Communication is complicated, and the sense of ownership disappears.

Yet management decided to unify everyone under a single Scrum process, complete with daily stand-ups, sprints, retrospectives, and poker planning.

That did not fix the problem of the 20-person team. Even my high-functioning team began to falter under the weight of unnecessary rituals.

It was a powerful lesson: the process isn’t inherently good or bad but must serve the people doing the work.

Starting Mergify with a Blank Slate

When we started Mergify, we wanted to avoid the traps of over-engineering our processes. We began with almost no structure—just Slack messages and quick syncs to stay aligned. As the team grew, we added a daily stand-up. For a remote-first company, these short, synchronous check-ins were critical for maintaining a shared understanding, even as most of our work remained asynchronous.

Instead of following Agile dogma, we opted for a Kanban approach. Tasks moved naturally across the board with minimal friction. We didn’t bother with two-week sprints or strict velocity tracking; we let the workflow dictate the process.

And it worked; for a while.

Why Lightweight Isn’t Always Enough

Over time, cracks began to appear. One issue was ownership: who was responsible for creating cards on the Kanban board? Engineers who weren’t involved in defining tasks felt disconnected from the problem, treating the cards as instructions rather than opportunities to solve meaningful challenges. The person creating the card and the person doing the work weren’t always on the same page.

Another challenge was the endless backlog without a clear sense of what we were building or why; tasks accumulated, and the act of moving cards felt less like progress and more like treading water. The team craved a greater sense of accomplishment—a way to see their impact beyond the daily grind.

Evolving to a Project-Driven Workflow

To address these issues, we introduced a project-driven layer to our workflow. Projects became our new organizing principle: scoped pieces of work that could be completed in two to four weeks. Each project was defined by three key elements: a brief, a lead, and a deadline.

1. The Brief:

The brief outlined the problem, the goals, and the context for the project. It provided enough structure to guide the engineer while leaving room for creativity. Engineers weren’t just implementers—they were collaborators, shaping the solution as they worked.

2. The Lead:

Every project had a designated lead who was responsible for tracking progress and ensuring the work stayed on course. This wasn’t about assigning blame; it was about having a clear point of contact who could raise blockers, answer questions, and coordinate efforts.

3. The Deadline:

Deadlines were less about pressure and more about focus. They encouraged engineers to make trade-offs, prioritize effectively, and avoid over-engineering. If something couldn’t be completed within the timeframe, we adjusted the scope or deferred less critical elements.

What We Gained

The shift to project-driven work transformed how we operated. It gave engineers a sense of ownership and allowed us to ship faster, avoiding the dreaded “tunnel effect” where nothing tangible gets delivered for months. It also helped us align our priorities, ensuring that every project contributed meaningfully to our goals.

This system wasn’t just about productivity but about creating a culture where engineers felt empowered and connected to their work. It reinforced our belief that processes should serve people, not the other way around.

A few people I talked about our system thought it resembled the Shape Up methodology from Basecamp. I think it does have similarities, except that we’re a small company, meaning we don’t have enough teams to model it exactly yet. But that’s definitely a methodology that resonates with us.

The Shape Up methodology

Final Thoughts

Looking back, the changes we made weren’t just about fixing problems—they were about staying true to our values. At Mergify, we believe in autonomy, ownership, and simplicity, and our workflow reflects those principles.

If you’re struggling with your own project management processes, ask yourself: do they serve your team’s needs, or are they just there because “that’s how it’s done”? (I wrote about a related frustration in The Problem with OKRs Isn’t OKRs.) The best workflows aren’t the most popular—they’re the ones that align with your culture and empower your people to do their best work.

At Mergify, we’re proud of the system we’ve built, and we’re excited to keep evolving it as we grow.

SaaS and Work-based Pricing

Tue, 19 Nov 2024 00:00:00 GMT

Despite the rising popularity of work-based pricing in SaaS, Mergify is sticking with seat-based pricing—for now. Here’s why we believe it’s the right choice for our product and our customers, and their ability to budget with confidence

The Work-Based Pricing Trend

Following the latest SaaS pricing trends is tempting, especially when they align so well with a powerful concept: customers pay in proportion to the value they receive. In recent months, “work-based” pricing has gained traction across the industry, especially in AI-driven applications where you pay per task completed. It’s a straightforward exchange: resolve a customer’s problem and receive $1.

Simple, clear, and highly attractive.

When we first considered revisiting Mergify’s pricing for next year, work-based pricing seemed like a model worth exploring. Imagine a setup where every task accomplished by Mergify correlated directly to the value we delivered to our customers. More value equals more payment, a transparent exchange that clients find easy to understand. But, as we dove deeper, we realized that the nature of Mergify’s work doesn’t fit so neatly into this model.

And that might be true for your SaaS as well.

The Value Clarity of Work-Based Pricing

Work-based pricing makes perfect sense for certain SaaS products. Consider the AI-driven customer support solutions rolling out recently, like Salesforce’s $2-per-conversation approach or Intercom’s Finn and its 0.99$ per resolved conversation. The pricing here is inherently appealing because it aligns precisely with the customer’s perception of value: for every problem resolved, they see a clear, direct benefit to their end users, who leave satisfied and engaged.

This clarity makes it easy for a customer to decide—they’re paying to solve a specific pain point for their users, and each solved interaction has a measurable outcome. The more conversations are solved, the more they pay; it feels like a no-brainer.

Customers see exactly where their money is going, and it scales beautifully alongside their growth: the more users they have, the more problems they have, and the more value you can provide. But also: the more users they have, the more money they have, so they’re happy with giving you a part of it. Everything can grow at the same rate, from the customer’s business size to the value your SaaS provides.

This reminds me of the early days of cloud computing when Amazon Web Services introduced usage-based billing. The more your business grew, the more infrastructure you used, and the higher the bill. It made perfect sense, and that was one of the reasons for its success and early adoption by startups.

Why Work-Based Pricing Doesn’t Always Fit

At Mergify, we aimed to find the same direct correlation between our value and our pricing model. However, our reality is different: the value we provide is primarily to software engineers who use Mergify to streamline their CI/CD workflows. It’s an incredibly powerful tool, but it doesn’t impact our customers’ end-users in an obvious way. This makes work-based pricing challenging because we’re one step removed from the end-user experience — and, therefore, from the business.

We brainstormed possible approaches. Perhaps we could charge per pull request made by developers. After all, that’s a major part of what Mergify automates and where it provides value. But we immediately saw a problem: this model could encourage the wrong behavior. If every pull request carries a charge, teams might try to reduce the number of pull requests to save on costs, potentially compromising code quality. As engineers ourselves, we value clean, atomic commits and easy reviews. A per-pull-request charge could discourage these practices, creating friction between the optimal workflow and our pricing model.

Another challenge with work-based pricing is the difficulty it presents for customers when predicting their usage. Most organizations set budgets a year in advance, and it’s nearly impossible for a team to accurately estimate the number of pull requests or jobs they’ll need in a year. This unpredictability makes budgeting stressful and challenging, especially for engineers seeking approval for software expenses. With seat-based pricing, customers know their costs upfront, which aligns much better with annual budget planning.

We also considered a per-job charge based on CI (Continuous Integration) runs, but this quickly ran into similar issues. Running a high volume of CI jobs often results in better software quality. Charging per job could lead to reduced testing—a problematic incentive in an industry where quality matters deeply. CI providers charge per job because of the required computing power, but in Mergify’s case, we don’t incur comparable costs. So, a per-job charge wouldn’t reflect our real costs and could end up discouraging best practices.

The Challenge of Finding the Right Fit

In a perfect world, we’d find a model where Mergify’s pricing scaled directly with the customer’s perceived value. However, we found that our usage patterns, like many SaaS, do not align with the benefits of the work-based approach. To truly capture the value Mergify provides, we realize that seat-based pricing continues to be our best option, at least for now.

By charging per user (seat), we avoid influencing developer behavior, allowing them to use Mergify to improve their workflow without second-guessing how often they use it. We also want to avoid the stress of unpredictable costs. Seat-based pricing allows our clients to budget accurately and avoid unexpected expenses, aligning our pricing model with their planning cycles and offering peace of mind. As we continue to build Mergify, our goal remains to be the trusted tool in the hands of developers.

Right now, that means a seat-based model, which keeps our focus where it belongs—on supporting teams to do their best work, no strings attached.

This decision wasn't easy, and we might revisit it in the future. Pricing is a delicate balance between the customer's experience, the product's value, and the company's needs. I wrote more about our full pricing journey at Mergify — from copying GitHub's model to active user billing.

The Engineer’s Dilemma: What We Did Right at Mergify

Tue, 05 Nov 2024 00:00:00 GMT

In the early days of Mergify, the journey Mehdi and I embarked on wasn’t unique. In fact, it’s a tale as old as time for engineer founders: a couple of smart engineers, passionate about technology, with an exciting vision to revolutionize their space. We had everything we needed—or so we thought: the knowledge, the technical expertise, and the drive to build something incredible.

This is where the story of Mergify begins, but what often happens next is a classic mistake that many tech founders make. They build a beautiful, feature-packed product that doesn’t solve a real problem—or even worse, it solves a problem no one has. This is the hard lesson that too many engineers learn too late, and it could have easily been us.

But we managed to steer our ship in a different direction, and looking back, there are key things we did right. Let me take you through that journey.

The Temptation of the Tech

For any engineer, there’s nothing more fun than building. The thrill of creating a new feature, optimizing your product, or pushing out updates can become intoxicating. Mehdi and I felt this pull strongly when we started Mergify. We had big ideas, a packed roadmap, and technical solutions that we were eager to implement.

But here’s the thing: technology, while critical, is only a part of building a successful SaaS business. We could have easily fallen into the trap of focusing solely on the tech and neglecting the most important piece of the puzzle: the customer.

We both had to confront the reality that building an amazing product wasn’t enough. If we didn’t speak to our customers, understand their pain points, and really get to the heart of their problems, we were going to fail.

This is where so many engineer-founded startups stumble. They fall in love with their technology rather than falling in love with solving the customer’s problem. Luckily, we managed to recognize this early on.

Engineers Need to Talk to People

Talking to customers doesn’t come naturally to most engineers—it didn’t for us either—but it was a necessary step. While it’s tempting to stay behind your keyboard, tweaking code or adding features, the real magic happens when you step out and listen to what your customers are saying. What do they struggle with? What would make their lives easier? What’s keeping them up at night?

I remember one story about two bright engineers who reached out to me on LinkedIn, seeking advice. We decided to meet at a bar. They had spent three years working on a fantastic piece of technology but hadn’t seen any traction. Why? because they hadn’t built it with a customer in mind. The tech was solid, but it didn’t solve any real problem. They hadn’t spent time talking to users, and so the product existed in a vacuum.

You've got to start with the customer experience and work backwards to the technology — Steve Jobs (1997)

With Mergify, we knew that if we were going to build something with a lasting impact, we needed to constantly engage with our community and understand their problems. It wasn’t enough to have a great piece of technology—we had to have a great solution to a real problem.

Shifting Roles for Success

One of the smartest things Mehdi and I did early on was divide our roles clearly. We knew that if both of us were deep in the tech, Mergify would never succeed. So, while Mehdi stayed focused on building the product, I took on the role of sales, marketing, and customer interaction.

For an engineer, stepping into these roles can be uncomfortable at first. Sales? Marketing? Communication? These aren’t things they teach you in computer science class. But it was a necessary shift and one that paid off.

I drew from my experience selling self-published books. I knew that just because you write something doesn’t mean people will read it. You have to market it, spread the word, and get it into the hands of people who need it. The same principle applied to Mergify. We couldn’t just build features and expect users to come flocking. We had to sell it, promote it, and make it known. This is still something we need to do to this day.

The Hard Truth

The truth is, you can have the best technology in the world, but if you don’t have customers, it’s worthless.

I remember another encounter at a wedding, where the bride's father introduced me to his nephew—a tech entrepreneur. The moment I heard him describe his startup, I already knew what was wrong. “You’re not selling anything, are you?” I asked. The bride’s father looked at me, astonished by the boldness of my assumption. And sure enough, he wasn’t. He and his co-founder, both engineers, had spent their time adding features instead of learning how to sell their product. He admitted that I was not the first to tell them it was one recipe for a disaster.

At Mergify, we avoided that trap. We recognized early on that while the tech needed to be solid, the success of our business depended on our ability to market and sell it.

What We Did Right

So, what did we do right at Mergify? We talked to our customers, really talked to them. We asked questions, learned about their challenges, and made sure we were solving their pain points. We didn’t fall in love with our technology; we fell in love with the problem. And most importantly, we divided and conquered. Mehdi stayed on the tech while I trained myself in the art of sales, marketing, and product management.

These steps weren’t easy, requiring us to step outside our comfort zones, but they made all the difference. Five years later, Mergify isn’t just a successful SaaS company because we built great tech — it’s successful because we solved real problems for real people. (For a different angle on the same lesson, read Tech Is the Easy Part.)

And that’s the real lesson for any engineer founder: focus on the problem, not the tech, and you’ll go far.

There's (almost) no GitLab

Tue, 29 Oct 2024 00:00:00 GMT

Do you guys support GitLab? Is there any way this can work with GitLab? Does this support merge requests?

No, we don’t.

As I spent hours screening software engineer candidates those last weeks, I repeatedly answered the same question: where’s Mergify support for GitLab?

To put things into perspective, we mostly hire in the French market, and I think it deserves some context.

Culture Bias

I’ve known software engineers for more than 20 years in the country of the baguette, and something is pretty clear. We have amazing engineers, but they suck at understanding what ROI means. Most people have no conception of the value of time, and for most average French engineers, it’s OK to spend time on anything as long as it avoids spending money (or requesting a budget).

It’s not even a frugality thing; it really is just the inability to compute a basic return on investment and put a price on an hour of work.

In the context of software forges, that means something: French companies, from startups to scaleups, are heavily biased towards deploying GitLab Community Edition because it’s free. They would do this over a cheap hosting bare metal server. You can find such hosting for around $20/month.

At this price, the average French software engineer will not buy a GitHub license. They’d call that a rip-off.

I’ve heard it.

Fire at OVH datacenter in Strasbourg, March 2021

If you don’t factor in the time it takes to spin up and maintain the GitLab instance or the impact of having your server on fire, then, indeed, a price of $20/month is unbeatable.

I remember asking one young French engineer in a startup about their GitLab instance and how they would maintain it and manage its security compliance. The answer was straightforward:

We just run `apt upgrade` every night so we’re sure we’ve every security deployment installed.

YMMV.

The Market Share

In April 2024, the Mergify team spent a few days at Devoxx France, the country's largest developer conference with nearly 5,000 attendees. We talked to dozens of engineers, and roughly 50% of them were using GitLab at work. Some large teams were moving away from GitLab to GitHub, but for a large majority, we were weirdos for not supporting GitLab. Their view of the market share is biased toward the French market, where GitLab might indeed have a large usage, which might not tend to generate large revenue, though. Remember that the Community Edition of GitLab is free.

Mergify’s team at Devoxx France 2024

If we compare GitHub’s $2 billion revenue to GitLab’s $579 million revenue for 2024, this is a 1:4 ratio, which is already pretty huge. Sure, revenue is not usage, but considering that GitHub has Microsoft behind it and GitLab is reportedly looking for a buyer, the future looks way brighter for GitHub — something I explored in more depth in Is GitHub the Future?. And I'm not even talking about the fact that the vast majority of open-source projects use GitHub.

No Gauthier, I don’t think that this is how it works.

(LinkedIn post)

Innovation

Now that I’ve set the scenery, I feel it’s safe to answer about GitLab support for Mergify. The main reasons why Mergify’s Merge Queue is not looking for GitLab support anytime soon are:

Innovation happens on GitHub nowadays. Ten years ago, GitHub was behind on certain topics (hello CI), but they are now way ahead of the competition. GitHub is the place where most open-source is built and where you need to be if you’re building new products;
Most teams using GitLab CE have no intention to buy any software;
The market share of GitLab is small and probably shrinking.

I’ve nothing against GitLab, and their software might be pretty good. All we know is that they’re outsiders, while a lot of the tech market in Europe seems to think that betting on GitLab is the best go-to-market strategy.

I beg to differ.

What's going on with Dependabot?

Tue, 15 Oct 2024 00:00:00 GMT

I loved Dependabot. I’ve used it since Grey Baker started it in 2017. I’ve seen it grow from a one-person shop to being acquired by GitHub in 2019. It’s been a fantastic tool that created more than 5,000 pull requests on Mergify repositories. I remember the excitement of finally having a tool that would bring all the new fancy features and bug fixes of my dependencies to my project in a snap.

Dependabot allowed us to fix a lot of security updates introduced by dependencies, and to be aware of anything new being released in the libraries we use.

But today, we kicked Dependabot out. Dependabot let us down.

Security You Said?

The move to acquire Dependabot from GitHub was smart. It was one of the key features that started their security roadmap with the acquisition of Semmle and their CodeQL engine the same year. GitHub has become all about security over the last few years, which makes sense considering the cybersecurity segment's hyper-growth over the last and upcoming years.

However, as Dependabot matured under GitHub, cracks started to show in what was once a flawless experience. The tool that was supposed to streamline security and updates became a source of frustration.

It has major design flaws that GitHub does not seem to care about.

First, Dependabot can fail silently. That happened to us multiple times a year when Dependabot would just stop working and create pull requests to update our packages. You’d think that debugging such an issue would be possible by going into the Dependabot tab of your repository, but no. The log for this is actually hidden in Insight → Dependency graph → Dependabot. A strange and unintuitive location for such crucial information.

Once you find your log, you can then read it and debug it yourself.

That’s a major problem because there’s nothing warning you that Dependabot is broken. We are used to updating our packages regularly, so we’d know, but there’s nothing preventing your dependencies and security updates from getting stale for months without you noticing. Terrible experience.

Always Lagging Behind

We’re a Python shop. We leverage poetry to manage our dependencies, and we use the latest Python version in our containers.

As a Python shop, staying on the latest version helps us ensure security, performance, and compatibility. So we update it as soon as we can, usually a few days after it’s released.

And then Dependabot is broken.

And you have to wait weeks for GitHub to fix the problem.

The last few times, we had to update ourselves Dependabot itself, as shown here or even here. We’re basically doing GitHub’s job for free, maintaining the Dependabot database ourselves for all their customers.

We contacted GitHub support about this already, and they did not care at all. Their laconic answer was, “Wait for it to be updated.”

Sure, thank you. We’re the ones doing the updates.

I get it—maybe Fortune 500 companies don’t care about the latest Python micro releases. But for startups like ours? It’s a big deal.

So today, we got rid of Dependabot and replaced it with Renovate. It seems better maintained and supports a larger package ecosystem than Dependabot. So far, it has simplified our workflow and is not broken on a simple Python micro update. 🤞

We're also adding support for Renovate in Mergify Merge Protections, as we have done for Dependabot in the past. That will ensure you can write advanced rules for automating your GitHub workflows, including automatically merging your dependency update. 🦾 Let me know if you’re interested in trying it out!

Why You Need Product Engineers

Tue, 01 Oct 2024 00:00:00 GMT

A couple of weeks ago, I attended our quarterly MAHOS (Mergify All Hands On-Site)—an event where we gather the whole Mergify team together for a week—and gave a small speech about product engineers.

Mergify’s team, Sept. 2024

I was unsure if I coined the term product engineer at that time or if it already existed. After Googling, I found that I was not the only one who realized that we no longer need software engineers.

When we started our venture a few years ago and decided to hire engineers, we naturally looked for software engineers. We found great engineers. They learned a ton of stuff working with us over the last couple of years and became very efficient at producing code. Awesome. I wrote last month about how to become a great software engineer, and while I hold to this, the next step in your career, if you want to work in a product-oriented startup, is to become a product engineer.

What is a Product Engineer?

Is that just a software engineer building a product? Yes! But that is not what a software engineer does by default. Let me tell you an anecdote.

Last month, with my product owner hat on, I wrote a user story explaining one of the changes we needed to make in Mergify: feature X is enabled by default in the product, which is annoying because most users do need it. We need to 1. allow the users to enable or disable it and 2. make it disabled by default.

One of our software engineers picks the ticket and implements a solution. Here’s what they do:

The feature X can be enabled or disabled;
The feature X is disabled by default;
An error message warns the user constantly that feature X is currently disabled and that they need to enable it to have it work. There’s a giant red banner to warn users that feature X is disabled—until the user has enabled the feature.

When I see that, I’m really confused. The code is great, and the ticket is indeed implemented, but the last part is terrible from a user experience perspective. It forces the user to enable feature X to get rid of the warning, meaning we get back to a point where users have to enable feature X by default, even if they don’t need it, just because they’re confused.

Bad product design.

That is great software engineering work but terrible product engineering work. In no case did the engineer put themself in the user's shoes or try to understand why we needed that change.

This is where a product engineer must shine. They need to understand the value and the reason behind the code they write, taking into account the product, its roadmap, its priorities, etc. It requires the ability to do trade-offs by being pragmatic. They need to be obsessed with the customer and understand their problem. In a startup, you need to ship fast, meaning, again, doing trade-offs and being efficient and practical. They need to be detail-oriented, have a sense of ownership, and be on the look to create terrific experiences.

Writing software has never been so easy. With AI on the rise, writing actual code will have less and less value.

The core value of building software is going to be whatever AI is not yet able to do, which is empathy—connecting with and learning from other human beings’ needs.

How to Transform Engineers in Product Gurus

Do not follow Indeed’s advice, for sure.

Based on the description I wrote above, we implemented some changes and made good progress overall. The improvements we made were due to simple changes we made to the organization.

Connect engineers with customers. Doing support directly with customers, joining a demo call, spending time in a booth during an event, and talking to prospects. All those activities where engineering can interact with prospects and customers are very valuable;
Explain the why, not the how. As a product owner, you must explain why changes are being made and not how they should be made. The more context you feed into your user stories, the easier for an engineer is to make the right decision when building a feature or fixing a problem. It’s especially important when, as a product manager, you have a technical background and you could be tempted to dictate a solution.

There are many software engineers out there, but not many product engineers. We’ll be on the lookout for that when hiring in the future. And if you want to join a product-oriented startup in the future, make sure you change your mindset to not just writing code. 😉

How To Test Your API Integration

Tue, 24 Sep 2024 00:00:00 GMT

As I was publishing last week's post on whether GitHub is becoming obsolete or the future of development platforms, they decided to trigger a two-hour interruption on Mergify in retaliation.

Just kidding. I am sure they did not do that on purpose.

Read my post-mortem if you want the whole story. The summary is that they broke their API for several hours until people started to complain, and they finally rolled back their change. Bringing down our service in the meantime.

That event forces me to talk about APIs this week.

API Definitions Are Just Definitions

I won’t go into the definition of an API per se; it’d be boring. You can Google it if you need to.

The real question is what having an API means. Offering an API to your users means authorizing them to interact with your service. This implies many rules, such as the data model of your API, the behavior of your API, the rules of usage, etc. Some can be encoded in a computer-readable machine; others cannot. Engineers like to talk about contracts, and I think it’s an almost good analogy.

To describe this contract, you need multiple specifications.

Developers have been ecstatic over OpenAPI over the last decade as a go-to media for describing their API. I want here to emphasize how little this documents your API. It illustrates the data model used but does not encode much of the behavior the system might exhibit.

Hey, I can confirm that GitHub did not break its OpenAPI schema when it broke its API last week. Formidable.

However, based on the assumption that OpenAPI is enough, many engineers mock their API consumption based on that part of the contract and think they’re done.

In that situation, the minimum you should do is validate that your mocking follows the OpenAPI schema you’re using. Even that is not enough because sometimes the schema changes—and sometimes it’s just not respected.

Let’s take GitHub again as an example. Their API is so legacy that the JSON schemas were crafted manually — and there might still be, I don’t know. It’s fine; it’s better than nothing, and it’s not obvious to change a legacy API that’s been there for 15 years.

We know first-hand that their system does not always respect the GitHub API JSON Schema.

APIs Have Side-effects

Again, this approach is entirely based on the data model and is insufficient and of little value.

Most of an API's value is in the behavior it triggers. Unless your API is a basic CRUD and does storage only, it will have side effects that might or might not be visible through the API.

For example, creating an asynchronous job on any REST API will return nothing except a unique identifier, which can be used later to identify the work. You might receive the data via a webhook or have to poll the API to get the job’s status. This kind of behavior cannot be documented in OpenAPI as it’s not part of the data model; there’s nothing to tell you to expect a webhook.

API Invisible Parts

Now, let’s discuss all the invisible parts of running an API. There are many. The first that come to mind are RBAC, quota, and rate limits. Most APIs have to implement those items, and they also have a direct impact on the API behavior and access.

Those features will massively impact the quality and quantity of API use. Again, they are pretty hard to test in a black box. There’s no way you can easily mock a full RBAC implementation or real-life rate limits.

Testing the Hard Way

Having consumed many different APIs for the last five years on Mergify, and especially GitHub’s one that we know by heart, gave us a few ideas on how you can or cannot test.

Rule number one: do not mock. Record your tests.

We leverage vcrpy in Python to do that: the idea is to run your test in a record mode where real HTTP requests are done against a service. Once the recording is done, you can replay the test when running it locally or in the CI.

If any of your code tries to make a different HTTP call, the test will fail, and you will have to re-record it. This ensures that no change is made to the application without being noticed.

Now, that does prevent your application from being broken, but that does not prevent the API from breaking your app. The only way to do this is to regularly re-record all the tests and see if they break.

So, rule number two: re-record your tests regularly — every day if possible.

For example, we have a test that plays with GitHub pull request labels. When re-recording a test a few months ago, we noticed that if it failed. It turned out that GitHub changed its API to become case-sensitive overnight (that was not in the OpenAPI schema!).

In that case, we preferred to ask GitHub to fix their API rather than fix our code, but hey, your mileage may vary.

Rule number three: be ready to fix the code.

No amount of testing will cover all the edge cases. For example, requests quota or rate limit might be hit in real scenarios but not in testing, meaning you’ll have to handle those specific cases without being able to test. It’s fine — you can actually mock part of the responses here.

For this, we leverage Sentry to obtain evidence of the problem, replicate it in a test, and fix it. No amount of testing can fix all scenarios, so having a way to hotfix your code is a must.

In the end, mixing API test recording for safety and error tracking for fast action is the best combination we’ve seen for dealing with external systems.

If we map those rules to last week's incident, rule number three helped to fix the issue quickly, while rule number one would have technically caught it, and rule number two would have done so in less than 24 hours. Even if it turned out in our case that reality kicked in before testing.

So use that. And retry mechanisms.

I guess that’ll be for another post.

Is GitHub the Future or Becoming Obsolete?

Tue, 17 Sep 2024 00:00:00 GMT

Over the last few months, I stumbled upon a few articles on GitHub's history and future. I find those quite interesting. Greg Foster gives a quick history of the rise of GitHub over the years, while Scott Chacon, one of GitHub’s co-founder, retraces the history of GitHub from the inside.

The story is great, and I’ll leave it to you to read those posts if you want to learn more. I was alive at that time; I saw and lived it all. I started using Git in 2005, and I’m the 2644th of the 100 million GitHub users—I joined GitHub in 2008 when it was still in beta.

It’s definitely true that Git won, and especially GitHub. I’ll probably have to write about GitLab at some point (I eventually did) — but you won’t read any ranting here today. As Scott Chacon writes, GitHub had a taste and perfect timing and soon gained traction from the open-source community.

Ok, great, so GitHub wins. Where do we go now?

Microsoft

I believe that GitHub's growth was already on track before Microsoft bought it in 2018, but that move still changed everything. At that time, GitHub still lacked major features, such as a CI/CD system, and had a hard time being compared to GitLab. The following year that changed, and GitHub Actions started (thanks, Azure) and shook everything up.

If you read opinions about GitHub vs. GitLab, CodeCommit, Azure DevOps, or any other platform, you’ll mostly see engineers comparing user-visible features. And sure, this has value, and that might be your main criteria to pick one or the other if you’re a small team with full power over their choice.

However, this is not what GitHub and Microsoft are after anymore. Take a look at the roadmap of the last few years, and you’ll see a pattern: enterprise. Pushing code, creating pull requests, and any part of software engineers' day-to-day activities have been covered for the last 15 years.

They designed it, the industry adopted it, and GitHub has nothing in its roadmap to change that paradigm. They’re building on the momentum they raised.

Security, Copilot (AI), and compliance are items that the largest corporation needs to embrace a platform such as GitHub. This is only the beginning: this year, the GitHub sales organization underwent a reorganization to look more like Microsoft's sales organization. I suspect GitHub is now able to leverage even more Microsoft resources to push its platform to large corporations—which definitely makes sense. The link between GitHub and Azure is tightening, both technically and commercially.

For the best and the worst.

Relevance

How is GitHub still relevant if it does not innovate on developer workflow? Is it becoming legacy software?

There is certainly a disjunction between what developers and corporations expect, and at this point, if you look at the ratio between features, compliance, and price, there's no better alternative than GitHub. I don’t think this is going to change anytime soon.

For open-source projects, there might be alternatives, but if you’re pragmatic (and lazy), GitHub is the pick. There have been a number of projects trying to escape GitHub over the years, such as when Microsoft acquired it. The latter is still seen as an enemy of open source by some folks (I suspect this is PTSD caused by the Balmer era). More recently, another exodus was triggered by the announcement of Copilot and the fear that the training was done on free software. However, at this stage, it’s like emptying an ocean with a spoon, and the impact does not compensate for larger projects moving to GitHub (e.g., Python).

Steve Ballmer

On the other hand, GitHub is attracting more competitors. It has grown to a point where many startups want to disrupt it: one can look at Radicle and its decentralized approach, Pierre and its modern design, Diversion with its game-centric approach, or Palmier, who’s building a new kind of repository.

They might succeed, but the road is going to be long to get massive adoption — and migration.

There’s nothing replacing GitHub in the short term. We better deal with it.

Staying Competitive

Tue, 10 Sep 2024 00:00:00 GMT

Starting a company is always a challenge. You start small, and you might have to fight competitors that are way larger than you. They could crush you.

However, being small does not mean that you will necessarily lose against large companies.

Last week, I was wandering through a forum where I regularly hang out with other SaaS founders. We share founders’ issues, anything from how to run ad campaigns to how to fire people (sigh).

That day, one of the participants asked a question about staying competitive when large companies enter your market. They were facing a few challenges, the main one being that a multi-billion corporation was adding features to its product that would poach on their turf.

That resonated with me. Mergify went through the same hassle when GitHub launched its own merge queue features last year.

Being in this position is extremely risky. There are many horror stories out there of startups being killed by larger competitors. You’d just have to watch the latest OpenAI DevDay announcements over the last few months to see how many startups they would instantly disrupt with the announcement of a single feature (marketplace, custom GPTs, etc.).

Is Apple going to kill a bunch of companies with iOS18?

Now, that being said, being the underdog is not necessarily a bad thing and can turn out to be a strength. When survival is at stake, the smallest dog can become the more fierce one.

80/20

In 1668, Jean de La Fontaine wrote a famous fable, Le Lièvre et la Tortue. If you never heard of it, the TL;DR is: a confident hare mocks a slow tortoise and challenges her to a race. The hare, overconfident, takes a nap mid-race while the tortoise steadily continues and wins. The fable teaches the lesson that persistence and diligence can triumph over arrogance and haste.

There are multiple choices to do in this situation, but my feeling is that most large companies will implement the “20% features that do 80% of the job.” It makes sense to them economically. With little effort, they can enter the market and grasp a large amount of the share using their moat, branding, marketing, and existing customers.

They can leverage their vast resources and existing ecosystems to quickly dominate this space. However, they often leave gaps in niche or specialized needs because building the remaining 80% of features to cover every specific use case requires more effort and may not align with their broader goals.

Unfortunately, if you were also targeting the same 20% feature, this can hurt your business. You’ll stop seeing new customers and will lose existing ones as they remove the one-too-many vendor from their list.

However, there is a chance for you to stay ahead of the game. Like the hare in the tale, large companies will just take a gigantic nap once they’re done with what they expect to be enough. This is where you can shine.

A small company that builds beyond the basic 20% and focuses on solving complex or specialized problems can appeal to customers who require more tailored solutions, thus capturing a large share of a more focused market. While large companies serve the general market, small players can dominate specialized verticals. By implementing, e.g., 40% of the features for a particular scope, you could address 97% of the market, thus appearing as an expert and leader in your area.

Vertical

Another play that I like is to target different verticals. If you design presentation software and suddenly PowerPoint enters the market, it’s going to be pretty hard to win over the long run. Microsoft will win because they’re known (the “nobody gets fired for buying IBM” rule), and they can reach out to millions of customers — you can’t.

However, if you pivot to becoming the presentation software specialized in building convincing, AI-generated, prospect-centered sales decks automatically, then you can win an entire part of the broader market.

For sure, the initial market might be narrower, but by being focused, it’ll be easier to win it. As you’re winning it, you can expand into adjacent markets and grab more share of the global offering.

Strategy

For both strategies — expanding beyond the 80/20 or addressing a particular, the key to discovering which approach might suit you better is to talk to your existing customers. They are the ones having the answer to the questions “Is there a usage pattern they rely on?” or “Are they working in the same industry?”. Focusing on customer experience or specializing in integrations that large corporations won’t do is only identifiable if you speak to your users.

Small companies have the agility and execution speed that large companies lack, making them the best innovators.

I guess that’s my message to anyone out there being attacked by large corporations. Don’t throw the towel too soon. There might be different plays for you to continue growing and fighting back against the Goliath.

In the end, it’s not just about size. It’s about understanding where you can uniquely provide value and delivering that value better and faster than anyone else.

Reflecting on the Journey of "Nom d'un Pipeline !"

Tue, 30 Jul 2024 00:00:00 GMT

"Nom d'un Pipeline !" (translation for English readers: "What a Pipeline!") has been an incredible journey, filled with learning, growth, and connecting with some of the brightest minds in the French tech scene. As a French-based startup for over five years, with most of our customers and audience in the US, we noticed a significant gap in the knowledge and maturity around CI/CD among French engineering teams. This realization motivated us to create a platform to elevate the understanding and practices of CI/CD in our home country, leading to the birth of "Nom d'un Pipeline !"

The Birth of an Idea

Starting a podcast was a new adventure for me. With the help of Mergify’s marketing team, we built everything from the ground up and began reaching out to potential guests. We aimed to find the right candidates and teams to feature in our 45-minute episodes, sharing their journeys and insights around continuous integration, continuous deployment, and quality assurance.

Highlights from Season 1

All episodes were amazing, and the content we built was really valuable. I think several episodes stood out for their depth and impact:

Mathieu Leroux-Huet discussed performance, offering valuable insights into optimizing systems for better efficiency.
Cyril Rohr from RunsOn shared how improving GitHub Actions runner can significantly enhance CI workflows speed and cost.
Olivier Pillaud-Tirard from ManoMano detailed how they have improved their CI over the last couple of years, providing a roadmap for other teams to follow.

These episodes were not only informative but also showcased the diverse approaches and innovations happening within the French tech community.

Overcoming Challenges

Producing a podcast comes with its own set of challenges. One of the biggest hurdles we faced was recording remotely. There were times when technical issues disrupted the recordings, but we managed to overcome these obstacles with perseverance and technical adjustments. While I would love to record in a studio, living in Toulouse makes it challenging since none of my guests are local. Despite these difficulties, the remote setup has allowed us to connect with a broader range of guests.

Positive Feedback and Growing Momentum

The feedback we've received has been overwhelmingly positive. Listeners appreciate the insights and real-world experiences shared by our guests. Knowing that the show provides value and helps the French tech scene advance in CI/CD practices is incredibly rewarding. Recently, we passed the 2,000 views/listens mark, indicating growing momentum after just eight months. This milestone is a testament to the show's impact and the increasing interest in CI/CD topics.

Personal and Professional Growth

On a personal level, hosting "Nom d'un Pipeline !" has been a delightful experience. I discovered that I genuinely enjoy talking to people and learning about their teams and tech stacks. It's been an eye-opening journey that has enriched my understanding of CI/CD and connected me with some brilliant minds in the industry.

Acknowledging Our Guests

I want to extend my heartfelt thanks to all our guests for their confidence and for sharing their journeys: Clément, Sofiyan, Romaric, Aurélien, Frédéric, Olivier, Dan, Thomas, Mathieu, Cyril and François.

Your contributions have been invaluable in making this podcast a success.

Looking Ahead to Season 2

As we wrap up the first season, we are already gearing up for season 2, set to launch in September. We are excited to bring new hosts from major French companies like Doctolib and Alan, promising even more insightful content and engaging discussions. While we don't have specific plans for season 2 yet, we are committed to continuing our mission of educating and inspiring the French tech community about CI/CD.

"Nom d'un Pipeline !" has been an extraordinary journey, and I'm proud of what we've accomplished so far. We hope that this podcast continues to serve as a valuable resource for the French tech scene, helping teams to improve their continuous integration, testing, continuous deployment, and overall development workflows — including tackling tough topics like the challenges of merge queues. Stay tuned for more exciting episodes and insights in the upcoming season!

If you understand French, don’t forget to subscribe to the podcast on your favorite podcast platform or on YouTube!

The Challenges of Merge Queues

Tue, 23 Jul 2024 00:00:00 GMT

Merge queues are a tough concept to grasp, and over the last five years at Mergify, we've spent countless hours educating developers about their importance and utility. We've published numerous blog posts, written extensive documentation, and even gone to conferences to teach software engineers what a merge queue is. This process of spreading awareness has been a rewarding yet challenging endeavor.

One of our developers, Charly Laurent, gave an insightful talk on the subject, highlighting how merge queues can revolutionize CI/CD processes. You can check out his talk here:

Understanding Merge Queues

Merge queues are not an obvious choice for most teams, and they often require a shift in the balance between safety and speed of delivery. Deploying a merge queue means prioritizing quality over quantity, which is not an easy decision for many development teams, who might be pressured to ship fast.

For example, without a merge queue, teams often merge untested code. This is due to outdated test runs, meaning that they are deploying code that might not work. Without a merge queue, there is no way to prevent merging pull requests with outdated tests and breaking the CI for everyone — which is exactly why you should stop merging your pull requests manually in the first place. One of our customers faced this exact issue, which meant they needed the equivalent of a full-time engineer dedicated to tracking issues in the main branch that broke the CI.

Most platform engineers find the concept of moving the post-merge tests to pre-merge tests challenging.

The Trade-offs

This blog post from Vercel captures this common misunderstanding and the trade-off around merge queues and their CI costs and latency:

Despite the majority of commits being safe to merge after the local CI checks complete on their pull request, the merge queue will incur running the cost of running the CI again every time.

While this is true, the problem here lies in the word "majority." The definition of "majority" can vary significantly across teams. If a minority of pull requests break the main branch after merging, it can cause considerable downtime and require substantial effort from CI engineers to restore stability. We've seen teams come to Mergify with a 30% failure rate on their main branch. While a merge queue won't magically improve the failure rate, it ensures that it doesn't worsen, even if it means a small decrease in merge speed. That ensures that the effort invested in improving the CI is not wasted the day after.

Another perspective from Vercel states:

With merge queues, changes from developers depend on changes from other developers even if they are unrelated to each other, and this makes it hard to scale monorepo merge times with more developers.

This concern is valid for merge queues that don't support monorepo and queue parallelization. However, most modern merge queues (GitHub's own being an exception) do allow for optimization in these scenarios.

Vercel’s blog post concludes with:

With this workflow in place, the merge queue can be safely removed because checks will still always be run before users ever see the deployment.

This reflects the workflow of many teams that don't use a merge queue: merge, run tests on main, then deploy. However, this approach doesn't solve the issue of merging something that breaks the main branch. During the downtime, teams have to identify the culprit, revert changes, and ensure everything works, causing delays and frustration. Bad developer experience ensues.

Teams like Uber recognized this problem six years ago and started building their merge queues. Similarly, in OpenStack, we had a system supporting multiple repositories with Zuul over ten years ago.

Build New Solutions

Considering the merge queue adoption issues, we've spent the last few months reworking our merge queue system to simplify deployment and enhance user experience. We know for a fact that developers appreciate the reliability it brings to CI processes, but we also observe the difficulty of discovering and integrating the system. By deploying a merge queue, teams can eliminate the need for a "check that main works before deployment" step because this is done before the actual merge.

One notable example is a team that previously needed a full-time engineer to manage CI issues due to frequent breaks in the main branch. After adopting Mergify's merge queue, they drastically reduced these disruptions, allowing their engineers to focus on more productive tasks.

The Road Ahead

Merge queues are not without their challenges, and the trade-offs between safety and speed are not always apparent. However, we believe in their potential to transform development workflows. We're on the verge of redefining the merge queue concept at Mergify, and we think it has far greater potential than what has been realized over the past decade. I’ll be happy to write about that soon and share what we’ve built.

Navigating SQL Migrations with Confidence: Introducing sql-compare

Tue, 16 Jul 2024 00:00:00 GMT

As long as I can remember, SQL has been a cornerstone of my engineering journey. My early days at university were filled with monotonous Oracle-based SQL courses, which I found uninspiring. Knowing I would likely never use Oracle, I shifted my focus to MySQL. Over time, I discovered the limitations of MySQL and was introduced to PostgreSQL, thanks to Dimitri. I even organized a few meetups in Paris and encouraged Dimitri to publish "The Art of PostgreSQL," arguably the best book on SQL (I reviewed it here). Eventually, I embraced PostgreSQL wholeheartedly.

SQL databases are a timeless technology that continues to evolve. From Timescale to pgvector, new advancements are continually emerging. However, one persistent challenge has been managing database migrations. Modifying your data model is crucial for evolving your application, but it’s often a daunting task. At Mergify, like many companies, we’ve faced this challenge head-on.

We've tried various solutions, from custom Python scripts to using migra, an open-source project that is unfortunately no longer maintained. Each solution had its drawbacks, leading us to a crossroads where we had to decide on our next move.

The Initial Struggle

At Mergify, PostgreSQL is the backbone of our data handling, from managing the state of GitHub objects to maintaining our event log. From the beginning, we’ve interacted with the database exclusively using an ORM, choosing SQLAlchemy for its maturity, framework agnosticism, and support for asynchronous I/O since version 2.0.0.

Given our frequent production deployments, a robust CI/CD pipeline is essential to handle database evolution smoothly. Every schema modification must be rigorously tested and automatically applied to the production database, adhering to the principles outlined in Martin Fowler's "Evolutionary Database Design." Version-controlling each database artifact and scripting every change as a migration are critical steps in this process.

We chose Alembic to manage our database migrations. Maintained by the SQLAlchemy team, Alembic is a command-line tool that can automatically create migration scripts from your SQLAlchemy models. Each script is version-controlled alongside your source code. Alembic applies these migrations to the database, recording the revision number in the alembic_version table to ensure only new migrations are applied subsequently. This command is typically executed in the continuous delivery pipeline to keep the production database up-to-date.

A Naive Beginning

Our initial approach to testing migration scripts was straightforward: create two databases—one using SQLAlchemy models and the other using only the migration scripts—and ensure they have identical schemas. This involved:

Creating PostgreSQL servers using Docker: On a new server, create two empty databases.
Generating schemas: Use the first database to create artifacts with SQLAlchemy models, and use Alembic to run migration scripts on the second database.
Comparing schemas: Dump each database schema into SQL files using pg_dump and compare them using Python’s filecmp and difflib builtin libraries.

Here’s an example command to dump a database schema into an SQL file:

pg_dump \
    --dbname=postgresql://user:password@host:port/database \
    --schema-only \
    --exclude-table=alembic_version \
    --format=p \
    --encoding=UTF8 \
    --file /path/to/dump.sql

To compare the files:

assert filecmp.cmp(schema_dump_creation_path, schema_dump_migration_path, shallow=False)

If the test fails, use difflib to display the differences:

def filediff(path1: pathlib.Path, path2: pathlib.Path) -> str:
    with path1.open() as f1, path2.open() as f2:
        diff = difflib.unified_diff(
            f1.readlines(),
            f2.readlines(),
            path1.name,
            path2.name,
        )
        return "Database dump differences: \n" + "".join(diff)

While effective, this test had limitations, such as sensitivity to column order. PostgreSQL doesn’t easily allow changing column positions, necessitating consistent column order in models and production databases.

The Complexity Grows

As our models grew more complex, our naive test struggled to keep up. Consider the following example:

class Base:
    updated_at: orm.Mapped[datetime.datetime] = orm.mapped_column(
        sqlalchemy.DateTime(timezone=True),
        server_default=sqlalchemy.func.now(),
    )

class User(Base):
    id: orm.Mapped[int] = orm.mapped_column(
        sqlalchemy.BigInteger,
        primary_key=True,
    )

In this setup, the updated_at column is added to every child model, such as User. Adding a new column to User, like name, would misalign the order, causing schema mismatches.

To address this, we needed to compare schemas while ignoring column order. We explored various tools:

Alembic: Can compare schemas to generate migration scripts but misses some differences.
Migra: An unmaintained tool that compares database schemas effectively.
SQL dumps: The most reliable format but challenging to parse and compare directly.

Building the Solution: sql-compare

It was clear that our current solutions were insufficient. We needed a hero to rescue us from the perils of SQL migration management, so we developed sql-compare.

sql-compare is a Python library that uses sqlparse to parse SQL files and compare schemas, ignoring irrelevant differences like comments, whitespace, and column order. This new tool became an integral part of our workflow, catching migration issues that other tools might miss.

The main challenge was filtering and grouping tokens by column definition before sorting them. Despite these complexities, sql-compare emerged victorious, enabling us to ensure seamless migrations and maintain schema integrity.

The Journey Forward

We’ve open-sourced sql-compare to help others facing similar challenges. You can try it by running pip install sql-compare. We plan to enhance sql-compare, such as creating functions to retrieve all schema differences for better test results. If you have suggestions or want to contribute, feel free to submit issues or pull requests on our GitHub repository.

Conclusion

Managing database migrations is a complex but essential task for evolving applications. With sql-compare, we found our solution, ensuring seamless migrations, maintaining schema integrity, and continuing to deliver high-quality software. Our journey through the challenges of SQL migrations has taught us valuable lessons, and with sql-compare, we’re better equipped to face the future.

The Biggest Mistake We Made Building Mergify: Navigating the Hiring Minefield

Tue, 02 Jul 2024 00:00:00 GMT

This is part 2 of the "Biggest Mistakes" series. Read part 1: Navigating the Payment System Nightmare.

Building a successful startup is a journey filled with unexpected challenges, and hiring the right people is undoubtedly one of the most daunting tasks. As a tech engineer with no HR background, I’ve faced numerous hiring pitfalls that have taught me invaluable lessons. Reflecting on our journey at Mergify, it's clear that navigating the complexities of hiring has been one of the biggest challenges we've encountered. Here’s what we’ve learned.

Who to Hire: The Temptation of Cheap Labor

When you’re a startup with limited resources, it’s tempting to hire interns or apprentices, especially when they come at a low or even zero cost, thanks to government sponsorships. In France, this is an attractive option many startups pursue, and we were no different. Initially, we hired interns and apprentices, believing that this would provide us with much-needed help without straining our budget.

However, we quickly realized that while interns can be a valuable addition, they often lack the expertise we needed to tackle complex tasks. As founders, we required skilled assistance, not just extra hands for minor tasks. The overhead of breaking down projects into manageable tasks and guiding interns through them often resulted in a net loss of productivity. While we did encounter some exceptional interns who became valuable team members, the general rule is that relying on almost-free resources like interns is unlikely to provide the expertise needed in the early days of your startup.

How to Hire: The Challenge of Assessing Candidates

Hiring is more than just finding someone with the right skills; it’s about finding the right fit for your team and company culture. Despite the plethora of online resources on evaluating candidates, the reality of assessing someone over a Zoom call in just a few hours is incredibly challenging.

We've had both terrific and terrible hires, and in each case, we believed they were a perfect fit at the time.

After conducting hundreds of interviews and hiring more than a dozen people, we’ve learned one golden rule: if you have any doubts about a candidate, don’t hire them. It’s better to wait for the right person than to rush and hire someone who might not fit well with your team.

Finding Candidates: The Perils of Recruitment

The search for talent is a constant struggle. In France, we experimented with various solutions, from Welcome to the Jungle (a nightmare) to talent.io (effective). We also engaged a few headhunting firms, which unfortunately turned out to be a costly mistake. These firms often sent us unsuitable candidates and still kept their fees. The legal obligations in France favor the headhunters, not the employers, making it a risky and expensive endeavor.

For example, we had a candidate that we hired and then paid the fees to the headhunting firm. The employee would stop their trial period and leave. That meant the headhunting firm should send new candidates our way to replace the one that left during the trial period. However, there was no incentive for them to do so as they’d been paid already. Therefore, they didn’t care. We found our candidate in a different manner, meaning the fee was then lost for us. Considering that fees can be 10–20% of the employee's yearly salary, that’s a large amount of money to throw out the window.

Networking and recommendations remain some of the best ways to find talent, but they don’t scale well and often have timing issues; you’d find the right candidate, but they’re not available, or a friend would knock at the door, and you’d not have any budget to hire them.

Additionally, we realized that marketing our company effectively during hiring talks is crucial. Initially, our pitch didn’t resonate, and most candidates would ignore us. By improving our presentation and emphasizing our company culture and values, we started attracting genuinely interested candidates.

After years, we discovered that you want to polarize your candidate early in the process and during their employment to ensure they get 100 % onboard with your venture. Depending on your founder profile, that might be something you do naturally. As tech founders, we were not particularly good at that, but we learned our way through.

The Remote Work Dilemma

Building a remote team has advantages, like accessing a broader talent pool, but it also comes with significant challenges. At Mergify, we embraced remote work and wrote extensively about our experience. Remote work works well with senior staff, but junior employees often struggle without in-person guidance. Sharing the company vision and brainstorming ideas are also more effective in person, which is why we regularly organize in-person meetings.

Regular team-building events, video calls, and asynchronous communication help bridge the gap, but they can’t completely replace the spontaneous interactions that foster innovation. Remote work is great for finding talent, but in-person connections remain essential for a cohesive and innovative team environment.

I would not consider remote work a mistake, but we underestimated its impact on the company.

Lessons Learned and Rules Established

From our hiring mistakes, we’ve developed a few key rules:

Avoid hiring remote junior staff if you are working remotely. They need more guidance and in-person interaction.
Leverage in-person connections for innovation. Remote work makes this challenging, especially for junior staff.
Share a lot of context to drive innovation and execution. Overcommunicate.
Be cautious of headhunters and their fees. Consider delaying payment until the trial period ends.
Avoid working with multiple headhunting firms to avoid finding your candidate with one when you have already paid the other.
Learn to pitch your company effectively. Highlight your values and culture to attract the right candidates.
If you have any doubts about a candidate, don’t hire them.
Don’t hire interns and trainees until you have enough senior staff to mentor them. Consider them a small cost, not a huge win.

Navigating the hiring process is complex and fraught with potential pitfalls, but by learning from our mistakes and establishing clear rules, we've been able to build a stronger, more effective team.

If you’re building a startup, remember that the right hires can make all the difference, and taking the time to find them is well worth the effort.

The Biggest Mistake We Made Building Mergify: Navigating the Payment System Nightmare

Tue, 11 Jun 2024 00:00:00 GMT

In 2018, we embarked on an exciting journey with Mergify, our brainchild aimed at simplifying GitHub pull request workflows. One of the first crucial decisions we faced was choosing a payment processor. Stripe, with its developer-centric approach, seemed like the perfect fit. Within a few days, I had mastered the Stripe API and built the foundational billing system for Mergify. For a while, everything ran smoothly as we scaled our user base. However, we soon encountered a significant roadblock: handling VAT in Europe.

European VAT is notoriously complex, with countless edge cases that can quickly become a nightmare for any business. Invoicing internationally from France presented additional challenges, leading us to the conclusion that outsourcing our invoicing would be the best course of action.

The GitHub Marketplace Misstep

In 2019, the GitHub Marketplace appeared to be an attractive solution. It promised to streamline invoicing while exposing Mergify to a broader audience. Although GitHub took a 15% cut (later reduced to 5%), we were not focused on optimizing margins at that stage.

GitHub Marketplace

However, this decision turned out to be a colossal mistake.

Issues with payments began to surface almost immediately. Problems with GitHub’s payment system meant we often had to ask customers to contact GitHub support, creating a frustrating experience for them. Our inability to manage payments directly, such as retrying failed transactions, was a significant drawback. It became evident that while the GitHub Marketplace was a great tool for acquiring new customers, it was far from ideal for handling payments.

As a glaring example, if a GitHub customer switched from credit card billing to invoice, they would lose access to all marketplace apps, including Mergify. This could abruptly cut off our service, leading to dissatisfied customers and lost revenue. By 2020, we decided to completely transition away from the GitHub Marketplace for payments, migrating our customers back to Stripe. This move eliminated the invoicing problems caused by GitHub and allowed us to regain control over our billing process.

The Paddle Predicament

Despite our move back to Stripe, the VAT issue remained unresolved. In our quest for a better solution, we discovered Paddle, a platform that promised to handle VAT by becoming the merchant of record for our transactions. We quickly integrated Paddle into our system, hopeful it would be the solution we needed. Unfortunately, this decision soon proved to be another costly mistake.

Paddle

Paddle's API was far less sophisticated than Stripe’s, and we found ourselves grappling with numerous limitations and workarounds to integrate our billing system. The added complexity and subpar user experience led us to conclude that Paddle was not the right fit for Mergify.

The Turning Point: Handling VAT Ourselves

Realizing that there was no perfect third-party solution, we decided to tackle the VAT problem head-on. In 2020, we took the plunge and developed our own VAT handling system using Stripe and Python. We detailed this process in a blog post sharing our approach and challenges.

Fortunately, Stripe was also working on solving the VAT issue. By the end of 2021, they released their comprehensive tax product, simplifying VAT and other tax processes. This allowed us to finally switch fully to Stripe, discarding our custom code in favor of their robust solution.

Lessons Learned

The most significant lesson from our journey is that payment processing is not a mere detail—it’s an integral part of the user experience. Even now, I spend several hours each month resolving payment issues, from credit card problems to ensuring invoices are correctly fed into various supplier systems. While automation can handle many aspects, the unique methods and systems of each customer often require personalized solutions.

Our experience underscores the importance of keeping things simple on your side and minimizing friction for your customers. Ensuring a smooth and reliable payment process is crucial for maintaining customer satisfaction and loyalty.

In hindsight, we should have approached the payment system with the same rigor and attention to detail as the rest of our product.

It's a lesson we learned the hard way, but one that has ultimately strengthened Mergify and our commitment to providing the best possible service for our users.

This is part 1 of the "Biggest Mistakes" series. Read part 2: Navigating the Hiring Minefield.

Sponsoring Conferences

Thu, 06 Jun 2024 00:00:00 GMT

Last week, I wrote about my experience attending conferences.

Over the last year, we've tried to expose Mergify at conferences to reach out to developers. We’ve done various conferences in Europe and the US—the largest being QCon San Francisco 2023 and Devoxx France 2024. We sponsored those events and ran booths for several days all day long to talk to engineers.

Mergify booth at QCon San Francisco 2023

The pattern we’ve seen has been interesting. First, QCon San Francisco was the smallest it’s been over the last few years, as far as I can tell. While 1,400 people were expected, counting the number of people seated in the keynote session revealed less than half of that were present. We talked to tens of engineers without great success. It turns out that trying to sell your tool for an early startup like Mergify is not efficient at all in such a place. Companies tend to do that when they are way larger to raise brand awareness and penetrate the market more efficiently.

At our stage, this was a lot of money spent for barely any gain.

As Mehdi, my cofounder and CTO says, “no great engineer will go to a conference to find the next tool they’ll need.” Indeed, I don’t believe any good engineer should wait 6 months for the next conference they will attend to find a product to their technical problems.

Doing market education, as we tried, over a booth, is utopian. Here’s the typical dialogue that would happen:

– Engineer attending the conference: “Hi! What does Mergify do?”
– Mergify staff: “We offer merge queues for your GitHub repository. Do you know about them?”
– Engineer attending the conference trying not to lose face: “Yeah, for sure!”
– Mergify staff: “Do you use one in your team?”
– Engineer: “No, we don’t need one.”
– Mergify: "How so? You're happy merging outdated PR or running a lot of CI on every PR?"
– Engineer: "We… don't… well… err… what are we talking about exactly?…"

The truth is, 95% of the engineers we talked to have no clue what a merge queue is. Actually, 80% of them don’t know anything about Git besides the basics (i.e., commit and push), and chatting for 10 minutes over a booth is not a good place to educate them.

Speaking at conferences is a way better strategy, as the advent of the Developer Evangelist role has demonstrated over the last years. If well executed, it’s cheaper and can have a far better outcome than sponsoring an event.

You could imagine that sponsoring an event buys you a ticket to speak, but it’s not the case by default. Some conferences allow you to buy speaking time in special, dedicated rooms, for example, but you usually don’t get any special treatment over the regular CfP.

I really need to talk about that CfP game.

One year of Mergify

Thu, 12 Mar 2020 00:00:00 GMT

It has been close to a year now that I've incorporated my new company, Mergify. I've been busy, and I barely wrote anything about it so far. Now is an excellent time to take a break and reflect a bit on what happened during those last 12 months.

What problem does Mergify solve?

Mergify is a powerful automation engine for GitHub pull requests. It allows you to automate everything — and especially merging. You write rules, and it handles the rest.

For example, let's say you want your pull request to be merged, e.g., once your CI passes and the pull request has been approved. You just write such a rule, and our engine merges the pull request as soon as it's ready.

We also deal with more advanced use cases. For instance, we provide a merge queue so your pull requests are merged serially and tested by your CI one after another — avoiding any regression in your code.

Our goal is to make pull request management and automation easy. You can use your bot to trigger a rebase of your pull requests, or a backport to a different branch, just with a single comment.

A New Adventure

Mergify is the first company that I ever started. I did run some personal businesses before, created non-profit organizations, built FOSS projects — but I never created a company from scratch, even less with an associate.

Indeed, I've chosen to build the company with my old friend Mehdi. We've known each others for 7 years now, and have worked together all that time on different open-source projects. Having worked with each other for so long has probably been a critical factor in the success of our venture so far.

I had little experience sharing the founding seats with someone, and tons of reading seemed to indicate that it would be a tough ride. Picking the right business partner(s) can be a hard task. Luckily, after working so much time together, Mehdi and I both know our strengths and weaknesses well enough to be able to circumvent them. 😅

On the other hand, we both have similar backgrounds as software engineers. That does not help to cover all the hats you need to wear when building a company. Over time, we found arrangements to cover most of those equally between us.

We don't have any magical advice to give on this. As in every relationship, communication is the key, and the #1 factor of success.

Getting Users

I don't know if we got lucky, but we got users and customers pretty early. We used a few cooperative projects as guinea pigs first, and they were brave enough to try our service and give us feedback. No repository has been harmed during this first phase!

Then, as soon as we managed to get our application on the GitHub Marketplace, we saw a steady number of users coming to us.

This has been fantastic as it allowed us to get feedback rapidly. We set up a form asking users for feedback after they used Mergify for a couple of weeks. What we hear is that users were happy, that the documentation was confusing and that some features were buggy or missing. We planned all of those ideas as our future work in our roadmap, using the principles we described a few months ago.

We tried various strategies to get new users, but so far, organic growth has been our #1 way of onboarding new users. Like many small startups out there, we're not that good at marketing and executing strategies.

We provide our service for free for open-source projects We are now powering many organizations, such as Mozilla, Amazon Web Services, Ceph and Fedora.

Working with GitHub

Working with GitHub has been… complicated. When you build an application for a marketplace, your business is entirely dependent on the platform you develop for — both in terms of features and quality of service.

In our case, we hit quite many bugs with GitHub. Their support has mostly been fast to answer, but some significant issues are still opened months later. The truth is that the GitHub API could deserve more love and care from GitHub. For example, their GraphQL API is a work in progress for years and miss out many essential features.

We dealt and still deal with all those issues. It obviously impacts our operations and decreases our overall velocity. We regularly have to find new ways to sidestep GitHub limitations.

You have no idea how we wished for GitHub to be open-source. The idea of not having access to their code and understand how it works is so frustrating that we publish our engine as an open-source project. That allows all of our users to see how it works and even propose enhancements.

Automate all the way

We're a tiny startup, and we decided to bootstrap our company. We never took any funding. From the beginning, it has been clear to us that we had to think and act like we had no resources. We're built around a scarcity mindset. Every decision we make is based on the assumption that we basically are very limited in terms of money and time.

We basically act like any wrong choice we do could (virtually) kill the company. We only do what is essential, we ship fast, and we automate everything.

For example, we have built our whole operation about CI/CD systems, and pushing any new fix or feature in production is done in a matter of minutes. It's not uncommon for us to push a fix from our phone, just by reviewing some code or editing a file.

Growth

We're extremely happy with our steady growth and more users using our service. We now manage close to 30k repositories and merge 15k pull requests per month for our users.

That's a lot of mouse clicks saved!

If you want to try Mergify yourself, it's a single click log-in using your GitHub account. Check it out!

Python Logging with Datadog

Mon, 03 Feb 2020 00:00:00 GMT

At Mergify, we generate a pretty large amount of logs. Every time an event is received from GitHub for a particular pull request, our engine computes a new state for it. Doing so, it logs some informational statements about what it's doing — and any error that might happen.

This information is precious to us. Without proper logging, it'd be utterly impossible for us to debug any issue. As we needed to store and index our logs somewhere, we picked Datadog as our log storage provider.

Datadog offers real-time indexing of our logs. The ability to search our records that fast is compelling as we're able to retrieve log about a GitHub repository or a pull request with a single click.

To achieve this result, we had to inject our Python application logs into Datadog. To set up the Python logging mechanism, we rely on daiquiri, a fantastic library I maintained for several years now. Daiquiri leverages the regular Python logging module, making its a no-brainer to set up and offering a few extra features.

We recently added native support for the Datadog agent in daiquiri, making it even more straightforward to log from your Python application.

Enabling log on the Datadog agent

Datadog has extensive documentation on how to configure its agent. This can be summarized to adding logs_enabled: true in your agent configuration. Simple as that.

You then need to create a new source for the agent. The easiest way to connect your application and the Datadog agent is using the TCP socket. Your application will write logs directly to the Datadog agent, which will forward the entries to Datadog backend.

Create a configuration file in conf.d/python.d/conf.yaml with the following content:

Setting up `daiquiri`

Once this is done, you need to configure your Python application to log to the TCP socket configured in the agent above.

The Datadog agent expects logs in JSON format being sent, which is what daiquiri does for you. Using JSON allows to embed any extra fields to leverage fast search and indexing. As daiquiri provides native handling for extra fields, you'll be able to send those extra fields without trouble.

First, list daiquiri in your application dependency. Then, set up logging in your application this way:

import daiquiri

daiquiri.setup(
  outputs=[
    daiquiri.output.Datadog(),
  ],
  level=logging.INFO,
)

This configuration logs to the default TCP destination localhost:10518 — though you can pass the host and port argument to change that. You can customize the outputs as you wish by checking out daiquiri documentation. For example, you could also include logging to stdout by adding daiquiri.output.Stream(sys.stdout) in the output list.

Using `extra`

When using daiquiri, you're free to use logging.getLogger to get your regular logging object. However, by using the alternative daiquiri.getLogger function, you're enabling the native use of extra arguments — which is quite handy. That means you can pass any arbitrary key/value to your log call, and see it up being embedded in your log data — up to Datadog.

Here's an example:

import daiquiri

[…]

log = daiquiri.getLogger(__name__)
log.info("User did something important", user=user, request_id=request_id)

The extra keyword argument passed to log.info will be directly shown as attributes in Datadog logs:

All those attributes can then be used to search or to display custom views. This is really powerful to monitor and debug any kind of service.

A log object per object

When passing extra arguments, it is easy to make mistakes and forget some. This especially can happen when your application wants to log information for a particular object.

The best pattern to avoid this is to create a custom log object per object:

import daiquiri

class MyObject:
    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.log = daiquiri.getLogger("MyObject", x=self.x, y=self.y)

    def do_something(self):
        try:
            self.call_this()
        except Exception:
            self.log.error("Something bad happened")

By using the self.log object as defined above, there's no way for your application to miss some extra fields for an object. All your logs will look in the same style and will end up being indexed correctly in Datadog.

Log Design

The extra arguments from the Python loggers are often dismissed, and many developers stick to logging strings with various information included inside. Having a proper explanation string, plus a few extra key/value pairs that are parsable by machines and humans, is a better way to do logging. Leveraging engines such as Datadog allow to store and query those logs in a snap.

This is way more efficient than trying to parse and grep strings yourselves!

Dependencies Handling in Python

Mon, 02 Sep 2019 00:00:00 GMT

Dependencies are a nightmare for many people. Some even argue they are technical debt. Managing the list of the libraries of your software is a horrible experience. Updating them — automatically? — sounds like a delirium.

Stick with me here as I am going to help you get a better grasp on something that you cannot, in practice, get rid of — unless you're incredibly rich and talented and can live without the code of others.

First, we need to be clear of something about dependencies: there are two types of them. Donald Stuff wrote better than I would about the subject years ago. To make it simple, one can say that they are two types of code packages depending on external code: applications and libraries.

Libraries Dependencies

Python libraries should specify their dependencies in a generic way. A library should not require requests 2.1.5: it does not make sense. If every library out there needs a different version of requests, they can't be used at the same time.

Libraries need to declare dependencies based on ranges of version numbers. Requiring requests>=2 is correct. Requiring requests>=1,<2 is also correct if you know that requests 2.x does not work with the library. The problem that your version range specification is solving is the API compatibility issue between your code and your dependencies — nothing else. That's a good reason for libraries to use Semantic Versioning whenever possible.

Therefore, dependencies should be written in setup.py as something like:

from setuptools import setup

setup(
    name="MyLibrary",
    version="1.0",
    install_requires=[
        "requests",
    ],
    # ...
)

This way, it is easy for any application to use the library and co-exist with others.

Applications Dependencies

An application is just a particular case of libraries. They are not intended to be reused (imported) by other libraries of applications — though nothing would prevent it in practice.

In the end, that means that you should specify the dependencies the same way that you would do for a library in the application's setup.py.

The main difference is that an application is usually deployed in production to provide its service. Deployments need to be reproducible. For that, you can't solely rely on setup.py: the requested range of the dependencies are too broad. You're at the mercy of random version changes at any time when re-deploying your application.

You, therefore, need a different version management mechanism to handle deployment than just setup.py.

pipenv has an excellent section recapping this in its documentation. It splits dependency types into abstract and concrete dependencies: abstract dependencies are based on ranges (e.g., libraries) whereas concrete dependencies are specified with precise versions (e.g., application deployments) — as we've just seen here.

Handling Deployment

The requirements.txt file has been used to solve application deployment reproducibility for a long time now. Its format is usually something like:

requests==3.1.5
foobar==2.0

Each library sees itself specified to the micro version. That makes sure each of your deployment is going to install the same version of your dependency. Using a requirements.txt is a simple solution and a first step toward reproducible deployment. However, it's not enough.

Indeed, while you can specify which version of requests you want, if requests depends on urllib3, that could make pip install urllib 2.1 or urllib 2.2. You can't know which one will be installed, which does not make your deployment 100% reproducible.

Of course, you could duplicate all requests dependencies yourself in your requirements.txt, but that would be madness!

There are various hacks available to fix this limitation, but the real saviors here are pipenv and poetry. The way they solve it is similar to many package managers in other programming languages. They generate a lock file that contains the list of all installed dependencies (and their own dependencies, etc.) with their version numbers. That makes sure the deployment is 100% reproducible.

Check out their documentation on how to set up and use them!

Handling Dependencies Updates

Now that you have your lock file that makes sure your deployment is reproducible in a snap, you've another problem. How do you make sure that your dependencies are up-to-date? There is a real security concern about this, but also bug fixes and optimizations that you might miss by staying behind.

If your project is hosted on GitHub, Dependabot is an excellent solution to solve this issue. Enabling this application on your repository creates automatically pull requests whenever a new version of the library listed in your lock file is available. For example, if you've deployed your application with redis 3.3.6, Dependabot will create a pull request updating to redis 3.3.7 as soon as it gets released. Furthermore, Dependabot supports requirements.txt, pipenv, and poetry!

Automatic Deployment Update

You're almost there. You have a bot that is letting you know that a new version of a library your project needs is available.

Once the pull request is created, your continuous integration system is going to kick in, deploy your project, and runs the test. If everything works fine, your pull request is ready to be merged. But are you really needed in this process?

Unless you have a particular and personal aversion on specific version numbers —"Gosh I hate versions that end with a 3. It's always bad luck."— or unless you have zero automated testing, you, human, is useless. This merge can be fully automatic.

This is where Mergify comes into play. Mergify is a GitHub application allowing to define precise rules about how to merge your pull requests. Here's a rule that I use in every project:

pull_requests_rules:
  - name: automatic merge from dependabot
    conditions:
      - author~=^dependabot(|-preview)\[bot\]$
      - label!=work-in-progress
      - "status-success=ci/circleci: pep8"
      - "status-success=ci/circleci: py37"
    actions:
      merge:
        method: merge

As soon as your continuous integration system passes, Mergify merges the pull request for you.

You can then automatically trigger your deployment hooks to update your production deployment and get the new library version installed right away. This leaves your application always up-to-date with newer libraries and not lagging behind several years of releases.

If anything goes wrong, you're still able to revert the commit from Dependabot — which you can also automate if you wish with a Mergify rule.

Beyond

This is to me the state of the art of dependency management lifecycle right now. And while this applies exceptionally well to Python, it can be applied to many other languages that use a similar pattern — such as Node and npm.

More GitHub workflow automation

Tue, 16 Oct 2018 00:00:00 GMT

The more you use computers, the more you see the potentials for automating everything. Who doesn't love that? By building Mergify those last months, we've decided it was time bring more automation to the development workflow.

Mergify's first version was a minimal viable product around automating the merge of pull requests. As I wrote a few months ago, we wanted to automate the merge of pull requests when it was ready to be merged. For most projects, this is easy and consists of a simple rule: "it must be approved by a developer and pass the CI".

Evolving on Feedback

For the first few months, we received a lot of feedback from our users. They were enthusiastic about the product but were frustrated by a couple of things.

First, Mergify would mess up with branch protections. We thought that people wanted the GitHub UI to match their rules. As I'll explain later, it turns out to be only partially true, and we found a workaround.

Then, Mergify's abilities were capped by some of the limitations of the GitHub workflow and API. For example, GitHub would only allow rules per branch, whereas our users wanted to have rules applied based on a lot of different criteria.

Building the Next Engine

We rolled up our sleeves and started to build that new engine. The first thing was to get rid of the GitHub branch protection feature altogether and leveraging the Checks API to render something useful to the users in the UI. You can now have a complete overview of the rules that will be applied to your pull requests in the UI, making it easy to understand what's happening.

Then, we wrote a new matching engine that would allow matching any pull requests based on any of its attributes. You can now automate your workflow with a finer-grained configuration.

What Does It Look Like?

Here's a simple rule you could write:

pull_request_rules:
  - name: automatic merge on approval and CI pass
    conditions:
     - "#approved-reviews-by>=1"
     - status-success=continuous-integration/travis-ci/pr
     - label!=work-in-progress
    actions:
      merge:
        method: merge

With that, any pull request that has been approved by a collaborator, passes the Travis CI job and does not have the label work-in-progress will be automatically merged by Mergify.

You could use even more actions to backport this pull request to another branch, close the pull request or add/remove labels. We're starting to see users building amazing workflow with that engine!

We're thrilled by this new version we launched this week and glad we're getting amazing feedback (again) from our users.

When you give it a try, drop me a note and let me know what you think about it!

How I stopped merging broken code

Tue, 03 Jul 2018 00:00:00 GMT

It's been a while since I moved all my projects to GitHub. It's convenient to host Git projects, and the collaboration workflow is smooth.

I love pull requests to merge code. I review them, I send them, I merge them. The fact that you can plug them into a continuous integration system is great and makes sure that you don't merge code that will break your software. I usually have Travis-CI setup and running my unit tests and code style check.

The problem with the GitHub workflow is that it allows merging untested code.

What?

Yes, it does. If you think that your pull requests, all green decorated, are ready to be merged, you're wrong.

You see, pull requests on GitHub are marked as valid as soon as the continuous integration system passes and indicates that everything is valid. However, if the target branch (let's say, master) is updated while the pull request is opened, nothing forces to retest that pull request with this new master branch. You think that the code in this pull request works while that might have become false.

So it might be that what went into your master branch now breaks this not-yet-merged pull request. You've no clue. You'll trust GitHub and press that green merge button, and you'll break your software. For whatever reason, it's possible that the test will break.

The good news is that's something that's solvable with the strict workflow that Mergify provides. There's a nice explanation and example in Mergify's blog post You are merging untested code that I advise you to read. What Mergify provides here is a way to serialize the merge of pull requests while making sure that they are always updated with the latest version of their target branch. It makes sure that there's no way to merge broken code.

That's a workflow I've now adopted and automatized on all my repositories, and we've been using such a workflow for Gnocchi for more than a year, with great success. Once you start using it, it becomes impossible to go back!

Stop merging your pull requests manually

Wed, 20 Jun 2018 00:00:00 GMT

If there's something that I hate, it's doing things manually when I know I could automate them. Am I alone in this situation? I doubt so.

Nevertheless, every day, they are thousands of developers using GitHub that are doing the same thing over and over again: they click on this button:

This does not make any sense.

Don't get me wrong. It makes sense to merge pull requests. It just does not make sense that someone has to push this damn button every time.

It does not make any sense because every development team in the world has a known list of pre-requisite before they merge a pull request. Those requirements are almost always the same, and it's something along those lines:

Is the test suite passing?
Is the documentation up to date?
Does this follow our code style guideline?
Have N developers reviewed this?

As this list gets longer, the merging process becomes more error-prone. "Oops, John just clicked on the merge button while there were not enough developer that reviewed the patch." Rings a bell?

In my team, we're like every team out there. We know what our criteria to merge some code into our repository are. That's why we set up a continuous integration system that runs our test suite each time somebody creates a pull request. We also require the code to be reviewed by 2 members of the team before it's approbated.

When those conditions are all set, I want the code to be merged.

Without clicking a single button.

That's exactly how Mergify started.

Mergify is a service that pushes that merge button for you. You define rules in the .mergify.yml file of your repository, and when the rules are satisfied, Mergify merges the pull request.

No need to press any button.

Take a random pull request, like this one:

This comes from a small project that does not have a lot of continuous integration services set up, just Travis. In this pull request, everything's green: one of the owners reviewed the code, and the tests are passing. Therefore, the code should be already merged: but it's there, hanging, chilling, waiting for someone to push that merge button. Someday.

With Mergify enabled, you'd just have to put this .mergify.yml a the root of the repository:

rules:
  default:
    protection:
      required_status_checks:
        contexts:
          - continuous-integration/travis-ci
      required_pull_request_reviews:
        required_approving_review_count: 1

With such a configuration, Mergify enables the desired restrictions, i.e., Travis passes, and at least one project member reviewed the code. As soon as those conditions are positive, the pull request is automatically merged.

We built Mergify as a free service for open-source projects. The engine powering the service is also open-source.

Now go check it out and stop letting those pull requests hang out one second more. Merge them!

If you have any question, feel free to ask us or write a comment below! And stay tuned — as Mergify offers a few other features that I can't wait to talk about!

A safe GitHub workflow with Pastamaker

Fri, 15 Dec 2017 00:00:00 GMT

When the Gnocchi project decided to move to GitHub, we developers had to move from a Gerrit based workflow to a GitHub pull-request one.

This has been challenging in some ways. We were satisfied with the workflow we had using Gerrit and Zuul for testing so we decided to adapt GitHub to our requirements.

We know that Zuul now supports GitHub. However, that implies having your own testing infrastructure, something we can't afford. Instead, we rely on Travis, like most open-source projects hosted on GitHub.

The workflow

The workflow we wanted to have was the following:

A contributor creates a pull-request on GitHub.
The pull-request is tested by Travis.
The pull-request is reviewed by approved projects members.
If the tests pass and two reviewers have approved the pull-request, then it
can be merged.

This sounds simple, but it is actually not that simple.

First, when Travis tests the pull-request, it checks what has been sent by the contributor. If the contributor created a pull-request on top of an outdated version of the base branch, that's what will be tested by Travis during the initial pull-request creation.

Even if the pull-request has been created using the tip of the base branch, as time passes, the base branch will progress. However, the pull-request created by your contributor will not get those new commits – unless rebased manually.

That means the Travis tests result is now outdated invalid. Still, GitHub and Travis will both show you that this pull-request passed all tests – yes it did but with an old base branch from a while back!

If you added new tests in the meantime in your base branch, it's possible that this pull-request does not work anymore. Pressing the merge button might just break your project!

To help with that problem, GitHub recently added a button that allows you to base branch into the pull-request. That allows, in one click, to get the pull-requested updated with the base branch (e.g., master) and retested by Travis.

Still, this means that if you have ten pull-requests, you need to:

Merge base branch into PR#1
Wait for Travis to pass
Wait for two reviewers to approve
Merge PR#1
All other nine pull-requests are not out of date. You need to do start back at operation 1. for each pull-request.

This is very tedious to do manually, especially when your projects has tons of pull-requests.

This is why Mehdi Abaakouk created Pastamaker.

Pastamaker to the rescue

Pastamaker is a small Web application that implements the described workflow. Once connected to your GitHub project, it will set the proper permissions to protect it for accidental manual merge and force the workflow above to be followed.

Pastamaker listens for GitHub and Travis events to track the state of each pull-request. If it detects that a pull-request has been approved by two reviewers and that the initial Travis test run passed, it will merge the base branch if needed in it, wait for Travis to pass again, and then finally merge it.

If multiple pull-requests are approved at the same time and are candidates for a merge, it will order them, update once at a time, wait for Travis results and merge them if they pass. It essentially automates the workflow described above.

Pastamaker exposes its data via a simple dashboard, which allows seeing all the pull-requests for your project in a snap.

Pastamaker offers a lot of tiny other details that make the developers lives easier, such as posting the job result with direct links to the jobs logs in the pull-request – so you're informed as soon as they pass or fail and can fix them right away!

Pastamaker is obviously open-source, and we would love to see you give it a try!