The $100,000 Mistake

How We Spent 6 Months Building a CI Tool Nobody Asked For

A Journey into Startup Reality

Not every startup success story begins with a garage, two co-founders, and an overnight explosion of users. And not every failure is a dramatic, fiery crash. Some of the most valuable lessons happen in the quieter moments—when you’ve built something, spent months refining it, and then realized you were solving the wrong problem all along.

This is the story of CI Optimizer, a product we believed would transform the way companies managed their CI/CD costs. We spent six months designing, building, and testing it—only to ultimately kill it before launch.

Why? Because we made one fundamental mistake.

This three-part series is not just about the technical challenges of optimizing CI/CD or the intricacies of pricing cloud infrastructure. It’s about the hard reality of building a product, talking to customers, and realizing you missed the mark.

If you’re an entrepreneur, a product builder, or just someone fascinated by the messy, unpredictable world of startups, this series is for you.

Let’s start at the beginning.

How It Started

At the end of 2022, Mergify was on a roll.

We had spent the past year growing steadily, tripling our revenue, refining our Merge Queue product, and deepening our place in the DevOps ecosystem. Our customers were engaged, our product-market fit felt strong, and our team fired on all cylinders.

But as the year drew to a close, something in the air felt different.

Conversations with prospects were shifting. Instead of discussing new features and scaling up their usage, they were hesitant. Startups—our core audience—were tightening their budgets. Investors were slowing down. Some of our customers simply disappeared.

First, it was the crypto companies. Then, real estate tech. One by one, they went silent—not because they didn’t love our product, but because their businesses were collapsing. The market was crashing. Funding was drying up.

The message became clear:

“We love Merge Queue, but our budget is frozen.”

We knew we couldn’t just sit back and hope the market would bounce back. We needed to adapt. That’s how startups survive.

And that’s when we had what we thought was a brilliant idea.

The Birth of CI Optimizer

One undeniable truth about software engineering is that CI/CD is expensive.

Every build, every test, every deployment—it all costs money.

At scale, those costs grow exponentially, often without teams fully understanding where their budget is going. Developers push a change, run a full test suite, and move on. But in the background, cloud bills are racking up, and finance teams are left wondering where all that money is going.

So we thought:

“What if we built a tool that gave teams complete visibility into their CI/CD costs? What if we could identify waste, eliminate unnecessary builds, and optimize pipelines automatically?” What if we could help companies save money on their CI without slowing them down?”

It sounded like a no-brainer. We’d build a smart system that could analyze CI/CD usage and recommend cost-saving adjustments—maybe even automate them.

This is something we could even use ourselves to save on our CI/CD bills.

This wasn’t just an idea—we were convinced we had struck gold.

Building the Future of CI/CD Cost Optimization

By early 2023, we had begun prototyping.

The first step? Connecting to GitHub Actions. GitHub, like many CI/CD providers, is well known for not providing a good report on cost analysis (still true as of today). Since GitHub is where all of our customers are, this made sense.

We needed to pull in detailed CI/CD usage data and break down the cost per minute of every build. Our system would scan pipelines, report metrics, identify inefficiencies, and provide actionable insights—like which jobs were wasting the most money.

It felt like a natural extension of what I had worked on years earlier at Datadog, where I had pushed for replacing CPU seconds with dollar values in profiling tools. The goal was simple: make CI/CD costs tangible, trackable, and optimizable.

We saw an opportunity to build something that would fit neatly into engineering workflows. The logic was airtight:

✅ Developers hate waiting for builds.

✅ Developers need to get more budget.

✅ We could solve both problems at once.

Or so we thought.

What We Hoped to Prove

Our plan for early 2023 was straightforward:

1️⃣ Build an MVP that could accurately measure CI/CD costs at a granular level.

2️⃣ Talk to real users and validate whether CI/CD engineers cared about cost optimization.

3️⃣ Launch a first version of CI Optimizer and start onboarding teams.

We expected engineers to tell us:

“This is amazing! We’ve been waiting for something like this!”

But that’s not what happened.

Instead, we hit an unexpected roadblock that completely changed the course of the project.