From Failure to Focus
How CI Insights Was Born
Startups love to talk about iteration. Failing fast. Learning from mistakes. But when you’re six months into a project that doesn’t ship — not once, but twice — that mantra starts to feel a bit too real.
At Mergify, we recently spent almost a year building two separate products around CI/CD. Both had potential. Both looked promising. And both ended up in the graveyard.
I have written about this in three posts already:
But here’s the thing: that journey was the best thing that could’ve happened to us because it led to CI Insights, our newest product, which we’re shipping for real this time.
When R&D Goes Off-Road
If you’ve followed the first parts of this series, you know the backstory. In 2023, after years of growing Mergify’s Merge Queue product, we started exploring new ideas.
The first was CI Optimizer — a tool to help teams reduce CI/CD costs. But we quickly learned something important: engineers aren’t the ones who care about CI spend. That’s a FinOps conversation. And FinOps teams weren’t our users.
Then came CI Issues, a project aimed at tracking flaky tests and infrastructure problems. This time we had interest. Teams did struggle with these problems. But we made the mistake of diving into R&D without doing proper design work. We built a complex system — and it worked — but it was so hard to deploy and operate that we never felt confident letting users in.
So once again, we shelved it.
Two product attempts, zero releases.
But what survived both efforts was a deeper understanding of the pain engineers feel every day around CI.
The Real Problem: CI Is a Black Box
Across all our conversations, one theme kept showing up: visibility.
Teams weren’t desperate to reduce CI costs. They weren’t obsessed with infrastructure failures. But they were all asking the same questions:
Why is our CI pipeline so slow?
Which PRs are consistently the bottleneck?
Which tests are flaky?
Where are we wasting time?
Nobody had good answers. CI is treated like a utility — flip the switch and hope the light turns on. But when it doesn’t, or when it flickers, most teams don’t have the tools to understand why.
We realized what was missing wasn’t a FinOps tool or a smart test tracker — it was observability.
CI Insights: The Missing Layer
With all the groundwork we’d laid — our CI connectors, our data pipelines, our internal dashboards — we already had most of the pieces. We just needed to reframe the problem.
So we did.
CI Insights is the observability layer for your CI.
It helps teams:
Spot flaky jobs and tests
Identify long-running or unstable jobs
Understand where their pipeline is slowing them down
Track trends over time across teams and repos
It’s not about cost savings. It’s not about blaming the CI tool. It’s about clarity — understanding what’s going on, so teams can ship faster and with less frustration.
This Time, We Built It Right
We learned from our previous mistakes.
This time, we:
Started with real use cases from customers and our own internal needs
Designed first, coded second
Focused on value over complexity
Shipped it to users. Yes. Really.
We’ve been using CI Insights ourselves for months now. It already helped us catch flaky jobs, detect broken test workflows, and reduce merge queue delays.
Now, we’re rolling it out to early users — and so far, the feedback has been 🔥.
The Bigger Picture
CI Insights is more than just a tool. It’s a shift in how we think about CI.
It’s not just a thing that “runs your tests.” It’s a critical part of your development workflow. And it deserves the same kind of visibility, metrics, and tooling that you already have for production systems.
We’re building CI Insights to be the best obervability tool for CI — because engineers deserve better tools.
What’s Next?
We’re just getting started. The roadmap is full. We’re onboarding users slowly and shaping the product based on real feedback.
If CI is a black box for your team — if you’re tired of guessing why things are slow — we’d love to hear from you.