coding — jd:/dev/blog

So I Will Never Write Code Again

Tue, 10 Feb 2026 00:00:00 GMT

A year ago, I thought AI-assisted coding was going to be a nice productivity boost. Generate a Python script with ChatGPT, copy-paste it somewhere, save twenty minutes. I figured that was the next five years: small wins, gradual improvement.

Then last August, I wrote a feature where Copilot did about 80% of the work. I thought: okay, it's getting closer.

Since January, I haven't written a single line of code.

I want to be precise: I've produced a lot of code. More than ever, probably. But I didn't write any of it. I steer. I review. I architect. I don't type.

And I don't feel the urge to go back.

This might sound like grief. I've been coding for 25 years. I wrote C for a window manager, Lisp for Emacs, Python for everything else. For most of my career, coding was a thing that defined me. Losing that should feel like losing a part of myself.

But it doesn't. It feels like relief.

For years, I was frustrated. I had more ideas than I could build. The bottleneck was never thinking, it was typing. Translating architecture into syntax, aligning parentheses, naming variables, fighting linters. The fun was in the solving, not the writing. And now the writing part is handled.

I still enjoy reading code. It's like reading a good book. Understanding how pytest works internally, tracing through a complex system, that remains satisfying. But when the goal is to produce, AI beats everything.

This is actually the second time I've stepped away from code. The first was when I became CEO. That time, it was forced. I didn't choose to stop. I just ran out of hours. There was always one more meeting, one more hire, one more decision that pushed coding to the evening, then to the weekend, then to never.

That was grief. A slow, reluctant surrender.

This time is different. I'm not being pushed away. I'm choosing to work at a higher layer. The same way I once chose Python over C, because life is short and the abstraction was worth it. AI is just the next rung.

The creativity doesn't stop. If anything, it accelerates. You still design systems, still make architectural choices, still think about data models and trade-offs. You just don't spend hours translating those decisions into semicolons. The craft moves up a level, and that's fine.

I know this will be harder for others. My colleague Rémy wrote about whether AI is killing craftsmanship. For engineers who defined themselves by the elegance of their code, by the perfectly named function, by the satisfaction of a clean diff, this shift feels like losing something sacred.

I get it. Writing C was a beautiful puzzle. Lisp was genuinely fun. And I still think learning to code by hand matters, the same way learning assembly helps you understand memory even if you never write it professionally.

But I'm not going to fight a paradigm shift out of nostalgia. The ride was great. The next one looks better.

I think the flow state people mourn isn't gone. It's just moving. Steering AI toward clean architecture, making the right system-level decisions, reviewing output with deep context, that has its own rhythm. The interruptions are still too frequent today (too many permission prompts), but the direction is clear. The flow will come back. It'll just be at a different altitude.

If you're a senior engineer feeling this shift approaching, here's what I'd say: the grief you're expecting might not be grief at all. The bottleneck was never the thinking. It was the typing. And the thinking is still yours.

AI Won’t Kill Juniors. It Will Expose Seniors.

Wed, 21 Jan 2026 00:00:00 GMT

The tech industry has a new consensus: AI will kill junior engineering jobs. Look at any discussion thread, and you’ll find the same narrative. Juniors are doomed. They’ll never learn to code properly. The entry-level pipeline is broken.

I’m not so sure. When I look at junior engineers today, I see people who are used to learning. They came up through boot camps, YouTube tutorials,and constantly shifting frameworks. Adapting is what they do. They might struggle for a year or two, but they’ll figure it out.

The engineers I’m worried about are the senior ones.

Sure, not all of them. But the ones who plateaued at “code craftsman” and never moved up.

I’ve seen it play out already. A standup where someone proudly reports they spent the day fixing a batch of bugs and shipping a couple of pull requests. The rest of the team glances at each other. They’re thinking: *that’s ten minutes of Claude Code. Why did you spend eight hours in your IDE?*

This isn’t new. We’ve seen it before. When bash gave way to Perl. When Java replaced C for most applications. Every paradigm shift leaves some people behind. Maybe 10%, maybe 20%, clinging to the old way because it’s what they know.

But AI is different. The shift is faster. The impact is more massive. And the reach is exponential.

Here’s the pattern I see. When I started programming, you’d learn assembly. Then you’d switch to C because life is short. Then Python, because life is really short. Each jump felt like cheating to the previous generation, and each one freed you to think at a higher level.

AI is the next rung on that ladder. I hope schools are teaching this now: learn to write code by hand first (you need to understand what you’re abstracting), then switch to AI-assisted development. Just like you learned assembly to understand memory, then moved on. Though knowing how slow institutions adapt, I’m not holding my breath.

The engineers who get this are thriving. Staff engineers, principal engineers, people whose job was already 70% architecture, cross-team coordination, and system design. They only coded 30% of the time anyway. Now they use AI to multiply that 30% and have even more impact. For them, AI is a force multiplier on an already leveraged role.

But there’s another group. Senior engineers, five to ten years in, who still think their job is writing code 90% of the time. They never thought deeply about data models. Never cared much about architecture. Never moved toward the work that would make them staff or principal.

Their entire value was "writing proper, clean code that runs well and passes the linter." They never invested in the skills that make a great software engineer — communication, system thinking, judgment.

That value just evaporated.

And here’s what makes it worse: working with AI is fundamentally communication work. The engineers who thrive are the ones who already know how to share context, explain problems to colleagues, and filter signal from noise across teams.

I’ve watched engineers struggle with AI because they won’t invest in communication. They type “fix this bug” without the stack trace, without the constraints, without explaining how production differs from their local setup. They keep the context in their head because explaining feels costly. The result is garbage, and they blame the tool.

What they don’t see: AI compounds. The more context you feed it about your project, the better it gets. But that requires upfront investment in articulation. If you spent your career avoiding that investment with humans, you’ll prevent it with AI too.

I don’t have a clean solution. The engineers who won’t adapt will stagnate. They might find work in industries that are slow to change. But it won’t be a great career. It never is when you’re holding onto the last paradigm.

The engineers at risk aren’t the ones who don’t know enough yet. They’re the ones who stopped growing at the wrong layer. Juniors will climb. The question is whether the seniors stuck in the middle will climb with them.

Building Features One Prompt at a Time

Tue, 26 Aug 2025 00:00:00 GMT

A few weeks ago, we released a new feature at Mergify: autoqueue.

It automatically adds pull requests into the merge queue when they’re ready. No more custom automation rules, no more fiddling with YAML — it just works, straight from the merge queue settings.

Here’s the kicker: I coded it.

Yes, me. The CEO. The guy who hasn’t touched production code in years. The guy who usually spends his days on calls, not in GitHub.

And I did it in less than an hour a day, over three weeks, with the help of AI.

Why I Even Tried This

I’ve used Copilot casually before (mostly autocomplete in Emacs), but this time I wanted to go all-in.

Why? Curiosity, mostly. And time constraints. As a CEO, I have close to zero time to code, and this feature wasn’t urgent. So I thought: why not see what happens if I vibe-code it with AI?

How It Worked

The way I interacted with Claude 4 via GitHub Copilot was simple: I explained the feature like I’d explain it to my team in a product story. I added some technical constraints (“use unit tests, not functional ones”).

Then I let the AI go.

It just felt like coding blindfolded.

It wrote the code. I tweaked less than 5% of it. Once it was done, I sent it for review. I pasted my coworkers’ review feedback back into it. It rewrote. I guided. It iterated.

Did it nail it on the first try? No. Sometimes it forgot instructions. Sometimes it “lost context” after a few iterations and tried to reinvent the test setup it had already learned. That was frustrating — like explaining to a junior dev, except this junior dev has goldfish memory.

But eventually, it worked. The code was merged. Released. In production. Done.

What Surprised Me

I only changed about 5% of the lines myself.
Nobody on the team noticed it was “AI-coded.”
It handled six years of legacy code surprisingly well.
Two years ago this wouldn’t have been possible — the progress is insane.

What It Means

This isn’t about me playing engineer again for nostalgia. It’s about what’s coming.

The quality and quantity bar is about to rise dramatically. AI isn’t just autocomplete anymore; it’s co-construction.

You can ship faster. You can tackle features you don't fully understand at the start. You can guide at a high level and let the AI grind the details. A few months later, I took this even further — to the point where I stopped writing code entirely.

But it also raises new challenges. For instance:

How do juniors review AI-generated PRs?

How do teams trust code written by something that forgets your instructions after 10 turns?

(That’s probably another blog post.)

For now, though, I’ll just say this:

I vibe-coded a real feature into existence in less than an hour a day.

It felt like cheating. And I’m amazed.

Why We Still Care About Quality

Tue, 24 Jun 2025 00:00:00 GMT

I recently read Linear’s excellent blog post on why quality is so rare, and it resonated deeply with me. Craft, quality, care — these aren’t buzzwords. They’re a way of working, a way of thinking, and frankly, the only way I’ve ever known how to build things.

Linear: Why is quality so rare?

For me, it started with open source. When you put your code out in the open, you naturally want to make it good. Maybe even beautiful. I started more than 20 years ago, polishing my Debian packages, making sure they were clean, understandable, and useful. Later I poured that same mindset into building awesomewm, striving to write the best C code I could — because that code was me, visible to anyone curious enough to look.

Open source taught me that quality is not an accident. It’s a habit. And a commitment.

Even though Mergify is no longer open source, the ethos never left. We still build like our code is going to be read by thousands, because, well, it is at least read by our folks. Our team ships work we’re proud of. Whether we win a deal or not, it’s common to hear people tell us:

The quality of Mergify stands out.

That never gets old.

I know I’m not alone in this. Mehdi, my cofounder, and I have been building together for over 15 years. It’s in our DNA: we hate mediocrity. We won’t ship something that we wouldn’t use ourselves — joyfully.

That said, I’ve also seen the flip side. Back when I worked on OpenStack, a massive open-source project, there was a lot of code… and not always a lot of care. Many contributors came from companies that didn’t value quality — and it showed. Open source can be beautiful, but when it’s driven by quantity instead of pride, it becomes exhausting. I hated that part.

Quality isn’t just aesthetic. It’s a business strategy. Linear nailed that in their post. When you build something that feels right — fast, polished, thoughtful — users notice. They stay. They tell others. We’ve seen this at Mergify: our growth has been fueled not just by features but by how those features feel to use.

But quality is more than just a great UI or bug-free code.

It’s also:

A fast, reliable, intuitive product.
Clean code that enables long-term agility.
Thoughtful defaults and edge-case handling.
Being able to say “no” when something adds complexity without enough payoff.

Getting there isn’t easy. You need judgment — to know what’s worth doing and what can wait. That comes with experience and the humility to know you’ll never get everything right. We aim for 80/20, not 100/0. Sometimes that means leaving the last 20% for another day — or maybe never. Not because we don’t care, but because we care about the whole system staying healthy and fast.

Quality isn’t free. But it pays back. In speed, trust, and joy.

So yes, it’s a choice. One you make every day.

You can take the shortcut, or you can make something that lasts.

We still choose the latter.

“It’s Complicated” Is Not an Excuse

Tue, 11 Mar 2025 00:00:00 GMT

I spend a lot of time talking to engineers.

I ask them about design choices, technical decisions, and why something is built a certain way. I try to understand why this feature is so cumbersome to use, why this API is so convoluted, or why the user experience feels unnecessarily difficult.

And more often than not, the response I get is:

💬 “Well… it’s complicated.”

Sure. Everything is complicated.

That’s why you’re here. That’s why you’re an engineer.

But “it’s complicated” should never be an excuse for bad design.

Imagine If Other Professions Worked Like This

Let’s take a bakery, for example.

You walk in and ask for a loaf of bread.

The baker hands you a cup of flour and some water.

🫤 “Uhh… I was expecting actual bread.”

💬 “Well… it’s complicated.”

💬 “We’d have to mix the dough, let it rise, bake it for a while…”

💬 “That’s a lot of steps, so we just decided to give you the raw ingredients instead.”

This is exactly how software feels someday.

When users interact with your product, they don’t want to assemble the damn bread. They just want something that works.

Your job as an engineer is to handle complexity—not push it onto the user.

The Difference Between Good and Bad Engineering

Look, I get it. Engineering is hard.

Making things simple is difficult.

Abstracting complexity takes effort.

But great engineers don’t just write code—they design experiences.

A bad engineer builds something difficult and says, “Well, it’s complicated.”

A great engineer builds something difficult and makes it look simple. (More on what makes a great software engineer.)

🔹 Bad engineering forces users to deal with complexity.

🔹 Good engineering hides the complexity behind smart design.

Take Apple, for example. You know what’s actually complicated?

🔹 Compressing a 4K video into a tiny file.

🔹 Rendering realistic lighting effects in real-time on an iPhone.

🔹 Syncing all your messages, contacts, and photos seamlessly across devices.

But do Apple users ever have to think about any of that?

No. It just works.

That’s good engineering.

Stop Saying “It’s Complicated”—Start Making It Simple

When you hear yourself saying, “It’s complicated”, stop for a second and think:

🛑 Are you solving a hard problem in the simplest way possible?

🛑 Or are you just passing the complexity to the user?

If it’s the latter, you haven’t finished the job yet.

Because real engineering isn’t about making things work.

It’s about making things work… simply.

How To Test Your API Integration

Tue, 24 Sep 2024 00:00:00 GMT

As I was publishing last week's post on whether GitHub is becoming obsolete or the future of development platforms, they decided to trigger a two-hour interruption on Mergify in retaliation.

Just kidding. I am sure they did not do that on purpose.

Read my post-mortem if you want the whole story. The summary is that they broke their API for several hours until people started to complain, and they finally rolled back their change. Bringing down our service in the meantime.

That event forces me to talk about APIs this week.

API Definitions Are Just Definitions

I won’t go into the definition of an API per se; it’d be boring. You can Google it if you need to.

The real question is what having an API means. Offering an API to your users means authorizing them to interact with your service. This implies many rules, such as the data model of your API, the behavior of your API, the rules of usage, etc. Some can be encoded in a computer-readable machine; others cannot. Engineers like to talk about contracts, and I think it’s an almost good analogy.

To describe this contract, you need multiple specifications.

Developers have been ecstatic over OpenAPI over the last decade as a go-to media for describing their API. I want here to emphasize how little this documents your API. It illustrates the data model used but does not encode much of the behavior the system might exhibit.

Hey, I can confirm that GitHub did not break its OpenAPI schema when it broke its API last week. Formidable.

However, based on the assumption that OpenAPI is enough, many engineers mock their API consumption based on that part of the contract and think they’re done.

In that situation, the minimum you should do is validate that your mocking follows the OpenAPI schema you’re using. Even that is not enough because sometimes the schema changes—and sometimes it’s just not respected.

Let’s take GitHub again as an example. Their API is so legacy that the JSON schemas were crafted manually — and there might still be, I don’t know. It’s fine; it’s better than nothing, and it’s not obvious to change a legacy API that’s been there for 15 years.

We know first-hand that their system does not always respect the GitHub API JSON Schema.

APIs Have Side-effects

Again, this approach is entirely based on the data model and is insufficient and of little value.

Most of an API's value is in the behavior it triggers. Unless your API is a basic CRUD and does storage only, it will have side effects that might or might not be visible through the API.

For example, creating an asynchronous job on any REST API will return nothing except a unique identifier, which can be used later to identify the work. You might receive the data via a webhook or have to poll the API to get the job’s status. This kind of behavior cannot be documented in OpenAPI as it’s not part of the data model; there’s nothing to tell you to expect a webhook.

API Invisible Parts

Now, let’s discuss all the invisible parts of running an API. There are many. The first that come to mind are RBAC, quota, and rate limits. Most APIs have to implement those items, and they also have a direct impact on the API behavior and access.

Those features will massively impact the quality and quantity of API use. Again, they are pretty hard to test in a black box. There’s no way you can easily mock a full RBAC implementation or real-life rate limits.

Testing the Hard Way

Having consumed many different APIs for the last five years on Mergify, and especially GitHub’s one that we know by heart, gave us a few ideas on how you can or cannot test.

Rule number one: do not mock. Record your tests.

We leverage vcrpy in Python to do that: the idea is to run your test in a record mode where real HTTP requests are done against a service. Once the recording is done, you can replay the test when running it locally or in the CI.

If any of your code tries to make a different HTTP call, the test will fail, and you will have to re-record it. This ensures that no change is made to the application without being noticed.

Now, that does prevent your application from being broken, but that does not prevent the API from breaking your app. The only way to do this is to regularly re-record all the tests and see if they break.

So, rule number two: re-record your tests regularly — every day if possible.

For example, we have a test that plays with GitHub pull request labels. When re-recording a test a few months ago, we noticed that if it failed. It turned out that GitHub changed its API to become case-sensitive overnight (that was not in the OpenAPI schema!).

In that case, we preferred to ask GitHub to fix their API rather than fix our code, but hey, your mileage may vary.

Rule number three: be ready to fix the code.

No amount of testing will cover all the edge cases. For example, requests quota or rate limit might be hit in real scenarios but not in testing, meaning you’ll have to handle those specific cases without being able to test. It’s fine — you can actually mock part of the responses here.

For this, we leverage Sentry to obtain evidence of the problem, replicate it in a test, and fix it. No amount of testing can fix all scenarios, so having a way to hotfix your code is a must.

In the end, mixing API test recording for safety and error tracking for fast action is the best combination we’ve seen for dealing with external systems.

If we map those rules to last week's incident, rule number three helped to fix the issue quickly, while rule number one would have technically caught it, and rule number two would have done so in less than 24 hours. Even if it turned out in our case that reality kicked in before testing.

So use that. And retry mechanisms.

I guess that’ll be for another post.

How to Be a Great Software Engineer

Tue, 03 Sep 2024 00:00:00 GMT

I did not write for the last few weeks as I enjoyed taking a break. Ha! That’s probably the first point I could write about being a great software engineer: taking breaks.

Nevermind.

What do I know, after all? I’m not a software engineer anymore. I’m a CEO, god damn it.

Not my role model.

At least being a CEO gives me some excuse to dispense some pieces of advice regularly. It turns out that over the last couple of years, I had to become a manager of people — and many people in my team are software engineers. The only thing I knew about management so far was being managed, which taught me many things, such as:

how to manage;
how not to manage;
how to be managed.

I don’t want to talk about the first two points here, but I’d like to write about the latter. I regularly have to give feedback to people on my team, and they often rely on the Great Engineer Framework that I built in my mind.

It’s time to write that down.

Expectations

Since I started my career as a software engineer 20 years ago, I always wondered how to improve. My initial appeal for this career was typing code on a keyboard, so I decided that the best way to become a great engineer was to be the best at technical stuff.

I coded days and nights, learned everything I could, and became amazing. I Debian-packaged hundreds of software, wrote C code for a window manager, Linux and CPython, wrote CIEDE2000 color space computation functions in Lisp, wrote thousands of lines of Python to do crazy stuff, implemented XML binding for the X11 protocol, built a scalable time-series database based on object storage, etc; you name it. I did many tech-crazy things and thought I was a great engineer.

It turns out I was only 33% good. As I grew into the tech and startup ecosystem, I started to understand whatever was around me, the industry, the business, the people. And I soon realized that this was not enough, even if I was among the best engineers you could probably find (sorry for bragging).

Aspects

After a few years, I built a mental model that I still use nowadays to give feedback to engineers in my team, based on 3 aspects that you must master to become a great engineer — like the 10x engineer they all talk about:

Tech
Business value
Collaboration

Tech

I just discussed tech. You have to be amazing at it, which means you have to dig deep into it. As my co-founder Mehdi says, great engineers pull the strings. This means that you’re not just there to paper over the problem; you’re here to understand it fully, to grasp it entirely, from top to bottom, and to fix it forever because you understand it.

Many junior engineers are not able to do that. They just tinker with their code until “well, it works, tests pass, whatever.” The rise of AI tooling is supporting that, and engineers working this way will have to step up their game, or they’ll disappear.

It takes a large amount of time to achieve this expertise, and as common sense says, maybe 10,000 hours. This is actually a major issue for people switching to tech after another career; 10,000 hours of coding 25 hours a week (if you just do it on the job) in a typical 45-week year is more than 8 years before starting to “know what you’re talking about.” If you start at 18 years old, tinkering with computers 60 hours a week for fun, you’ll be pretty good at it by 21. I know that’s not fair, but I see this as a major roadblock for hiring tech talents coming from a career change.

So: do tech. Don’t stop until you understand everything of what you are responsible for. I remember 15 years ago being screened by a recruiter at Google who’d ask me what happened when I typed google.com in my Web browser. Being able to explain everything, from the keyboard input to the DNS requests and TCP headers of the packet sent, to the HTTP server made me pass without a blink.

Business Value

This sounds totally stupid, and I might be slightly biased by my French experience, but there are too many engineers who do not understand business value. It actually took me a few years to understand this, probably because I was only focused on tech. Let me give you a good anecdote to illustrate this.

Ten years ago, I was called by a senior manager to help with a Python project in a media company. I go to the meeting and meet the manager. He comes from a famous French tech school — one where they learn the C standard library from scratch in their first year — and so do most of the engineers in his team. They’re managing hundreds of servers, and after evaluating various software to do that (Puppet, Ansible, etc) they didn’t find anything that suited 100% of their needs, so they built their own. They invested hundreds of hours in it, and now they’d need help maintaining it.

It turns out that what they needed was probably Ansible plus a custom plugin to reach 98% of their need in a tenth of the time, but they didn’t see it that way. They just built an entire tech project, not related to their core business, from the ground up, investing hours of time. This is the kind of similar experience I talked about in my previous post, Solving Build vs Buy. I skipped that project and moved on to other things. I had no interest in maintaining a project that was not providing core value to the business. That’d have been a great way to be ditch as soon as somebody smarter in the company realized the amount of wasted time that project might have been.

This kind of behaviour applies everywhere. Engineers would spend hours trying to implement perfect systems that will scale to millions of users. While the business might have no user — yet. Engineers would spend hours building a feature or solving a problem that would impact 0.1% of users. It’s true that engineering might not be entirely responsible for the roadmap directly, but they are responsible for the time they spend on how far they go into implementing systems and features.

We live in a world where economy is the driver, which means you have to maximum throughput and minimize input. Input is your coding time, and throughput is the (extra) money the company that hires you can make with you work.

Collaboration

I could probably summarize this aspect with just:

If you want to go fast, go alone. If you want to go far, go together.

That’s entirely true. Considering the team is a problem for many engineers who are frustrated by members of their team. It does take time to deal with people, and they are not as easy to understand as computers. However, in the long run, they are the best way to provide you with leverage to achieve amazing things. Maybe another secret of 10x engineers, who knows?

Therefore, you’ll need to understand the dynamics that make your teamwork. You have to make sure your work is not isolated and is not the only correct piece of code hidden in its own corner. You need to connect both your piece of software and your brain to other pieces of software and people. I know it requires a lot of effort for some people, especially because it can feel annoying and inefficient to talk or write things for other engineers to understand what you’re achieving.

But until we can all Neuralink, yes, you’ll have to pause and do something that seems like a waste of time: talking to your teammates, your manager, or customers.

Always remember: this is an investment. In the long run, it will pay off.

Conclusion

Those three aspects are always the ones that I use to drive my feedback on performance reviews to engineers in the team. Not everyone is always 10/10 in every aspect, so it eases providing feedback and drives them to where they should improve next. They are probably not exhaustive, but they are a great way to spot great and inadequate behaviors.

Navigating SQL Migrations with Confidence: Introducing sql-compare

Tue, 16 Jul 2024 00:00:00 GMT

As long as I can remember, SQL has been a cornerstone of my engineering journey. My early days at university were filled with monotonous Oracle-based SQL courses, which I found uninspiring. Knowing I would likely never use Oracle, I shifted my focus to MySQL. Over time, I discovered the limitations of MySQL and was introduced to PostgreSQL, thanks to Dimitri. I even organized a few meetups in Paris and encouraged Dimitri to publish "The Art of PostgreSQL," arguably the best book on SQL (I reviewed it here). Eventually, I embraced PostgreSQL wholeheartedly.

SQL databases are a timeless technology that continues to evolve. From Timescale to pgvector, new advancements are continually emerging. However, one persistent challenge has been managing database migrations. Modifying your data model is crucial for evolving your application, but it’s often a daunting task. At Mergify, like many companies, we’ve faced this challenge head-on.

We've tried various solutions, from custom Python scripts to using migra, an open-source project that is unfortunately no longer maintained. Each solution had its drawbacks, leading us to a crossroads where we had to decide on our next move.

The Initial Struggle

At Mergify, PostgreSQL is the backbone of our data handling, from managing the state of GitHub objects to maintaining our event log. From the beginning, we’ve interacted with the database exclusively using an ORM, choosing SQLAlchemy for its maturity, framework agnosticism, and support for asynchronous I/O since version 2.0.0.

Given our frequent production deployments, a robust CI/CD pipeline is essential to handle database evolution smoothly. Every schema modification must be rigorously tested and automatically applied to the production database, adhering to the principles outlined in Martin Fowler's "Evolutionary Database Design." Version-controlling each database artifact and scripting every change as a migration are critical steps in this process.

We chose Alembic to manage our database migrations. Maintained by the SQLAlchemy team, Alembic is a command-line tool that can automatically create migration scripts from your SQLAlchemy models. Each script is version-controlled alongside your source code. Alembic applies these migrations to the database, recording the revision number in the alembic_version table to ensure only new migrations are applied subsequently. This command is typically executed in the continuous delivery pipeline to keep the production database up-to-date.

A Naive Beginning

Our initial approach to testing migration scripts was straightforward: create two databases—one using SQLAlchemy models and the other using only the migration scripts—and ensure they have identical schemas. This involved:

Creating PostgreSQL servers using Docker: On a new server, create two empty databases.
Generating schemas: Use the first database to create artifacts with SQLAlchemy models, and use Alembic to run migration scripts on the second database.
Comparing schemas: Dump each database schema into SQL files using pg_dump and compare them using Python’s filecmp and difflib builtin libraries.

Here’s an example command to dump a database schema into an SQL file:

pg_dump \
    --dbname=postgresql://user:password@host:port/database \
    --schema-only \
    --exclude-table=alembic_version \
    --format=p \
    --encoding=UTF8 \
    --file /path/to/dump.sql

To compare the files:

assert filecmp.cmp(schema_dump_creation_path, schema_dump_migration_path, shallow=False)

If the test fails, use difflib to display the differences:

def filediff(path1: pathlib.Path, path2: pathlib.Path) -> str:
    with path1.open() as f1, path2.open() as f2:
        diff = difflib.unified_diff(
            f1.readlines(),
            f2.readlines(),
            path1.name,
            path2.name,
        )
        return "Database dump differences: \n" + "".join(diff)

While effective, this test had limitations, such as sensitivity to column order. PostgreSQL doesn’t easily allow changing column positions, necessitating consistent column order in models and production databases.

The Complexity Grows

As our models grew more complex, our naive test struggled to keep up. Consider the following example:

class Base:
    updated_at: orm.Mapped[datetime.datetime] = orm.mapped_column(
        sqlalchemy.DateTime(timezone=True),
        server_default=sqlalchemy.func.now(),
    )

class User(Base):
    id: orm.Mapped[int] = orm.mapped_column(
        sqlalchemy.BigInteger,
        primary_key=True,
    )

In this setup, the updated_at column is added to every child model, such as User. Adding a new column to User, like name, would misalign the order, causing schema mismatches.

To address this, we needed to compare schemas while ignoring column order. We explored various tools:

Alembic: Can compare schemas to generate migration scripts but misses some differences.
Migra: An unmaintained tool that compares database schemas effectively.
SQL dumps: The most reliable format but challenging to parse and compare directly.

Building the Solution: sql-compare

It was clear that our current solutions were insufficient. We needed a hero to rescue us from the perils of SQL migration management, so we developed sql-compare.

sql-compare is a Python library that uses sqlparse to parse SQL files and compare schemas, ignoring irrelevant differences like comments, whitespace, and column order. This new tool became an integral part of our workflow, catching migration issues that other tools might miss.

The main challenge was filtering and grouping tokens by column definition before sorting them. Despite these complexities, sql-compare emerged victorious, enabling us to ensure seamless migrations and maintain schema integrity.

The Journey Forward

We’ve open-sourced sql-compare to help others facing similar challenges. You can try it by running pip install sql-compare. We plan to enhance sql-compare, such as creating functions to retrieve all schema differences for better test results. If you have suggestions or want to contribute, feel free to submit issues or pull requests on our GitHub repository.

Conclusion

Managing database migrations is a complex but essential task for evolving applications. With sql-compare, we found our solution, ensuring seamless migrations, maintaining schema integrity, and continuing to deliver high-quality software. Our journey through the challenges of SQL migrations has taught us valuable lessons, and with sql-compare, we’re better equipped to face the future.

A Journey of Embracing Linters

Tue, 09 Jul 2024 00:00:00 GMT

Recently, I found myself in a spirited debate with one of our front-end developers at Mergify. This discussion, revolving around the usage of linters, reminded me of my long and storied history with these "advisor tools." Having been confronted with linters for the past 25 years, I believe it's time to share some of that accumulated wisdom.

My first encounter with a linter was with use strict in Perl. Although I can't recall the specifics of what it did, I do remember it being an essential tool for writing better code. Later on, I encountered the gcc -W and -pedantic options, which I enabled religiously in all my projects. These early experiences set the stage for my ongoing relationship with linters.

Warnings

Fast forward to today, my recent discussion centered around eslint and enabling all the checks for the Playwright plugin, treating every drift as an error rather than a warning. This distinction is crucial: an error causes the CI to fail, while a warning merely generates noise. Not all linters have this warning level, but in my experience, warnings will be disruptive if left unaddressed in your development workflow. An error should be a clear-cut issue: either ignore it or fix it.

Having unresolved warnings in your CI logs creates ambiguity and inefficiency. Make a decision. Commit to it.

The original warning that triggered our discussion.

Picking Errors

Despite not being a JavaScript expert, my 25 years of experience with various linters gives me some perspective on this matter. Our debate also touched on which approach to use with respect to linters, either:

stick to the recommended and default settings;
being stricter by promoting certain warnings to errors for checks we deemed useful;
enable everything to be an error and explicitly ignore checks that don't apply to our project or are considered incorrect.

Every linter, from gcc -W flags to ruff in Python, starts with a set of "recommended" settings. These are designed to throw a manageable number of errors on a typical project, making the linter easy to adopt for teams. This doesn't mean the disabled options are bad; they are simply considered "too much for beginners" and can be enabled later.

This incremental approach is how we adopted mypy at Mergify. The default typing checks are relatively light, allowing us to enable it without much friction. We spent a few weeks fixing typing issues, caught a few bugs in the process, and were satisfied. Gradually, we enabled more checks until we reached the point of enabling strict = true (a nostalgic nod to Perl) and caught even more (potential) bugs.

On the flip side, having a poorly calibrated set of default recommendations is why I never adopted pylint. Running pylint on our otherwise impeccable Python code, which passes ruff with most checks enabled, results in 13,000 errors for 140,000 SLOC. (I wrote about similar code quality tools in The Best Flake8 Extensions.) This is an insurmountable barrier for any developer. The prospect of ignoring all these non-critical errors, such as missing docstrings or line lengths, seems daunting.

Eslint and Playwright

Returning to eslint and Playwright, we used the following code to enable all Playwright rules as errors:

...Object.keys(playwrightPlugin.configs['flat/recommended'].plugins.playwright.rules).reduce(
  (acc, rule) => {
    acc[`playwright/${rule}`] = 'error';
    return acc;
  },
  {}
),

This approach ensures we don't miss any linting recommendations from the Playwright team. With Dependabot automatically updating our dependencies, new errors introduced by updates appear in brand-new pull requests, allowing us to improve our code continuously.

In conclusion, "recommended" settings in linters are designed for ease of adoption, striking a balance between "best practices" and "practicality," but they are not guidelines for what you should follow.

Striving for perfection (assuming your linter is robust and not crazy) is always the goal. Make deliberate choices about which checks to ignore, and remember that linters are here to help you write better, more reliable code.

Atomic lock-free counters in Python

Mon, 06 Jan 2020 00:00:00 GMT

At Datadog, we're really into metrics. We love them, we store them, but we also generate them. To do that, you need to juggle with integers that are incremented, also known as counters.

While having an integer that changes its value sounds dull, it might not be without some surprises in certain circumstances. Let's dive in.

The Straightforward Implementation

class SingleThreadCounter(object):
	def __init__(self):
    	self.value = 0
        
    def increment(self):
        self.value += 1

Pretty easy, right?

Well, not so fast, buddy. As the class name implies, this works fine with a single-threaded application. Let's take a look at the instructions in the increment method:

>>> import dis
>>> dis.dis("self.value += 1")
  1           0 LOAD_NAME                0 (self)
              2 DUP_TOP
              4 LOAD_ATTR                1 (value)
              6 LOAD_CONST               0 (1)
              8 INPLACE_ADD
             10 ROT_TWO
             12 STORE_ATTR               1 (value)
             14 LOAD_CONST               1 (None)
             16 RETURN_VALUE

The self.value +=1 line of code generates 8 different operations for Python. Operations that could be interrupted at any time in their flow to switch to a different thread that could also increment the counter.

Indeed, the += operation is not atomic: one needs to do a LOAD_ATTR to read the current value of the counter, then an INPLACE_ADD to add 1, to finally STORE_ATTR to store the final result in the value attribute.

If another thread executes the same code at the same time, you could end up with adding 1 to an old value:

Thread-1 reads the value as 23
Thread-1 adds 1 to 23 and get 24
Thread-2 reads the value as 23
Thread-1 stores 24 in value
Thread-2 adds 1 to 23
Thread-2 stores 24 in value

Boom. Your Counter class is not thread-safe. 😭

The Thread-Safe Implementation

To make this thread-safe, a lock is necessary. We need a lock each time we want to increment the value, so we are sure the increments are done serially.

import threading

class FastReadCounter(object):
    def __init__(self):
        self.value = 0
        self._lock = threading.Lock()
        
    def increment(self):
        with self._lock:
            self.value += 1

This implementation is thread-safe. There is no way for multiple threads to increment the value at the same time, so there's no way that an increment is lost.

The only downside of this counter implementation is that you need to lock the counter each time you need to increment. There might be much contention around this lock if you have many threads updating the counter.

On the other hand, if it's barely updated and often read, this is an excellent implementation of a thread-safe counter.

A Fast Write Implementation

There's a way to implement a thread-safe counter in Python that does not need to be locked on write. It's a trick that should only work on CPython because of the Global Interpreter Lock.

While everybody is unhappy with it, this time, the GIL is going to help us. When a C function is executed and does not do any I/O, it cannot be interrupted by any other thread. It turns out there's a counter-like class implemented in Python: itertools.count.

We can use this count class as our advantage by avoiding the need to use a lock when incrementing the counter.

If you read the documentation for itertools.count, you'll notice that there's no way to read the current value of the counter. This is tricky, and this is where we'll need to use a lock to bypass this limitation. Here's the code:

import itertools
import threading

class FastWriteCounter(object):
    def __init__(self):
        self._number_of_read = 0
        self._counter = itertools.count()
        self._read_lock = threading.Lock()

    def increment(self):
        next(self._counter)

    def value(self):
        with self._read_lock:
            value = next(self._counter) - self._number_of_read
            self._number_of_read += 1
        return value

The increment code is quite simple in this case: the counter is just incremented without any lock. The GIL protects concurrent access to the internal data structure in C, so there's no need for us to lock anything.

On the other hand, Python does not provide any way to read the value of an itertools.count object. We need to use a small trick to get the current value. The value method increments the counter and then gets the value while subtracting the number of times the counter has been read (and therefore incremented for nothing).

This counter is, therefore, lock-free for writing, but not for reading. The opposite of our previous implementation

Measuring Performance

After writing all of this code, I wanted to make sure how the different implementations impacted speed. Using the timeit module and my fancy laptop, I've measured the performance of reading and writing to this counter.

Operation

SingleThreadCounter

FastReadCounter

FastWriteCounter

increment

176 ns

390 ns

169 ns

value

26 ns

529 ns

I'm glad that the performance measurements in practice match the theory 😅. Both SingleThreadCounter and FastReadCounter have the same performance for reading. Since they use a simple variable read, it makes absolute sense.

The same goes for SingleThreadCounter and FastWriteCounter, which have the same performance for incrementing the counter. Again they're using the same kind of lock-free code to add 1 to an integer, making the code fast.

Conclusion

It's pretty obvious, but if you're using a single-threaded application and do not have to care about concurrent access, you should stick to using a simple incremented integer.

For fun, I've published a Python package named fastcounter that provides those classes. The sources are available on GitHub. Enjoy!