github — jd:/dev/blog

Properly managing your .gitignore file

Mon, 02 Dec 2019 00:00:00 GMT

There's not a single month where I don't have to explain this. I thought it'd be a good opportunity to write about this .gitignore file so everyone is up to date on this magic file.

The purpose of `.gitignore`

The .gitignore file is meant to be a list of files that Git should not track. It resides at the root directory of your repository. It can be a list of file path relative to the repository, or a list of wildcard. The file format and location is fully documented in Git documentation.

For example, this is a valid content for a .gitignore file:

foo
bar/*

When you're using Git commands such as git add, all the files matching what's listed in .gitignore are ignored. That makes sure you don't commit a file that should not be there by mistake. In the example above, any file in the bar directory or any file named foo will be completely ignored by all Git commands.

Awesome!

What's the problem with it?

Soon, developers realize that their directory is cluttered with temporary files. It might be from their build system, their editors or some test files they wrote.

So what do they do? They add those files to .gitignore for their project. You end up with a .gitignore file that contains entries like:

*~
.vscode
*.DS_Store
.idea

With that, you're sure to ignore backup files from vim, folders from MacOS and temporary files from Visual Studio code, etc.

Don't do this. Not everybody uses your editor or favorite pet tool, and nobody cares. The repository you're working in is shared with a lot of others developers. Sending pull requests to just add this kind of entry to ignore files generated by your pet editor is wrong and annoying.

Wait, how do I ignore my editor files then?

If you read through Git documentation, the answer lies there: Git has a global ignore file that works for EVERY repository on your system. No need to hack each repository. By default, it's in ~/.config/git/ignore. Here's mine:

.#*
*.swp
.DS_Store
.dir-locals.el
.dir-locals-2.el

That's enough to ignore my editors and OS files in all my repositories so I don't git add wrong files by mistake. You can tweak this global file location by changing by tweaking core.excludesFile in your Git configuration.

So what should I put in .gitignore?

You should put in .gitignore all files and patterns that are generated by the build system of your project, or any file that it might output while running.

For example, for a Python project, it's common to have this:

*.pyc
__pycache__

With this, it makes sure that nobody is committing compiled Python files.

Thanks for reading through this. I hope you'll write better .gitignore files in the future. 🤞

Dependencies Handling in Python

Mon, 02 Sep 2019 00:00:00 GMT

Dependencies are a nightmare for many people. Some even argue they are technical debt. Managing the list of the libraries of your software is a horrible experience. Updating them — automatically? — sounds like a delirium.

Stick with me here as I am going to help you get a better grasp on something that you cannot, in practice, get rid of — unless you're incredibly rich and talented and can live without the code of others.

First, we need to be clear of something about dependencies: there are two types of them. Donald Stuff wrote better than I would about the subject years ago. To make it simple, one can say that they are two types of code packages depending on external code: applications and libraries.

Libraries Dependencies

Python libraries should specify their dependencies in a generic way. A library should not require requests 2.1.5: it does not make sense. If every library out there needs a different version of requests, they can't be used at the same time.

Libraries need to declare dependencies based on ranges of version numbers. Requiring requests>=2 is correct. Requiring requests>=1,<2 is also correct if you know that requests 2.x does not work with the library. The problem that your version range specification is solving is the API compatibility issue between your code and your dependencies — nothing else. That's a good reason for libraries to use Semantic Versioning whenever possible.

Therefore, dependencies should be written in setup.py as something like:

from setuptools import setup

setup(
    name="MyLibrary",
    version="1.0",
    install_requires=[
        "requests",
    ],
    # ...
)

This way, it is easy for any application to use the library and co-exist with others.

Applications Dependencies

An application is just a particular case of libraries. They are not intended to be reused (imported) by other libraries of applications — though nothing would prevent it in practice.

In the end, that means that you should specify the dependencies the same way that you would do for a library in the application's setup.py.

The main difference is that an application is usually deployed in production to provide its service. Deployments need to be reproducible. For that, you can't solely rely on setup.py: the requested range of the dependencies are too broad. You're at the mercy of random version changes at any time when re-deploying your application.

You, therefore, need a different version management mechanism to handle deployment than just setup.py.

pipenv has an excellent section recapping this in its documentation. It splits dependency types into abstract and concrete dependencies: abstract dependencies are based on ranges (e.g., libraries) whereas concrete dependencies are specified with precise versions (e.g., application deployments) — as we've just seen here.

Handling Deployment

The requirements.txt file has been used to solve application deployment reproducibility for a long time now. Its format is usually something like:

requests==3.1.5
foobar==2.0

Each library sees itself specified to the micro version. That makes sure each of your deployment is going to install the same version of your dependency. Using a requirements.txt is a simple solution and a first step toward reproducible deployment. However, it's not enough.

Indeed, while you can specify which version of requests you want, if requests depends on urllib3, that could make pip install urllib 2.1 or urllib 2.2. You can't know which one will be installed, which does not make your deployment 100% reproducible.

Of course, you could duplicate all requests dependencies yourself in your requirements.txt, but that would be madness!

There are various hacks available to fix this limitation, but the real saviors here are pipenv and poetry. The way they solve it is similar to many package managers in other programming languages. They generate a lock file that contains the list of all installed dependencies (and their own dependencies, etc.) with their version numbers. That makes sure the deployment is 100% reproducible.

Check out their documentation on how to set up and use them!

Handling Dependencies Updates

Now that you have your lock file that makes sure your deployment is reproducible in a snap, you've another problem. How do you make sure that your dependencies are up-to-date? There is a real security concern about this, but also bug fixes and optimizations that you might miss by staying behind.

If your project is hosted on GitHub, Dependabot is an excellent solution to solve this issue. Enabling this application on your repository creates automatically pull requests whenever a new version of the library listed in your lock file is available. For example, if you've deployed your application with redis 3.3.6, Dependabot will create a pull request updating to redis 3.3.7 as soon as it gets released. Furthermore, Dependabot supports requirements.txt, pipenv, and poetry!

Automatic Deployment Update

You're almost there. You have a bot that is letting you know that a new version of a library your project needs is available.

Once the pull request is created, your continuous integration system is going to kick in, deploy your project, and runs the test. If everything works fine, your pull request is ready to be merged. But are you really needed in this process?

Unless you have a particular and personal aversion on specific version numbers —"Gosh I hate versions that end with a 3. It's always bad luck."— or unless you have zero automated testing, you, human, is useless. This merge can be fully automatic.

This is where Mergify comes into play. Mergify is a GitHub application allowing to define precise rules about how to merge your pull requests. Here's a rule that I use in every project:

pull_requests_rules:
  - name: automatic merge from dependabot
    conditions:
      - author~=^dependabot(|-preview)\[bot\]$
      - label!=work-in-progress
      - "status-success=ci/circleci: pep8"
      - "status-success=ci/circleci: py37"
    actions:
      merge:
        method: merge

As soon as your continuous integration system passes, Mergify merges the pull request for you.

You can then automatically trigger your deployment hooks to update your production deployment and get the new library version installed right away. This leaves your application always up-to-date with newer libraries and not lagging behind several years of releases.

If anything goes wrong, you're still able to revert the commit from Dependabot — which you can also automate if you wish with a Mergify rule.

Beyond

This is to me the state of the art of dependency management lifecycle right now. And while this applies exceptionally well to Python, it can be applied to many other languages that use a similar pattern — such as Node and npm.

More GitHub workflow automation

Tue, 16 Oct 2018 00:00:00 GMT

The more you use computers, the more you see the potentials for automating everything. Who doesn't love that? By building Mergify those last months, we've decided it was time bring more automation to the development workflow.

Mergify's first version was a minimal viable product around automating the merge of pull requests. As I wrote a few months ago, we wanted to automate the merge of pull requests when it was ready to be merged. For most projects, this is easy and consists of a simple rule: "it must be approved by a developer and pass the CI".

Evolving on Feedback

For the first few months, we received a lot of feedback from our users. They were enthusiastic about the product but were frustrated by a couple of things.

First, Mergify would mess up with branch protections. We thought that people wanted the GitHub UI to match their rules. As I'll explain later, it turns out to be only partially true, and we found a workaround.

Then, Mergify's abilities were capped by some of the limitations of the GitHub workflow and API. For example, GitHub would only allow rules per branch, whereas our users wanted to have rules applied based on a lot of different criteria.

Building the Next Engine

We rolled up our sleeves and started to build that new engine. The first thing was to get rid of the GitHub branch protection feature altogether and leveraging the Checks API to render something useful to the users in the UI. You can now have a complete overview of the rules that will be applied to your pull requests in the UI, making it easy to understand what's happening.

Then, we wrote a new matching engine that would allow matching any pull requests based on any of its attributes. You can now automate your workflow with a finer-grained configuration.

What Does It Look Like?

Here's a simple rule you could write:

pull_request_rules:
  - name: automatic merge on approval and CI pass
    conditions:
     - "#approved-reviews-by>=1"
     - status-success=continuous-integration/travis-ci/pr
     - label!=work-in-progress
    actions:
      merge:
        method: merge

With that, any pull request that has been approved by a collaborator, passes the Travis CI job and does not have the label work-in-progress will be automatically merged by Mergify.

You could use even more actions to backport this pull request to another branch, close the pull request or add/remove labels. We're starting to see users building amazing workflow with that engine!

We're thrilled by this new version we launched this week and glad we're getting amazing feedback (again) from our users.

When you give it a try, drop me a note and let me know what you think about it!

A safe GitHub workflow with Pastamaker

Fri, 15 Dec 2017 00:00:00 GMT

When the Gnocchi project decided to move to GitHub, we developers had to move from a Gerrit based workflow to a GitHub pull-request one.

This has been challenging in some ways. We were satisfied with the workflow we had using Gerrit and Zuul for testing so we decided to adapt GitHub to our requirements.

We know that Zuul now supports GitHub. However, that implies having your own testing infrastructure, something we can't afford. Instead, we rely on Travis, like most open-source projects hosted on GitHub.

The workflow

The workflow we wanted to have was the following:

A contributor creates a pull-request on GitHub.
The pull-request is tested by Travis.
The pull-request is reviewed by approved projects members.
If the tests pass and two reviewers have approved the pull-request, then it
can be merged.

This sounds simple, but it is actually not that simple.

First, when Travis tests the pull-request, it checks what has been sent by the contributor. If the contributor created a pull-request on top of an outdated version of the base branch, that's what will be tested by Travis during the initial pull-request creation.

Even if the pull-request has been created using the tip of the base branch, as time passes, the base branch will progress. However, the pull-request created by your contributor will not get those new commits – unless rebased manually.

That means the Travis tests result is now outdated invalid. Still, GitHub and Travis will both show you that this pull-request passed all tests – yes it did but with an old base branch from a while back!

If you added new tests in the meantime in your base branch, it's possible that this pull-request does not work anymore. Pressing the merge button might just break your project!

To help with that problem, GitHub recently added a button that allows you to base branch into the pull-request. That allows, in one click, to get the pull-requested updated with the base branch (e.g., master) and retested by Travis.

Still, this means that if you have ten pull-requests, you need to:

Merge base branch into PR#1
Wait for Travis to pass
Wait for two reviewers to approve
Merge PR#1
All other nine pull-requests are not out of date. You need to do start back at operation 1. for each pull-request.

This is very tedious to do manually, especially when your projects has tons of pull-requests.

This is why Mehdi Abaakouk created Pastamaker.

Pastamaker to the rescue

Pastamaker is a small Web application that implements the described workflow. Once connected to your GitHub project, it will set the proper permissions to protect it for accidental manual merge and force the workflow above to be followed.

Pastamaker listens for GitHub and Travis events to track the state of each pull-request. If it detects that a pull-request has been approved by two reviewers and that the initial Travis test run passed, it will merge the base branch if needed in it, wait for Travis to pass again, and then finally merge it.

If multiple pull-requests are approved at the same time and are candidates for a merge, it will order them, update once at a time, wait for Travis results and merge them if they pass. It essentially automates the workflow described above.

Pastamaker exposes its data via a simple dashboard, which allows seeing all the pull-requests for your project in a snap.

Pastamaker offers a lot of tiny other details that make the developers lives easier, such as posting the job result with direct links to the jobs logs in the pull-request – so you're informed as soon as they pass or fail and can fix them right away!

Pastamaker is obviously open-source, and we would love to see you give it a try!

Sending GitHub pull-request from your shell

Wed, 24 May 2017 00:00:00 GMT

I've always been frustrated by the GitHub workflow. A while back I
wrote how Gerrit workflow was superior to GitHub pull-request system. But it seems that GitHub listened and they improved the pull-request system these last years to include reviews, and different workflow implementation, e.g. requiring continuous integration tests to pass before merging a patch.

All those improvements great helped the Gnocchi team to consider moving to GitHub when leaving OpenStack. Our first days have been great and I cannot say we miss Gerrit much for now.

The only tool that I loved and miss is git-review. It allows pushing a branch of update easily to Gerrit.

Unfortunately, in the GitHub world, things are different. To send a pull-request you have to execute a few steps which are:

Clone the target repository
Push your local branch to your repository
Create a pull-request from your pushed local branch to the target branch

If you want to update later your pull-request, you either have to push new commits to your branch or, more often, edit your patches and force push your branch to your forked repository so you can ask for a new review of your pull-request.

I'm way too lazy to do all of that by hand, so I had a tool for a few years that I used based on hub, a command-line tool that interacts with GitHub API. Unfortunately, it was pretty simple and did not have all the feature I wanted.

Which pushed me to write my own tool, humbly entitled git-pull-request. It allows to send a pull-request to any GitHub project just after you just cloned it. So there's no need to manually fork the repository, send branches, etc.

Once you created a branch and committed to it, just run git pull-request and everything we'll be done for you automatically.

## First pull-request creation
$ git clone https://github.com/gnocchixyz/gnocchi.git
$ cd gnocchi
$ git checkout -b somefeature
<edit files>
$ git commit -a -m 'I did some changes'
$ git pull-request
Forked repository: https://github.com/jd/gnocchi
Force-pushing branch `somefeature' to remote `github'
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 562 bytes | 0 bytes/s, done.
Total 5 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/jd/gnocchi.git
 + 73a733f7...1be2bf29 somefeature -> somefeature (forced update)
Pull-request created: https://github.com/gnocchixyz/gnocchi/pull/33

If you need to update your pull-request with new patches, just edit your branch and call git pull-request again. It'll re-push your branch and will not create a pull-request if one already exists.

<edit some more files>
$ git commit --amend -a
$ git pull-request
Forked repository: https://github.com/jd/gnocchi
Force-pushing branch `somefeature to remote `github'
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 562 bytes | 0 bytes/s, done.
Total 5 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/jd/gnocchi.git
 + 73a733f7...1be2bf29 somefeature -> somefeature (forced update)
Pull-request already exists at: https://github.com/gnocchixyz/gnocchi/pull/33

This tool was definitely the missing piece to smooth my GitHub workflow, so I'm glad I took some time to write it. I hope you'll enjoy it and will send me awesome pull-requests, so go check it out. This program is written in Python and uses the GitHub API.

And feel free to request new fancy features!

Tracking OpenStack contributions in GitHub

Tue, 19 Aug 2014 00:00:00 GMT

I've switched my Git repositories to GitHub recently, and started to watch my contributions statistics, which were very low considering I spend my days hacking on open source software, especially OpenStack.

OpenStack hosts its Git repositories on its own infrastructure at git.openstack.org, but also mirrors them on GitHub. Logically, I was expecting GitHub to track my commits there too, as I'm using the same email address everywhere.

It turns out that it was not the case, and the help page about that on GitHub describes the rule in place to compute statistics. Indeed, according to GitHub, I had no relations to the OpenStack repositories, as I never forked them nor opened a pull request on them (OpenStack uses Gerrit).

Starring a repository is enough to build a relationship between a user and a repository, so this is was the only thing needed to inform GitHub that I have contributed to those repositories. Considering OpenStack has hundreds of repositories, I decided to star them all by using a small Python script using pygithub.

And voilà, my statistics are now including all my contributions to OpenStack!

Rant about Github pull-request workflow implementation

Fri, 10 May 2013 00:00:00 GMT

One of my recent innocent tweet about Gerrit vs Github triggered much more reponses and debate that I expected it to. I realize that it might be worth explaining a bit what I meant, in a text longer than 140 characters.

I'm having a hard time now contributing to projects not using Gerrit. Github isn't that good.

— Julien Danjou (@juldanjou) May 8, 2013

The problems with Github pull-requests

I always looked at Github from a distant eye, mainly because I always disliked their pull-request handling, and saw no value in the social hype it brings. Why?

One click away isn't one click effort

The pull-request system looks like an incredible easy way to contribute to any project hosted on Github. You're a click away to send your contribution to any software. But the problem is that any worthy contribution isn't an effort of a single click.

Doing any proper and useful contribution to a software is never done right the first time. There's a dance you will have to play. A slowly rhythmed back and forth between you and the software maintainer or team. You'll have to dance it until your contribution is correct and can be merged.

But as a software maintainer, not everybody is going to follow you on this choregraphy, and you'll end up with pull-request you'll never get finished unless you wrap things up yourself. So the gain in pull-requests here, isn't really bigger than a good bug report in most cases.

This is where the social argument of Github isn't anymore. As soon as you're talking about projects bigger than a color theme for your favorite text editor, this feature is overrated.

Contribution rework

If you're lucky enough, your contributor will play along and follow you on this pull-request review process. You'll make suggestions, he will listen and will modify his pull-request to follow your advice.

At this point, there's two technics he can use to please you.

Technic #1: the Topping

Github's pull-requests invite you to send an entire branch, eclipsing the fact that it is composed of several commits. The problem is that a lot of one-click-away contributors do not masterize Git and/or do not make efforts to build a logical patchset, and nothing warns them that their branch history is wrong. So they tend to change stuff around, commit, make a mistake, commit, fix this mistake, commit, etc. This kind of branch is composed of the whole brain's construction process of your contributor, and is a real pain to review. To the point I quite often give up.

Without Github, the old method that all software used, and that many software still use (e.g. Linux), is to send a patch set over e-mail (or any other medium like Gerrit). This method has one positive effect, that it forces the contributor to acknowledge the list of commits he is going to publish. So, if the contributor he has fixup commits in his history, they are going to be seen as first class citizen. And nobody is going to want to see that, neither your contributor, nor the software maintainers. Therefore, such a system tend to push contributors to write atomic, logical and self-contained patchset that can be more easily reviewed.

Technic #2: the History Rewriter

This is actually the good way to build a working and logical patchset using Git. Rewriting history and amending problematic patches using the famous git rebase --interactive trick.

The problem is that if your contributor does this and then repush the branch composing your pull-request to Github, you will both lose the previous review done, each time. There's no history on the different versions of the branch that has been pushed.

In the old alternative system like e-mail, no information is lost when reworked patches are resent, obviously. This is far better because it eases the following of the iterative discussions that the patch triggered.

Of course, it would be possible for Github to enhance this and fix it, but currently it doesn't handle this use case correctly..

A quick look at OpenStack workflow

It's not a secret for anyone that I've been contributing to OpenStack as a daily routine for the last 18 months. The more I contribute, the more I like the contribution workflow and process. It's already well and longly described on the wiki, so I'll summarize here my view and what I like about it.

Gerrit

To send a contribution to any OpenStack project, you need to pass via Gerrit. This is way simpler than doing a pull-request on Github actually, all you have to do is do your commit(s), and type git review. That's it. Your patch will be pushed to Gerrit and available for review.

Gerrit allows other developers to review your patch, add comments anywhere on it, and score your patch up or down. You can build any rule you want for the score needed for a patch to be merged; OpenStack requires one positive scoring from two core developers before the patch is merged.

Until a patch is validated, it can be reworked and amended locally using Git, and then resent using git review again. That simple. The historic and the different version of the patches are available, with the whole comments. Gerrit doesn't lose any historic information on your workflow.

Finally, you'll notice that this is actually the same kind of workflow projects use when they work by patch sent over e-mail. Gerrit just build a single place to regroup and keep track of patchsets, which is really handy. It's also much easier for people to actually send patch using a command line tool than their MUA or git send-email.

Gate testing

Testing is mandatory for any patch sent to OpenStack. Unit tests and functionnals test are run for each version of each patch of the patchset sent. And until your patch passes all tests, it will be impossible to merge it.

Yes, this implies that all patches in a patchset must be working commits and can be merged on their own, without the entire patchset going in! With such a restricution, it's impossible to have "fixup commits" merged in your project and pollute the history and the testability of the project.

Once your patch is validated by core developers, the system checks that there is not any merge conflicts. If there's not, tests are re-run, since the branch you are pushing to might have changed, and if everything's fine, the patch is merged.

This is an uncredible force for the quality of the project. This implies that no broken patchset can ever sneak in, and that the project pass always all tests.

Conclusion: accessibility vs code review

In the end, I think that one of the key of any development process, which is code review, is not well covered by Github pull-request system. It is, along with history integrity, damaged by the goal of making contributions easier.

Choosing between these features is probably a trade-off that each project should do carefully, considering what are its core goals and the quality of code it want to reach.

I tend to find that OpenStack found one of the best trade-off available using Gerrit and plugging testing automation via Jenkins on it, and I would probably recommend it for any project taking seriously code reviews and testing.

My latest contributions to the Emacs' world

Tue, 01 Mar 2011 00:00:00 GMT

I spend too much time writing Emacs Lisp code these days. Unfortunately, the more I do the more I find new useful tools to improve my work-flow and save time for doing more Lisp. D'oh.

I did not work on any big thing these last weeks, so I'm thinking it's a good time to talk about the various code and patches I sent to multiple Emacs packages.

el-get

el-get, a fabulous tool that installs and handles all the external Emacs packages I use. A friendly war started on the development list about autoloads handling. The discussion was overall pointless, since we had a very hard time to communicate our ideas, and we did not understand each others several times.

In the end, el-get now supports autoload correctly and do not load automatically all your packages, improving the startup time, and using the Emacs way to do things. Which is always better, obviously.

git-commit-mode

I've started to use git-commit-mode some times ago. I usually use git-commit with the -v option to see what I'm committing. I though it would be useful to color the diff with diff-mode, so I wrote a patch just to do that, which
was merged today by Florian.

magit

Some weeks ago, I decided to give a try to magit, and loved it. I am not always using it, but for basic operations it is very useful. But I really soon found some things I did not like and therefore send patches to enhance it.

First, I've added a patch to honor status.showUntrackedFiles which I use in my home directory. In the mean time, I've also added a patch to allow adding an arbitrary file.

Yesterday, I sent another pull request, not closed for now, which adds the possibility to visit files in another window from a diff file, and the support for add-change-log-entry directly from the displayed diff. Useful for these old projects still using ChangeLog files but accessible through git (hi Emacs & Gnus!).

Gnus

Nothing remarkable, but I write a couple of fixes and enhancements to the Sieve manage mode, to the Gravatar code and cleaned-up some very very old code. Also added the possibility to
set list-identifier as a group parameter.

Org-mode

I spent most of my time working on my jd/agenda-format branch, which is soon to be merged. I've also just got developer access to the Org-mode patch work and repository, so I'll be able to break things even more! ;-)

ERC

I fixed the bug that annoyed me for a long time. Now erc-track does not reset the last channel status on window visibility changes not made by the user.