jd:/dev/blog

Rant about Github pull-request workflow implementation

2013-05-10T17:55:00Z

One of my recent innocent tweet about Gerrit vs Github triggered much more reponses and debate that I expected it to. I realize that it might be worth explaining a bit what I meant, in a text longer than 140 characters.

I'm having a hard time now contributing to projects not using Gerrit. Github isn't that good.
— Julien Danjou (@juldanjou) May 8, 2013

The problems with Github pull-requests

I always looked at Github from a distant eye, mainly because I always disliked their pull-request handling, and saw no value in the social hype it brings. Why?

One click away isn't one click effort

The pull-request system looks like an incredible easy way to contribute to any project hosted on Github. You're a click away to send your contribution to any software. But the problem is that any worthy contribution isn't an effort of a single click.

Doing any proper and useful contribution to a software is never done right the first time. There's a dance you will have to play. A slowly rhythmed back and forth between you and the software maintainer or team. You'll have to dance it until your contribution is correct and can be merged.

But as a software maintainer, not everybody is going to follow you on this choregraphy, and you'll end up with pull-request you'll never get finished unless you wrap things up yourself. So the gain in pull-requests here, isn't really bigger than a good bug report in most cases.

This is where the social argument of Github isn't anymore. As soon as you're talking about projects bigger than a color theme for your favorite text editor, this feature is overrated.

Contribution rework

If you're lucky enough, your contributor will play along and follow you on this pull-request review process. You'll make suggestions, he will listen and will modify his pull-request to follow your advice.

At this point, there's two technics he can use to please you.

Technic #1: the Topping

Github's pull-requests invite you to send an entire branch, eclipsing the fact that it is composed of several commits. The problem is that a lot of one-click-away contributors do not masterize Git and/or do not make efforts to build a logical patchset, and nothing warns them that their branch history is wrong. So they tend to change stuff around, commit, make a mistake, commit, fix this mistake, commit, etc. This kind of branch is composed of the whole brain's construction process of your contributor, and is a real pain to review. To the point I quite often give up.

A typical case: 3 commits to build a 4 lines long file.

Without Github, the old method that all software used, and that many software still use (e.g. Linux), is to send a patch set over e-mail (or any other medium like Gerrit). This method has one positive effect, that it forces the contributor to acknowledge the list of commits he is going to publish. So, if the contributor he has fixup commits in his history, they are going to be seen as first class citizen. And nobody is going to want to see that, neither your contributor, nor the software maintainers. Therefore, such a system tend to push contributors to write atomic, logical and self-contained patchset that can be more easily reviewed.

Technic #2: the History Rewriter

This is actually the good way to build a working and logical patchset using Git. Rewriting history and amending problematic patches using the famous git rebase --interactive trick.

The problem is that if your contributor does this and then repush the branch composing your pull-request to Github, you will both lose the previous review done, each time. There's no history on the different versions of the branch that has been pushed.

In the old alternative system like e-mail, no information is lost when reworked patches are resent, obviously. This is far better because it eases the following of the iterative discussions that the patch triggered.

Of course, it would be possible for Github to enhance this and fix it, but currently it doesn't handle this use case correctly..

Exercise for the doubtful readers: good luck finding all revisions of my patch in the pull-request #157 of Hy.

A quick look at OpenStack workflow

It's not a secret for anyone that I've been contributing to OpenStack as a daily routine for the last 18 months. The more I contribute, the more I like the contribution workflow and process. It's already well and longly described on the wiki, so I'll summarize here my view and what I like about it.

Gerrit

To send a contribution to any OpenStack project, you need to pass via Gerrit. This is way simpler than doing a pull-request on Github actually, all you have to do is do your commit(s), and type git review. That's it. Your patch will be pushed to Gerrit and available for review.

Gerrit allows other developers to review your patch, add comments anywhere on it, and score your patch up or down. You can build any rule you want for the score needed for a patch to be merged; OpenStack requires one positive scoring from two core developers before the patch is merged.

Until a patch is validated, it can be reworked and amended locally using Git, and then resent using git review again. That simple. The historic and the different version of the patches are available, with the whole comments. Gerrit doesn't lose any historic information on your workflow.

Finally, you'll notice that this is actually the same kind of workflow projects use when they work by patch sent over e-mail. Gerrit just build a single place to regroup and keep track of patchsets, which is really handy. It's also much easier for people to actually send patch using a command line tool than their MUA or git send-email.

Gate testing

Testing is mandatory for any patch sent to OpenStack. Unit tests and functionnals test are run for each version of each patch of the patchset sent. And until your patch passes all tests, it will be impossible to merge it. Yes, this implies that all patches in a patchset must be working commits and can be merged on their own, without the entire patchset going in! With such a restricution, it's impossible to have "fixup commits" merged in your project and pollute the history and the testability of the project.

Once your patch is validated by core developers, the system checks that there is not any merge conflicts. If there's not, tests are re-run, since the branch you are pushing to might have changed, and if everything's fine, the patch is merged.

This is an uncredible force for the quality of the project. This implies that no broken patchset can ever sneak in, and that the project pass always all tests.

Conclusion: accessibility vs code review

In the end, I think that one of the key of any development process, which is code review, is not well covered by Github pull-request system. It is, along with history integrity, damaged by the goal of making contributions easier.

Choosing between these features is probably a trade-off that each project should do carefully, considering what are its core goals and the quality of code it want to reach.

I tend to find that OpenStack found one of the best trade-off available using Gerrit and plugging testing automation via Jenkins on it, and I would probably recommend it for any project taking seriously code reviews and testing.

OpenStack Design Summit Havana, from a Ceilometer point of view

2013-04-25T14:49:45Z

Last week was the OpenStack Design Summit in Portland, OR where we, developers, discussed and designed the new OpenStack release (Havana) coming up.

The summit has been wonderful. It was my first OpenStack design summit -- even more as a PTL -- and bumping into various people I've never met so far and worked with online only was a real pleasure!

Me and Nick ready to talk about Ceilometer new features.

Nick Barcet from eNovance, our dear previous Ceilometer PTL, and myself, talked about Ceilometer and presented the work that bas been done for Grizzly, with some previews of what we'll like to see done for its Havana release. You can take a look at the slides if you're curious.

Design sessions

Ceilometer had his design sessions during the last days of the summit. We noted a lot of things and commented during the sessions in our Etherpads instances.

The first session was a description of Ceilometer core architecture for interested people, and was a wonderful success considering that the room was packed. Our Doug Hellmann did a wonderful job introducing people to Ceilometer and answering question.

Doug explaining Ceilometer architecture.

The next session was about getting feedbacks from our users. We had a lot of surprise to discover wonderful real use-cases and deployments, like the CERN using Ceilometer and generating 2 GB of data per day!

The following sessions ran on Thursday and were much more about new features discussion. A lot ot already existing blueprints were discussed and quickly validated during the first morning session. Then, Sandy Walsh introduced the architecture they use inside StackTach, so we can start thinking about getting things from it into Ceilometer.

API improvements were discussed without surprises and with a good consensus on what needs to be done. The four following sessions that occupied a lot of the days were related to alarming. All were lead by Eoghan Glynn, from RedHat, who did an amazing job presenting the possible architectures with theirs pros and cons. Actually, all we had to do was to nod to his designs and acknowledge the plan on how to build this.

That last two sessions were about discussing advanced models for billing where we got some interesting feedback from Daniel Dyer from HP, and then were a quick follow-up of the StackTach presentation from the morning session.

Havana roadmap

The list of blueprints targetting Havana is available and should be finished by next week. If you want to propose blueprints, you're free to do so and inform us about it so we can validate it. The same applies if you wish to implement one of them!

API extension

I do think the API version 2 is going to be heavily extended during this release cycle. We need more feature, like the group-by functionnality.

Healthnmon

In parallel of the design sessions, discussions took place in the unconference room with the Healthnmon developers to figure out a plan in order to merge some of their efforts into Ceilometer. They should provide a component to help Ceilometer supports more hypervisors than it currently does.

Alarming

Alarming is definitely going to be the next big project for Ceilometer. Today, Eoghan and I started building blueprints on alarming, centralized in a general blueprint.

We know this is going to happen for real and very soon, thanks to the engagements of eNovance and RedHat who are commiting resources to this amazing project!

Hy, Lisp in Python

2013-04-03T22:01:20Z

I've meant to look at Hy since Paul Tagliamonte started to talk to me about it, but never took a chance until now. Yesterday, Paul indicated it was a good time for me to start looking at it, so I spent a few hours playing.

But what's Hy?

Python is very nice: it has a great community and a wide range of useful libraries. But let's face it, it misses a great language.

Hy is an implementation of a Lisp on top of Python.

Technically, Hy is built directly with a custom made parser (for now) which then translates expressions using the Python AST module to generate code, which is then run by Python. Therefore, it shares the same properties as Python, and is a Lisp-1 (i.e. with a single namespace for symbols and functions).

If you're interested to listen Paul talking about Hy during last PyCon US, I recommend watching his lightning talk. As the name implies, it's only a few minutes long.

Does it work?

I've been cloning the code and played around a bit with Hy. And to my greatest surprise and pleasure, it works quite well. You can imagine writing Python from there easily. Part of the syntax smells like Clojure's, which looks like a good thing since they're playing in the same area.

You can try a Hy REPL in your Web browser if you want.

Here's what some code look like:

(import requests)
 
(setv req (requests.get "http://hy.pault.ag"))
(if (= req.status_code 200)
  (for (kv (.iteritems req.headers))
    (print kv))
  (throw (Exception "Wrong status code")))

This code would ouput:

('date', 'Wed, 03 Apr 2013 12:09:23 GMT')
('connection', 'keep-alive')
('content-encoding', 'gzip')
('transfer-encoding', 'chunked')
('content-type', 'text/html; charset=utf-8')
('server', 'nginx/1.2.6')

As you can see, it's really simple to write Lispy code that really uses Python idioms.

There's obviously still a lots of missing features in Hy. The language if far from complete and many parts are moving, but it's really promising, and Paul's doing a great job implementing every idea.

I actually started to hack a bit on Hy, and will try to continue to do so, since I'm really eager to learn a bit more about both Lisp and Python internals in the process. I've already send a few patches on small bugs I've encountered, and proposed a few ideas. It's really exciting to be able to influence early a language design that I'll love to use! Being a recent fan of Common Lisp, I tend to grab the good stuff from it to add them into Hy.

Announcing Climate, the OpenStack capacity leasing project

2013-03-25T17:49:23Z

While working on the XLcloud project (HPC on cloud) it appeared clear to us that OpenStack was missing a critical component towards resource reservations.

A capacity leasing service is something really needed by service providers, especially in the context of cloud platforms dedicated to HPC style workload. Instead of building something really specific, the decision has been made to build a new standalone OpenStack components aiming to provide this kind of functionnality to OpenStack. In the spirit of others OpenStack components, it will be extensible to fullfil a large panel of needs around this problematic.

The project is named Climate, and is hosted on StackForge. It will follow the standard OpenStack development modal. This service will be able to handle a calendar of reservations for various resources, based on various criteria.

The project is still at its early design stage, and we plan to have a unconference session during the next OpenStack summit in Portland to present our plans and ideas for the future!

Ceilometer bug squash day #2

2013-03-04T11:55:00Z

The Ceilometer team is pleased to announce that tomorrow Tuesday 5th March 2013 will be the second bug squash day for Ceilometer.

We wrote an extensive page about how you can contribute to Ceilometer, from updating the documentation, to fixing bugs. There's a lot you can do. We've good support for Ceilometer built into Devstack, so installing a development platform is really easy.

The main goal for this bug day will be to put Ceilometer in the best possible shape before the grizzly-rc1 release arrives (14th March 2013). This version of Ceilometer should be the last one before the final Grizzly release, so it's a pretty important one.

We'll be hanging out on the #openstack-metering IRC channel on Freenode, as usual, so feel free to come by and join us!

OpenStack Ceilometer and Heat projects graduated

2013-02-27T14:49:45Z

The OpenStack Technical Committee have voted these last weeks about graduation of Heat and Ceilometer, to change their status from incubation to integrated.

The details of the discussion can be found in the TC IRC meetings logs for the brave. The results are:

Approve graduation of Heat (to be integrated in common Havana release)?: yes: 10, abstain: 1, no: 1
Approve graduation of Ceilometer (to be integrated in common Havana release)?: yes: 11, abstain: 1

Therefore both projects have been graduated from Incubation to Integrated status. That means that Heat and Ceilometer will be released as part as OpenStack for the next release cycle Havana, due in Autumn 2013.

For people being curious, we the Ceilometer team put up a nice wiki page about our status and what we think we were ready to jump. For the curious, The OpenStack Technical Committee charter has some explanations about the incubation and integration process.

What about Grizzly?

Both projects will be released with Grizzly too, obviously, since they already follow the release process of OpenStack.

What about core?

The question that has been raised several times to me is if that means the projects are becoming Core projects. The answer is no, because how to become a Core project is still under discussion and is more a matter for the Board of Directors than the Technical Committee. But this is definitely a step in this direction.

Anyway, from a technical point of view, this means both projects are now onboard with other OpenStack components so you can enjoy them!

Cloud tools for Debian

2013-02-13T13:21:45Z

Recently, I've worked on the cloud utilities that are provided as standard in Ubuntu, and I ported them to Debian. Let's see how that brings Debian to the cloud!

Basics of a cloud image

When starting an instance on a IaaS platform, your instance image is raw, un-configured. Therefore, you need to have a way to configure it automagically at boot time, based on what you want to do with it. Usually, IaaS platforms provides for this a metadata server, like Amazon EC2 does. It's a special HTTP server listening on a special and hard-coded IP address that your instance can request to know basic information about itself, like its hostname, and retrieve basic user metadata to auto-configure itself. You can check the documentation about the OpenStack metadata service for more information.

Also, image have a predefined size at upload time. So when you run it on a platform, the disk size you request is usually bigger than the size of your image disk: you mayneed to resize and grow your image to use the full disk space that is allocated to your instance.

Needed tools

To run a cloud platform, and especially Amazon EC2 or OpenStack, you need to configure and update your image based on the context you're started in. This also includes extending your template image disk to use the full available disk size provided to the running instance.

Ubuntu provides a set of cloud utils, which is actually composed of different source packages (cloud-init, cloud-utils and clout-initiramfs-tools).

Combined, these 3 packages will allow you to run a number of step, from disk resize at boot time to Puppet configuration handling.

So Ubuntu got this working right a long time ago, but unfortunately, Debian was really late on that.

Until now.

I've worked on getting these into Debian, and you can now find these 3 packages adapted and uploaded to Debian sid.

All you need to do, is to build a Debian image and then run:

apt-get install cloud-init cloud-tools cloud-initiramfs-growroot

And voilà: at the next reboot, your instance will extend its root partition size to the full available disk size, and ask the metadata server to configure things like its hostname.

The packages sources are available on Debian's git server for cloud-utils and cloud-initramfs-tools and you can build them yourself until the packages are processed by ftp-master and get out of the NEW queue. cloud-init on the other hand is directly available in sid.

One of next steps would probably be to build or enhance a tool like vmbuilder to be able to build cloud-compatible Debian images with a simple command line.

Going to FOSDEM 2013

2013-01-29T22:01:20Z

For the first time, I'll be at FOSDEM 2013 in Brussels on Sunday 2nd February 2013.

You'll find me probably hanging out in the cloud devroom where I'll talk about Ceilometer with my fellow developers Nick Barcet and Eoghan Glynn.

I also hope I'll find time to take a peek at some other talks, like PostgreSQL's ones that Dimitri Fontaine will handle.

See you there!

Extending Swift with middleware: example with ClamAV

2013-01-22T09:37:45Z

In this article, I'm going to explain you how you can extend Swift, the OpenStack Object Storage project, so it performs extra action on files at upload or at download time.

We're going to build an anti-virus filter inside Swift. The goal is to refuse uploaded data if they contain a virus. To help us with virus analyses, we'll use ClamAV.

WSGI, paste and middleware

In a pipeline, a black cat can become a white cat with the help of some middleware.

To do our content analysis, the best place to hook in the Swift architecture is at the beginning of every request, on swift-proxy, before the file is actually stored on the cluster. Swift proxy uses, like many other OpenStack projects, paste to build his HTTP architecture.

Paste uses WSGI and provides an architecture based on a pipeline. The pipeline is composed of a succession of middleware, ending with one application. Each middleware has the chance to look at the request or at the response, can modify it, and then pass it to the following middleware. The latest component of the pipeline is the real application, and in this case, the Swift proxy server.

If you've already deployed Swift, you encountered a default pipeline in the swift-proxy.conf configuration file:

[pipeline:main]
pipeline = catch_errors healthcheck cache ratelimit tempauth proxy-logging proxy-server

This is a really basic pipeline with a few middleware. The first one catches error, the second one is in charge to return 200 OK response if you send a GET /healthcheck request on your proxy server. The third one is in charge of caching, the fourth one is used for rate limiting, the fifth for authentication, the sixth one for logging, and the final one is the actual proxy server, in charge of proxying the request to the account, container, or object servers (the others components of Swift). Of course, we could remove or add any of the middleware here at our convenience.

Be aware that the order matters: for example, if you put healthcheck after tempauth, you won't be able to access the /healthcheck URL without being authenticated!

ClamAV

If you don't know ClamAV, it's an antivirus engine designed for detecting trojans, viruses, malware and other malicious threats. Wwe're going to use it to scan every incoming file. To build the middleware, we'll use the Python binding pyclamd. The API is quite simple, see:

>>> import pyclamd
>>> pyclamd.init_unix_socket('/var/run/clamav/clamd.ctl')
>>> print pyclamd.scan_stream(pyclamd.EICAR)
{'stream': 'Eicar-Test-Signature(44d88612fea8a8f36de82e1278abb02f:68)'}
>>> print pyclamd.scan_stream("safe!")
None

Anatomy of a WSGI middleware

Your WSGI middleware should consist of a callable object. Usually this is done with a class implementing the __call__ method. Here's a basic boilerplate:

class SwiftClamavMiddleware(object):
    """Middleware doing virus scan for Swift."""
 
    def __init__(self, app, conf):
        # app is the final application
        self.app = app
 
    def __call__(self, env, start_response):
        return self.app(env, start_response)
 

def filter_factory(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
 
    def clamav_filter(app):
        return SwiftClamavMiddleware(app, conf)
    return clamav_filter

I'm not going to expand more on why this is built this way, but if you want to have more info on this kind of filter middleware, you can read their documentation on Paste.

This middleware will just do nothing as it is. It's going to simply pass all requests it receives to the final application, and returns the result.

Testing our basic middleware

Now is a really good time to add unit tests. I hope you didn't think we were going to write code without some tests, right? It's really easy to test a middleware, as we're going to use WebOb for that.

import unittest
from webob import Request
 

class FakeApp(object):
    def __call__(self, env, start_response):
        return Response(body="FAKE APP")(env, start_response)
 

class TestSwiftClamavMiddleware(unittest.TestCase):
 
    def setUp(self):
        self.app = SwiftClamavMiddleware(FakeApp(), {})
 
    def test_simple_request(self):
        resp = Request.blank('/',
                             environ={
                                 'REQUEST_METHOD': 'GET',
                             }).get_response(self.app)
        self.assertEqual(resp.body, "FAKE APP")

We create a FakeApp class, that represents a fake WSGI application. You could also use a real application, or write a fake application looking like the one you want to test. It'll require more time, but your tests will be closer to the reality.

Here we write the simplest test we can for our middleware. We're just sending a GET / request to it, so it passes the request to the final application and returns the result. It is transparent, it does nothing.

Now, with that solid base we'll able to add more features and test these features incrementally.

Plugging ClamAV in

With our base ready, we can start thinking about how to plug ClamAV in. What we want to check here, is the content of the file when it's uploaded. If we refer to the OpenStack object storage API , a file upload is done via a PUT request, so we're going to limit the check to that kind of requests. Obviously, more checks could be added, but we'll keep things simple here for the sake of comprehensibility.

With WSGI, the content of the request is available in env['wsgi.input'] as an object implementing a file interface. We'll scan that stream with ClamAV to check for viruses.

import pyclamd
from webob import Response
 

class SwiftClamavMiddleware(object):
    """Middleware doing virus scan for Swift."""
 
    def __init__(self, app, conf):
        pyclamd.init_unix_socket('/var/run/clamav/clamd.ctl')
        # app is the final application
        self.app = app
 
    def __call__(self, env, start_response):
        if env['REQUEST_METHOD'] == "PUT":
            # We have to read the whole content in memory because pyclamd
            # forces us to, but this is a bad idea if the file is huge.
            scan = pyclamd.scan_stream(env['wsgi.input'].read())
            if scan:
                return Response(status=403,
                                body="Virus %s detected" % scan['stream'],
                                content_type="text/plain")(env, start_response)
        return self.app(env, start_response)
 

def filter_factory(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
 
    def clamav_filter(app):
        return SwiftClamavMiddleware(app, conf)
    return clamav_filter

That's it. We only check for PUT requests and if there's a virus in the file, we return a 403 Forbidden error with the name of the detected virus, bypassing entirely the rest of the middleware chain and the application handling.

Then, we can simply test it.

import unittest
from cStringIO import StringIO
from webob import Request, Response
 

class FakeApp(object):
    def __call__(self, env, start_response):
        return Response(body="FAKE APP")(env, start_response)
 

class TestSwiftClamavMiddleware(unittest.TestCase):
    def setUp(self):
        self.app = SwiftClamavMiddleware(FakeApp(), {})
 
    def test_put_empty(self):
        resp = Request.blank('/v1/account/container/object',
                             environ={
                                 'REQUEST_METHOD': 'PUT',
                             }).get_response(self.app)
        self.assertEqual(resp.body, "FAKE APP")
 
    def test_put_no_virus(self):
        resp = Request.blank('/v1/account/container/object',
                             environ={
                                 'REQUEST_METHOD': 'PUT',
                                 'wsgi.input': StringIO('foobar')
                             }).get_response(self.app)
        self.assertEqual(resp.body, "FAKE APP")
 
    def test_put_virus(self):
        resp = Request.blank('/v1/account/container/object',
                             environ={
                                 'REQUEST_METHOD': 'PUT',
                                 'wsgi.input': StringIO(pyclamd.EICAR)
                             }).get_response(self.app)
        self.assertEqual(resp.status_code, 403)

The first test test_put_empty simulates an empty PUT request. The second one, test_put_no_virus simulates a regular PUT request but with a simple file containing no virus.

Finally, the third and last test simulates the upload of a virus using the EICAR test file. This is a special test file that is recognized as a virus, even if it's not real one. Very handy for testing virus detection software!

Configuring Swift proxy

Our middleware is ready! We can configure Swift's proxy server to use it. We need to add the following lines to our swift-proxy.conf to teach it how to load the filter:

[filter:clamav]
paste.filter_factory = swiftclamav:filter_factory

We'll assume that our Python modules is named swiftclamava here. Now that we've defined our filter and how to load it, we can use it in our pipeline:

[pipeline:main]
pipeline = catch_errors healthcheck cache ratelimit tempauth clamav proxy-logging proxy-server

Just before reaching the proxy-server, and after the user being authenticated, the content will be scanned for viruses. It's important here to put this after authentication for example, because otherwise we may scan content that will get rejected by the authtemp module, thus scanning for nothing!

Beyond scanning

And voilà, we now have a simple middleware testing uploaded content and refusing infected files. We could enhance it with various other things, like configuration handling, but I'll let that as an exercise for the interested readers.

We didn't exploited it here, but note that you can also manipulate request headers and modify them if needed. For example, we could have added a header X-Object-Meta-Scanned-By: ClamAV to indicates that the file has been scanned by ClamAV.

You should now be able to build your own middleware doing whatever you want with uploaded data. Happy hacking!

Overriding cl-json object encoding

2013-01-11T21:14:21Z

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that.

Identifying the problem

CL-JSON & CLOS

CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:

(defclass kitten ()
  ((tail :initarg :tail)))
 
(json:encode-json-to-string (make-instance 'kitten :tail 'black))

will produce:

{"tail":"black"}

Still using CL-JSON, we can also decode the JSON object to a CLOS object:

(slot-value
 (json:with-decoder-simple-clos-semantics
   (json:decode-json-from-string "{\"tail\":\"black\"}"))
 :tail)

That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article.

Postmodern

Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects.

With this, we can easily store our kitten into a table.

(defclass kitten ()
  ((tail :initarg :tail))
  (:metaclass postmodern:dao-class))

If we try to encode this to JSON, it will produce the exact same result seen previously.

The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol.

So this code:

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))
 
(postmodern:deftable kitten
  (postmodern:!dao-def))
 
(postmodern:connect-toplevel …)
 
(postmodern:create-table 'kitten)
 
(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

will return:

"{"tail":"null"}"

Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string.

This is not at all what we want here!

Overriding encode-json

CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:

(defmethod encode-json ((o standard-object)
                        &optional (stream *json-output*))
  "Write the JSON representation (Object) of the CLOS object O to
STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (stream-object-member-encoder stream) o)))

All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))
 
(export 'kitten)
 
;; Switch package just to define the new method
(in-package :json)
(defmethod encode-json ((o cl-user:kitten)
                        &optional (stream json:*json-output*))
  "Write the JSON representation (Object) of the postmodern DAO CLOS object
O to STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (lambda (key value)
                 (as-object-member (key stream)
                   (encode-json (if (eq value :null) nil value) stream)))
               o)))
 
;; Go back into our package
(in-package :cl-user)
 
(postmodern:deftable kitten
  (postmodern:!dao-def))
 
(postmodern:connect-toplevel …)
 
(postmodern:create-table 'kitten)
 
(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil.

Now if we try to encode another kitten, we'll get:

{"tail":null}

which is far better for our JavaScript data consumers!

In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation. The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp.

The ability to teach any library about how it should handle your class just by defining a new method is really handy!

Integrating cl-irc and cl-async

2013-01-04T12:51:23Z

Recently, I've started programming in Common Lisp.

My idea here is to use cl-irc, an IRC library into an event loop. This can be really useful, for example to trigger action based on time, using timers.

Creating a connection

The first step is to create a basic cl-irc:connection object on our own. This can be achieved easily with this:

(require :cl-irc)
 
(defun connect (server)
  (cl-irc:make-connection :connection-type 'cl-irc:connection
                                              :client-stream t
                                              :network-stream ?
                                              :server-name server))

This will return a cl-irc:connection object, logging to stdout (:client-stream t) and having the server name server. Note that the server name could be any string.

You probably noticed the ? I used as :network-stream value. This is not a real and working value: this should be a stream established to the IRC server you want to chat with. This is where we'll need to use cl-async:tcp-connect to establish a TCP connection.

As you can read in this function's documentation, all we need to pass is the server address, two callbacks for read and general events, and the :stream option to get a stream rather than a socket.

So you would do something like:

(require :cl-irc)
(require :cl-async)
 
(defun connection-socket-read (socket stream)
  (format t "We should read the IRC message from ~a ~%" stream))
 
(defun connection-socket-event (ev)
  (format t "Socket event: ~a~%" ev))
 
(defun connect (server &optional (port 6667))
  (cl-irc:make-connection :connection-type 'cl-irc:connection
                          :client-stream t
                          :network-stream (as:tcp-connect server port
                                                          #'connection-socket-read
                                                          #'connection-socket-event
                                                          :stream t)
                          :server-name server))
 
(as:start-event-loop (lambda () (connect "irc.oftc.net")))

If you run this program, it will connect to the OFTC IRC server, and then notice you each time the server is sending you a message.

Therefore our problem here is how we you treat the message read from the stream in connection-socket-read and handle them in the name of our connection object you used? We can't link both together at this point.

We can't build a closure, because as the time we use as:tcp-connect we don't have the cl-irc:connection instance. Also we can't change easily the read-cb parameter of our network-stream established by as:tcp-connect, simply because cl-async doesn't use to do allow that.

Building a closure

So one solution here is to hack cl-irc:make-connection so we can build an cl-irc:connection instance without providing in advance the network-stream, allowing us to build a closure including the cl-irc:connection to read event for. This is what we're going to do in the connect function.

(require :cl-irc)
(require :cl-async)
(require :flexi-streams)
 
(defun connection-socket-read (connection)
  (loop for message = (cl-irc::read-irc-message connection)
        while message
        do (cl-irc:irc-message-event connection message)))
 
(defun connection-socket-event (ev)
  (format t "Socket event: ~a~%" ev))
 
(defun connect (server port nickname
                &key
                  (username nil)
                  (realname nil)
                  (password nil))
  ;; Build an instance of cl-irc:connection, without any network/output stream
  (let* ((connection (make-instance 'cl-irc:connection
                                    :user username
                                    :password password
                                    :server-name server
                                    :server-port port
                                    :client-stream t))
         ;; Use as:tcp-connect to build our network stream, and build a
         ;; closure calling `connection-socket-read' with our `connection'
         ;; as arguments
         (network-stream (as:tcp-connect server port
                                         (lambda (socket stream)
                                           (declare (ignore socket stream))
                                           (connection-socket-read connection))
                                         #'connection-socket-event
                                         :stream t)))
    ;; Set the network stream on the connection
    (setf (cl-irc:network-stream connection) network-stream)
    ;; Set the output stream on the connection
    (setf (cl-irc:output-stream connection)
         ;; This is grabbed from cl-irc:make-connection
          (flexi-streams:make-flexi-stream
           network-stream
           :element-type 'character
           :external-format '(:utf8 :eol-style :crlf)))
 
    ;; Now handle the IRC protocol authentication pass
    (unless (null password)
      (cl-irc:pass connection password))
    (cl-irc:nick connection nickname)
    (cl-irc:user- connection (or username nickname) 0 (or realname nickname))
    connection))
 
(as:start-event-loop (lambda () (connect "irc.oftc.net" 6667 "jd-blog")))

And here we are! If we run this, we're now using an event loop to run cl-irc. Each time the socket has something to read, the function connection-socket-read will be called on the non-blocking mode socket. If there's no message to be read, then the function will exit and the loop will continue to run.

Using timers

You can now modify the last line with this:

(defun say-hello (connection)
  (cl-irc:privmsg connection "#jd-blog" "Hey I read your blog!")
  (as:delay (lambda () (say-hello connection)) :time 60))
 
(as:start-event-loop (lambda ()
                       (let ((connection (connect "irc.oftc.net" 6667 "jd-blog")))
                         (cl-irc:join connection "#jd-blog")
                         (say-hello connection))))

This will connect to the IRC server, join a channel and then say the same sentence every minute.

Challenge accomplished!

And I'd like to thank Andrew Lyon, the author of cl-async, who has been incredibly helpful with my recent experimentations in this area.

Ceilometer bug squash day #1

2012-12-24T17:45:00Z

In order to start the year in a good mood, what's the best than squashing some bugs on OpenStack?

Therefore, the Ceilometer team is pleased to announce that it organizes a bug squashing day on the Friday 4th January 2013.

The main goal on this bug day will be put Ceilometer in the best possible shape before the grizzly-2 milestone arrives (10th January 2013). This version of Ceilometer will aim to keep compatibility with Folsom, so early deployers can enjoy some of our new features before upgrading to Grizzly. After that date, we'll start merging more extensive changes.

We'll be hanging out on the #openstack-metering IRC channel on Freenode, as usual, so feel free to come by and join us!

Logitech Unifying devices support in UPower

2012-11-16T10:44:00Z

A few months ago, I wrote about my reverse engineering attempt to Logitech Unifying devices. Back then, I concluded my post with big hopes on the future after receiving a document with some part of the specification of the HID++ 2.0 from Logitech.

A couple of weeks ago, some of my summer work has been merged to UPower, adding battery support for some Logitech devices.

HID++

As I discovered late in my first reverse engineering attempt, Logitech developed a custom HID protocol named HID++. This protocol exists in two versions, 1.0 and 2.0. Some devices talk with version 1 of the protocol (like my M705 mouse) and some others talk with version 2 of the protocol (like my K750 keyboard).

Recently, I've been able to be in touch with a Logitech engineer who worked on the Linux support for the Unifying receiver, and he has been really helpful and exposed me some details about this protocol.

Logitech made the decision to publish their HID++ specification publicly about a year ago, but still didn't do it. The internal review needed to publish such documents hasn't be done yet. The only published draft is just an extract of the specification, with even some typo in it as I discovered.

Some other documents have been recently published, but I didn't have the time to review them. They contains HID++ 1.0 specifications and some details I asked for about the K750 keyboard.

UPower support

It took me sometime to get a full understanding of the protocol, its different version etc. After reverse engineering my K750 keyboard, I've also reverse engineered the data stream used to get my M705 mouse battery status. I've also received some information about the HID++ 1.0 protocol, so I've been able to discover a bit more on what the packets mean. Most of my discoveries are now used to do proper #define in up-lg-unifying.c so the code makes more sense.

My first patch implements a new property for UPower devices, named luminosity, that use with K750 keyboard to report the light level received. The second patch add support for Logitech Unifying devices (over USB only) and should work with at least Logitech M705 and K750 devices. This should be available with the next version of UPower, which should be 0.9.19.

So far, Logitech has been kind enough to help me understanding part of the protocol and even sent me a few devices so I can play and test my work with them. Unfortunately, this will probably requires some work and time, and so far Logitech was not able to help with that.

There should be enough information out there to at least add support for battery to HID++ 2.0 devices, and probably a few other things too. I hope I'd get the time do this at some point, but feel free to beat me in this race!

OpenStack France meetup #2

2012-11-06T11:37:00Z

I was at the OpenStack France meetup 2 yesterday evening.

This has been a wonderful evening, talking about OpenStack and all with around 30-40 people. I and Nick Barcet presented Ceilometer and have received some good feedbacks about it. We should also thanks Nebula, who sponsored the evening, and Erwan Gallen since it was nicely organized, and free beers are always enjoyable.

For people interested, the slides of our Ceilometer presentations are available. This is a lighter and fresher version of the slides used by Nick and Doug at the OpenStack Design Summit.

Inside Synaps, a CloudWatch-like implementation for OpenStack

2012-10-22T11:16:00Z

A few days ago, Samsung released the source code of Synaps, an implementation of the Amazon Web Service CloudWatch API for OpenStack.

Being a developer on the Ceilometer project, I've been curious to look on this project and how it could overlap with Ceilometer or other projects like Heat.

What is CloudWatch?

CloudWatch is a monitoring system provided by Amazon on its Web Services platform to monitor services. This allows you get notifications and trigger an action on certain threshold.

For example, this can be used to scale your architecture by monitoring the number of requests you get on it and its general load by starting new servers.

Synaps

Synaps is written in around 7k lines of Python (with 28 % of which are comments), reuses at least one common module of OpenStack (openstack.common.cfg) and copy some modules from Nova. One thing that strikes me, is that there seems to be only a few unit tests compared to most OpenStack projects. Also, many parts of the code and documentation contains text written in korean, which won't be very helpful for most people! :-) It uses some external technologies, like Storm, Cassandra to store its persistent data and Pandas to do data analysis.

The API server provides an EC2 compatible API only: no OpenStack specific API. This is probably not a bad thing for now, since I am not aware of any work in this direction. The API access directly the Cassandra back-end for read operation, but relies on RPC to do writes. This way, a set of daemon handles the write using the Storm part of Synaps and do data aggregation. The authentication only supports LDAP, but it should still be possible to add a driver for Keystone.

A Java and a Python SDK are provided to record metrics into Synaps, but there's not enough documentation for it to be useful.

Overlap with Heat

For now, there's not a lot of overlap with Heat, because Heat does not implement completely the CloudWatch API. Heat actually still misses a lot of the CloudWatch functions. But as soon as it will implement the CloudWatch API completely, the overlap will be complete with Synaps in this regard.

One divergence point however, is that Heat uses RPC to access data from the storage back-end via its engine (the central daemon), whereas Synaps directly connects to Cassandra. Also, Heat relies on SQLAlchemy, like most OpenStack projects needing a database.

Overlap with Ceilometer

One of the goal of Ceilometer is to provide data probes and pollsters for all OpenStack components (Nova, Swift, Quantum…) whereas Synaps let the OpenStack users to put any kind of metric inside it, and therefore doesn't provide anything for now.

But the storage of metrics is the main common point between Synaps and Ceilometer. Synaps chose only one technology, Cassandra, to store its metrics, whereas Ceilometer took care of building an abstraction layer for the storage engine. Ceilometer currently allows an operator to use SQL or MongoDB, but Cassandra could likely be added.

Data metric consolidation is done by Synaps. This makes sense, since Synaps don't need to have the full data history to trigger alarms. On the opposite, Ceilometer needs to have a full history to allow things like billing, and don't do any aggregation on data.

Also, in Synaps, the data analysis is done using Pandas. This means the data used are retrieved from the Cassandra back-end, and then transformed by Pandas inside Synaps in something else. It's likely that in such a case, Synaps should use CQL to achieve that. Ceilometer manipulates the data near their storage: it means that the computation are done by back-end to be efficient (SQL, mapreduce…).

Conclusion

Considering Samsung open-sourced Synaps late in the development process, I don't feel like they aimed to have it becoming a core component. This is always sad, because the effort put into this implementation are big and it would have probably little to add some abstraction layers to follow what other OpenStack projects do. But this takes time and energy, and it's understandable that Samsung didn't want to achieve this in a short time frame.

There's a part of the code and architecture that overlaps with Ceilometer and Heat. Ceilometer is becoming a specialized point to store data metrics from any source: so it's sad, but understandable, that Synaps did not tried to reuse it. Fortunately, Heat is working with Ceilometer to achieve exactly that. This means OpenStack would have only one metrics storage point, used for billing, for monitoring and alarming.

Therefore, I think Synaps is an implementation of CloudWatch that should be looked at as an inspiration for Heat and Ceilometer to build a better and more integrated solution!

Ceilometer 0.1 released

2012-10-12T17:45:00Z

After 6 months of development, we are proud to release the first release of Ceilometer, the OpenStack Metering project. Ceilometer. This is a first and amazing milestone for us: we follow all other projects by releasing a version for Folsom!

Using Ceilometer, you should now be able to meter your OpenStack cloud and retrieve its usage to build statistics or bill your customer!

You can read our announcement on the OpenStack mailing list.

Architecture

We spent a good amount of time defining and refining our architecture.

One of its important point, is that it has been designed to work without modifying any of the existing core components. Patching OpenStack components in an intrusive way to meter them was not an option for now, simply because we had no legitimacy to do so. This may change in the future, and this will likely be discussed next week during the OpenStack Summit.

Meters

Initially, we defined a bunch of meters we'd like to have for a first release, and in the end, most of them are available. Some of them are still missing, like OpenStack Object Storage (Swift) ones, mainly due to lack of interest from the involved parties so far.

Anyhow, with this first release, you should be able to meter your instances, their network usage, memory, CPU. Images, networks and volumes and their CRUD operations are metered too. For more detail, you can read the complete list of implemented meters.

REST API

The HTTP REST API has been partially implemented. The provided methods should allow basic integration with a billing system.

DreamHost is using Ceilometer in their deployment architecture and coupling it with their billing system!

Towards Grizzly

We don't have a clear and established road-map for Grizzly yet.

We already have a couple of patches waiting in the queue to be merged, like the use of Keystone to authenticate API request and the removal of Nova DB access.

On my side, these last days I've been working on a small debug user interface for the API. Ceilometer API server will return this interface if your do an API request from a browser (i.e. requesting text/html instead of application/json).

I hope this will help to discover Ceilometer API more easily for new comers and leverage it to build powerful tools!

Anyhow, we have tons of idea and work to do, and I'm sure the upcoming weeks will be very interesting. Also, we hope to be able to become an OpenStack incubated project soon. So stay tuned!

Gnus notifications

2012-08-29T18:08:00Z

Today, I've merged my Gnus notifications module inside Gnus git repository. This way, it will be available for everybody in Emacs 24.2.

This module allows you to be notified via notifications-notify (the Emacs implementation of the Freedesktop desktop notifications) on new messages received in Gnus. It can also retrieves contacts photo via gravatar.el and google-contacts.el to include them in the notification.

To enable it in Emacs > 24.1, you just have to add the following line to your Gnus configuration file:

(add-hook 'gnus-after-getting-new-news-hook 'gnus-notifications)

If you want to download it and use it stand-alone for a previous Emacs version, you can fetch the latest file revision and load it before adding the previously given line.

Sony Vaio Z Debian Linux support

2012-08-11T15:52:00Z

I had to install Debian Wheezy on a brand new Sony Vaio Z laptop with the new Ivy Bridge architecture (SVZ1311C5E). I'll talk about this here, because it's always nice to know that new hardware works quite fine (or not) under Debian.

The laptop is delivered with Window 7, which I decided to remove entirely anyway, and replace with Debian. I've installed it with Linux 3.2 and then ran Linux 3.4, 3.5 and 3.6-rc1.

USB booting

Don't ask me why, nor an Ubuntu or Debian USB installation booted, blocked at SYSLINUX at best, or at a black screen. This does not work. I had to use PXE to install Debian.

Storage

The only thing that can be surprising, is that the 128 GB SSD storage is actually made of 2 64 GB Samsung SSD aggregated in a RAID 0 using Intel Rapid Storage Technology, previously known as Intel Matrix. This is supported by Linux using the dm-raid module. So this is a fake RAID, and you anyway can see the both drives as sda and sdb under Linux.

Unfortunately, this kind of RAID is not supported correctly by GRUB, and I was unable to install it this way. Therefore, I decided to remove entirely this fake RAID (which is possible via the BIOS) and use a Linux software md RAID 0 instead, plus crypto on top of it. That I know well and I trust. :)

Graphics

The Intel HD Graphics 4000 works fine. I'm alsmo using the HDMI output, which works fine. There's some GPU hanging (as seen on screen and in kernel logs) in Linux up to 3.4, but with versions 3.5 and above, I didn't see any problem so far.

Sound

The Intel HDA sound card works pretty well, both for playing and recording. The main problem is that I hear a constant noise on the speakers, but tweaking the ALSA mixers ends it at some point. There's still probably a bug, not yet resolved in Linux 3.6-rc1.

Keyboard

The keyboard works fine, and the back-light too, via the sony-laptop kernel module. Wonderful.

Touchpad

Touchpad works fine.

Fingerprint

It does not work, and is not supported according to my research. Not that I care about, but don't count on it. It's an AuthenTec AES1660.

Webcam

It works perfectly.

USB

Well, USB 3.0 does not work. I had to disable XHCI in the BIOS and use the 2 ports as standard USB 2.0, otherwise I would just get errors from the kernel. Still not working with Linux 3.6-rc1, and I've no clue to debug, and do not use USB 3.0 yet, so…

WiFi

The WiFi module (based on iwlwifi) works fine. The only problem with NetworkManager is that the sony-laptop offers a second rfkill switch and NM does not know how to handle it correctly. A bug is opened about this and I hope to be able to write a patch or something at some point. Also, there seems to be some quality issue with the iwlwifi driver and 802.11n at this point. I'm losing connection quite often when the signal drops below 40 %. Loading the module with 11n_disable=1 helps a lot.

Ethernet

The gigabit Realtek Ethernet controller works perfectly.

Card reader

Works perfectly.

Ceilometer, the OpenStack metering project

2012-07-27T12:37:00Z

For the last months, I've been working on a metering project for OpenStack, so it's time to talk a bit about it.

OpenStack is a growing cloud platform providing IaaS. A problem easily identified by everyone building a public cloud platform is that nothing is provided to retrieve the platform usage data. Some data are available in some places, but not everything is, and you have to do a lot of processing from the various components to get something useful in the end. But in order to bill customers that are using your public cloud platform, you need to do his.

In this regard, a lot of companies running public OpenStack based infrastructure wrote their own solution to cover this functional areas, and to become able to bill theirs customers.

To avoid everybody doing and maintaining such a stack in their corners, the Ceilometer has been created.

The project aims to cover the metering aspect of the OpenStack components, pulling usage data from every components and storing them into a single place. It then offer a retrieving point for this data via a REST API.

The initial specifications have been written in April this year, and actual implementation started in May. The project is currently worked on by me, Dreamhost and Canonical.

We already have designed an architecture that we are implementing, and we hope to release a first usable version with Folsom.

I did a presentation of this project yesterday at XLCloud, which has been very well received.

If you are interested in helping us and contributing, feel free to join us during one of our weekly IRC meeting or fix some bugs. :-)

Emacs configuration published

2012-07-24T14:15:00Z

I've finally published my Emacs configuration.

This took me a while, since I had personal information inside (like passwords). Recently, I've been able to move them away and can now publish everything in my Git repository.

It's probably not yet usable from scratch, since I didn't include the bootstrap code for el-get. But you can at least lurk and grab some ideas or lines of code. And do not hesitate to ask me anything about it!

Note that I'm using Emacs development version (trunk), so it's possible that some things do not work with (old) released Emacs versions.

ERC notifications

2012-07-21T13:32:00Z

Today, I've merged my erc notifications module inside Emacs trunk. This way, it will be available for everybody in Emacs 24.2.

This module allows you to be notified via notifications-notify (the Emacs implementation of the Freedesktop desktop notifications) on private message received on IRC, or when your nickname is mentioned on a channel.

To enable it in Emacs > 24.1, you just have to add the following line to your configuration file:

(add-to-list 'erc-modules 'notifications)

If you want to download it and use it stand-alone for a previous Emacs version, you can fetch the latest file revision and load it before adding the previously given line.

Logitech K750 keyboard and Unifying Receiver Linux support

2012-07-09T17:48:00Z

A year ago, I bought a Logitech Wireless Solar Keyboard K750. I'm particularly picky on keyboards, but this one is good. It has an incredible useful feature: while being wireless, it has no need for disposable or rechargeable batteries, it uses solar power!

My problem is that there's obviously no way to know the battery status from Linux, the provided application only working on Windows.

And one dark night, while fragging on QuakeLive, my keyboard stopped working: it had no battery left. This activity being quite energy consuming, it emptied the whole battery.

Someone should write code to get the battery status and light meter from Linux: challenge accepted!

How the keyboard works

This keyboard, like many of the new wireless devices from Logitech, uses the Unifying interface. It's an USB receiver that can be attached up to 6 differents devices (mouse, keyboards…). On old Linux kernel, the Unifying receiver is recognized as only one keyboard and/or one mouse device.

Recently, a driver called hid-logitech-dj has been added to the Linux kernel. With this driver, each device attached to the receiver is recognized as one different device.

What the Logitech application does

The Logitech application under Windows works that way: you launch it, and it displays the battery charge level. On the keyboard, there's a special "light" button (up right). When pressed, a LED will light up on the keyboard: green if the keyboard is receiving enough light and is charging, red if the keyboard does not receive enough light and is therefore discharging. Pushing this same button while the application is running will makes the light meter activated: the application will tell you how much lux your keyboard is receiving.

Let's reverse engineer this

As far as I know, there's nothing in the USB HID protocol that handles this kind of functionality (battery status, light meter…) in a standard way. So the first task to accomplish is, unfortunately, to reverse engineer the program.

I discovered a bit too late that Drew Fisher did a good presentation on USB reverse engineering at 28c3. You might want to take a look at it if you want to reverse engineer on USB. I did not need it, but I learned a few things.

Anyway, my plan was the following: run the Logitech application inside a virtual machine running Windows, give it direct access to the USB keyboard, and sniff what happens on the USB wire.

To achieve that, you need a virtual machine emulator that can do USB pass-through. Both KVM and VirtualBox can do that, but VirtualBox works much better with USB and allow hot(un)plugging of devices, so I used it.

To sniff what happens on the USB, you need to load the usbmon Linux kernel module. Simply doing modprobe usbmon will work. You can then use Wireshark which know how to use usbmon devices and understand the USB protocol.

USB stuff you need to know

You don't need to know much about USB to understand what I'll write about below, but for the sake of comprehensibility I'll write a couple of things here before jumping in.

To communicate with an USB device, we communicate with one of its endpoints. Endpoints are regrouped into an interface. Interfaces are regrouped into a configuration. A device might contains one or several configurations.

There's also several types of packets in the USB wire protocol, and at least two of them interest us there, they are:

Interrupt packets, a packet send spontaneously;
Controls packets, used for command and status operations.

All of this and more is well (and better) explained in the chapter 13 of Linux Device Drivers, Third Edition.

Sniffed data

Once everything was set-up, I ran my beloved Wireshark. There's a an URB of type interrupt sent each time you press any key with some data in it. Therefore I advise you to plug another keyboard (or use the laptop keyboard if you're doing this on a laptop), otherwise you'll get crazy trying to sniff the keyboard you're typing on.

At this point, just launching the application does a bunch of USB traffic. Pressing the "light" button on the keyboard makes even more USB packets coming in and out. Here's the interesting packets that I noticed once I excluded the noise:

When pressing the "light" button, an URB of type interrupt is sent by the keyboard to the computer;
An URB control packet is sent by the computer to the keyboard in response;
Regularly URB interrupt packets are sent just after.

With all this, the next step was clear: understand the packets and reproduce that exchange under Linux.

What the packets mean

The "go for the light meter" packet

The packet sent from the computer to the keyboard is the following.

Frame 17: 71 bytes on wire (568 bits), 71 bytes captured (568 bits)
    Frame Length: 71 bytes (568 bits)
    Capture Length: 71 bytes (568 bits)
USB URB
    URB id: 0xffff880161997240
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_CONTROL (0x02)
    Endpoint: 0x00, Direction: OUT
        0... .... = Direction: OUT (0)
        .000 0000 = Endpoint value: 0
    Device: 6
    URB bus id: 1
    Device setup request: relevant (0)
    Data: present (0)
    URB sec: 1340124450
    URB usec: 495643
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 7
    Data length [bytes]: 7
    [Response in: 18]
    [bInterfaceClass: HID (0x03)]
    URB setup
        bmRequestType: 0x21
            0... .... = Direction: Host-to-device
            .01. .... = Type: Class (0x01)
            ...0 0001 = Recipient: Interface (0x01)
    bRequest: SET_REPORT (0x09)
    wValue: 0x0210
        ReportID: 16
        ReportType: Output (2)
    wIndex: 2
    wLength: 7
0000  40 72 99 61 01 88 ff ff 53 02 00 06 01 00 00 00   @r.a....S.......
0010  22 ad e0 4f 00 00 00 00 1b 90 07 00 8d ff ff ff   "..O............
0020  07 00 00 00 07 00 00 00 21 09 10 02 02 00 07 00   ........!.......
0030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0040  10 01 09 03 78 01 00                              ....x..

What's here interesting is the last part representing the data. wLength says that the length of the data is 7 bytes, so let's take a look at those 7 bytes: 10 01 09 03 78 01 00.

Well, actually, you can't decode them like that, unless you're a freak or a Logitech engineer. And I have actually no idea what they mean. But sending this to the keyboard will trigger an interesting thing: the keyboard will start sending URB interrupt with some data without you pressing any more key.

The "light meter and battery values" packet

This is most interesting packet. This is the one sent by the keyboard to the host and that contains the data we want to retrieve.

Frame 1467: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
    Frame Length: 84 bytes (672 bits)
    Capture Length: 84 bytes (672 bits)
USB URB
    URB id: 0xffff88010c43c380
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_INTERRUPT (0x01)
    Endpoint: 0x83, Direction: IN
        1... .... = Direction: IN (1)
        .000 0011 = Endpoint value: 3
    Device: 2
    URB bus id: 6
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1334953309
    URB usec: 728740
    URB status: Success (0)
    URB length [bytes]: 20
    Data length [bytes]: 20
    [Request in: 1466]
    [Time from request: 0.992374000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
Leftover Capture Data: 1102091039000c061d474f4f4400000000000000

0000  80 c3 43 0c 01 88 ff ff 43 01 83 02 06 00 2d 00   ..C.....C.....-.
0010  5d c5 91 4f 00 00 00 00 a4 1e 0b 00 00 00 00 00   ]..O............
0020  14 00 00 00 14 00 00 00 00 00 00 00 00 00 00 00   ................
0030  02 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................
0040  11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00   ....9....GOOD...
0050  00 00 00 00                                       ....

This packets come in regularly (1 per second) on the wire for some time once you sent the "go for the light meter" packet. At one point they are emitted less often and do not contain the value for the light meter anymore, suggesting that the control packet sent earlier triggers the activation of the light meter for a defined period.

Now you probably wonder where the data are in this. They're in the 20 bytes leftover in the capture data part, indicated by Wireshark, at the end of the packet: 11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00 00 00 00 00.

Fortunately, it was easy to decode. Knowing we're looking for 2 values (battery charge and light meter), we just need to observe and compare the packet emitted on the wire with the values displayed by the Logitech Solar App.

To achieve this, I looked both at the Logitech Solar App and Wireshark while bringing more and more light near the keyboard, increasing the lux value received by the meter on the Solar App, and saw that the fields represented in blue (see below) where changing in Wireshark. Since 2 bytes were changing, I guessed that it was coded on 16 bits, and therefore it was easy to correlate the value with the Solar App.

[ ....9....GOOD....... ]
11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00 00 00 00 00
4 bytes - 1 byte for battery charge - 2 bytes for light meter - 2 bytes - 4 bytes for GOOD - 7 bytes

In this example, the battery has a charge of 0x39 = 57 % and the light meter receives 0x0c = 12 lux of light. It's basically dark, and that makes sense: it was night and the light was off in my office, the only light being the one coming from my screen.

I've no idea what the GOOD part of the packet is about, but it's present in every packet and it's actually very handy to recognize such a packet. Therefore I'm considering this as some sort of useful mark for now.

For the other bytes, they were always the same (0x11 0x2 0x9 0x10 at the beginning, 7 times 0x00 at the end). The 2 bytes between the light meter and GOOD probably mean something, but I've no idea what for now.

Building our solar app

Now we've enough information to build our own very basic solar application. We know how to triggers the light meter, and we know how to decode the packets.

We're going to write a small application using libusb. Here's a quick example. It's not perfect and does not check for error codes, be careful.

/* Written by Julien Danjou <julien@danjou.info> in 2012 */
 
#include &lt;linux/hid.h&gt;
 
#include &lt;libusb.h&gt;
#include &lt;stdio.h&gt;
#include &lt;stdlib.h&gt;
#include &lt;string.h&gt;
 
int main(void)
{
    libusb_context *ctx;
    libusb_init(&ctx);
    libusb_set_debug(ctx, 3);
 
    /* Look at the keyboard based on vendor and device id */
    libusb_device_handle *device_handle = libusb_open_device_with_vid_pid(ctx, 0x046d, 0xc52b);
 
    fprintf(stderr, "Found keyboard 0x%p\n", device_handle);
 
    libusb_device *device = libusb_get_device(device_handle);
 
    struct libusb_device_descriptor desc;
 
    libusb_get_device_descriptor(device, &desc);
 
    for(uint8_t config_index = 0; config_index < desc.bNumConfigurations; config_index++)
    {
        struct libusb_config_descriptor *config;
 
        libusb_get_config_descriptor(device, config_index, &config);
 
        /* We know we want interface 2 */
        int iface_index = 2;
        const struct libusb_interface *iface = &config->interface[iface_index];
 
        for (int altsetting_index = 0; altsetting_index < iface->num_altsetting; altsetting_index++)
        {
            const struct libusb_interface_descriptor *iface_desc = &iface->altsetting[al>tsetting_index];
 
            if (iface_desc->bInterfaceClass == LIBUSB_CLASS_HID)
            {
                libusb_detach_kernel_driver(device_handle, iface_index);
                libusb_claim_interface(device_handle, iface_index);
 
                unsigned char ret[65535];
 
                unsigned char payload[] = "\x10\x02\x09\x03\x78\x01\x00";
 
                if(libusb_control_transfer(device_handle,
                                           LIBUSB_REQUEST_TYPE_CLASS | LIBUSB_RECIPIENT_INTERFACE,
                                           HID_REQ_SET_REPORT,
                                           0x0210, iface_index, payload, sizeof(payload) - 1, 10000))
                {
                    int actual_length = 0;
 
                    while(actual_length != 20 || strncmp((const char *) &ret[9], "GOOD", 4))
                        libusb_interrupt_transfer(device_handle,
                                                  iface_desc->endpoint[0].bEndpointAddress,
                                                  ret, sizeof(ret), &actual_length, 100000);
 
                    uint16_t lux = ret[5] << 8 | ret[6];
 
                    fprintf(stderr, "Charge: %d %%\nLight: %d lux\n", ret[4], lux);
                }
 
                libusb_release_interface(device_handle, iface_index);
                libusb_attach_kernel_driver(device_handle, iface_index);
            }
        }
    }
 
    libusb_close(device_handle);
    libusb_exit(ctx);
}

What the program is doing is the following:

Request for the Unifying Receiver device based on vendor and product ID
Get the HID interface
Detach the HID interface from the kernel driver
Claim the interface
Send a control packets, were parameters are defined using the same data we captured earlier
Read interrupt packets coming in until we receive one we recognize (length 20 containing the "GOOD" string)
Decode the content (battery charge & light meter)
Release the interface
Reattach the kernel driver to the interface

This gives the following:

Found keyboard 0x0x24ec8e0
Charge: 64 %
Light: 21 lux

Challenge accomplished!

To be continued

Unfortunately, this approach has at least one major drawback. We have to disconnect the Logitech Unifying Receiver from the kernel. That means that while we're waiting for the packet, we're dropping packets corresponding to other events from every connected device (key presses, pointer motions…).

In order to solve that, I sent a request for help on the linux-input mailing list. That way, I learned that Logitech is actually using the HID++ protocol to communicate with the devices using the Unifying Receiver. Lars-Dominik Braun managed to get the HID++ specifications from Logitech and published them with their authorization.

This opens a whole new world. With that document, I may be able to understand the part I reverse engineered and convert this to a more useful and generic library using the hidraw interface (so we don't have to disconnect the devices from the kernel driver).

Making the jump: working freelance

2012-07-02T10:55:00Z

For the last 10 years, I've been working on many Free Software projects. From Debian to OpenStack, through awesome, Emacs, XCB and many more. This obviously allowed me to enhance my technical skills, but it also taught me about Free Software and Open Source development processes, and how to work with and close to the community.

Working for almost 6 years at Easter-eggs taught me how to work in an autonomous manner, how to lead and manage a project. And how to run a company, thanks to the cooperative status of this great one.

These are the reasons why I decided to leave my latest job and run my own company to work as a freelance consultant & developer specialized in Free Software, starting today.

Therefore, I am now able and available to provide expertise and development on Free Software, including upstream contribution. Especially on projects I already worked on recently, like OpenStack.

How to make Twitter's Bootstrap tabs bookmarkable

2012-06-29T12:21:00Z

I've been using Twitter's bootstrap library recently to build this Web site, and wondered how to be able to use the bootstrap-tab Javascript plugin in a bookmark friendly manner.

I ended up with a simple solution. These are my first steps in Javascript and front-end manipulation, and it's really not my area of expertise, so don't be harsh.

function bootstrap_tab_bookmark (selector) { if (selector == undefined) {
    selector = ""; }
 
    /* Automagically jump on good tab based on anchor */
    $(document).ready(function() {
        url = document.location.href.split('#');
        if(url[1] != undefined) {
            $(selector + '[href=#'+url[1]+']').tab('show');
        }
    });
 
    var update_location = function (event) {
        document.location.hash = this.getAttribute("href");
    }
 
    /* Update hash based on tab */
    $(selector + "[data-toggle=pill]").click(update_location);
    $(selector + "[data-toggle=tab]").click(update_location);
}

All you need is to use and call this function with a selector (only useful if you have several tabs/pills divisions) when the document is ready.

The first part takes care of showing the good tab based on the hash contained in the URL. The second part takes care of changing the document location to add the current tab to it when the user clicks.

OpenStack Swift eventual consistency analysis & bottlenecks

2012-04-23T12:06:00Z

Swift is the software behind the OpenStack Object Storage service.

This service provides a simple storage service for applications using RESTful interfaces, providing maximum data availability and storage capacity.

I explain here how some parts of the storage and replication in Swift works, and show some of its current limitations.

If you don't know Swift and want to read a more "shallow" overview first, you can read John Dickinson's Swift Tech Overview.

How Swift storage works

If we refer to the CAP theorem, Swift chose availability and partition tolerance and dropped consistency. That means that you'll always get your data, they will be dispersed on many places, but you could get an old version of them (or no data at all) in some odd cases (like some server overload or failure). This compromise is made to allow maximum availability and scalability of the storage platform.

But there are mechanisms built into Swift to minimize the potential data inconsistency window: they are responsible for data replication and consistency.

The official Swift documentation explains the internal storage in a certain way, but I'm going to write my own explanation here about this.

Consistent hashing

Swift uses the principle of consistent hashing. It builds what it calls a ring. A ring represents the space of all possible computed hash values divided in equivalent parts. Each part of this space is called a partition.

The following schema (stolen from the Riak project) shows the principle nicely:

In a simple world, if you wanted to store some objects and distribute them on 4 nodes, you would split your hash space in 4. You would have 4 partitions, and computing hash(object) modulo 4 would tell you where to store your object: on node 0, 1, 2 or 3.

But since you want to be able to extend your storage cluster to more nodes without breaking the whole hash mapping and moving everything around, you need to build a lot more partitions. Let's say we're going to build 2¹⁰ partitions. Since we have 4 nodes, each node will have 2¹⁰ ÷ 4 = 256 partitions. If we ever want to add a 5^th node, it's easy: we just have to re-balance the partitions and move 1⁄4 of the partitions from each node to this 5^th node. That means all our nodes will end up with 2¹⁰ ÷ 5 ≈ 204 partitions. We can also define a weight for each node, in order for some nodes to get more partitions than others.

With 2¹⁰ partitions, we can have up to 2¹⁰ nodes in our cluster. Yeepee.

For reference, Gregory Holt, one of the Swift authors, also wrote an explanation post about the ring.

Concretely, when building one Swift ring, you'll have to say how much partitions you want, and this is what this value is really about.

Data duplication

Now, to assure availability and partitioning (as seen in the CAP theorem) we also want to store replicas of our objects. By default, Swift stores 3 copies of every objects, but that's configurable.

In that case, we need to store each partition defined above not only on 1 node, but on 2 others. So Swift adds another concept: zones. A zone is an isolated space that does not depends on other zone, so in case of an outage on a zone, the other zones are still available. Concretely, a zone is likely to be a disk, a server, or a whole cabinet, depending on the size of your cluster. It's up to you to chose anyway.

Consequently, each partitions has not to be mapped to 1 host only anymore, but to N hosts. Each node will therefore store this number of partitions:

number of partition stored on one node = number of replicas × total number of partitions ÷ number of node

Examples:

We split the ring in 2¹⁰ = 1024 partitions. We have 3 nodes. We want 3 replicas of data.
→ Each node will store a copy of the full partition space: 3 × 2¹⁰ ÷ 3 = 2¹⁰ = 1024 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 5 nodes. We want 3 replicas of data.
→ Each node will store 2¹¹ × 3 ÷ 5 ≈ 1129 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 6 nodes. We want 3 replicas of data.
→ Each node will store 2¹¹ × 3 ÷ 6 = 1024 partitions.

Three rings to rule them all

In Swift, there is 3 categories of thing to store: account, container and objects.

An account is what you'd expect it to be, a user account. An account contains containers (the equivalent of Amazon S3's buckets). Each container can contains user-defined key and values (just like a hash table or a dictionary): values are what Swift call objects.

Swift wants you to build 3 different and independent rings to store its 3 kind of things (accounts, containers and objects).

Internally, the two first categories are stored as SQLite databases, whereas the last one is stored using regular files.

Note that this 3 rings can be stored and managed on 3 completely different set of servers.

Data replication

Now that we have our storage theory in place (accounts, containers and objects distributed into partitions, themselves stored into multiple zones), let's go the replication practice.

When you put something in one of the 3 rings (being an account, a container or an object) it is uploaded into all the zones responsible for the ring partition the object belongs to. This upload into the different zones is the responsibility of the swift-proxy daemon.

But if one of the zone is failing, you can't upload all your copies in all zones at the upload time. So you need a mechanism to be sure the failing zone will catch up to a correct state at some point.

That's the role of the swift-{container,account,object}-replicator processes. This processes are running on each node part of a zone and replicates their contents to nodes of the other zones.

When they run, they walk through all the contents from all the partitions on the whole file system and for each partition, issue a special REPLICATE HTTP request to all the other zones responsible for that same partition. The other zone responds with information about the local state of the partition. That allows the replicator process to decide if the remote zone has an up-to-date version of the partition.

In case of account and containers, it doesn't check at the partition level, but check each account/container contained inside each partition.

If something is not up-to-date, it will be pushed using rsync by the replicator process. This is why you'll read that the replication updates are "push based" in Swift documentation.

# Pseudo code describing replication process for accounts
# The principle is exactly the same for containers
for account in accounts:
    # Determine the partition used to store this account
    partition = hash(account) % number_of_partitions
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-account-server process
        version_of_account = zone.send_HTTP_REPLICATE_for(account):
        if version_of_account < account.version()
            account.sync_to(zone)

This replication process is O(number of account × number of replicas). The more your number of account will increase and the more you will want replicas for your data, the more the replication time for your accounts will grow. The same rule applies for containers.

# Pseudo code describing replication process for objects
for partition in partitions_storing_objects:
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-object-server process
        verion_of_partition = zone.send_HTTP_REPLICATE_for(partition):
        if version_of_partition < partition.version()
            # Use rsync to synchronize the whole partition
            # and all its objects
            partition.rsync_to(zone)

This replication process is O(number of objects partitions × number of replicas). The more your number of objects partitions will increase, and the more you will want replicas for your data, the more the replication time for your objects will grow.

I think this is something important to know when deciding how to build your Swift architecture. Choose the right number the number of replicas, partitions and nodes.

Replication process bottlenecks

File accesses

The problem, as you might have guessed, is that to replicate, it walks through every damn things, things being accounts, containers, or object's partition hash files. This means it need to open and read (part of) a every file your node stores to check that data need or not to be replicated!

For accounts & containers replication, this is done every 30 seconds by default, but it will likely take more than 30 seconds as soon as you hit around 12 000 containers on a node (see measurements below). Therefore you'll end up checking consistency of accounts & containers on each all node all the time, using obviously a lot of CPU time.

For reference, Alex Yang also did an analysis of that same problem.

TCP connections

Worst, the HTTP connections used to send the REPLICATE commands are not pooled: a new TCP connection is established each time something has to be checked against the same thing stored on a remote zone.

This is why you'll see in the Swift's Deployment Guide this lines listed under "general system tuning":

# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
 
# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144

In my humble opinion, this is more an ugly hack than a tuning. If you don't activate this and if you have a lot of containers on your node, you'll end up soon with thousands of connections in TIME_WAIT state, and you indeed risk to overload the IP conntrack module.

Container deletion

We also should talk about container deletion. When a user deletes a container from its account, the container is marked as deleted. And that's it. It's not deleted. Therefore the SQLite database file representing the container will continue to be checked for synchronization, over and over.

The only way to have a container permanently deleted is to mark an account as deleted. This way the swift-account-reaper will delete all its containers and, finally, the account.

Measurement

On a pretty big server, I measured the replications to be done at a speed of around 350 {account,container,object-partitions}/second, which can be a real problem if you chose to build a lots of partition and you have a low number_of_node ⁄ number_of_replicas ratio.

For example, the default parameters runs the container replication every 30 seconds. To check replication status of 12 000 containers stored on one node at the speed of 350 containers/seconds, you'll need around 34 seconds to do so. In the end, you'll never stop checking replication of your containers, and the more you'll have containers, the more your inconsistency window will increase.

Conclusion

Until some of the code is fixed (the HTTP connection pooling probably being the "easiest" one), I warmly recommend to chose correctly the different Swift parameters for your setup. The replication process optimization consists in having the minimum amount of partitions per node, which can be done by:

decreasing the number of partitions
decreasing the number of replicas
increasing the number of node

For very large setups, some code to speed up accounts and containers synchronization, and remove deleted containers will be required, but this does not exist yet, as far as I know.

The opinions expressed here represent my own and not those of my employer.

First release of PyMuninCli

2012-04-17T11:06:00Z

Today I release a Python client library to query Munin servers.

I wrote it as part of some experiments I did a few weeks ago. I discovered there was no client library to query a Munin server. There's PyMunin or python-munin which help developing Munin plugins, but nothing to access the munin-node and retrieve its data.

So I decided to write a quick and simple one, and it's released under the name of PyMuninCli, providing the munin.client Python module.

mod_defensible 1.5 released

2012-04-03T15:48:00Z

Apache 2.4 being out, I noticed that my good old mod_defensible did not compile anymore.

The changes in the new Apache 2.4 API were small for its concern, so it was pretty easy to update this software to make it compile again.

Honestly, I'm not sure that this module is really used into the wild, but I still think that it can serve as a good prototype for doing other things so I like keeping it around. :-)

All this has been triggered by the Apache 2.4 arrival into Debian experimental. Therefore I've updated the mod_defensible package to use the new dh_apache2, and imported it into Git at the same time.

xpyb 1.3 released

2012-03-22T22:06:00Z

It took a while to get it out, but finally, 3 years after the latest release (1.2), the version of 1.3 of xpyb (the XCB Python bindngs) is out.

This version has a lot of improvement, and major bug fixes (memory corruption and memory leak were tracked down and fixed).

One amazing feature that is now shipped with that release, is my code to export the xpyb API to other Python modules, allowing to draw with Pycairo in Python using XCB.

Here is an example of a Python program that draws a spiral in a window using xpyb and Pycairo. You need xpyb >= 1.3 and Pycairo >= 1.10 to make this works.

import cairo
import xcb
from xcb.xproto import *
 
WIDTH, HEIGHT = 600, 600
 
def draw_spiral(ctx, width, height):
    """Draw a spiral with lines!"""
    wd = .02 * width
    hd = .02 * height
 
    width -= 2
    height -= 2
 
    ctx.move_to (width + 1, 1-hd)
    for i in range(9):
  ctx.rel_line_to (0, height - hd * (2 * i - 1))
  ctx.rel_line_to (- (width - wd * (2 *i)), 0)
 ctx.rel_line_to (0, - (height - hd * (2*i)))
 ctx.rel_line_to (width - wd * (2 * i + 1), 0)
 
    ctx.set_source_rgb (0, 0, 1)
    ctx.stroke()
 
# Connect to the X server
conn = xcb.connect()
# Get the X server setup
setup = conn.get_setup()
# Generate X ID for our X "objects"
window = conn.generate_id()
pixmap = conn.generate_id()
gc = conn.generate_id()
# Create a new window
conn.core.CreateWindow(setup.roots[0].root_depth, window,
                       # Parent is the root window
                       setup.roots[0].root,
                       0, 0, WIDTH, HEIGHT, 0, WindowClass.InputOutput,
                       setup.roots[0].root_visual,
                       CW.BackPixel | CW.EventMask,
                       [ setup.roots[0].white_pixel, EventMask.ButtonPress | EventMask.EnterWindow | EventMask.LeaveWindow | EventMask.Exposure ])
 
# Create a pixmap: it will be used to draw with cairo
conn.core.CreatePixmap(setup.roots[0].root_depth, pixmap, setup.roots[0].root,
                       WIDTH, HEIGHT)
 
# We just need a GC to copy later the pixmap on the window, so create one
# very simple
conn.core.CreateGC(gc, setup.roots[0].root, GC.Foreground | GC.Background,
                   [ setup.roots[0].black_pixel, setup.roots[0].white_pixel ])
 
# Create a cairo surface
surface = cairo.XCBSurface (conn, pixmap,
                            setup.roots[0].allowed_depths[0].visuals[0], WIDTH, HEIGHT)
# Create a cairo context with that surface
ctx = cairo.Context(surface)
 
# Paint everything in white
ctx.set_source_rgb (1, 1, 1)
ctx.set_operator (cairo.OPERATOR_SOURCE)
ctx.paint()
 
# Draw our spiral
draw_spiral (ctx, WIDTH, HEIGHT)
 
# Map the window on the screen so it gets visible
conn.core.MapWindow(window)
 
# Flush all X requests to the X server
conn.flush()
 
while True:
    try:
        event = conn.wait_for_event()
    except xcb.ProtocolException, error:
        print "Protocol error %s received!" % error.__class__.__name__
        break
    except:
        break
 
    # ExposeEvent are received when we need to refresh the content of the
    # window, so we copy the content of the pixmap (where cairo drew) in the
    # window
    if isinstance(event, ExposeEvent):
        conn.core.CopyArea(pixmap, window, gc, 0, 0, 0, 0, WIDTH, HEIGHT)
    # You click, I quit.
    elif isinstance(event, ButtonPressEvent):
        break
    conn.flush()

Seeing the complexity it is to draw something simple with this technology, I somehow understand why nobody bothered to release or use the code during the last 3 years.

But hey, now that it's out, you can build the next Python based desktop environment with bleeding edge technologies. :-)

10 years as a Debian developer

2012-02-24T08:55:00Z

Ten years ago, I joined the Debian project as a developer.

At that time, I was 18 and in my first year at university, hanging out with the TuxFamily system administrators, which included 3 french Debian developers (sjg, igenibel and creis).

I was learning Debian packaging while working on VHFFS, and decided to package one or two non-yet-packaged software for Debian. My friends pushed me into the NM process, and less than 2 months later I was a Debian developer. One have to admit that back in the days, the NM process was really fast if you were able to reply to the questions quickly. :-) I think I became the youngest developer among Debian's ones.

That was my first steps in a Free Software project, and it was really exciting.

In 10 years, I've been doing a lot of different things for Debian. Sure, I've been using it all the years long, but let's recap a bit what I did, from what I recall.

My first Debian only project was apt-build around 2003, and later rebuildd in 2007.

I built the Xen packaging team in 2005, I've been a Stable Release Manager for a year in 2006, and did heavy bug squashing to release Etch that same year.

I also was an Application Manager in 2006 and managed the application of 2 Debian developers (Jose Parrella and Damián Viano).

I admit I've been less active in Debian after 2007, mainly because I was busy working on awesome, GNU Emacs and others software.

Since 2011, I joined the OpenStack packaging team and I'm working on OpenStack on a (almost) daily basis.

I don't know how many packages I touched, managed or updated, but that should be one or two hundreds. I still maintain 53 of them.

After all, the adventure has been really pleasant, and I had the chance to work with and meet fabulous and smart people. I always liked this project and what it's trying to do.

After all these years, I'm definitively staying! See you in another 10 years, folks! :)

Google Calendar notifications using pynotify

2012-01-03T18:55:00Z

I use Google Calendar to manage my calendars, and I really missed something to warn me whenever I have an appointment with an alert set.

So here is an example of a Python program to do such a thing. It is written using the Google Data APIs Python client library and pynotify.

I'll detail the code here, so you can build your own and adapt it to your needs.

First, we need to import GTK+ and pynotify, and initialize it.

import gtk
import pynotify
pynotify.init(sys.argv[0])

Then, we need to import gdata Calendar API and connect to the calendar. I'll use the simple email/password way to login, which is clearly not the best, but it's also the simplest. Feel free to use OAuth 2.0. :-)

calendar_service = gdata.calendar.service.CalendarService()
calendar_service.email = 'mygooglelogin'
calendar_service.password = 'mygooglepassword'
calendar_service.ProgrammaticLogin()

Now we're ready to request stuff and notify! First, request the events from the default calendar.

feed = calendar_service.GetCalendarEventFeed()

Now we can iterate over feed and do various checks.

for event in feed.entry:
    # If the event status is not confirmed, go to the next event.
    if event.event_status.value != "CONFIRMED":
        continue
    # Now iterate over all the event dates (usually it has one)
    for when in event.when:
        # Parse start and end time
        try:
            start_time = datetime.datetime.strptime(when.start_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
            end_time = datetime.datetime.strptime(when.end_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
        except ValueError:
            # ValueError happens on parsing error. Parsing errors
            # usually happen for "all day" events since they have
            # not time, but we do not care about this events.
            continue
        now = datetime.datetime.now()
        # Check that the event hasn't already ended
        if end_time > now:
            # Check each alert
            for reminder in when.reminder:
                # We handle only reminders with method "alert"
                # and whose start time minus the reminder delay has passed
                if reminder.method == "alert" \
                        and start_time - datetime.timedelta(0, 60 * int(reminder.minutes)) < now:
                    # Build the notification
                    notification = pynotify.Notification(summary=event.title.text,
                                                         message=event.content.text)
                    # Set an icon from the GTK+ stock icons
                    notification.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_DIALOG_INFO,
                                                                              gtk.ICON_SIZE_LARGE_TOOLBAR))
                    notification.set_timeout(0)
                    # Show the notification
                    notification.show()

Running this program, you should see a notification if an appointment has an alert to be raised at that time.

This should be enough to start to build something.

If you don't want to program this into Python, you might want to take a look at gcalcli.

Using GTK+ stock icons with pynotify

2011-12-27T11:55:00Z

It took me a while to find this, so I'm just blogging it so other people will be able to find it.

I wanted to send a desktop notification using pynotify, but using a GTK+ stock icons.

With the following snippet, I managed to do it.

import pynotify
pynotify.init("myapp")
import gtk
n = pynotify.Notification(summary="Summary", message="Message!")
n.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_HARDDISK, gtk.ICON_SIZE_LARGE_TOOLBAR))
n.show()

Note that the use of a Label is just to have a widget instanciated to use the render_icon() method. It could be any widget type as far as I understand.

My OpenStack work

2011-12-16T17:34:00Z

Like I already wrote here last week, I've been heavily working on OpenStack for the last weeks.

My first assignment was to package OpenStack for Debian. The packages already present in unstable were mainly done by Thomas Goirand, who based its work on the one done in Ubuntu. Therefore, the packages where not in a very good shape for Debian.

Today Ghe Rivero and I (members of the OpenStack Debian packaging team) managed to push the OpenStack Essex 2 milestone into unstable with great success. You can now test and deploy OpenStack Essex 2 very easily!

Packaging OpenStack made me write several patches, mainly related to packaging, patches which were all accepted and merged by upstream. This is nice because most of the OpenStack Debian packages lost their debian/patches directories now!

Finally, I've finished to implement one blueprint I really missed: the ability to boot from an ISO image using libvirt. The code still needs a review, but it should be included in the Essex 3 milestone if everything's right.

New job, new blog

2011-12-07T13:47:00Z

It has been a while since I blogged but I've been very busy, with my new job and this new blog!

New job!

I quitted my job last September, and found another one that I started in October. I'm now the lead developer of eNovance Labs, where I work on the OpenStack project. So far, this allowed me to contribute heavily to the Debian packaging of OpenStack.

New blog!

In the meantime, I took some time to redesign my personal homepage and this blog, which is now using Hyde, the Python equivalent of Jekyll, which is in Ruby. Since I dislike Ruby (sorry), I preferred to use a Python based generator, and I admit Hyde is really cool. Since I really suck at Web design, this one is obviously based on Twitter's bootstrap

Google Contacts for Emacs

2011-09-26T13:01:00Z

I finally finished a thing I was really missing: accessing my Google Contacts from Emacs.

That's now possible, thanks to my new google-contacts.el package.

It includes searching for any contact and displaying the result in a window. You can also jump to a contact from Gnus by pressing a key, and complete e-mail addresses while composing a mail.

OAuth 2.0 for Emacs

2011-09-23T18:01:00Z

This week, I've finished my OAuth 2.0 client implementation for GNU Emacs.

I have imported it into GNU ELPA so Emacs 24 users will be soon able to install it using the new Emacs packaging system.

OAuth 2.0 can be used to access, among others, Google APIs or the Facebook Graph API.

Quitting my job

2011-08-29T10:57:00Z

After more than 5 years at Easter-eggs as a system engineer, I'll be leaving my job soon.

It has been a fabulous adventure, also due to the "cooperative" nature of the company. I've enjoyed worked here, with great people. I do wish them luck for the future. Looking at the numerous things I did for the past years, it has been quite productive!

Therefore, I'll be looking for a new job in the next weeks, which will probably keep me busy a bit. :-)

Python sets comparisons

2011-05-17T14:01:00Z

This week I lost some time playing with Python's sets.

After digging into Python source code, I finally discovered there is what seems to be little bug. Anyway, it has been "fixed" in Python 3, fortunately. I did not find if it was reported somewhere, but since it's fixed, it's not a big deal.

Python 2.7.1+ (default, Apr 20 2011, 10:53:33) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class A(object):
...     def __eq__(self, other):
...             return True
... 
>>> A() == A()
True
>>> [A()] == [A()]
True
>>> set([A()]) == set([A()])
False

This clearly did not make any sense to me. I've then tested under Python 3.2:

Python 3.2.1a0 (default, May  4 2011, 19:59:25) 
[GCC 4.6.1 20110428 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class A(object):
...     def __eq__(self, other):
...             return True
... 
>>> set([A()]) == set([A()])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'A'

At least, raising an error is saner. It actually helped me to understand what I needed to do to have my sets working correctly with Python 2:

Python 2.7.1+ (default, Apr 20 2011, 10:53:33) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class A(object):
...     def __eq__(self, other):
...             return True
...     def __hash__(self):
...             return 123456789
... 
>>> set([A()]) == set([A()])
True

Why not Lua

2011-04-26T18:53:00Z

Since my latest announcement of the Lua workshop, I received a couple of emails asking why I discourage the use of Lua.

Actually, I already wrote out many of the things I dislike about Lua. I won't come back on this technical issues here, but since Lua 5.2 is not yet released (it's still at alpha stage), they are still relevant nowadays.

Stack based API is harder

The ease of integration of Lua into a C program is one of the point of Lua. They claim it's very easy to integrate Lua into your C application, because it does not use pointer, nor reference counting, nor anything that requires a minimum amount of skills to be used.

It uses a virtual stack based approach. You push or pop things on a stack, and refers to them using a relative or absolute index.

In order to people who never wrote Lua code to understand, here's a quick example on how this work. The L pointer is a Lua environment.

/* Create a table on the stack: index 1 */
lua_newtable(L);
/* Push a string on the stack: index 2 */
lua_pushstring(L, "hello");
/* Push a number on the stack: index 3 */
lua_pushnumber(L, 123);
/* Set newtable["hello"] = 123 */
lua_settable(L, -3);

You first push a table (in Lua, a table is almost equivalent to what you'd call a hash table in other language), then push the key, the value, and do the assignment operation. In the settable, we use -3 as index, meaning the "3rd item on the stack counting from top". We could also have written lua_settable(L, 1), since the table is also the first item on the stack from the bottom.

So far, so good.

Problems arise when you do more complicated stuff. My previous example is what you would typically find in a tutorial, but of course, real life is different, and usually more complex. If you cut the things in different parts, it can start to be more complicated.

Let's take a look at the following:

/* Create a table on the stack: index 1 */
lua_newtable(L);
/* Push a string on the stack: index 2 */
lua_pushstring(L, "hello");
/* Push a number on the stack: index 3 */
lua_pushnumbe(L, mycomputingfunction());
/* Set newtable["hello"] = 123 */
lua_settable(L, -3);

Here, we do exactly the same thing, but we do not push 123 directly: we compute it.

And here's the trick: if your computing function is also using the Lua stack, things can become very messy. As long as your computing function use the stack cleanly by pushing and poping all its item, and returning the stack in the same state it was before, you're safe. The problem is that in a complex program, you also write bugs. You do not chose to, but you do. And sometimes, you forget to pop one of the item you fetched from a table.

Imagine that mycomputingfunction is:

int
mycomputingfunctiong(void)
{
  /* Just push the table we want to fetch
     the number from on the stack */
  pushatableonstack(L);
  lua_pushstring(L, "mykey");
  lua_gettable(L, -2);
  return lua_tonumber(L, -1);
}

This function works perfectly. It pushes a table, then a key ("mykey"), then fetches mytable["mykey"] and pops the key (lua_gettable does push value/pop key itself), and then returns the numeric value of the last item (the fetched one) of the stack.

However, this function has a bug: it does not pop the table! This does not prevent the function to work. It does not raise a segmentation fault. It does not show any problem under gdb. It does not show any leak under Valgrind. It does now show any problem under any standard C debugging tool.

But when you'll start using it, your program will start to do weird things, and you'll have to spend a huge amount of time debugging it manually, dumping the stack content at each step of your program to watch out what's wrong.

Another bad thing, that can happen, is some code poping accidentally an item from the stack, or worst, from an empty stack. This does not raise any error on the Lua side, but will break your program in very unfunny way.

Even if I've been very meticulous writing awesome, but we hit that problem regularly.

The easiest workaround is to use lua_settop(L, 0) to reset the stack to 0 element. Doing this regularly (like after each program event or treatment) can remove left-over items and avoid the never ending stack grow you may experience if your left-over items continue to pile up. Did I tell you I dislike work-around?

You could also use lua_call(), which would avoid such an error, but this would require a huge amount of indirection, and would make write more (useless) code.

This kind of problem does not exists with pointer based API. If you screw things up, the problem will cause a segmentation fault or leak memory, or cause things you can (easily) debug with standard tools like gdb or Valgrind.

No reference counting is a pain in the ass

Userdata objects are variable Lua size objects embedding a C struct you define. It's the equivalent of an object in object oriented language.

Lua does not provide any reference counting for the userdata objects. That means you can push this objects on the stack, use them, but they cannot directly reference each others. If you have a "car" userdata and a "wheel" one, the car cannot hold directly a reference to the wheel. This is not possible because userdata are allocated and garbage collected by Lua, and there's no way to increase the reference counting yourself.

So the common hack is to store the wheel into a table as a value, and store the table index as an integer into the car data structure.

This obviously makes memory leaks tracking harder, add huge level of reference indirection in usage (still more code), and does not make the whole process less error prone (at least in my opinion).

No paradigm makes you lose time

Lua is proud to come with no paradigm and to provide metatables. I already showed 3 years ago that it has big flaws.

To me, this ain't no good. Lua is not functional, nor it is object oriented.

Most people, including me, want one of this paradigm, or any else. Plain old imperative is not enough.

So you'll start to build more, or to use something like LOOP, which implements an object model. You'll implement your paradigm. I say life is too short to (re)write a paradigm.

In awesome we wanted to have an object oriented approach (this is kind of typical in such a graphical application context), so we tried to build one. To me, this started to be a show stopper when I realized that I've ended writing Python object model into Lua while developing awesome (which aims to be a window manager, not a language). This is one of the reason I stopped hacking on Lua things.

I liked Python object model and wanted to have it in Lua, and spending time rewriting Python is just not worth it. I probably should have chose Python, not Lua. YMMV.

Embedding may not be a good choice

This is not Lua related, but I want to mention it. Googling for "embedding vs extending" will probably tell you more about why you should double check that you really need to embed Lua rather than to extend it.

Being small is not an excuse

One common argument to choose Lua is that it has a small footprint. Yeah, that's true, but that's useless. Bummer! When I program, I don't have any resource usage pressure. People who have such pressure are either paranoids or playing in the world of embedded computers. This is also a no more existing conception since quad core processors equiped phones are coming into the market. I'm rather confident that what we used to call embedded devices are just dead and are now plain computers. But as usual, YMMV.

So start to forget about it, run in your underpants and yelling "yay we killed that shit!", and then use real computers stuff. :-)

Even if benchmarks show how Lua is damn fast, remember what a benchmark proves: that you can do useless things very fast.

Too few extension modules

This is not directly Lua's fault, but there's too few extension modules for Lua. The community is quite small compared to other big languages' ones.

So think twice

before you choose Lua (or any other language). My recommendations these days would be not to embed, but to extend. If you really have no choice and need to embed a language into your application, GNU Guile is probably worth considering, because it's a Scheme and therefore a functional language :-), and because it can provides also different languages.

Including Lua.

Lua workshop at Fabelier/tmplab

2011-04-14T14:04:00Z

It seems I'll be at the Lua workshop at Fabelier/tmplab on April 28th 2011, where I'll try to present and talk about Lua, how to use it, and why you should probably not use it. ;-)

Using advanced filter with mod_authnz_ldap

2011-04-04T14:02:00Z

As you may know, Apache's mod_authzn_ldap allows to authenticate users in Apache HTTP server using an LDAP server. Unfortunately, it has a little implementation flaw.

The filter used to authenticate the user is built by abusing the RFC 2255 which specifies the LDAP URL format. This format has an "attribute" field which is normally used to specify which attributes should be returned. But mod_authzn_ldap uses this attribute to compare with the username given by the client. That means that you have to have an attribute in your LDAP entries which matches the username, and you have to use it in the "attribute" part of the URL to get things working.

Therefore, I wrote a patch to add a format string in the LDAP URL in order to user the provided username in the filter, and ignore the attribute part of the URL, which has no use in such a context anyway.

The bug has been opened in ASF Bugzilla and has number #51005, with the patch. The patch is backward compatible with the current configuration format, which is not the best choice in theory, but probably the more pragmatic.

I've no clue on the typical delay for patches in clusion in Apache HTTP server, so le t's just wait'n see.

Org contacts now part of org-contrib

2011-03-18T18:08:00Z

Thanks to my recent promotion allowing me to commit directly in Org-mode, I've moved Org-contacts into the contrib directory of the Orgmode distribution.

Second OrgCamp Paris

2011-03-11T17:01:00Z

I'll be at the OrgCamp April 2011 in Paris next month. However, I'm not yet sure of what I'll talk about this time.

My latest contributions to the Emacs' world

2011-03-01T14:36:00Z

I spend too much time writing Emacs Lisp code these days. Unfortunately, the more I do the more I find new useful tools to improve my work-flow and save time for doing more Lisp. D'oh.

I did not work on any big thing these last weeks, so I'm thinking it's a good time to talk about the various code and patches I sent to multiple Emacs packages.

el-get

el-get, a fabulous tool that installs and handles all the external Emacs packages I use. A friendly war started on the development list about autoloads handling. The discussion was overall pointless, since we had a very hard time to communicate our ideas, and we did not understand each others several times.

In the end, el-get now supports autoload correctly and do not load automatically all your packages, improving the startup time, and using the Emacs way to do things. Which is always better, obviously.

git-commit-mode

I've started to use git-commit-mode some times ago. I usually use git-commit with the -v option to see what I'm committing. I though it would be useful to color the diff with diff-mode, so I wrote a patch just to do that, which was merged today by Florian.

magit

Some weeks ago, I decided to give a try to magit, and loved it. I am not always using it, but for basic operations it is very useful. But I really soon found some things I did not like and therefore send patches to enhance it.

First, I've added a patch to honor status.showUntrackedFiles which I use in my home directory. In the mean time, I've also added a patch to allow adding an arbitrary file.

Yesterday, I sent another pull request, not closed for now, which adds the possibility to visit files in another window from a diff file, and the support for add-change-log-entry directly from the displayed diff. Useful for these old projects still using ChangeLog files but accessible through git (hi Emacs & Gnus!).

Gnus

Nothing remarkable, but I write a couple of fixes and enhancements to the Sieve manage mode, to the Gravatar code and cleaned-up some very very old code. Also added the possibility to set list-identifier as a group parameter.

Org-mode

I spent most of my time working on my jd/agenda-format branch, which is soon to be merged. I've also just got developer access to the Org-mode patch work and repository, so I'll be able to break things even more! ;-)

ERC

I fixed the bug that annoyed me for a long time. Now erc-track does not reset the last channel status on window visibility changes not made by the user.

Handling my music collection with git-annex

2011-02-23T17:44:00Z

git-annex is a recent tool allowing to manage files with git without having theirs contents checked into git.

I've been looking at it with some interest in the past few weeks in order to use it to handle my music collection on the various computers I use.

My laptop just got a new SSD drive which is quite small and cannot handle the whole collection. By using git-annex, I'm now able to only check out part of my music collection and keeping it synchronized with all my others computers.

Using git-annex is pretty trivial once you understand the concepts.

I've set up a a git repository in my Music directory, like that:

% git init
Initialized empty Git repository in /home/jd/Music

Then I initialized git annex:

% git annex init "Music on keller"

So now I'm able to import all my date into git annex. git-annex offers several backends to store data. WORM is the default, but is not a good choice for file that could be modified, and music files can be, at least when editing the metadata. Therefore, I chose the slow but more reliable SHA1 backend.

% git annex add --backend=SHA1 .

That can be long, so you should be patient. Then you should commit the fact you annexed files:

% git commit -a -m "Imported my music collection"
[master 82d060f] Imported my music collection

Now, it's done. You can go on another computer and clone the repo:

% git clone ssh://keller/~/Music
% cd Music
% git annex get "Incubus - Light Grenades"

And then let the magic happens. It will retrieve the whole directory for you. Do not forgot to commit the fact that you got those album on that machine!

The concept used by git-annex is quite simple. All files are replaced by symlinks pointing to .git-annex/SHA1:filechecksum.log. This file contains lines, whose format is:

retrieval-timestamp is-file-present uuid-of-the-repository

Indicating if the file is present in a repository. This logs file are commited into the repository, whereas the real content is stored in the .git/annex directory and therefore not commited.

I recommend to take a look to the walkthrough page to get more idea and what you can and cannot do with git-annex.

Announcing Org-contacts

2011-02-08T12:22:00Z

When I started to use Emacs, I got hooked by many stuff like Gnus and Org-mode. One thing I quickly started to hate is how the Lisp code can be old and unmaintained. That especially applies to BBDB, which has been unmaintained for years, and has very very very old and obsolete code.

Therefore I've decided to develop my own BBDB replacement based on my lovely Org-mode. It's called org-contacts, and it allows you to handle your contact like anything you would handle in Org. This way you can manage them the way you want, without any preset fields or any assumptions like BBDB has.

I had the chance to present it at the Paris OrgCamp a couple of weeks ago, and due to the enthusiastic audience I had, I'm now releasing it to the wide Internet.

Naquadah theme for Emacs

2011-01-31T16:31:00Z

I often post Emacs screenshots on this blog, and consequently receive a bunch of request for my Emacs theme. Therefore I decided to publish it.

Emacs snapshot Ubuntu packages

2011-01-24T15:13:00Z

Emacs snapshot packages for Ubuntu are now available, thanks to Dimitri Fontaine.

OrgCamp Paris 2011 review

2011-01-23T13:26:00Z

Yesterday afternoon, I was at the first OrgCamp in Paris.

It was my first attendance to a BarCamp, and I really liked it. It's basically the first geek event I do not find boring nor useless.

There was about 18-20 persons participating, which was quite high, since we all initially though we would have been only 5.

We had several presentations of various features and personal usages of BBDB replacement named org-contacts (I'll probably talk about it on this blog in another post later).

The only downside was that Bastien (the new Org-mode maintainer) was not able to come and join us. On the other side, there were so much to tell for a first time, I did not have so much time to code. I only have been able to fix one bug reported during my agenda presentation.

In the end, the overall atmosphere was very enthusiastic and friendly, which was extremely pleasant. The #org-mode-fr IRC channel has been created on Freenode, following this event. Feel free to join us.

Since people liked it so badly, it seems there should be another barcamp in the next months. Stay tuned.

Code fontification with Gnus and Org-mode

2011-01-20T14:45:00Z

I've added code fontification using Org src blocks inside Gnus.

This interprets the block as an Org buffer and fontify it accordingly if org-src-fontify-natively it set to t.

Thanks to Sébastien Vauban for the original idea and implementation. Now it works out of the box without any customization.

OrgCamp 2011

2011-01-17T13:51:00Z

I'll be at the OrgCamp 2011 in Paris next Saturday.

Emacs snapshot Debian packages

2011-01-12T15:54:00Z

I've decided to take over the maintenance of the unofficial emacs-snapshot Debian packages that were maintained by Romain Francoise.

They are available on a dedicated page.

Color contrast correction

2010-11-23T16:14:00Z

I finally took some time to finish my color contrast corrector.

It's now able to compare two colors and to tell if they are readable when used as foreground and background color for text rendering. If they are too close, the code corrects both colors so to they'll become distant enough to be readable.

To do that, it uses color coordinates in the CIE Lab* colorspace. This allows to determine the luminance difference between 2 colors very easily by comparing the L component of the coordinates. The default threshold used to determine readability based on luminance difference is 40 (on 100), which seems to give pretty good results so far.

Then it uses the CIE Delta E 2000 formula to obtain the distance between colors. A distance of 6 is considered to be enough for the colors to be distinctive in our case, but that can be adjusted anyway. That depends on reader's eyes.

If both the color and luminance distances are big enough, the color pair is considered readable when used upon each other.

If these criteria are not satisfied, the code simply tries to correct the color by adjusting the L (luminance) component of the colors so their difference is 40. Optionally, the background color can be fixed so only the foreground color would be adjusted; this is especially handy when the color background is not provided by any external style, but it the screen one (like the Emacs frame background in my case).

Here is an example result generated over 10 pairs of random colors. Left colors are randomly generated, and right colors are the corrected one.

bg: DarkSeaGreen4 fg: gray67 ->                             bg: #4a6b4b fg: #cccccc
bg: SlateGray4 fg: forest green ->                          bg: #9faec0 fg: #005700
bg: grey13 fg: grey36 ->                                    bg: #131313 fg: #6c6c6c
bg: MediumPurple2 fg: honeydew ->                           bg: #9e78ed fg: #f0fff0
bg: grey43 fg: chartreuse3 ->                               bg: #5e5e5e fg: #79de25
bg: linen fg: DeepPink2 ->                                  bg: linen fg: DeepPink2
bg: CadetBlue4 fg: blue1 ->                                 bg: #6c9fa4 fg: #0000e1
bg: gray33 fg: NavajoWhite3 ->                              bg: #525252 fg: #cfb58c
bg: chartreuse1 fg: RosyBrown3 ->                           bg: #9cff38 fg: #b28282
bg: medium violet red fg: DeepPink1 ->                      bg: #9c0060 fg: #ff55b9

All this has been written in Emacs Lisp. The code is now available in Gnus (and therefore in Emacs 24) in the packages color-lab and shr-color.

A future work would be to add support for colour blindness.

As a side note, several people pointed me at the WCAG formulas to determine luminance and contrast ratio. These are probably good criteria to choose your color when designing a user interface. However, they are not enough to determine if displayed color will be readable. This means you can use them if you are a designer, but IMHO they are pretty weak for detecting and correcting colors you did not choose.

Elisp color manipulation routines

2010-11-20T22:11:00Z

Last week, I spent some time implementing various color manipulation routines. The ultimate goal was to find a way to determine if a text in a certain color was readable on a background with a different color.

Something I failed to do so far, despite my research in the area.

However, since I think my code could be useful for other people, I've set up a tiny git repository with the routines I wrote.

The funniest one to implement was CIEDE2000. I verified my code with the data given in the specifications and can assure it's correct. :-)

Org-mode and holidays

2010-11-15T16:11:00Z

Org-mode has a nice option which allows you to show week-end days in a different color in your agenda. That means that Saturday and Sunday (when I do not work) are fontified with org-agenda-date-weekend.

But there are other days I do not work, like my vacations or holidays.

Therefore, I've wrote a patch to add `org-agenda-day-face-function' which is optionally called to determine what should be the face used to fontify a day. This allows me to use the same face for holidays and for week-end days, like for last Thursday which was an holiday in France.

That patch has been merged in Org last week.

Google Maps for Emacs: moving, caching and home

2010-11-08T12:33:00Z

Last week, I worked on my Google Maps for Emacs extension. I've introduced a new format handling for locations which include the longitude and latitude. The initial format was just a string describing the location, which was obviously too limited.

It now prints coordinates of the different elements when the mouse is over the map, with other information.

It also center the map on M-x google-maps and set a default zoom level. This is something which was not set because it's not a good idea to set center coordinates in order to see all points on the map automatically. But you can still remove the centering by pressing "C". On the other hand, setting it automatically allows to move the map easily, and I think that what most users want to do.

I've also added a "place my home on the map" feature, accessible by pressing "h" on any map. That adds a marker according to the location set in Emacs using the calendar- variables.

This feature is also available under [[http://orgmode.org][Org]] by pressing C-u C-c M-l, which shows the location of your appointment with your home on the map too.

Finally, you also get caching so it does not request images you already seen, which makes the moving nicer and faster to use, and prompt history.

Icon category support in Org-mode

2010-11-04T13:11:00Z

My latest patch for Org mode has been accepted by Carsten today. It adds support for custom category icons in all views, like agenda or todo.

You just need to configure org-agenda-category-icon-alist and it will work out of the box.

Transparent GIF support in Emacs 24

2010-11-02T11:43:00Z

Last week, I wrote a couple of patches to add support for transparency when Emacs is displaying GIF images.

Until now, it was displaying the color used to define transparency in the file data. Now it displays the image correctly by using the frame color as the transparency color, like it's done for other image formats.

The patches have not been merged yet, but will probably be soon.

No more dashes in Emacs 24 mode-line

2010-10-20T13:01:00Z

We all know the good old Emacs mode-line you got under every window. Since the beginning (a long time ago), it starts and ends with dashes. I've proposed a patch to remove them.

Before:

After:

This has been merged in Emacs 24. You won't see any more ugly dashes in graphical mode.

Enhancing Emacs mouse avoidance

2010-10-19T19:23:00Z

Recent Emacs versions have a wonderful capacity to hide the mouse pointer as soon as you type and insert characters in a buffer. This is controlled by the `make-pointer-invisible' variable, which is set to t by default.

However that does not hide the pointer when simply moving the cursor on screen. Therefore, I've started to use `mouse-avoidance-mode', which make the mouse pointer jump if your cursor hits it.

Unfortunately, if your cursor hits the invisible mouse pointer, `mouse-avoidance-mode' makes it jump too, because it does not know it is invisible.

Well, it did not know. Now it does, thanks to my patches which have been merged in Emacs 24. Using the new function frame-pointer-invisible-p, one can know if the mouse pointer has been hidden by Emacs. Therefore I enhanced `mouse-avoidance-mode' to use it, and everything is alright now. :-)

Why notmuch is not much good

2010-10-07T13:22:00Z

I've recently got a mail from one of my faithful reader, asking why not considering notmuch.

Actually, I think notmuch already exists in a better way, and that's called IMAP and Sieve.

What notmuch does, is tagging your mail with tags (obviously), based on filtering rules you write. The big downside is that you have to tag all your mails on your computer.

And if you use several computers, you'll have to tag several times your mails. And you'll have to find a way to maintain your rules to be identical on all your computers. That does not scale.

Using Sieve for mail filtering, one can do already that actually, and much more.

A notmuch rule like:

notmuch tag +intel from:intel.com and not tag:intel

Can be written as a Sieve rule like:

if address :all :contains "From" "intel.com" {
    addflag "intel";
}

The flags extension for Sieve is explained in RFC5232.

The Sieve based solution has the advantage of being treated server side, and therefore not subject to multiple or different MUA usages. It's also fast, if you use a good IMAP server like Dovecot, which has indexing, etc.

Furthermore, Sieve can obviously do a lot more than tagging, like splitting into different mailboxes, filtering with regexp usage, vacation, etc.

And if you want to fetch your mail locally, you can synchronize the IMAP box entirely with any software able to (like OfflineIMAP).

Now, what's probably missing, is a correct support for IMAP flags on various MUA around. But that's not something notmuch helps to solve either. :-)

Gnus and Gravatar support

2010-09-25T09:38:00Z

This last couple of days I've been dedicated making Gnus… fresher.

I've decided to give a whirl on Gravatar support. I already tried the gravatar.el lying on the Interweb, but well, it was crap: it used wget to fetch pictures, therefore was totally synchronous. Reading each mail was slower. The cache did not even have TTL, as far as I recall.

So, I've now wrote gravatar.el implementing the Gravatar API. Asynchronously of course. With cache, TTL, etc. Perfect. :-)

Then I've composed gnus-gravatar.el, implementing a washing function adding Gravatar for From field and/or Cc/To fields, like done for picons.

As I was expecting, the patch was badly received by the GNU guys, which start talking about thinks like external resources, privacy, non-free software, etc. Boring.

Fortunately, Lars allowed me to push the patch in git so everybody can give it a try. I'm now waiting for feedbacks in order to know if I will have to maintain this patch outside Gnus, or not.

Here's the mandatory screen-shot.

Gnus news is good news!

2010-09-23T10:21:00Z

As I already wrote too many times, I've started to use Gnus 6 months ago, and never looked back.

At that time, I joined the ding mailing list in order to ask some dumb questions and, once, send a patch. There were very low activity on that list.

Until Lars, the original Gnus author, came back.

Three weeks ago, he started to wrote a new wash function to render HTML mails properly, with pictures. It's named `gnus-html', and is (for now) based on w3m (but not on emacs-w3m, which is not part of Emacs).

Last week, I've sent a set of patches to replace the usage of curl by the standard `url-retrieve' function to fetch images, plus various enhancement. It seems that my work was good enough that Lars offered me write access to the git repository. I can therefore mess up the Gnus entirely. Hurrah!

I've continued to work on gnus-html and recently merged a set of patches improving image retrieval (which is now done in parallel) and starting to use `url-cache' to cache image for a defined period of time. Of course, I found a bunch of tiny bug and special case while reading RSS feeds and various HTML mails, and fixed them all along.

Lars added a libxml binding for Emacs 24, providing the `html-parse-string' function. His future plan seems to be the abandon of w3m in favor of a native parsing via libxml to render HTML, and therefore, HTML mails.

I should also mention the new `nnimap' back-end; Gnus has been designed to read NNTP newsgroups, and not mails. Consequently, it had a very poor behaviour when used with a back-end such has (IMAP)[http://en.wikipedia.org/wiki/Internet_Message_Access_Protocol].

Lars took a week to rewrite entirely our dear `nnimap' back-end, and make it act in a more expected way. There's still a bunch of bug and code to write, but it is at least usable and seems faster than the old code.

Last thing I did was to rewrite the icon support in the group buffer. When I started to use Gnus, I was curious and tried to configure this. I never managed to make it work, and now know and understand why it was broken. So I ended rewriting entirely, and now it works. I never though I would understand, fix, and commit this code when reading the Gnus documentation this winter, but hell yeah, I did.

Now I've still several little project to improve things in all sort of area. We'll see what I'll do next. :-)

Emacs, Org, whatever the weather!

2010-09-08T22:44:00Z

Another week, another Emacs extension!

I had (once again) a wonderful idea: what if I could have the weather forecasts in my Org agenda? Wouldn't that be wonderful?

My quest started by looking for a service offering a good weather forecast API. I found nothing simple as the hidden Google Weather API, which is nice, but… not documented. Not at all. Not a single line. Nah.

Then, I wrote a google-weather extension, implementing a basic Emacs Lisp API to retrieve data from the Google service:

ELISP> (google-weather-data->forecast (google-weather-get-data "Paris"))
(((9 8 2010)
  (low "53")
  (high "63")
  (icon "http://www.google.com/ig/images/weather/rain.gif")
  (condition "Rain"))
 ((9 9 2010)
  (low "53")
  (high "69")
  (icon "http://www.google.com/ig/images/weather/chance_of_rain.gif")
  (condition "Scattered Showers"))
 ((9 10 2010)
  (low "54")
  (high "72")
  (icon "http://www.google.com/ig/images/weather/partly_cloudy.gif")
  (condition "Partly Cloudy"))
 ((9 11 2010)
  (low "55")
  (high "75")
  (icon "http://www.google.com/ig/images/weather/partly_cloudy.gif")
  (condition "Partly Cloudy")))

My API even implements data caching, which is nice to speed up the agenda display.

By the way, I think my next job will be to hack on the url-cache feature of Emacs, which is utterly buggy and has probably never be used. But that's another topic.

Finally, I just had to write another module on top of that to export the forecasts to Org. A screen shot is probably better than a long and boring explanation, so here's the result.

My only regret is that the icons provided by Google are ugly squares, so I did not want to use them. On the other hand, I did not found any icon set that would have all the icons Google provides (around 20). So I felt back on the icon naming specification to map the Google images to standard images. Any better idea would be welcome, of course.

All the information can be found on the Google Weather for Emacs extension homepage.

Emacs and OfflineIMAP

2010-09-03T15:08:00Z

I recently decided to use OfflineIMAP to synchronize my mails on my laptop. It's a great piece of software, and allows me to read my mail while I'm offline.

I use it with Gnus, of course. But I lacked a proper way to integrate OfflineIMAP with it, so I decided to write a little Emacs extension to run and monitor OfflineIMAP directly from Emacs.

Here comes offlineimap.el, an Emacs extension to run OfflineIMAP directly within Emacs. It will display OfflineIMAP output in a buffer, and optionally shows the current OfflineIMAP operation in the mode line.

By default the status is in the mode line only if you are in the Gnus group buffer. But that's customizable, of course, since this is Emacs!

If you are using el-get, there's already a recipe to install it!

Emacs, Google Maps and BBDB

2010-08-18T00:00:00Z

Today's fun idea was to put all my contacts stored into BBDB on a Google Maps' map, using my Google Maps extension for Emacs.

With the help of a few lines of Lisp glue:

(google-maps-static-show
 :markers
 (mapcar
  (lambda (address-entry)
    `((,(concat
         (mapconcat
          'identity
          (elt address-entry 1) ", ") ", "
          (elt address-entry 2) ", "
          (elt address-entry 3) ", "
          (elt address-entry 4) ", "
          (elt address-entry 5)))))
  (mapcan
   (lambda (record)
     ;; We need to copy the returned list, because mapcan will modify it later
     (copy-list (bbdb-record-addresses record)))
   (bbdb-records))))

we can make that:

It's really simplistic, but I did not need more to have fun. :-) This could be extended to set a specific marker and/or color for each contact, with a legend. I'll let that as an exercise for my readers.

Updating muse-el in Debian

2010-08-15T20:43:00Z

The Debian package of Emacs Muse was maintained by Michael W. Wolson, who is also the upstream author of that software.

He announced months ago that Muse needed a new upstream maintainer. That's not something I'm willing to do, since I really think Muse has been superseded by Org mode nowadays.

However, I'm still using Muse to maintain this blog with my muse-blog extension, since Org still lacks some infrastructure to maintain and publish a blog.

Therefore, I adopted the Emacs Muse Debian package and updated it to the latest version!

Update on rainbow-mode

2010-08-10T00:00:00Z

rainbow-mode had a big success and good feedbacks when I released it for the first time a couple of months ago.

Several users asked to me request its inclusion into Emacs. Therefore, some days ago, I proposed to merge it inside Emacs trunk. My request has been denied, but the mode has been added to the Emacs 24 package repository.

In the mean time, I've added support for hsl() and hsla() support, and added CSS 3/SVG color names.

Porting D-Bus to XCB: story of a failure

2010-07-29T00:00:00Z

Even if I recently stated I lost some of my faith in XCB, I still sometimes hack things to add support for it.

These last days, I've worked on a D-Bus port from Xlib to XCB. The port was quite straight forward, since there's only a little piece of D-Bus using X, which is dbus-launch.

I though D-Bus was a good candidate, since it's part of the Freedesktop initiative. Therefore, I was expecting a warm welcome and some enthusiasm from a fellow project.

My contribution got one useful review, and a cold reply from Thiago Macieira (a KDE/Qt/Nokia developer):

No, sorry, I don't agree.. I've just checked and my Solaris machine doesn't have XCB. Please do not remove the X11 code. You may add the XCB code, but you cannot remove the X11 code.

This is not really the kind of answer I expected, actually. I then reworked the code to please Thiago, and added some #ifdef to add XCB support to D-Bus, with a fallback to libx11 where XCB would not be available.

Havoc Pennington replied:

Given that libX11 now uses xcb as backend, I don't understand the value of porting to use libxcb directly when there isn't an issue of round trips or other stuff. It will just make #ifdef hell, while the X11 API is an API that works on both xcb and non-xcb platforms. Maybe people should be thinking about porting xcb to non-Linux platforms? The X protocol should be the same on other UNiX, so xcb in theory ought to work fine if you just compiled it on Solaris/BSD, same as GTK or dbus or Qt would work fine.

The last part "Maybe people should be thinking about porting xcb to non-Linux platforms?" is still unclear to me, even though I asked Havoc to explain what he meant.

Finally, Thiago refused to merge the patch:

[…] thanks for the patch, but like Havoc I am unsure of the value. We can't drop the X11 codepaths now because too many systems exist without XCB. Adding the XCB codepaths only made it more complex, even though you did a good job.

I can't disagree with that conclusion: using both XCB and X11 make the code unreadable for little gain. That's why I did replace libx11 by XCB directly in the first version of the patch. On the other hand, D-Bus people does not seems to really care about making their software evolve in the right direction, even if that requires users to upgrade their systems.

I think D-Bus using and depending on XCB would have been a good point to push adoption of XCB. Unfortunately, it seems you can't even rely of projects of the same initiative (i.e. Freedesktop) to work together to make things a little bit better.

After 5 years of existence, XCB is still not so obvious to people, and making it adopt is going to be a challenge for the next years. The upside is that new X.org 7.6 will bring XCB with it, as part of the katamari.

M-x google-maps

2010-06-28T00:00:00Z

Since I have started to use Org-mode, I though it was missing something to have appointment locations on a map. Of course, it's easy to get a LOCATION property from an entry, and then browse-url on Google Maps.

But it is too easy for me, so once again I said: challenge accepted! I will bring Google Maps into Emacs!

After several hours of work, the google-maps.el project shows a map!

It fully implements the Google Static Maps API and the Google Maps Geocoding API.

You can type M-x google-maps and type some place to see it marked on map. Of course you can do much more, as seen in the screen shot above.

I've also completed all of this with a small org-location-google-maps.el which simply show a Google Maps' map for the location of an event in Org mode by pressing C-c M-l in an Org buffer or in the Org agenda.

Announcing rainbow-mode

2010-06-16T00:00:00Z

While customizing Emacs this last weeks, I had the need to customize also the color theme.

Color themes are always a pain in the ass to edit, because you're supposed to read color strings like #aabbcc and guess what colors they represent.

This is why I wrote [[http://julien.danjou.info/rainbow-mode.html][rainbow-mode]], a minor mode for Emacs that will highlight strings that represents color, using the color they represent.

This support hexadecimal syntax, HTML color name, X color names and rgb() CSS syntax.

Desktop notification support for Emacs

2010-06-09T00:00:00Z

This last weeks, I've worked on implementing the Desktop Notification Specification into Emacs.

It allows sending desktop notification in a very simple way.

(notifications-notify
    :title "You've got mail!"
    :body "There's 34 mails unread"
    :app-icon "~/.emacs.d/icons/mail.png"
    :urgency 'low)

It supports the protocol signals (NotificationClosed and ActionInvoked) and the two main methods (Notify and CloseNotification).

The methods specification are implemented entirely (hints, replaces, actions, icon, etc).

The signals are supported via callbacks function provided on the notification creation.

It have been merged into Emacs trunk today.

2010-06-09  Julien Danjou  <julien@danjou.info>
 
 * net/notifications.el: New file.

This also allowed me to discover, raise and fix a bug in the D-Bus binding of Emacs, which will be probably fixed in trunk soon.

Announcing erc-track-score

2010-06-07T00:00:00Z

A couple of months ago, I've started using ERC to hang out on IRC.

I've read all the pages on EmacsWiki about it, just to see how far I could customize it.

I must admit that I was not disappointed, even if I expected to be. It's quite a nice software, and once well configured it's more convenient that my old [[http://www.irssi.org][irssi]] setup.

While browsing EmacsWiki, I read an interesting idea about channel scoring/temperature on the [[http://www.emacswiki.org/emacs/ErcChannelTracking#toc9][erc-track]] page. The idea is to see if it's worth jumping to an IRC channel to see what people are talking about.

Challenge accepted!

I sat up and started to dig though ERC source code to find the information I needed about variables and functions.

I finally did write something nice, which I called erc-track-score. And yet another piece of software I wrote for my lovely Emacs!

How does it work? Ha-ha, I was sure you would ask. You're so predictable, dude! Read the following, and you'll know everything you ever wanted to know about it since the moment you read the title of that blog entry.

Which probably turned you on.

Nasty you.

First of all, the score of a channel starts at zero. Zero means "seriously, don't bother, nothing is happening here".

Upon each new message arrival, the score is incremented by 1. If a new message contains a keyword, your nickname or is sent by a pal, the score is increased by configurable values, by default between 2 and 20 points, depending on the match type. On the other hand, when a message is send by some fool, the score is decreased by 1 by default.

Obviously, if the score is going negative, you really should not jump to the channel.

Finally, the score is permanently and slowly brought back to 0. By default, the score is decreased by 1 point every 10 seconds.

Overall, reading the score should gives you a good idea of the channel temperature.

I'm still not sure what is the best formula to compute the score, but so far the default values seem quite good. We'll see.

Thoughts and rambling on the X protocol

2010-06-01T00:00:00Z

Two years ago, while working on awesome, I joined the Freedesktop initiative to work on XCB. I had to learn the arcane of the X11 protocol and all the mysterious and old world that goes with it.

Now that I've swum all this months in this mud, I just feel like I need to share my thoughts about what become a mess over the decades.

When I was unborn…

…the Toto band were releasing their song "Africa" and some smart guys were working on a windowing system: the X Window System (this is its full name) which therefore has a (too) long history. The latest version of its protocol, the 11th one, has been designed in the 80's. You can learn more about the history in the Wikipedia article about X.

In 2010, we still listen disco music and we still use various protocols designed in the 80's and even before X. Music have evolved, protocols have evolved, and so did X11.

The problem is that X11 did not evolve that well. The guys at MIT-and-some other-places-with-very-smart-people-in-it created X version 1 in 1984, and updated it until X version 11 (the one we're still using) in 1987. Eleven version in 3 years, that was following the "release early, release often" model. But I don't know why, it just stopped to happen for the last 23 years (that's not totally true: they added (and then deprecated) many extensions.)

I don't know what changes have been made in the first 11 major versions of the X protocol, but I'm rather sure we should have deserve a couple of major version updates this last 2 decades.

In my humble opinion, X11 was not designed to live 23 years. But hey, I'm not blaming anyone here: I was 4 years old and playing Lego ® when they released this latest version of the X protocol, so there is little chance I'd have done something better.

We won't fix. We'll work-around.

That is probably one of the guideline of the X protocol for the last years. And don't misread me: I'm not bashing anyone thereafter.

Since the X11 protocol was aging, the X guys started to add extensions. They added tons of them over the years. This, in application of one of the early principles of X:

It is as important to decide what a system is not as to decide what it is. Do not serve all the world's needs; rather, make the system extensible so that additional needs can be met in an upwardly compatible fashion.

All of them with no exception were added because, bad luck, the X11 protocol did not anticipated the things that happened in the last 23 years, like video, OpenGL, multiple monitors, or the pleasure to draw oval windows. Some of this extensions are still in use, while some of them have been dropped.

While this is not a bad thing to extends the protocol, it seems like a bad thing to try to fix the protocol with for example the XFixes extension, even with all the good intentions Keith Packard might have in his greatness.

Actually it's even worst than you think

The X11 protocol (without extensions) defines about 120 types of requests: create a window, move a window, etc.

Nowadays, there's at least 25 % of them which are useless: usage of server-side font, or the drawing of squares and polygon, are unused by any modern application or toolkit. All of this is superseded by requests from extensions, like the XRender one.

The handling of multiple monitors displays has totally been screwed up. X11 has been designed to work in Zaphod mode (independent monitors). But Xinerama, and nowadays XRandR have replaced it up: recent X servers (released after ~2007) does not support Zaphod mode anymore, even if it's a core piece of the X11 protocol.

Worst: on many requests, there's limitation or design flaws, like described in this document: Why X Is Not Our Ideal Window System by DEC researchers.

We'll add more broken standard on top of that

Following its early principle, X does not define policies but only mechanisms, which seems like a good thing,

Consequently, people started writing specifications to determine a number of stuff and dogmas: ICCCM. That was 22 years ago in 1988. It's useless to add that many things in this specification are now obsolete, useless, or that it misses many modern stuff.

I was not the only one to think that. The people from what will be the major desktop environments, KDE and GNOME, saw that too in the 90's while I was learning to count. So they wrote EWMH, another standard that comes on top of ICCCM and extends it with nifty features like maximization, full screen mode, etc.

The problem is that this standard has also been written by narrow-minded people who at that time, were working on GNOME or KDE (and maybe others). This desktop environments were having and still have some strong concepts of how should work a desktop: "it should have work-spaces", "a window is only on one workspace", "we only see a workspace at a time", "you do not have multiple screens", etc.

Dude, we don't care: we have toolkits!

This vision of how the desktop should work have now been written in marble in all applications and libraries implementing EWMH, like GTK+ or Qt.

Nowadays, everybody forgot about all of this standards. Toolkits have implemented this, circumvented the X11 protocol limitation and flaws, and nobody wants to look back.

Like all standards, obviously some people implemented them badly. This had some side effects, like [OpenOffice acting like a pager].

We don't look back? Worst, we forgot where we came from!

With all these layers of bad designed standards, the desktop continued to evolve for more than a decade. They continued to add more standard, the more recent ones being based on D-Bus like the Desktop Notification Specification or the latest Status Notifier Specification developed by KDE.

The Status Notifier is a new implementation of the good old system tray based on D-Bus instead of the X11 mechanisms, and adding the possibility to show the system tray with something else than icons.

This specification draft saw an important issue and design flaw raised by Wolfgang Draxinger in this thread on the XDG mailing-list. What Wolfgang points out, is that X is network-oriented, and D-Bus is not. Therefore, making the Status Notifier specification to use D-Bus to pass system tray messages around is a bad idea, since running a X application from host A on host B will draw the system tray on the wrong host!

Apparently, reading the thread, this does not fear some of the KDE people:

of course this is a bizarre corner case not worth much thought. at least that's what you'll think until you actually run into it yourself (be it because you are testing something or because you are setting up some weird kiosk environment).

What Oswald describes as a corner case is an actual common use case for many of us. Of course, YMMV.

From my point of view, this is a step back in the wrong direction. But we can conclude that the network part of X is now worthless, to at least KDE.

I used to believe in XCB

When I joined Freedesktop, it was to work on XCB, the X C Binding. XCB is a nice, clean, 21st century technology based API to play with the X11 protocol. Its code is auto generated based on XML file describing the protocol.

In comparison, Xlib is 80's obfuscated code with almost no comments and hard-coded things. Only a few people can understand some of its corner like its i18n or XKB implementations. And all its code is synchronous.

For people not knowing it yet, X is a network protocol where you send request (like a GET in HTTP) and then get a response. Xlib forces the application to wait for the reply to its request, so the application is blocked until the X server sends the reply to the request. XCB on the other hand does not block and allows the application to send a batch of requests, do some other stuff in the mean time, and then gets the replies.

It's like your Web browser would send one request at a time to a Web server, and would wait until you downloaded all the images one by one to display the page.

In cases where X and all its clients are on the same host, the latency is small and not really visible, therefore the gain for XCB to be asynchronous is small. On slow network however, the gain can be huge, as proved in the rewrite of xlsclients with XCB by Peter Harris.

One of the long standing goal of the XCB folks is to kick-out Xlib, to increase speed and hides latency in X11 applications. That requires to port many libraries, because almost none of them (Cairo being an exception) supports XCB.

From where I stand, I don't really see if the work is worth it now. The desktop world is trusted by GNOME and KDE, meaning GTK+ and Qt. It seems none of this toolkits are interested to work on XCB, neither on the X protocol. They probably put hard effort in bypassing X limitation and flaws, and they now sit on top of crap of workarounds and broken-by-design-standard implementation. It seems to me they don't want to go back in the layers and improves things.

They're too high to go back down and they don't see what the gain would be.

EFL was the first toolkit to have an XCB back-end with the work of Vincent Torri. Unfortunately, the back-end is not maintained and nobody cares about it. Last time I tried it, it did not compile at all.

X12?

There's a page called X12 on the Freedesktop wiki, listing all the things that should be fixed some days. Unfortunately, the list continues to grow up an no one talks about working on X12.

On the other hand, there's a handset of people trying to work when they will have time on XKB2, the second version of the "let's-try-to-fix-up-the keyboard-part-of-the-protocol-we-wrote-23-years-ago-a-second-time" extension.

To me, it does not seem X12 will happen in the next decade neither.

Alternative?

Do we got alternative to X? There's Wayland, but it's far from being usable. There's DirectFB, but that's not very portable. None seems candidate to replace X some days to me.

Anyhow, none of the main toolkits around support this alternative. GTK+ once supported DirectFB, but as far as I know, it is not supported nor works nowadays, as stated by Josselin Mouette. This is why recent versions of the Debian installer have migrated to X for the graphic part, thanks to Cyril Brulebois work.

Conclusion

XCB has been around for more than half-a-decade, and very few people showed interested in it. As far as I can see, nobody is interested to use the X protocol and everybody tries to encapsulate it in some higher-level API as soon as possible to stop seeing it. This leads to poorly written application and toolkits, with a lot of ugly hack.

All of that also means that starting to write applications and graphical toolkits based on XCB would be a very interesting project, but that would lead to spend too much time learning to circumvent the X protocol flaws, things that have been done in years by predecessors like Qt and GTK+.

Major toolkits implementations have almost nothing to win in going back in the dark water of X. I guess most of their folks prefer to work on shiny 3D effects based on your GPS location, rather than redefining better basis for everyone.

The manpower available in the X world is very small. Debian lacking of X maintainers is just the summit of that. There is very smart and very competent and skilled guys in the X world, as you can see by simply reading blog posts on Planet Freedesktop for example (me excluded). Unfortunately, there's not enough of them to cover all the things involved in X: input devices, graphics devices, new protocol extension specification and so on. The X server is really late, and it seems most of the developers prefers to work on the server itself than on the protocol behalf. Which is understandable.

I'm curious to see where all of that will lead in the upcoming years. I've been walking in the X world hallways for about 3 years now, and I feel desktop alternatives to KDE and GNOME will all die sooner or later. The time were you could choose between a dozen "modern" window managers has passed away.

After all, maybe that is simply Darwinism applied to computer software.

Making startup-notification XCB native

2010-05-24T18:05:00Z

I'm trying to work on XCB this week. And today I've started to accomplish the second step of a long term goal: making an X11 only library using XCB as its primary interface instead of Xlib.

Last year, I had extended the API of startup-notification to support XCB as a back-end. This had been made possible by factorizing some code, duplicating the X11 code and translating it into equivalent XCB.

Today, I've accomplished the second step, being dropping the Xlib code inside startup-notification to keep only the XCB one.

For this, I used the x11-xcb library, which is available when Xlib is compiled with XCB as its transport, which is nowadays the standard.

This library provides the function XGetXCBConnection, which can convert a Display pointer to a xcb_connection_t pointer. Consequently, it's now possible to write and execute XCB based code and being compatible with Xlib.

I've made some benchmark of my work for the occasion, in order to measure what the gain is.

The first table described 1000 launches of a fake application (a modified version of the startup-notification test suite actually). The X server is local (the latency is very minimal then). The gain is computed between the same back-end type for the total time. Full XCB is the "version" I'm working on.

Version - Back-end	User time (seconds)	Kernel time (seconds)	Total time (seconds)	Gain
0.10 - libx11	3.20	7.42	12.989	-
0.10 - libxcb	2.76	7.36	12.414	-
Full XCB - libx11	2.74	7.50	12.380	4.6 %
Full XCB - libxcb	2.72	7.16	12.037	3.0 %

The user time and kernel time are provided but are not really interesting. XCB does not offers a big gain in CPU execution time, but is more about latency. Anyhow, there's always a gain with XCB.

This second table describe the same test but running only 100 times over a slow network.

Version - Back-end	Total time (seconds)	Gain
0.10 - libx11	76	-
0.10 - libxcb	35	-
Full XCB - libx11	72	5.2 %
Full XCB - libxcb	33	5.7%

The gain is relatively small, about 5 %. But anyhow, there's still a gain. Note that the difference between the execution time of the same test written in XCB and Xlib is just huge. I've tried to optimize the Xlib test, but I did not manage to win more seconds.

In conclusion, considering that startup-notification is only used when an application launches another application, the perceivable gain might be even smaller. But anyhow, I think it's worth it.

Announcing muse-blog

2010-05-19T17:00:00Z

Digging into the fabulous world of Emacs and Lisp, I wanted to use it to build my personal Web site and my blog.

I already moved from ikiwiki to [[http://mwolson.org/projects/EmacsMuse.html][Emacs Muse]] for my HTML pages some weeks ago.

Muse provides an extension to maintain a journal, called muse-journal. Unfortunately, it was far to fulfill all my needs, and I decided that it would be a good exercise to write a better extension.

Consequently, I started to wrote my own extension, which I named muse-blog.

And this is now what is used to build this blog. :-)

Entering the Emacs world

2010-05-17T18:36:00Z

In February 2009, my friend dim tried to force me using Emacs. I know a couple of people using it and Gnus for reading their mail, and it always made me curious.

At that time, more than a year ago, Emacs 22 and Gnus did not seem usable from my point of view.

But around mid February, with the help of dim, I tried again to start using Emacs.

Actually, this was not something new for me. I (very basically) used Emacs between 2000 and 2006. In 2006, when I finished the university and started working at vim enthusiasts. They taught me how to use it in various ways, and I started to know more about vim than Emacs, so I switched.

This time, I started by configuring it, but reading the manual and also learning a bit of Lisp. It took me several weeks, but step by step I learned many, many things. And I must admit, I liked it.

I've configured and starting to use some very important mode, like Gnus, Org mode or even ERC.

I'll probably talk about various Emacs related things in the near future, since I already wrote more than a thousand lines of Lisp in the last 2 months.

Anyhow, I'd just conclude by asserting that my new Emacs/Gnus/Org/ERC setup beats my old vim/mutt/nothing/irssi to the death with a baseball bat. :-)

On media players: 2 years after

2010-01-22T18:36:00Z

Two years ago, I switched from my beloved xmms to audacious.

During this 2 years with Audacious, I suffered a bit. It was working quite fine, but I saw no big progress around it. Life happened, and I had to use a network system to play music. I started to use PulseAudio over TCP, but it does not work well, and does not work at all with Audacious (and even if the plugin is provided by upstream). So I decided to dump it.

And some days ago I discovered Sonata, a [MPD](http://sonata.berlios.de/] client. I never liked MPD so far because all clients I found were lame.

But I really like Sonata. It allows me to listen music the way I still want: load everything in one playlist, listen everything randomly or type a song/artist to jump to it directly in the current playlist. It even has some nice feature (lyrics, so I'll be able to song out loud, covers, tag editing…) and is written in Python and GTK+ (some days I may even hack it!).

You can rest in peace x11amp :-p

Python cairo and XCB support

2009-12-22T00:00:00Z

cairo has a Python binding (pycairo) since a long time, and some months ago a Python binding for XCB (xpyb) has been released.

Pycairo has no support for creating Xlib surfaces. You can get a Xlib surface from PyGTK and then use Pycairo to draw on it, but there's no way to create one directly.

What I've done is make Pycairo aware of xpyb so it can creates directly an XCB surface from a XCB connection and a drawable.

As said in my mail to the XCB list, I'm now waiting for a review before pushing this upstream. :-)

For the first time, I guess, XCB has beat Xlib support! ;-)

Teething troubles

2009-12-20T00:00:00Z

It's not that often that I start something from scratch. It's an amazing feeling to start a new project, to start writing something new. I like that. It's creation, it's an artistic part of our computing stuff. I feel like a code artist.

And what I like even more is that little feeling that you are going in an unknown land. Some area in this tech world where nobody ever came before you, or only a few pioneers.

That's the sensation I got starting to using Cython, Python 3 and various other tools. I just spent half of my time trying to fix problems, rather than working on my code. Problems in autoconf macro not knowing Python 2.6 or Python 3.1. Problems and limitations in Cython. And problem in Python.

That last one was a hard one. I'm still a beginner in the Python world: I barely know anything. And I was trying to use something nobody never did: building an embedded Python with a set of built-in modules.

I spent hours trying to find why one type of module importing was badly failing. I finally found the answer thanks to a guy. who has the same problem A guy ? No. A pioneer. What do I say? A hero. He's been my week-hero! Thank you Miguel Lobo because you found the bug I chased for hours and because you even reported it as issue 1644818, including a patch! How not damn wonderful is that?

I will not bore you with the technical details of that bug, since nobody cares. Nobody cares, even the Python guys, since that bug has been opened for 3 years, and nobody even reviewed in that time. I found an old thread about that bug where some guys were wanking about how they should do the review, because Miguel pushed for several weeks to have a review, back in 2007.

But that bug was in my way. I had to do something. So I prepared my mail reader, mounted my web browser and here I was for a uniq quest: getting a Python bug fixed.

At that point, if you did not stop reading earlier, you might get very excited. Don't be, spoiler, it's still not fixed. You'll have to wait the end of the season and see all the episodes I'll have to write to get the end of the story!

Let's continue.

I had to create an account on the Python bug tracking system. That was a trivial task for a man like me (you bet). Then, I launched a verbal attack, something you rarely see in a bug tracking system. Something I knew would awake any developer caring about their software.

Julien Danjou: Is there any chance to see this bug fixed someday?

I had the deep feeling that my quest was starting here. How many days would I have to wait until I get an answer? Time was passing. Minutes were ticking while I was waiting, sat in a comfortable sofa in a softly lighted room. It seemed like all my life was shorter than the delay I had to wait to get an answer.

After waiting for hours, suddenly, and only 15 minutes later, I got an answer:

Martin v. Löwis: Please ask on python-dev. I may be willing to revive my five-for-one offer.

Martin? Don't know that guy. Who is he? Who is he like? Will he fix that bug? What is this offer? So many question without an answer. But he asked to ask on python-dev, and I said: challenged accepted! I will write a mail to python-dev to get that bug fixed.

Which I did. I sent a short (but well written you know, I made efforts) "WTF?" to pyhon-dev.

And then the guy asked me to review 5 bugs so he will review and fix this one. And this is how I said that he was pissing me off for blackmailing me to fix a bug that was its "duty".

Therefore, this is the end of the story so far. Will that bug be fixed some day? There's a hope, because another guy jumped in and took the bug assignment.

To be continued.

My conclusion about all that story: that is a little rude to start something new, with new tools, and get quickly into teething troubles. It's even more harsh to enter a community because you just found bugs, and be not very well received when you ask to apply a 10 lines long fix somebody wrote 3 years ago to fix it.

I'll probably still use Python :-), but I get a darker image of its community now.

Courier to Dovecot migration

2009-10-02T00:00:00Z

This week, I've managed to migrate from courier-imap to dovecot at work. I always had a good experience with dovecot, and I still have one.

Dovecot performances are very good in comparison with courier. With that switch, we dropped the CPU usage of the server from 25 % to 10 %, and it's damn faster now. I have no idea why, but I think that it's better written looking at the code, and also that its usage of index files helps a lot.

We got no problem getting things work with public folders either, so the switch was almost painless.

The only problem we had is that Dovecot is too smart for some MUA. Consequently, we hit an 8 years old Mutt bug #969, which I also reported to the Debian BTS as #549204 with a not-well-tested-but-seems-to-work patch.

Thanks to Claws mail, we also found a bug in dovecot 1.2.5, which should be fixed soon. Dovecot upstream is very responsive and that's always something nice to know when you use a free software.

Various news: what happend during summer

2009-09-22T00:00:00Z

It's been a while since I blogged about something. So here's a bunch of things I've done the last month.

Holidays

Well, I've been in holidays one week. :-P

awesome

There have been a huge number of changes between 3.3 (released in June) and 3.4 (almost relesed). I wrote a small but very useful object layer on top of Lua, which adds a class/object system a bit like gobject. I've also replaced all the hooks by per-class/object signals. Finally, the awesome Lua basement are cleaner than they were before, and the extendability is improved. How nice.

We're trying to release 3.4 (rc2 should be out soon), but the development pace is a bit slower than a year before. We're basically almost 2 months late on what was our previous release rate. Not a big deal however.

I've started working on 3.5 slowly. It gonna get amazing new features too. :-)

Google Summer Of Code 2009

I've mentored Mariusz Ceier on XCB GSoC. He worked on adding Xinput2 and XKB extensions. And he managed to do this. His work should be imported ASAP, the discussion has started on XCB maling list last week.

In exchange, Google offered me (and to every mentor) an awful blue t-shirt! Thanks Google! :-P

TODO list management

2009-07-10T00:00:00Z

My fellow Debian developer Steve Kemp told us about his TODO list management.

While reading his post, I was constantly thinking "been there, been there buddy". Yeah, I've been.

I had the same problem since months, impossibility to track the things I had to do, being computer related stuff or real life ones. The bad thing is that until you write them down, you keep them in mind, and that's exhausting. You know you have, let's say 5, things to do, but unless you write this 5 items down in a TODO list, you will keep thinking about it once in a while. And that's a real lost time.

And that's totally inefficient: imagine you though "it'd be nice if I could buy a USB stick next time I buy some hardware". Well, unless you actually write this somewhere and have the habit to check the "To Buy" category of your TODO list, you're going to buy a replacement hard drive in a hurry some day, and forget about your USB stick.

I think the good practice, which I really recommend to everyone, is to write down as soon as possible what you think you have to do. Don't write it on a small paper you will lose, write it in a TODO list, a paper or electronic one, whatever, but write it, and stop thinking about it. When you'll have time, you'll get your TODO list from your pocket and give a look at it, doing what you can do at that moment. Once in a while, you check that list.

Personally, the tool I chose to handle my TODO list is a Palm Centro phone, which I got for only a hundred of euros. It runs the good old PalmOS, which basically know how to handle TODO list and plannings better than all phones I saw so far (and yes, probably better than your iPhone).

My choice was based on the fact that I've random ideas almost everywhere: that means while hacking, but also while walking in the street, while being in the train or while sleeping (yeah, already happened). And the only thing I always carry with me is my phone, in my pocket.

However, Steve choice may be nice if you have Internet access on your phone, which I haven't since it's too expensive for what it is, in my opinion. :-)

Upgrading to dovecot 1.2: hello Sieve!

2009-07-09T00:00:00Z

Last year, I told you I wanted to use Sieve to filter my mail. I did not switch, because of the lacking implementation of some Sieve features inside Dovecot, my preferred IMAP server.

After that disapointement, I kept my 8 years old mail setup, being fetchmail running on my workstation and throwing the mails in procmail, then using mutt locally to read the maildirs. But that's over.

I got a laptop to replace my workstation. It was not possible to continue using such a mail setup, since my laptop can be offline, and so would be my mails.

So I decided to upgrade Dovecot to 1.2. I used the dovecot-1.2-work Subversion branch of our lovely Debian maintainers, and built a Debian package for Lenny. The upgrade from 1.1 was almost painless, since the configuration file did not change heavily.

Then I started to write my little Sieve script. Sieve is a very nice language. Almost user friendly. So in 20 lines I rewrote all my procmail stuff, matching things like List-Id with regex to put the mails automagically in the right folder. I reconfigured mutt to use IMAP, and it works fine. I even reimported my old Maildir via IMAP using mutt too.

I am now a happy IMAP user.

For people wondering why I wanted to switch away from procmail to Sieve: the reason is that Sieve script can be uploaded remotely via managesieve. This means you do not need FTP/SSH/whatever access to put your script. You can, for example, use connect-sieve or the Sieve plugin for Thunderbird/Icedove.

Taking the other direction

2009-04-15T00:00:00Z

I've started to develop awesome more than 18 months ago, and somehow I feel it's time to stop a bit and think where we come from and where we are going to.

The motivation

I never though I'd be written a window manager one day. That seems kinda stupid when you see how many window manager there's around.

As many people, I've tested and have been using tons of window manager: Window Maker, Fluxbox, etc.

In August 2007, I was using fvwm since 2004 and was quite happy with it. I used the famous fvwm crystal as a configuration starter and then rewrote lots of stuff. Digging into ''fvwm'' configuration files was boring, and since I'm lazy, I never really configured it to fit entirely my needs.

The thing is that, in July 2007, my workstation died. I bought a new one based on the amd64 architecture. Too bad, with this new box, fvwm decided that it will not longer runs and was segfaulting almost every time I logged in.

I was really upset. Another failure in the window manager world. So I decided to get the yearly ride of testing many window managers. I went on the no more developed stuff like the *boxes, ion3, etc… but well, I did not like them, there were not powerful enough, too bugged or upstream was insane.

Then I found xmonad. The Haskell configuration file format made my cry. I did not want to learn Haskell, it seemed too obfuscated to me. At that time it was even not packaged for Debian, so I gave up. But I found

The jdwm

I just added a 'j' in front of dwm and started to hack it days and nights to add many feature I missed, like multi-head, etc… On 5th September 2007, I created a git repository to host my code.

That's gonna be… awesome.

Five days later, on 10th September, I finally found a name for my new pet: awesome, borrowed from Barney Stinson who heavily uses and abuses this word.

The 1.x branch

The first releases until December were noted 1.x. It was just a better dwm with a simple flat configuration file.. The configuration file used libconfig, but it was a very poor choice. And I was not able to put in into Debian because of name clash.

The 2.x branch

The 2.x branch came in January 2008 with a brand new configuration file format based on libconfuse, which was a bit more powerful. Many concepts and features that have been added in this branch are still used in the current 3.x branch.

At this time, between December 2007 and April 2008, the community was growing smoothly.

But as I said, awesome 2 was based on a flat configuration file. That raised a problem very soon: users expectation were growing and the development team (me and a couple of regular contributors) was unable to cope with them.

One of the event that started to change my mind was the support for titlebars.

When I've added titlebar support, it was minimal. It was on top of a window, with the window title. Dot. Then I've started to add a lot of options, like the application icon drawing, the position (left, right, bottom) etc.

And then users started to ask for more, like: "add titlebar on windows only when the window is floating".

That's ok, but that's complicated: that's again another option to do some stuff conditionally. And then, why don't add titlebar on windows when ?

The 3.x branch

Why

At that time, around April 2008, I'd totally stopped development. I was trying to find a solution which was simple and powerful. But after 2 weeks of thinking, I was not able to find anything else than: use a real language for configuration.

So, I've started prototyping awesome 3 using Lua. The choice was not obvious, and despite the problem Lua might suffer, it's one of the easiest language to integrate into an existing application.

But, let's go a little back: in January 2008, Arnaud Fontaine contacted me because he was interested to use awesome as one of its school project. He decided to port awesome from Xlib to XCB, a modern asynchronous X library.

His work took some time, but in May 2008, Arnaud did finished to port git master version of awesome to use XCB.

Consequently, I decided to start a new major branch, using XCB instead of Xlib (no change for users in this regard) and Lua instead of our previous flat configuration file format.

Development

It took me a while to get from here to there, but in September 2008, it was ready. We had a simple Lua API, and the XCB port was working perfectly.

It took us some time to release and have something totally working, because we had to work on XCB and contribute back to the project. It was really not ready to use by an application, but we did great work in this area and it's now really fine.

We're still here

Releases continue to happens, 3.1 around December 2008, and 3.2 around March 2009. 3.3 should be here in June.

One of the drawback we had, is that we moved many stuff from C to Lua. Why? Because writing things in Lua is quicker and easier to maintain than C, and makes thing more configurable for the user.

For example, the layout algorithm used to organize window were written in C until 3.2 came out. At that time, users had no choice than using a set of predefined layout to organize their windows.

Starting with 3.2, if they have minimal knowledge about geometry, they can start writing a layout function organising windows on the screen.

But this kind of API changes was a bit rough for users, since they had to port some part of their configuration file to the new API. The thing is that the project was still a teenager at that time, not really knowing were it will go. But I'm happy to announce that API breakage are more and more rare (so far only one minor between 3.2 and 3.3), and anyway always for the Good.

But I admit that it built a bad reputation around awesome 3.x during its first month of existence.

Future direction

I am currently working on 3.3 development. We have still many things to do. Time passing, we get more idea, and more users. And more users bring more ideas. We also have many more contributors, and some guys are even taking maintainer-ship of some code area.

Conclusion

My post title is "Taking the other direction" because I feel this way.

I've got that feeling that some approaches in projects like GNOME are sometimes bad. Please don't misread me, I know we are not playing in the same yard.

When adding a key shortcut for starting an application makes you dig into gconf, I wonder how this is a win for the user.

Well, it's probably a win for the end-user, but I surely am not one of them. And I don't intend to target them with my software, anyway.

And now, when I hear things like GNOME 3.0 and the "desktop shell" approach, that makes me smile. Guys, it was time, but have luck. What I see from here, is that any desktop control interface is wrong somehow, and that there's no approach that can fulfill all users wishes.

I think that we, the awesome development team (no pun intended) took the direction of building a frame-work window manager rather than a solution written in marble.

We (partially) solved the issue of UI ergonomic by not writing one and allowing the user to write his own. I don't say that's easy to do for most of users, but it's doable.

And I think it's worth it: I use window managers since I use Linux, around 1998. If something like ''awesome'' came 5 years ago, I'd be using it so far, because you can write Fluxbox or WindowMaker using awesome in a hundred of Lua code. And you can write your own version of it. And it starts in less than 3 seconds, supporting almost all standard desktop specification (ICCCM, EWMH, XDG, system tray, message notification, D-Bus, etc), whereas many of the window mangers do not.

You can even write and play space invaders.

Finally, I'm happy about the the road we took so far, and hope we will continue into that direction. The rants I read about our project are not that big, compared to the kudos we received.

OpenOffice is better as a pager than as a text processor

2009-02-11T00:00:00Z

Since several month, awesome users have reported a bug with OpenOffice.org. When using OOo and clicking on a menu, or using the mouse wheel to read a document, the currently selected tag (desktop) will change automagically to another one.

I've digged into awesome and found that awesome received a _NET_CURRENT_DESKTOP request. As defined by EWMH, this kind of request are sent by a pager to change the active desktop.

That was weird. Nobody is using a pager here. So, I just kicked my gdb out, attached it to OOo, breaking on XSendEvent call. And I got it:

Breakpoint 1, XSendEvent (dpy=0x1a00080, w=483, propagate=0, event_mask=1572864, event=0x7fff1fd70d70)
   at ../../src/SendEvent.c:46
(gdb) bt
#0  XSendEvent (dpy=0x1a00080, w=483, propagate=0, event_mask=1572864, event=0x7fff1fd70d70)
   at ../../src/SendEvent.c:46
#1  0x00007f8c0ab4193f in vcl_sal::WMAdaptor::switchToWorkArea ()
  from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#2  0x00007f8c0aafdbd8 in X11SalFrame::Show ()
  from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#3  0x00007f8c1378623c in Window::Show ()
  from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#4  0x00007f8c13785f40 in Window::Show ()
  from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#5  0x00007f8c1372cb54 in FloatingWindow::StartPopupMode ()
  from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#6  0x00007f8c1373c877 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#7  0x00007f8c1373ccf2 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#8  0x00007f8c1373ce84 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#9  0x00007f8c13795e7f in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#10 0x00007f8c13797e74 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#11 0x00007f8c13796748 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#12 0x00007f8c0aafe6f8 in X11SalFrame::HandleMouseEvent ()
  from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#13 0x00007f8c0ab040c2 in X11SalFrame::Dispatch ()
  from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#14 0x00007f8c0ab31625 in SalX11Display::Yield ()
  from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#15 0x00007f8c0ab356f3 in ?? () from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#16 0x00007f8c0ab2df1f in SalXLib::Yield () from /usr/lib/openoffice/basis3.0/program/libvclplug_genlx.so
#17 0x00007f8c135b050e in Application::Yield ()
  from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#18 0x00007f8c135b0587 in Application::Execute ()
  from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#19 0x00007f8c17517e80 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libsofficeapp.so
#20 0x00007f8c135b4b24 in ?? () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#21 0x00007f8c135b4bc5 in SVMain () from /usr/lib/openoffice/program/../basis-link/program/libvcllx.so
#22 0x00007f8c1754ca6c in soffice_main ()
  from /usr/lib/openoffice/program/../basis-link/program/libsofficeapp.so
#23 0x000000000040105b in main ()

I started digging more into the code, and this is what I finally found in salframe.cxx:

        // #i45160# switch to desktop where a dialog with parent will appear
        if( mpParent && mpParent->m_nWorkArea != m_nWorkArea )
            GetDisplay()->getWMAdaptor()->switchToWorkArea(mpParent->m_nWorkArea );

Beautiful! It even has a comment with a IssueZilla bug number. Let's go and see where it comes from.

After 10 minutes of research to find that fucking IZ, I finally found the link to the issue #45160. The bug is IMHO not related to OOo but to a window manager doing poor job.

I've found that an awesome user already reported an bug… err, wait, I mean an issue as issue #96684 (remember there's no bug in OOo, only issues) and I commented about it.

It seems OOo developers have agreed to fix that bug eventually.

startup-notification ported to XCB

2009-01-29T16:17:00Z

Since Tuesday, I've begun to work on XCB portage of the startup-notification library.

I've just completed the job, and send a bunch of patches to the XDG mailing list.

If the patches are merged, which I don't doubt, I'll be able to use this lib into awesome, which would be nice step to the Freedesktop standard compliance I like to make.

Rants about Lua

2008-12-30T11:23:00Z

I've started using Lua some months ago, while looking for a more powerful way to configure awesome. At this time, around March 2008, Lua seemed to be the best language to integrate inside the core system of awesome.

I still think that Lua was a good choice, but after 8 months, it shows some important drawbacks.

I'll try to keep my explanation simple and to make you understand everything, even if you do not know Lua.

I refer here to Lua version 5.1.

Design flaws

Length operator

Lua has a length operator on its objects, known as ''#''. It can be used to get the size of various objects.

> return #"lol"
> 3

This operator works for table, string, etc… It's possible to define this operator by setting a __len meta-method on a userdata value.

The problem is that you cannot redefine it on string or table objects, see:

> a = { "hello", "world" }
> return #a
2
> setmetatable(a, { __len = function () return 18 end })
> return #a
2

Indeed, looking at the Lua core code:

case OP_LEN: {
        const TValue rb = RB(i);
        switch (ttype(rb)) {
          case LUA_TTABLE: {
            setnvalue(ra, cast_num(luaH_getn(hvalue(rb))));
            break;
          }
          case LUA_TSTRING: {
            setnvalue(ra, cast_num(tsvalue(rb)->len));
            break;
          }
          default: {  / try metamethod /
            Protect(
              if (!call_binTM(L, rb, luaO_nilobject, ra, TM_LEN))
                luaG_typeerror(L, rb, "get length of");
            )
          }
        }
        continue;
      }

You clearly see that tables and strings always use the internal length operator, never the __len* meta-method.

That's for me a design problem, which will cause more trouble. We'll see later.

index and newindex metamethods

Lua defines two useful meta-methods, which are index and newindex. Both can be set on a table or any other object. index will be called upon each read access to an undefined key on an object, and newindex upon each write access.

> a = {}
> -- function are not defined, this is just an example
> setmetatable(a, { __index = myindexfunction, __newindex = mynewindexfunction })
> a[1] = "hello" -- This will call __newindex metamethod
> return a[2] -- This will call __index metamethods
> return a[1] -- This will NOT call __index

The last line does not call __index meta-method because a[1] does exists. This is a problem when you want to use table as object, because sometimes you want to monitor access to the table elements.

This can be easily worked around using a proxy system: you don't store things in the table you manipulate, but in another table.

> a = {}
> realtable = {}
> -- function are not defined, this is just an example:
> setmetatable(a, { __index = myindexfunction, __newindex = mynewindexfunction })

Where meta-methods are something like:

function myindexfunction(table, key)
   return realtable[key]
end
 
function mynewindexfunction(table, key, value)
   realtable[key] = value
end

This way, our a table will always be empty, and realtable will have the data. At every read or write access to a, the meta-methods will be called. This is very convenient and widely used hack.

But this has serious drawbacks: as we saw before, the length operator (#) cannot be redefined on a table. That means #a will always be 0, and you cannot get the table length anymore, except by defining another method or a special attribute.

Also, Lua has several functions in the table library that are used to manipulate table in a easy way. The problem is that standard functions like table.insert or table.remove do raw accesses to the table. Meaning that if you do table.insert(a, 1, 1) it will insert the value 1 at the key 1 into a, without calling the __newindex meta-methods, breaking all your beautiful object-oriented model.

Another solution is to use a userdata object, like done in the Lua newproxy function (which is under-documented).

The problem is that it breaks all the other functions that are waiting for a table as argument, because they see a userdata, not a table. So this time table.insert is now more usable, which somehow fixes the problem, but not in the right way IMHO. However, this allows to use the __len meta-method.

Development model

The development model of Lua is, from my point of view, non-existent.

There is no public version control system repository available, so there's no chance to really contribute to Lua. It seems only a defined set of people work on it and therefore, the development system is very closed to my eyes, in comparison of usual projects.

Conclusion

I still think Lua is a good choice, because it is very easy to integrate into any C program, and to expand to fulfill your needs. However, some bad design choices were made, and the poor and closed development model chosen does not allow to have a good overview of the future of Lua.

This has been well stated by the authors themselves.

Security bug found in Imlib2

2008-11-22T00:00:00Z

Yeah, I'm the proud discover of CVE-2008-5187.

It's my first time, it does mean something to me. ;-)

The eggtray problem

2008-10-03T00:00:00Z

I still don't know why but many GTK+ applications use something called eggtrayicon. As far as I know, eggtrayicon.c is a file written in 2002 by Anders Carlsson which implements the Freedesktop.org system tray procotol for GTK+ applications.

Problem is that this C file is used in dozens of programs and maybe more, and is a bit bugged. I've already send patches for mail-notification and Audacious. pidgin is the first fixed implementation I found and works quite well. Many other applications are probably affected.

That seems to me like a real problem. Multiple copy of bad code instead of using native GTK+ system tray implementation.

So please stop using this bad implementation…

Unexpected VARMon new release

2008-08-18T00:00:00Z

This has been 4 years since I released a new upstream release of VARMon, the DAC960 administration tool.

There was a bug first discovered in #401236. It has been fixed in Debian with an ugly fix, which did not work finally for a long time. Recently #491505 got opened too, which was the same as the previous one. But this time I got access to hardware, thanks to Christoph! And I finally fixed the bug. I've even be able to test the fixes I wrote years ago for all of the compilation warnings.

That's a shame that the problem was caused by dead code from the previous upstream, and that I did not realize that sooner. Kids, do not let dead debug code in your program at home.

So I've finally been able to release a new 1.2.1 version which maybe the last release for the next decade! ;-)

ATL1E support in 2.6.26-1

2008-07-31T11:59:00Z

Ben Armstrong opened an ITP for the ATL1E NIC driver, which is found on some Asus EeePC laptops. So, as suggested by Maximilian Attems, I provided a clean patch for this driver, made from a cherry-pick from the linux-netdev 2.6.27 tree. It has been committed into the 2.6.26-1 Debian kernel, which will be furnished with Lenny.

What's fun, is that in the mean time, I got a new computer at work. Wait, it's not fun yet. Because what I did not know is that it's made of an Asus P5Q motherboard which runs a NIC needing the ATL1E driver (and now you see why it's fun).

So I've just upgraded to 2.6.26-1-amd64 and I'm glad that my own work is useful to me (and will be probably be to others as well). :-)

EWMH and XRandR

2007-12-27T19:48:00Z

Today I decided to add some EWMH support to awesome. It now supports a bunch of this extensions quite nicely.

However, while reading the spec and writing the code, it appears that this forces a window manager to behave in only one way: have a poor desktop support, and no multi-head/XRandR/Xinerama support at all.

The main caveats are that in Xinerama/XRandR mode, you'll have only one root window. And the root window is where you must store the NET_WM X properties… So you cannot handle screens in a independant way like awesome does. That's really a shame.

There's also a big problem for window managers like awesome which are happy to draw several desktops at the same time. There's no support for stuff like that.

So far, I think EWMH is nice but is really too narrow-minded for software and people who want to think window management in a different way.

Kicking out Web spammers with DNSBL

2007-01-15T11:33:00Z

Every project has its story. Every war has its winner, and its casualties. They were 20 millions men, fighting for their freedom.

And you'll never know their story.

Because during last week, I was looking why my Web server was so heavily loaded. And I discovered that my blog was attacked by spammers trying to post comments. They were stopped by a great plug-in named spamplemousse, which use spam keywords and DNSBL to drop spam comments. However, this plug-in is written in PHP, like the rest of my blog, so it loads Apache and MySQL in a way that is no more acceptable: the page have still to be rendered for this !@#$ spammers.

Consequently, I decided to write a Apache 2.x module which will just drop a 403 Forbidden error page in the spammers' head using DNSBL servers. Here it is, and it is called mod_defensible.

I'm using it since 3 days now, and I got some pretty interesting result and less load on my Web server, so c'est tout bon.