linux — jd:/dev/blog

Using Gnocchi with Docker

Thu, 17 Aug 2017 00:00:00 GMT

I've recently started to look into Docker to build images ready to be used with Gnocchi in it. I found it would be a great way to distribute a working instance of Gnocchi.

To this end, we created the gnocchi-docker repository on GitHub. It contains:

a 11 lines long (only!) Dockerfile to build a Linux image containing Gnocchi;
a Dockerfile to create a Grafana image that will use Gnocchi as datasource (preconfigured);
a Dockerfile to create a collectd image that gather various metrics for your container in order to feed Gnocchi and have something to display in Grafana;
a docker-compose file that orchestrates and runs those containers.

If you don't know docker-compose, it's a tool to define and run applications using multiple containers. This is very handy in our case, as we need to start a few services, and therefore a few containers, to have our whole stack running.

If you just want to use and run Gnocchi in a snap using this, it's easy. First clone the repository:

$ git clone https://github.com/gnocchixyz/gnocchi-docker.git

Then, just ask docker-compose to start your stack of containers:

$ cd gnocchi-docker
$ docker-compose up

On the first run, docker-compose will build the various images (this should take only a few minutes) and then will start them.

Once everything is started, you can connect to Grafana by typing the URL http://<ip of your docker server>:3000 in your browser and using "admin" as username and "password" as password. Just click on the dashboard entitled "Gnocchi" and wait a few minutes: you will see the chart being drawn in real time!

The data fed into Gnocchi come from the collectd container, which gathers various metrics (CPU, network interface statistics, etc).

You can then edit the docker files as you like to add new features or test your code. The files are also a good basis if you want to deploy Gnocchi in production running Docker!

If you want to access and play with Gnocchi in command line, just install gnocchiclient and do the following:

$ export GNOCCHI_ENDPOINT=http://`docker-machine ip`:8041
$ gnocchi resource list
+----------+----------+------------+---------+----------------------+------------+----------+----------------+--------------+---------+
| id       | type     | project_id | user_id | original_resource_id | started_at | ended_at | revision_start | revision_end | creator |
+----------+----------+------------+---------+----------------------+------------+----------+----------------+--------------+---------+
| c31e4adc | collectd | None       | None    | collectd:fake-phy-   | 2017-08-17 | None     | 2017-08-17T12: | None         | admin   |
| -2cff-5f |          |            |         | host-719acbad336c    | T12:20:27. |          | 20:27.643790+0 |              |         |
| 78-8206- |          |            |         |                      | 643778+00: |          | 0:00           |              |         |
| f5ca66e4 |          |            |         |                      | 00         |          |                |              |         |
| 6cce     |          |            |         |                      |            |          |                |              |         |
+----------+----------+------------+---------+----------------------+------------+----------+----------------+--------------+---------+

You can now have fun creating new resources and metrics!

Feel free to contribute patches to the GitHub project too, obviously!

Logitech Unifying devices support in UPower

Fri, 16 Nov 2012 00:00:00 GMT

A few months ago, I wrote about my reverse engineering attempt to Logitech Unifying devices. Back then, I concluded my post with big hopes on the future after receiving a document with some part of the specification of the HID++ 2.0 from Logitech.

A couple of weeks ago, some of my summer work has been merged to UPower, adding battery support for some Logitech devices.

HID++

As I discovered late in my first reverse engineering attempt, Logitech developed a custom HID protocol named HID++. This protocol exists in two versions, 1.0 and 2.0. Some devices talk with version 1 of the protocol (like my M705 mouse) and some others talk with version 2 of the protocol (like my K750 keyboard).

Recently, I've been able to be in touch with a Logitech engineer who worked on the Linux support for the Unifying receiver, and he has been really helpful and exposed me some details about this protocol.

Logitech made the decision to publish their HID++ specification publicly about a year ago, but still didn't do it. The internal review needed to publish such documents hasn't be done yet. The only published draft is just an extract of the specification, with even some typo in it as I discovered.

Some other documents have been recently published, but I didn't have the time to review them. They contains HID++ 1.0 specifications and some details I asked for about the K750 keyboard.

UPower support

It took me sometime to get a full understanding of the protocol, its different version etc. After reverse engineering my K750 keyboard, I've also reverse engineered the data stream used to get my M705 mouse battery status. I've also received some information about the HID++ 1.0 protocol, so I've been able to discover a bit more on what the packets mean. Most of my discoveries are now used to do proper #define in up-lg-unifying.c so the code makes more sense.

My first patch implements a new property for UPower devices, named luminosity, that use with K750 keyboard to report the light level received. The second patch add support for Logitech Unifying devices (over USB only) and should work with at least Logitech M705 and K750 devices. This should be available with the next version of UPower, which should be 0.9.19.

So far, Logitech has been kind enough to help me understanding part of the protocol and even sent me a few devices so I can play and test my work with them. Unfortunately, this will probably requires some work and time, and so far Logitech was not able to help with that.

There should be enough information out there to at least add support for battery to HID++ 2.0 devices, and probably a few other things too. I hope I'd get the time do this at some point, but feel free to beat me in this race!

Sony Vaio Z Debian Linux support

Sat, 11 Aug 2012 00:00:00 GMT

I had to install Debian Wheezy on a brand new Sony Vaio Z laptop with the new Ivy Bridge architecture (SVZ1311C5E). I'll talk about this here, because it's always nice to know that new hardware works quite fine (or not) under Debian.

The laptop is delivered with Window 7, which I decided to remove entirely anyway, and replace with Debian. I've installed it with Linux 3.2 and then ran Linux 3.4, 3.5 and 3.6-rc1.

USB booting

Don't ask me why, nor an Ubuntu or Debian USB installation booted, blocked at SYSLINUX at best, or at a black screen. This does not work. I had to use PXE to install Debian.

Storage

The only thing that can be surprising, is that the 128 GB SSD storage is actually made of 2 64 GB Samsung SSD aggregated in a RAID 0 using Intel Rapid Storage Technology, previously known as Intel Matrix. This is supported by Linux using the dm-raid module. So this is a fake RAID, and you anyway can see the both drives as sda and sdb under Linux.

Unfortunately, this kind of RAID is not supported correctly by GRUB, and I was unable to install it this way. Therefore, I decided to remove entirely this fake RAID (which is possible via the BIOS) and use a Linux software md RAID 0 instead, plus crypto on top of it. That I know well and I trust. :)

Graphics

The Intel HD Graphics 4000 works fine. I'm alsmo using the HDMI output, which works fine. There's some GPU hanging (as seen on screen and in kernel logs) in Linux up to 3.4, but with versions 3.5 and above, I didn't see any problem so far.

Sound

The Intel HDA sound card works pretty well, both for playing and recording. The main problem is that I hear a constant noise on the speakers, but tweaking the ALSA mixers ends it at some point. There's still probably a bug, not yet resolved in Linux 3.6-rc1.

Keyboard

The keyboard works fine, and the back-light too, via the sony-laptop kernel module. Wonderful.

Touchpad

Touchpad works fine.

Fingerprint

It does not work, and is not supported according to my research. Not that I care about, but don't count on it. It's an AuthenTec AES1660.

Webcam

It works perfectly.

USB

Well, USB 3.0 does not work. I had to disable XHCI in the BIOS and use the 2 ports as standard USB 2.0, otherwise I would just get errors from the kernel. Still not working with Linux 3.6-rc1, and I've no clue to debug, and do not use USB 3.0 yet, so…

WiFi

The WiFi module (based on iwlwifi) works fine. The only problem with NetworkManager is that the sony-laptop offers a second rfkill switch and NM does not know how to handle it correctly. A bug is opened about this and I hope to be able to write a patch or something at some point. Also, there seems to be some quality issue with the iwlwifi driver and 802.11n at this point. I'm losing connection quite often when the signal drops below 40 %. Loading the module with 11n_disable=1 helps a lot.

Ethernet

The gigabit Realtek Ethernet controller works perfectly.

Card reader

Works perfectly.

Logitech K750 keyboard and Unifying Receiver Linux support

Mon, 09 Jul 2012 00:00:00 GMT

A year ago, I bought a Logitech Wireless Solar Keyboard K750. I'm particularly picky on keyboards, but this one is good. It has an incredible useful feature: while being wireless, it has no need for disposable or rechargeable batteries, it uses solar power!

My problem is that there's obviously no way to know the battery status from Linux, the provided application only working on Windows.

And one dark night, while fragging on QuakeLive, my keyboard stopped working: it had no battery left. This activity being quite energy consuming, it emptied the whole battery.

Someone should write code to get the battery status and light meter from Linux: challenge accepted!

How the keyboard works

This keyboard, like many of the new wireless devices from Logitech, uses the Unifying interface. It's an USB receiver that can be attached up to 6 differents devices (mouse, keyboards…). On old Linux kernel, the Unifying receiver is recognized as only one keyboard and/or one mouse device.

Recently, a driver called hid-logitech-dj has been added to the Linux kernel. With this driver, each device attached to the receiver is recognized as one different device.

What the Logitech application does

The Logitech application under Windows works that way: you launch it, and it displays the battery charge level. On the keyboard, there's a special "light" button (up right). When pressed, a LED will light up on the keyboard: green if the keyboard is receiving enough light and is charging, red if the keyboard does not receive enough light and is therefore discharging. Pushing this same button while the application is running will makes the light meter activated: the application will tell you how much lux your keyboard is receiving.

Let's reverse engineer this

As far as I know, there's nothing in the USB HID protocol that handles this kind of functionality (battery status, light meter…) in a standard way. So the first task to accomplish is, unfortunately, to reverse engineer the program.

I discovered a bit too late that Drew Fisher did a good presentation on USB reverse engineering at 28c3. You might want to take a look at it if you want to reverse engineer on USB. I did not need it, but I learned a few things.

Anyway, my plan was the following: run the Logitech application inside a virtual machine running Windows, give it direct access to the USB keyboard, and sniff what happens on the USB wire.

To achieve that, you need a virtual machine emulator that can do USB pass-through. Both KVM and VirtualBox can do that, but VirtualBox works much better with USB and allow hot(un)plugging of devices, so I used it.

To sniff what happens on the USB, you need to load the usbmon Linux kernel module. Simply doing modprobe usbmon will work. You can then use Wireshark which know how to use usbmon devices and understand the USB protocol.

USB stuff you need to know

You don't need to know much about USB to understand what I'll write about below, but for the sake of comprehensibility I'll write a couple of things here before jumping in.

To communicate with an USB device, we communicate with one of its endpoints. Endpoints are regrouped into an interface. Interfaces are regrouped into a configuration. A device might contains one or several configurations.

There's also several types of packets in the USB wire protocol, and at least two of them interest us there, they are:

Interrupt packets, a packet send spontaneously;
Controls packets, used for command and status operations.

All of this and more is well (and better) explained in the chapter 13 of Linux Device Drivers, Third Edition.

Sniffed data

Once everything was set-up, I ran my beloved Wireshark. There's a an URB of type interrupt sent each time you press any key with some data in it. Therefore I advise you to plug another keyboard (or use the laptop keyboard if you're doing this on a laptop), otherwise you'll get crazy trying to sniff the keyboard you're typing on.

At this point, just launching the application does a bunch of USB traffic. Pressing the "light" button on the keyboard makes even more USB packets coming in and out. Here's the interesting packets that I noticed once I excluded the noise:

When pressing the "light" button, an URB of type interrupt is sent by the keyboard to the computer;
An URB control packet is sent by the computer to the keyboard in response;
Regularly URB interrupt packets are sent just after.

With all this, the next step was clear: understand the packets and reproduce that exchange under Linux.

What the packets mean

The "go for the light meter" packet

The packet sent from the computer to the keyboard is the following.

Frame 17: 71 bytes on wire (568 bits), 71 bytes captured (568 bits)
    Frame Length: 71 bytes (568 bits)
    Capture Length: 71 bytes (568 bits)
USB URB
    URB id: 0xffff880161997240
    URB type: URB_SUBMIT ('S')
    URB transfer type: URB_CONTROL (0x02)
    Endpoint: 0x00, Direction: OUT
        0... .... = Direction: OUT (0)
        .000 0000 = Endpoint value: 0
    Device: 6
    URB bus id: 1
    Device setup request: relevant (0)
    Data: present (0)
    URB sec: 1340124450
    URB usec: 495643
    URB status: Operation now in progress (-EINPROGRESS) (-115)
    URB length [bytes]: 7
    Data length [bytes]: 7
    [Response in: 18]
    [bInterfaceClass: HID (0x03)]
    URB setup
        bmRequestType: 0x21
            0... .... = Direction: Host-to-device
            .01. .... = Type: Class (0x01)
            ...0 0001 = Recipient: Interface (0x01)
    bRequest: SET_REPORT (0x09)
    wValue: 0x0210
        ReportID: 16
        ReportType: Output (2)
    wIndex: 2
    wLength: 7
0000  40 72 99 61 01 88 ff ff 53 02 00 06 01 00 00 00   @r.a....S.......
0010  22 ad e0 4f 00 00 00 00 1b 90 07 00 8d ff ff ff   "..O............
0020  07 00 00 00 07 00 00 00 21 09 10 02 02 00 07 00   ........!.......
0030  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0040  10 01 09 03 78 01 00                              ....x..

What's here interesting is the last part representing the data. wLength says that the length of the data is 7 bytes, so let's take a look at those 7 bytes: 10 01 09 03 78 01 00.

Well, actually, you can't decode them like that, unless you're a freak or a Logitech engineer. And I have actually no idea what they mean. But sending this to the keyboard will trigger an interesting thing: the keyboard will start sending URB interrupt with some data without you pressing any more key.

The "light meter and battery values" packet

This is most interesting packet. This is the one sent by the keyboard to the host and that contains the data we want to retrieve.

Frame 1467: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
    Frame Length: 84 bytes (672 bits)
    Capture Length: 84 bytes (672 bits)
USB URB
    URB id: 0xffff88010c43c380
    URB type: URB_COMPLETE ('C')
    URB transfer type: URB_INTERRUPT (0x01)
    Endpoint: 0x83, Direction: IN
        1... .... = Direction: IN (1)
        .000 0011 = Endpoint value: 3
    Device: 2
    URB bus id: 6
    Device setup request: not relevant ('-')
    Data: present (0)
    URB sec: 1334953309
    URB usec: 728740
    URB status: Success (0)
    URB length [bytes]: 20
    Data length [bytes]: 20
    [Request in: 1466]
    [Time from request: 0.992374000 seconds]
    [bInterfaceClass: Unknown (0xffff)]
Leftover Capture Data: 1102091039000c061d474f4f4400000000000000

0000  80 c3 43 0c 01 88 ff ff 43 01 83 02 06 00 2d 00   ..C.....C.....-.
0010  5d c5 91 4f 00 00 00 00 a4 1e 0b 00 00 00 00 00   ]..O............
0020  14 00 00 00 14 00 00 00 00 00 00 00 00 00 00 00   ................
0030  02 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00   ................
0040  11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00   ....9....GOOD...
0050  00 00 00 00                                       ....

This packets come in regularly (1 per second) on the wire for some time once you sent the "go for the light meter" packet. At one point they are emitted less often and do not contain the value for the light meter anymore, suggesting that the control packet sent earlier triggers the activation of the light meter for a defined period.

Now you probably wonder where the data are in this. They're in the 20 bytes leftover in the capture data part, indicated by Wireshark, at the end of the packet: 11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00 00 00 00 00.

Fortunately, it was easy to decode. Knowing we're looking for 2 values (battery charge and light meter), we just need to observe and compare the packet emitted on the wire with the values displayed by the Logitech Solar App.

To achieve this, I looked both at the Logitech Solar App and Wireshark while bringing more and more light near the keyboard, increasing the lux value received by the meter on the Solar App, and saw that the fields represented in blue (see below) where changing in Wireshark. Since 2 bytes were changing, I guessed that it was coded on 16 bits, and therefore it was easy to correlate the value with the Solar App.

[ ....9....GOOD....... ] 11 02 09 10 39 00 0c 06 1d 47 4f 4f 44 00 00 00 00 00 00 00 4 bytes - 1 byte for battery charge - 2 bytes for light meter - 2 bytes - 4 bytes for GOOD - 7 bytes

In this example, the battery has a charge of 0x39 = 57 % and the light meter receives 0x0c = 12 lux of light. It's basically dark, and that makes sense: it was night and the light was off in my office, the only light being the one coming from my screen.

I've no idea what the GOOD part of the packet is about, but it's present in every packet and it's actually very handy to recognize such a packet. Therefore I'm considering this as some sort of useful mark for now.

For the other bytes, they were always the same (0x11 0x2 0x9 0x10 at the beginning, 7 times 0x00 at the end). The 2 bytes between the light meter and GOOD probably mean something, but I've no idea what for now.

Building our solar app

Now we've enough information to build our own very basic solar application. We know how to triggers the light meter, and we know how to decode the packets.

We're going to write a small application using libusb. Here's a quick example. It's not perfect and does not check for error codes, be careful.

/* Written by Julien Danjou <julien@danjou.info> in 2012 */

#include <linux/hid.h>

#include <libusb.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void)
{
    libusb_context *ctx;
    libusb_init(&ctx);
    libusb_set_debug(ctx, 3);

    /* Look at the keyboard based on vendor and device id */
    libusb_device_handle *device_handle = libusb_open_device_with_vid_pid(ctx, 0x046d, 0xc52b);

    fprintf(stderr, "Found keyboard 0x%p\n", device_handle);

    libusb_device *device = libusb_get_device(device_handle);

    struct libusb_device_descriptor desc;

    libusb_get_device_descriptor(device, &desc);

    for(uint8_t config_index = 0; config_index < desc.bNumConfigurations; config_index++)
    {
        struct libusb_config_descriptor *config;

        libusb_get_config_descriptor(device, config_index, &config);

        /* We know we want interface 2 */
        int iface_index = 2;
        const struct libusb_interface *iface = &config->interface[iface_index];

        for (int altsetting_index = 0; altsetting_index < iface->num_altsetting; altsetting_index++)
        {
            const struct libusb_interface_descriptor *iface_desc = &iface->altsetting[altsetting_index];

            if (iface_desc->bInterfaceClass == LIBUSB_CLASS_HID)
            {
                libusb_detach_kernel_driver(device_handle, iface_index);
                libusb_claim_interface(device_handle, iface_index);

                unsigned char ret[65535];

                unsigned char payload[] = "\x10\x01\x09\x03\x78\x01\x00";

                if(libusb_control_transfer(device_handle,
                                           LIBUSB_REQUEST_TYPE_CLASS | LIBUSB_RECIPIENT_INTERFACE,
                                           HID_REQ_SET_REPORT,
                                           0x0210, iface_index, payload, sizeof(payload) - 1, 10000))
                {
                    int actual_length = 0;

                    while(actual_length != 20 || strncmp((const char *) &ret[9], "GOOD", 4))
                        libusb_interrupt_transfer(device_handle,
                                                  iface_desc->endpoint[0].bEndpointAddress,
                                                  ret, sizeof(ret), &actual_length, 100000);

                    uint16_t lux = ret[5] << 8 | ret[6];

                    fprintf(stderr, "Charge: %d %%\nLight: %d lux\n", ret[4], lux);
                }

                libusb_release_interface(device_handle, iface_index);
                libusb_attach_kernel_driver(device_handle, iface_index);
            }
        }
    }

    libusb_close(device_handle);
    libusb_exit(ctx);
}

What the program is doing is the following:

Request for the Unifying Receiver device based on vendor and product ID
Get the HID interface
Detach the HID interface from the kernel driver
Claim the interface
Send a control packets, were parameters are defined using the same data we captured earlier
Read interrupt packets coming in until we receive one we recognize (length 20 containing the "GOOD" string)
Decode the content (battery charge & light meter)
Release the interface
Reattach the kernel driver to the interface

This gives the following:

Found keyboard 0x0x24ec8e0
Charge: 64 %
Light: 21 lux

Challenge accomplished!

To be continued

Unfortunately, this approach has at least one major drawback. We have to disconnect the Logitech Unifying Receiver from the kernel. That means that while we're waiting for the packet, we're dropping packets corresponding to other events from every connected device (key presses, pointer motions…).

In order to solve that, I sent a request for help on the linux-input mailing list. That way, I learned that Logitech is actually using the HID++ protocol to communicate with the devices using the Unifying Receiver. Lars-Dominik Braun managed to get the HID++ specifications from Logitech and published them with their authorization.

This opens a whole new world. With that document, I may be able to understand the part I reverse engineered and convert this to a more useful and generic library using the hidraw interface (so we don't have to disconnect the devices from the kernel driver).

Porting D-Bus to XCB: story of a failure

Thu, 29 Jul 2010 00:00:00 GMT

Even if I recently stated I lost some of my faith in XCB, I still sometimes hack things to add support for it.

These last days, I've worked on a D-Bus port from Xlib to XCB. The port was quite straight forward, since there's only a little piece of D-Bus using X, which is dbus-launch.

I though D-Bus was a good candidate, since it's part of the Freedesktop initiative. Therefore, I was expecting a warm welcome and some enthusiasm from a fellow project.

My contribution got one useful review, and a cold reply from Thiago Macieira (a KDE/Qt/Nokia developer):

No, sorry, I don't agree..
I've just checked and my Solaris machine doesn't have XCB.
Please do not remove the X11 code. You may add the XCB code, but you cannot remove the X11 code.

This is not really the kind of answer I expected, actually. I then reworked the code to please Thiago, and added some #ifdef to add XCB support to D-Bus, with a fallback to libx11 where XCB would not be available.

Havoc Pennington replied:

Given that libX11 now uses xcb as backend, I don't understand the
value of porting to use libxcb directly when there isn't an issue of
round trips or other stuff. It will just make #ifdef hell, while the
X11 API is an API that works on both xcb and non-xcb platforms.
Maybe people should be thinking about porting xcb to non-Linux
platforms? The X protocol should be the same on other UNiX, so xcb in
theory ought to work fine if you just compiled it on Solaris/BSD, same
as GTK or dbus or Qt would work fine.

The last part "Maybe people should be thinking about porting xcb to non-Linux platforms?" is still unclear to me, even though I asked Havoc to explain what he meant.

Finally, Thiago refused to merge the patch:

[…] thanks for the patch, but like Havoc I am unsure of the value. We can't
drop the X11 codepaths now because too many systems exist without
XCB. Adding the XCB codepaths only made it more complex, even though you did a good job.

I can't disagree with that conclusion: using both XCB and X11 make the code unreadable for little gain. That's why I did replace libx11 by XCB directly in the first version of the patch. On the other hand, D-Bus people does not seems to really care about making their software evolve in the right direction, even if that requires users to upgrade their systems.

I think D-Bus using and depending on XCB would have been a good point to push adoption of XCB. Unfortunately, it seems you can't even rely of projects of the same initiative (i.e. Freedesktop) to work together to make things a little bit better.

After 5 years of existence, XCB is still not so obvious to people, and making it adopt is going to be a challenge for the next years. The upside is that new X.org 7.6 will bring XCB with it, as part of the katamari.

Unexpected VARMon new release

Mon, 18 Aug 2008 00:00:00 GMT

This has been 4 years since I released a new upstream release of VARMon, the DAC960 administration tool.

There was a bug first discovered in #401236. It has been fixed in Debian with an ugly fix, which did not work finally for a long time. Recently #491505 got opened too, which was the same as the previous one. But this time I got access to hardware, thanks to Christoph! And I finally fixed the bug. I've even be able to test the fixes I wrote years ago for all of the compilation warnings.

That's a shame that the problem was caused by dead code from the previous upstream, and that I did not realize that sooner. Kids, do not let dead debug code in your program at home.

So I've finally been able to release a new 1.2.1 version which maybe the last release for the next decade! ;-)

ATL1E support in Linux 2.6.26-1

Thu, 31 Jul 2008 00:00:00 GMT

Ben Armstrong opened an ITP for the ATL1E NIC driver, which is found on some Asus EeePC laptops. So, as suggested by Maximilian Attems, I provided a clean patch for this driver, made from a cherry-pick from the linux-netdev 2.6.27 tree. It has been committed into the 2.6.26-1 Debian kernel, which will be furnished with Lenny.

What's fun, is that in the mean time, I got a new computer at work. Wait, it's not fun yet. Because what I did not know is that it's made of an Asus P5Q motherboard which runs a NIC needing the ATL1E driver (and now you see why it's fun).

So I've just upgraded to 2.6.26-1-amd64 and I'm glad that my own work is useful to me (and will be probably be to others as well). :-)