jd:/dev/blog

OpenStack Swift eventual consistency analysis & bottlenecks

2012-04-23T12:06:00Z

Swift is the software behind the OpenStack Object Storage service.

This service provides a simple storage service for applications using RESTful interfaces, providing maximum data availability and storage capacity.

I explain here how some parts of the storage and replication in Swift works, and show some of its current limitations.

If you don't know Swift and want to read a more "shallow" overview first, you can read John Dickinson's Swift Tech Overview.

How Swift storage works

If we refer to the CAP theorem, Swift chose availability and partition tolerance and dropped consistency. That means that you'll always get your data, they will be dispersed on many places, but you could get an old version of them (or no data at all) in some odd cases (like some server overload or failure). This compromise is made to allow maximum availability and scalability of the storage platform.

But there are mechanisms built into Swift to minimize the potential data inconsistency window: they are responsible for data replication and consistency.

The official Swift documentation explains the internal storage in a certain way, but I'm going to write my own explanation here about this.

Consistent hashing

Swift uses the principle of consistent hashing. It builds what it calls a ring. A ring represents the space of all possible computed hash values divided in equivalent parts. Each part of this space is called a partition.

The following schema (stolen from the Riak project) shows the principle nicely:

In a simple world, if you wanted to store some objects and distribute them on 4 nodes, you would split your hash space in 4. You would have 4 partitions, and computing hash(object) modulo 4 would tell you where to store your object: on node 0, 1, 2 or 3.

But since you want to be able to extend your storage cluster to more nodes without breaking the whole hash mapping and moving everything around, you need to build a lot more partitions. Let's say we're going to build 2¹⁰ partitions. Since we have 4 nodes, each node will have 2¹⁰ ÷ 4 = 256 partitions. If we ever want to add a 5^th node, it's easy: we just have to re-balance the partitions and move 1⁄4 of the partitions from each node to this 5^th node. That means all our nodes will end up with 2¹⁰ ÷ 5 ≈ 204 partitions. We can also define a weight for each node, in order for some nodes to get more partitions than others.

With 2¹⁰ partitions, we can have up to 2¹⁰ nodes in our cluster. Yeepee.

For reference, Gregory Holt, one of the Swift authors, also wrote an explanation post about the ring.

Concretely, when building one Swift ring, you'll have to say how much partitions you want, and this is what this value is really about.

Data duplication

Now, to assure availability and partitioning (as seen in the CAP theorem) we also want to store replicas of our objects. By default, Swift stores 3 copies of every objects, but that's configurable.

In that case, we need to store each partition defined above not only on 1 node, but on 2 others. So Swift adds another concept: zones. A zone is an isolated space that does not depends on other zone, so in case of an outage on a zone, the other zones are still available. Concretely, a zone is likely to be a disk, a server, or a whole cabinet, depending on the size of your cluster. It's up to you to chose anyway.

Consequently, each partitions has not to be mapped to 1 host only anymore, but to N hosts. Each node will therefore store this number of partitions:

number of partition stored on one node = number of replicas × total number of partitions ÷ number of node

Examples:

We split the ring in 2¹⁰ = 1024 partitions. We have 3 nodes. We want 3 replicas of data.
→ Each node will store a copy of the full partition space: 3 × 2¹⁰ ÷ 3 = 2¹⁰ = 1024 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 5 nodes. We want 3 replicas of data.
→ Each node will store 2¹¹ × 3 ÷ 5 ≈ 1129 partitions.

We split the ring in 2¹¹ = 2048 partitions. We have 6 nodes. We want 3 replicas of data.
→ Each node will store 2¹¹ × 3 ÷ 6 = 1024 partitions.

Three rings to rule them all

In Swift, there is 3 categories of thing to store: account, container and objects.

An account is what you'd expect it to be, a user account. An account contains containers (the equivalent of Amazon S3's buckets). Each container can contains user-defined key and values (just like a hash table or a dictionary): values are what Swift call objects.

Swift wants you to build 3 different and independent rings to store its 3 kind of things (accounts, containers and objects).

Internally, the two first categories are stored as SQLite databases, whereas the last one is stored using regular files.

Note that this 3 rings can be stored and managed on 3 completely different set of servers.

Data replication

Now that we have our storage theory in place (accounts, containers and objects distributed into partitions, themselves stored into multiple zones), let's go the replication practice.

When you put something in one of the 3 rings (being an account, a container or an object) it is uploaded into all the zones responsible for the ring partition the object belongs to. This upload into the different zones is the responsibility of the swift-proxy daemon.

But if one of the zone is failing, you can't upload all your copies in all zones at the upload time. So you need a mechanism to be sure the failing zone will catch up to a correct state at some point.

That's the role of the swift-{container,account,object}-replicator processes. This processes are running on each node part of a zone and replicates their contents to nodes of the other zones.

When they run, they walk through all the contents from all the partitions on the whole file system and for each partition, issue a special REPLICATE HTTP request to all the other zones responsible for that same partition. The other zone responds with information about the local state of the partition. That allows the replicator process to decide if the remote zone has an up-to-date version of the partition.

In case of account and containers, it doesn't check at the partition level, but check each account/container contained inside each partition.

If something is not up-to-date, it will be pushed using rsync by the replicator process. This is why you'll read that the replication updates are "push based" in Swift documentation.

# Pseudo code describing replication process for accounts
# The principle is exactly the same for containers
for account in accounts:
    # Determine the partition used to store this account
    partition = hash(account) % number_of_partitions
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-account-server process
        version_of_account = zone.send_HTTP_REPLICATE_for(account):
        if version_of_account < account.version()
            account.sync_to(zone)

This replication process is O(number of account × number of replicas). The more your number of account will increase and the more you will want replicas for your data, the more the replication time for your accounts will grow. The same rule applies for containers.

# Pseudo code describing replication process for objects
for partition in partitions_storing_objects:
    # The number of zone is the number of replicas configured
    for zone in partition.get_zones_storing_this_partition():
        # Send a HTTP REPLICATE command to the remote swift-object-server process
        verion_of_partition = zone.send_HTTP_REPLICATE_for(partition):
        if version_of_partition < partition.version()
            # Use rsync to synchronize the whole partition
            # and all its objects
            partition.rsync_to(zone)

This replication process is O(number of objects partitions × number of replicas). The more your number of objects partitions will increase, and the more you will want replicas for your data, the more the replication time for your objects will grow.

I think this is something important to know when deciding how to build your Swift architecture. Choose the right number the number of replicas, partitions and nodes.

Replication process bottlenecks

File accesses

The problem, as you might have guessed, is that to replicate, it walks through every damn things, things being accounts, containers, or object's partition hash files. This means it need to open and read (part of) a every file your node stores to check that data need or not to be replicated!

For accounts & containers replication, this is done every 30 seconds by default, but it will likely take more than 30 seconds as soon as you hit around 12 000 containers on a node (see measurements below). Therefore you'll end up checking consistency of accounts & containers on each all node all the time, using obviously a lot of CPU time.

For reference, Alex Yang also did an analysis of that same problem.

TCP connections

Worst, the HTTP connections used to send the REPLICATE commands are not pooled: a new TCP connection is established each time something has to be checked against the same thing stored on a remote zone.

This is why you'll see in the Swift's Deployment Guide this lines listed under "general system tuning":

# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1

# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144

In my humble opinion, this is more an ugly hack than a tuning. If you don't activate this and if you have a lot of containers on your node, you'll end up soon with thousands of connections in TIME_WAIT state, and you indeed risk to overload the IP conntrack module.

Container deletion

We also should talk about container deletion. When a user deletes a container from its account, the container is marked as deleted. And that's it. It's not deleted. Therefore the SQLite database file representing the container will continue to be checked for synchronization, over and over.

The only way to have a container permanently deleted is to mark an account as deleted. This way the swift-account-reaper will delete all its containers and, finally, the account.

Measurement

On a pretty big server, I measured the replications to be done at a speed of around 350 {account,container,object-partitions}/second, which can be a real problem if you chose to build a lots of partition and you have a low number_of_node ⁄ number_of_replicas ratio.

For example, the default parameters runs the container replication every 30 seconds. To check replication status of 12 000 containers stored on one node at the speed of 350 containers/seconds, you'll need around 34 seconds to do so. In the end, you'll never stop checking replication of your containers, and the more you'll have containers, the more your inconsistency window will increase.

Conclusion

Until some of the code is fixed (the HTTP connection pooling probably being the "easiest" one), I warmly recommend to chose correctly the different Swift parameters for your setup. The replication process optimization consists in having the minimum amount of partitions per node, which can be done by:

decreasing the number of partitions
decreasing the number of replicas
increasing the number of node

For very large setups, some code to speed up accounts and containers synchronization, and remove deleted containers will be required, but this does not exist yet, as far as I know.

The opinions expressed here represent my own and not those of my employer.

First release of PyMuninCli

2012-04-17T11:06:00Z

Today I release a Python client library to query Munin servers.

I wrote it as part of some experiments I did a few weeks ago. I discovered there was no client library to query a Munin server. There's PyMunin or python-munin which help developing Munin plugins, but nothing to access the munin-node and retrieve its data.

So I decided to write a quick and simple one, and it's released under the name of PyMuninCli, providing the munin.client Python module.

mod_defensible 1.5 released

2012-04-03T15:48:00Z

Apache 2.4 being out, I noticed that my good old mod_defensible did not compile anymore.

The changes in the new Apache 2.4 API were small for its concern, so it was pretty easy to update this software to make it compile again.

Honestly, I'm not sure that this module is really used into the wild, but I still think that it can serve as a good prototype for doing other things so I like keeping it around. :-)

All this has been triggered by the Apache 2.4 arrival into Debian experimental. Therefore I've updated the mod_defensible package to use the new dh_apache2, and imported it into Git at the same time.

xpyb 1.3 released

2012-03-22T22:06:00Z

It took a while to get it out, but finally, 3 years after the latest release (1.2), the version of 1.3 of xpyb (the XCB Python bindngs) is out.

This version has a lot of improvement, and major bug fixes (memory corruption and memory leak were tracked down and fixed).

One amazing feature that is now shipped with that release, is my code to export the xpyb API to other Python modules, allowing to draw with Pycairo in Python using XCB.

Here is an example of a Python program that draws a spiral in a window using xpyb and Pycairo. You need xpyb >= 1.3 and Pycairo >= 1.10 to make this works.

import cairo
import xcb
from xcb.xproto import *

WIDTH, HEIGHT = 600, 600

def draw_spiral(ctx, width, height):
    """Draw a spiral with lines!"""
    wd = .02 * width
    hd = .02 * height

    width -= 2
    height -= 2

    ctx.move_to (width + 1, 1-hd)
    for i in range(9):
    ctx.rel_line_to (0, height - hd * (2 * i - 1))
    ctx.rel_line_to (- (width - wd * (2 *i)), 0)
    ctx.rel_line_to (0, - (height - hd * (2*i)))
    ctx.rel_line_to (width - wd * (2 * i + 1), 0)

    ctx.set_source_rgb (0, 0, 1)
    ctx.stroke()

# Connect to the X server
conn = xcb.connect()
# Get the X server setup
setup = conn.get_setup()
# Generate X ID for our X "objects"
window = conn.generate_id()
pixmap = conn.generate_id()
gc = conn.generate_id()
# Create a new window
conn.core.CreateWindow(setup.roots[0].root_depth, window,
                       # Parent is the root window
                       setup.roots[0].root,
                       0, 0, WIDTH, HEIGHT, 0, WindowClass.InputOutput,
                       setup.roots[0].root_visual,
                       CW.BackPixel | CW.EventMask,
                       [ setup.roots[0].white_pixel, EventMask.ButtonPress | EventMask.EnterWindow | EventMask.LeaveWindow | EventMask.Exposure ])

# Create a pixmap: it will be used to draw with cairo
conn.core.CreatePixmap(setup.roots[0].root_depth, pixmap, setup.roots[0].root,
                       WIDTH, HEIGHT)

# We just need a GC to copy later the pixmap on the window, so create one
# very simple
conn.core.CreateGC(gc, setup.roots[0].root, GC.Foreground | GC.Background,
                   [ setup.roots[0].black_pixel, setup.roots[0].white_pixel ])

# Create a cairo surface
surface = cairo.XCBSurface (conn, pixmap,
                            setup.roots[0].allowed_depths[0].visuals[0], WIDTH, HEIGHT)
# Create a cairo context with that surface
ctx = cairo.Context(surface)

# Paint everything in white
ctx.set_source_rgb (1, 1, 1)
ctx.set_operator (cairo.OPERATOR_SOURCE)
ctx.paint()

# Draw our spiral
draw_spiral (ctx, WIDTH, HEIGHT)

# Map the window on the screen so it gets visible
conn.core.MapWindow(window)

# Flush all X requests to the X server
conn.flush()

while True:
    try:
        event = conn.wait_for_event()
    except xcb.ProtocolException, error:
        print "Protocol error %s received!" % error.__class__.__name__
        break
    except:
        break

    # ExposeEvent are received when we need to refresh the content of the
    # window, so we copy the content of the pixmap (where cairo drew) in the
    # window
    if isinstance(event, ExposeEvent):
        conn.core.CopyArea(pixmap, window, gc, 0, 0, 0, 0, WIDTH, HEIGHT)
    # You click, I quit.
    elif isinstance(event, ButtonPressEvent):
        break
    conn.flush()

Seeing the complexity it is to draw something simple with this technology, I somehow understand why nobody bothered to release or use the code during the last 3 years.

But hey, now that it's out, you can build the next Python based desktop environment with bleeding edge technologies. :-)

10 years as a Debian developer

2012-02-24T08:55:00Z

Ten years ago, I joined the Debian project as a developer.

At that time, I was 18 and in my first year at university, hanging out with the TuxFamily system administrators, which included 3 french Debian developers (sjg, igenibel and creis).

I was learning Debian packaging while working on VHFFS, and decided to package one or two non-yet-packaged software for Debian. My friends pushed me into the NM process, and less than 2 months later I was a Debian developer. One have to admit that back in the days, the NM process was really fast if you were able to reply to the questions quickly. :-) I think I became the youngest developer among Debian's ones.

That was my first steps in a Free Software project, and it was really exciting.

In 10 years, I've been doing a lot of different things for Debian. Sure, I've been using it all the years long, but let's recap a bit what I did, from what I recall.

My first Debian only project was apt-build around 2003, and later rebuildd in 2007.

I built the Xen packaging team in 2005, I've been a Stable Release Manager for a year in 2006, and did heavy bug squashing to release Etch that same year.

I also was an Application Manager in 2006 and managed the application of 2 Debian developers (Jose Parrella and Damián Viano).

I admit I've been less active in Debian after 2007, mainly because I was busy working on awesome, GNU Emacs and others software.

Since 2011, I joined the OpenStack packaging team and I'm working on OpenStack on a (almost) daily basis.

I don't know how many packages I touched, managed or updated, but that should be one or two hundreds. I still maintain 53 of them.

After all, the adventure has been really pleasant, and I had the chance to work with and meet fabulous and smart people. I always liked this project and what it's trying to do.

After all these years, I'm definitively staying! See you in another 10 years, folks! :)

Google Calendar notifications using pynotify

2012-01-03T18:55:00Z

I use Google Calendar to manage my calendars, and I really missed something to warn me whenever I have an appointment with an alert set.

So here is an example of a Python program to do such a thing. It is written using the Google Data APIs Python client library and pynotify.

I'll detail the code here, so you can build your own and adapt it to your needs.

First, we need to import GTK+ and pynotify, and initialize it.

import gtk
import pynotify
pynotify.init(sys.argv[0])

Then, we need to import gdata Calendar API and connect to the calendar. I'll use the simple email/password way to login, which is clearly not the best, but it's also the simplest. Feel free to use OAuth 2.0. :-)

calendar_service = gdata.calendar.service.CalendarService()
calendar_service.email = 'mygooglelogin'
calendar_service.password = 'mygooglepassword'
calendar_service.ProgrammaticLogin()

Now we're ready to request stuff and notify! First, request the events from the default calendar.

feed = calendar_service.GetCalendarEventFeed()

Now we can iterate over feed and do various checks.

for event in feed.entry:
    # If the event status is not confirmed, go to the next event.
    if event.event_status.value != "CONFIRMED":
        continue
    # Now iterate over all the event dates (usually it has one)
    for when in event.when:
        # Parse start and end time
        try:
            start_time = datetime.datetime.strptime(when.start_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
            end_time = datetime.datetime.strptime(when.end_time.split(".")[0], "%Y-%m-%dT%H:%M:%S")
        except ValueError:
            # ValueError happens on parsing error. Parsing errors
            # usually happen for "all day" events since they have
            # not time, but we do not care about this events.
            continue
        now = datetime.datetime.now()
        # Check that the event hasn't already ended
        if end_time > now:
            # Check each alert
            for reminder in when.reminder:
                # We handle only reminders with method "alert"
                # and whose start time minus the reminder delay has passed
                if reminder.method == "alert" \
                        and start_time - datetime.timedelta(0, 60 * int(reminder.minutes)) < now:
                    # Build the notification
                    notification = pynotify.Notification(summary=event.title.text,
                                                         message=event.content.text)
                    # Set an icon from the GTK+ stock icons
                    notification.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_DIALOG_INFO,
                                                                              gtk.ICON_SIZE_LARGE_TOOLBAR))
                    notification.set_timeout(0)
                    # Show the notification
                    notification.show()

Running this program, you should see a notification if an appointment has an alert to be raised at that time.

This should be enough to start to build something.

If you don't want to program this into Python, you might want to take a look at gcalcli.

Using GTK+ stock icons with pynotify

2011-12-27T11:55:00Z

It took me a while to find this, so I'm just blogging it so other people will be able to find it.

I wanted to send a desktop notification using pynotify, but using a GTK+ stock icons.

With the following snippet, I managed to do it.

import pynotify
pynotify.init("myapp")
import gtk
n = pynotify.Notification(summary="Summary", message="Message!")
n.set_icon_from_pixbuf(gtk.Label().render_icon(gtk.STOCK_HARDDISK, gtk.ICON_SIZE_LARGE_TOOLBAR))
n.show()

Note that the use of a Label is just to have a widget instanciated to use the render_icon() method. It could be any widget type as far as I understand.

My OpenStack work

2011-12-16T17:34:00Z

Like I already wrote here last week, I've been heavily working on OpenStack for the last weeks.

My first assignment was to package OpenStack for Debian. The packages already present in unstable were mainly done by Thomas Goirand, who based its work on the one done in Ubuntu. Therefore, the packages where not in a very good shape for Debian.

Today Ghe Rivero and I (members of the OpenStack Debian packaging team) managed to push the OpenStack Essex 2 milestone into unstable with great success. You can now test and deploy OpenStack Essex 2 very easily!

Packaging OpenStack made me write several patches, mainly related to packaging, patches which were all accepted and merged by upstream. This is nice because most of the OpenStack Debian packages lost their debian/patches directories now!

Finally, I've finished to implement one blueprint I really missed: the ability to boot from an ISO image using libvirt. The code still needs a review, but it should be included in the Essex 3 milestone if everything's right.

New job, new blog

2011-12-07T13:47:00Z

It has been a while since I blogged but I've been very busy, with my new job and this new blog!

New job!

I quitted my job last September, and found another one that I started in October. I'm now the lead developer of eNovance Labs, where I work on the OpenStack project. So far, this allowed me to contribute heavily to the Debian packaging of OpenStack.

New blog!

In the meantime, I took some time to redesign my personal homepage and this blog, which is now using Hyde, the Python equivalent of Jekyll, which is in Ruby. Since I dislike Ruby (sorry), I preferred to use a Python based generator, and I admit Hyde is really cool. Since I really suck at Web design, this one is obviously based on Twitter's bootstrap

Google Contacts for Emacs

2011-09-26T13:01:00Z

I finally finished a thing I was really missing: accessing my Google Contacts from Emacs.

That's now possible, thanks to my new google-contacts.el package.

It includes searching for any contact and displaying the result in a window. You can also jump to a contact from Gnus by pressing a key, and complete e-mail addresses while composing a mail.