This article is the final one of the series I wrote about scaling a large
number of connections in a Python application. If you don't remember what the
problem we're trying to solve is, here it is, coming from one of my followers':
I am proud to announce today the immediate release of Scaling Python, my second book
about Python! It talks about the distribution and performance of applications
written in Python, and how to build them properly!
It took me a year to build this entirely new product around Python. It's an
exciting moment and I am sure it will enjoy many of my dear readers that are
waiting for it for a while now!
A few days ago, I've recently been contacted by Godson Rapture
from Cool Python codes to answer a few questions about what I work on in
open source. Godson regularly interview developers and I invite you to check
out his website!
My day-to-day activities are still evolving around
the Python programming language, as I continue working on
the OpenStack project as part of my job
at Red Hat. OpenStack is still the biggest Python project
out there, and attract a lot of Python hackers.
Those last few years, however, things have taken a different turn for me when I
made the choice with my team to rework the telemetry stack architecture. We
decided to make a point of making it scale way beyond what has been done in the
project so far.
Packaging Python has been a painful experience for long. The history of the
various distribution that Python offered along the years is really bumpy, and
both the user and developer experience has been pretty bad.
When working with timestamps, one question that often arises is the precision
of those timestamps. Most software is good enough with a precision up to the
second, and that's easy. But in some cases, like working on metering, a finer
precision is required.
The first major version of the scalable timeserie database I work on,
Gnocchi was a released a few months ago. In this first
iteration, it took a rather naive approach to data storage. We had little ideas
about if and how our distributed back-ends were going to be heavily used, so we
stuck to the code of the first proof-of-concept written a couple of years ago.
Recently we got more feedbacks from our users, ran a few
benchmarks. That gave us enough feedback to start
investigating in improving our storage strategy.
Writing programs is fun, but making them fast can be a pain. Python programs
are no exception to that, but the basic profiling toolchain is actually not
that complicated to use. Here, I would like to show you how you can quickly
profile and analyze your Python code to find what part of the code you should
I've started to use Pocket a few months ago to store my
backlog of things to read. It's especially useful as I can use it to read
content offline since we still don't have any Internet access in places such
as airplanes or the Paris metro. It's only 2015 after all.
Recently, I've been fighting with the never ending issue of timezones. I never
thought I would have plunged into this rabbit hole, but hacking on OpenStack
and Gnocchi I felt into that trap easily is, thanks to Python.
I don't often write about tools I use when for my daily software development
tasks. I recently realized that I really should start to share more often my
workflows and weapons of choice.
One thing that I have a hard time enduring while doing Python code reviews, is
people writing utility code that is not directly tied to the core of their
business. This looks to me as wasted time maintaining code that should be
reused from elsewhere.
So today I'd like to start with
retrying, a Python package that you
can use to… retry anything.
I've recently been contacted by Johannes Hubertz,
who is writing a new book about Python in German called "Softwaretests mit
Python" which will be published by Open Source Press, Munich this
summer. His book will feature some interviews, and he was kind enough to let me
write a bit about software testing. This is the interview that I gave for his
book. Johannes translated to German and it will be included in Johannes' book,
and I decided to publish it on my blog today. Following is the original
A lot of people read up on good Python practice, and there's plenty of
information about that on the Internet. Many tips are included in the book I
wrote this year,
The Hacker's Guide to Python. Today I'd
like to show a concrete case of code that I don't consider being the state of
And done! It took me just 8 months to do this entire book project around
Python. From the first day I started writing to today, where I finally
publish and sell – almost entirely – myself this book. I'm really proud of
what I've achieved so far, as this was something totally new to me.
The Ceilometer project supports various
database backend that can be used as storage. Among them are
MongoDB, SQLiteMySQL, PostgreSQL,
HBase, DB2… All Ceilometer's code is unit
tested, but when dealing with external storage services, one cannot be sure
that the code is really working. You could be inserting data with an
incorrect SQL statement, or in the wrong table. Only having the real
database storage running and being used can tell you that.
I love to say that Python is a nice subset of Lisp, and I discover that it's
getting even more true as time passes. Recently, I've stumbled upon the
PEP 443 that describes a way to
dispatch generic functions, in a way that looks like what CLOS, the Common
Lisp Object System, provides.
I've been hacking on Python for a lot of years now, on various project. For
the last two years, I've been heavily involved in
OpenStack, which makes an heavy usage of Python.
Once you start working with a hundred of hackers, on several software and
libraries representing more than half a million source lines of Python,
things change. The scalability, testing and deployment problems inherent to
a cloud platform meddle with everything in designing components.
Doing code reviews is a great way to discover things that people might
struggle to comprehend. While proof-reading
OpenStack patches recently, I spotted that
people were not using correctly the various decorators Python provides for
methods. So here's my attempt at providing me a link to send them to in my
next code reviews. :-)
I've meant to look at Hy since
Paul Tagliamonte started to talk to me about
it, but never took a chance until now.
Yesterday, Paul indicated it was a good time for me to start looking at it,
so I spent a few hours playing.
It's not that often that I start something from scratch. It's an amazing
feeling to start a new project, to start writing something new. I like
that. It's creation, it's an artistic part of our computing stuff. I feel
like a code artist.