books — jd:/dev/blog

A Decade of Writing Books and Selling 25,000 Copies

Wed, 12 Jun 2024 00:00:00 GMT

Ten years ago, I embarked on a journey that profoundly shaped my career and personal growth. Writing my first book, The Hacker's Guide to Python (later updated and renamed Serious Python), marked the beginning of a series of literary endeavors that allowed me to share my knowledge, experiences, and passion for Python programming with a global audience. Today, I reflect on this journey, the lessons learned, and the incredible milestones achieved along the way.

First print of The Hacker’s Guide to Python in 2014

The Genesis: The Hacker's Guide to Python

In March 2014, I published my first book, The Hacker's Guide to Python. This book was born out of a desire to provide a comprehensive resource for Python developers, offering insights and techniques I had gathered over the years. The response was overwhelmingly positive, and it motivated me to continue writing and sharing my expertise. I sold over 3,000 copies of the book in a couple of years, which is a very good number in its category.

Challenges and Time Investment

Writing was an enormous undertaking that demanded significant time and effort. I spent around 150 hours on each book, covering various activities from writing and editing to marketing and publishing. The process was spread over a year most of the time.

One of the toughest challenges was constructing a coherent and comprehensive table of contents. This initial step was crucial, as it guided the entire writing process, making the subsequent task of filling in the blanks somewhat more manageable. Additionally, I had to balance my time between my day job as a software engineer and this side project, making time management a critical aspect of the endeavor.

Another significant difficulty was the proofreading process. I needed both technical and language reviews to ensure the content was accurate and well-written, considering English is not my native language. Finding reliable reviewers who could provide timely and constructive feedback was challenging. Despite reaching out to many contacts, only a fraction responded and contributed consistently.

Self-publishing also taught me marketing, one of the best skills I could have learned, and I’m still leveraging it to this day while working on Mergify.

Scaling Python and Serious Python

Following the success of my first book, I continued exploring new topics and challenges within the Python ecosystem. Scaling Python, published in 2017, delved into the complexities of scaling applications, a topic that resonated with many developers facing similar challenges. I distributed around 1,000 copies of this book.

Scaling Python cover

In 2019, after being approached by No Starch, I released Serious Python, a book aimed at helping developers write more efficient, maintainable, and scalable code. Both books received praise for their practical approach and in-depth coverage of advanced topics. Being backed by No Starch helped the book to be distributed widely, and made it reach 20,000 copies as of today.

Impact on My Career

Writing these books significantly impacted my career and established me as an authority in the Python community. When I joined Datadog in 2019, I remember seeing my books casually lying around at the entrance of the Paris office.

My books chilling in the Datadog Paris office entrance in 2019

This moment was a profound realization of the reach and influence of my work. Colleagues and peers often treated the content of my books as definitive guides. The books provided answers and insights so clearly that people often didn't feel the need to ask me questions about the topics I covered during interviews; they trusted my written word as a reliable source. This validation opened new opportunities and allowed me to connect with an extensive network of professionals who recognized and respected my expertise.

Connecting with the Community

Writing allowed me to talk to anyone, reach out to amazing hackers worldwide, and forge new friendships. Writing books was the best excuse to meet fantastic people and create new friendships. I discovered fantastic engineers and learned from their experiences while interviewing them. This journey has been incredibly rewarding, not just professionally but also personally, as I connected with a vibrant community of developers who share my passion for Python and open-source software.

Sharing the knowledge of my book on stage during PyCon FR 2017

I also saw my books being translated into multiple languages, including Chinese and Korean.

This feeling is awesome as it gives even more impact to your writing, knowing that your knowledge is spreading across the globe. Having your work translated and accessible to a wider audience is a great reward, and it emphasizes the importance and value of sharing knowledge on such a large scale.

The Joy of Writing

While writing is hard, it is also refreshing. Producing content that people love and are happy to recommend is a fantastic feeling. My golden rule was, and still is:

Produce content that you'd be happy consuming.

The rest then becomes history. This philosophy guided me through the writing process and ensured that my books remained valuable and relevant to readers.

A New Era of Writing

Reflecting on my writing journey, it resonates deeply with a post I wrote titled "I used to write.” In that post, I shared my journey from writing extensively in my early years to facing the challenges of balancing life and work, decreasing my writing output. The desire to return to the keyboard lingered, and despite the rise of AI-generated content, I realized that authentic, human writing still holds immense value.

Over the last year, I toyed with GPT, generating tons of content and using it to brainstorm, change sentences, and rewrite text. This experimentation reaffirmed my belief that AI could never truly replace the nuanced and creative process of human writing. As AI-generated content grows, the need for genuine, human-crafted writing becomes even more critical. This new writing era challenges us to strengthen our signal amidst the growing noise.

The past ten years have been an incredible journey of learning, teaching, and connecting with developers worldwide. I look forward to continuing this journey, exploring new topics, and sharing my insights through future books and blog posts.

Thank you for being a part of this journey.

I used to write

Tue, 21 May 2024 00:00:00 GMT

I did. I mean, I used to write a lot back in the day.

To give some context, I started my blog in 2003 — 21 years ago. (I later reflected on my writing career in A Decade of Writing Books and Selling 25,000 Copies.) It ran on Dotclear, an old blog engine. I switched to various static code generators over the years, spending hours migrating data from one format to another. I used Muse within Emacs, Org-mode, probably at some point, and finally, Ghost.

The cover of Serious Python

Then I stopped.

The number of publications I posted online decreased over the years as I spent more and more time coding, CEO’ing things at Mergify, and managing life. Writing slowly faded away, replaced with the mundane demandes of daily life. I closed my blog, and its content vanished in the limb of a file named jd-dev-blog.zip.

Sure, I thought about writing more. Or again. Over the last couple of years, every week, my stomach would ache, and my brain would melt under the weight of my thoughts. I wanted to yell so many things at the world, correct so many wrongs, and share so many learnings.

https://xkcd.com/386/

But the time to write disappeared, buried under the many priorities that other necessities. Still, the desire to return to the keyboard lingered.

Until the final nail in the coffin hit.

AI became mainstream.

Anyone could write anything in seconds. Copywriters were being replaced by bots. The cost savings of replacing humans would revolutionize entire industries. There was no point in writing anymore. My engineer brain decided that the problem was solved.

So I gave up. My mind gave up.

My desire to write would be instantly killed by the concept of ChatGPT. The existence of that malicious AI would stifle any urge to draft thoughts on virtual paper. The sheer thought of such an entity lurking in the digital shadows, analyzing and predicting every word, strikes a chilling fear into the heart of my creativity.

Until today.

Over the last year, I toyed with GPT, generating tons of content. I used it to brainstorm, change sentences, and rewrite text. The more I used it, the more I realized I was getting bored. Browsing the Internet, and social networks, I realized humans were replaced by AI.

No one would write anymore; everyone would just publish.

With its cold, calculating algorithms, AI reduced the rich tapestry of human expression to mere patterns and data points. People would throw a bare concept or even ask AI for one and demand it to produce text. Many of those publishers would not even take the time to tweak the AI, to feed it with the small amount of style and humanity they would have. Content would be farmed, from social media posts to SEO blog posts.

Over the last year, everything became bland. The once vibrant landscape of ideas has been replaced with mechanical mimicry.

My brain acknowledged that AI could never truly write. That revelation shifted my perspective, and I realized that writing wasn't dead. The noise would undoubtedly grow louder, but this only meant the signal would need to be stronger.

We are entering a new era for writing.

Well, at least, that’s what I hope.

I am a Software Engineer and I am in Charge

Tue, 22 Dec 2020 00:00:00 GMT

Fifteen years have passed since I started my career in IT — which is quite some time. I've been playing with computers for 25 years now, which makes me quite knowledgeable about the field, for sure.

However, while I was fully prepared to bargain with computers, I was not prepared to do so with humans. The whole career management thing was unknown to me. I had no useful skills to navigate within the enterprise organization. I had to learn the ropes the hard way, failing along the way. It hurts.

Almost ten years ago, I had the chance to meet a new colleague — Alexis Monville. Alexis was a team facilitator, and I started to work with him on many non-technical levels. He taught me a lot about agility and team organization. Working on this set of new skills changed how I envisioned my work and how I fit into the company.

I worked on those aspects of my job because I decided to be in charge of my career rather than keeping things boring. That was one of the best decisions I ever made. Growing the social aspect of my profession allowed me to develop and find aspiring jobs and missions.

Getting to that point takes a lot of time and effort, and it's pretty hard to do it alone. My friend Alexis wrote an excellent book titled I am a Software Engineer and I am in Charge. I'm proud to have been the first reviewer the book before it was released a few weeks ago.

Many developers out there are stuck in a place where they are not excited by their colleagues' work and whose managers do not appropriately recognize their achievement. It would be best for them if they did something about that.

This book is an excellent piece for engineers who wants to break the cycle of frustration. It covers many situations I encountered across my professional life those last years, giving good insights into how to solve them.

To paraphrase Alexis, the answers to your career management problems are not on StackOverflow — they're not technical issues. However, you can still solve them with the right tools. That's where I am a Software Engineer and I am in Charge shines. It gives you leads, solutions, and exercise to get out of this kind of situation. It helps increase your impact and satisfaction at work.

I love this book, and I wish I had access to it years ago. Developing technical leadership is not easy and requires a mindset shift. Having a way to bootstrap yourself with this is a luxury.

If you're a software engineer at the beginning of your career or struggling with your current professional situation, I profoundly recommend reading this book!

You'll get a fast track on your career, for sure.

The Art of PostgreSQL is out!

Wed, 28 Aug 2019 00:00:00 GMT

If you remember well, a couple of years ago, I wrote about Mastering PostgreSQL, a fantastic book written by my friend Dimitri Fontaine.

Dimitri is a long-time PostgreSQL core developer — for example, he wrote the extension support in PostgreSQL — no less. He is featured in my book Serious Python, where he advises on using databases and ORM in Python.

Today, Dimitri comes back with the new version of this book, named The Art of PostgreSQL.

I love the motto of this book: Turn Thousands of Lines of Code into Simple Queries. I have spent all my career working with code that talks to databases, and I can't count the number of times where I've seen people write lengthy, slow code in their pet language rather than a single well-thought SQL query which would do a better job.

This is exactly what this book is about.

That's why it's my favorite SQL book. I learned so many things from it. In many cases, I've been able to divide by 10 the size of the code I had to write in Python to implement a feature. All I had to do is to browse the book to discover the right PostgreSQL feature and write a single SQL query. The right query that does the job for me_._

Less code, fewer bugs, more happiness!

The book also features interviews with great PostgreSQL users and developers — hey, no wonder where Dimitri got this idea, right? ;-)

I loved those interviews. What's better than reading Kris Jenkins explaining how Clojure and PostgreSQL play nice together, or Markus Winand (from the famous use-the-index-luke.com) talking about the relationship developers have with their database. :-)

No need to say that you should get your hands on this right now. Dimitri just made a launch offer where he offers a 15% discount on the book until the end of this month! You can also read the free chapter to get an idea of what you'll get.

Last thing: it's DRM-free and money-back guaranteed. You can get this book with your eyes closed.

Serious Python released!

Thu, 17 Jan 2019 00:00:00 GMT

Today I'm glad to announce that my new book, Serious Python, has been released.

However, you wonder… what is Serious Python?

Well, Serious Python is the the new name of The Hacker's Guide to Python — the first book I published. Serious Python is the 4th update of that book — but with a brand a new name and a new editor!

For more than a year, I've been working with the editor No Starch Press to enhance this book and bring it to the next level! I'm very proud of what we achieved, and working with a whole team on this book has been a fantastic experience.

The content has been updated to be ready for 2019: pytest is now a de-facto standard for testing, so I had to write about it. On the other hand, Python 2 support was less a focus, and I removed many mentions of Python 2 altogether. Some chapters have been reorganized, regrouped and others got enhanced with new content!

The good news: you can get this new edition of the book with a 15% discount for the next 24 hours using the coupon code SERIOUSPYTHONLAUNCH on the book page.

The book is also released as part as No Starch collection. They also are in charge of distributing the paperback copy of the book. If you want a version of the book that you can touch and hold in your arms, look for it in No Starch shop, on Amazon or in your favorite book shop!

Scaling Python released

Tue, 05 Dec 2017 00:00:00 GMT

I am proud to announce today the immediate release of Scaling Python, my second book about Python! It talks about the distribution and performance of applications written in Python, and how to build them properly!

It took me a year to build this entirely new product around Python. It's an exciting moment and I am sure it will enjoy many of my dear readers that are waiting for it for a while now!

I've been able to build this using my last three years of experience working on The Hacker's Guide to Python – an amazing adventure.

Starting now, you can enjoy reading the book and learn a bit more about building distributed and scalable applications with Python. I really hope it'll help you bring your Python-fu to a new level, and that it will help you build great projects!

Since this is first days of sale, you will enjoy a 15% discount on all packages for the next 48 hours!

Scaling Python: the interviewees

Tue, 28 Nov 2017 00:00:00 GMT

The release date for Scaling Python is now very close! Today, I'd like to talk a bit about the interviews that I've run those last months that are featured in the book.

I'm glad that during those long weeks work, I have managed to find a Python expert on each of the major topic covered in the book. They will provide hindsight on the different subject covered and share their experience so you can benefit from it!

Without further delay, ladies and gentlemen, here they are:

# Mehdi Abaakouk

Mehdi is a French free software hacker, working at Red Hat, who has been using Linux for almost twenty years now. He works daily on OpenStack, the largest open source project using Python. He also regularly builds and contribute to distributed applications and is responsible for several widely used Python libraries – Cotyledon, oslo.messaging, etc.

In the book, Mehdi gives excellent tips on how to build distributed daemons.

Naoki Inada

Naoki is a Japanese software engineer, who happens to also be one of the CPython developers. He worked on several significant features in CPython, such as asyncio.

You'll be able to read Naoki opinion on Python and other programming languages when it comes to asynchronous workflows.

Chris Dent

Chris Dent has been using Python for more than 15 years now and is an expert on WSGI. He has an extensive knowledge about REST API – he is one of the early organizers of the OpenStack API working group.

Chris has, among other things, created Gabbi, a fabulous Python testing tool for. In Scaling Python, he provides best practice on building REST API.

Joshua Harlow

Joshua is a highly experienced engineer in distributed systems. He maintains a few Python libraries, such as Kazoo (ZooKeeper client) or TaskFlow (distributed tasks).

In the book, Joshua lays down principles that make Python application resilient and fault tolerant.

Alexys Jacob-Monier

Alexys is the CTO of 1000mercis and is part of the open-source software community for a few years now. He regularly gives speeches at Python conferences and talks about how to leverage Python when distributing applications.

Alexys talks about advanced techniques, e.g. using consistent hash rings, and how they should be applied.

Victor Stinner

Victor is a long time CPython core developer, working on the language itself for several years now. He is well known in the community for working on making CPython faster and leads several performance-oriented projects.

In Scaling Python, Victor talks about optimizations, profiling, and performance when using Python, and how to make the right decisions.

Jason Myers

Jason is a Python developer and an author – he wrote an entire book on SQLAlchemy, the famous Python SQL library. He worked on cloud computing platforms, as a Web developer, and as a data engineer.

In the book, we discuss with Jason about caching and RDBMS usage.

It was marvelous to have a chat with all those developers and pick their brain about different subjects. These contents broaden the scope and expand the view of the themes covered through the chapters. I can't thank them all enough!

If you want to be informed of the release of the book, subscribe in the following form! You'll be the first to be notified and to enjoy an exclusive offer. ;-)

Mastering PostgreSQL

Tue, 07 Nov 2017 00:00:00 GMT

A few months ago, my friend Dimitri Fontaine and I discussed writing books and sharing our knowledge. If you do not know Dimitri yet, he is an old-time PostgreSQL Major Contributor – meaning he writes code for the PostgreSQL software itself!

I interviewed Dimitri a few years ago in The Hacker's Guide to Python, where he shared his insight about writing proper Python application code with relational database management system.

All of this gave Dimitri the idea of writing his own book about PostgreSQL. And he released his book this week! If like me, you can't wait to read it, just scroll down below and grab a package with a 15% discount.

To celebrate the event, I went ahead and decided to ask him a few questions.

Hey Dimitri! So what made you start writing this book in the first place?

Dimitri: As a PostgreSQL consultant, I've met with many developers for whom SQL just didn't click. They then tend to consider SQL much as they would consider HTML: some string you need to build dynamically then send over to an external part of the system, either the browser or the database server.

As soon as you start on this path, SQL is more and more of a problem in your daily life and developer workflow. It doesn't integrate well with the usual testing and continuous integration tools, not to mention it's hard to review (as in code review) and hard to maintain.

At the end of the day, when using a relational database system, you have to
know your SQL.

As a developer, you need to be fluent in SQL and master window functions, common table expressions, recursive queries, time zone handling and advanced string and regexp processing functions, transaction behaviors and also how to build a query result set in JSON. And so much more.

For most developers, it's a daunting task. Just too much to learn when they have so many other things to take care of. So many lines of code to write to implement that new product idea. Well, as Dijkstra put it, lines of code are “spent” on writing a new feature. When you master SQL, you spend much less of those lines of code.

So I wrote Mastering PostgreSQL in Application Development to teach SQL to developers. Focusing on real use cases and authentic data set, so that it's easier to grasp all those advanced features. The book also addresses the tooling you need to integrate SQL as another programming language with a decent worfklow, from code review to unit testing, including regression testing and production debugging.

Who should read this book? What are the prerequisites to get the most of it?

Dimitri: The pre-requisites are quite easy to reach.

If you ever deployed an application that embeds SQL queries and talks to a
database server, you're in the target audience, the book is for you.

If you've never used PostgreSQL before, reading Mastering PostgreSQL in Application Development may convince you to have a look at that awesome piece of technology. My bet is on you switching to PostgreSQL and finding it much better at helping you in your daily work and challenges.

Even if you've been using MySQL all your life, you will learn about standard SQL features and how to use them in a way that applies to more than just PostgreSQL, so the book is going to help you in your daily life.

What's your next adventure now that this book is out?

Dimitri: There are more things that I want to do that a lifetime allows, and I am in the process of choosing what is going to be my next adventure. I feel so lucky to be able to have that problem to solve… and it still isn't the easiest one for me.

What I can tell you is that I have much more PostgreSQL knowledge to share after having been using, promoting, and contributing to this database server technology for about 20 years now. So if that first book sells well, I will get back to filling empty pages and deliver more contents to help developers making the best of SQL, to help developers on their road to Mastering PostgreSQL!

Thanks Dimitri!

I've just read the book and found it fantastic. It contains tons of tips on how to use PostgreSQL correctly, and I discovered SQL features I had no clue about. The book uses real data that you can fetch and play with. It provides the data and a Docker container with everything included so you can edit the query yourself and try it out. There is no better way to learn things than to play with the examples that are included, in just a few clicks!

The book also features a few interviews with SQL experts from the PostgreSQL community and from the development community, which gives great insight about how to use the software.

Dimitri is offering 15% off for my readers during the next 48 hours for any of the edition of the book. Just use the PYTHON-LOVES-POSTGRESQL coupon code in any of the following package:

.product img { max-width: 100%; margin-top: 10px; } .col-sm-1 { width: 8.33333333%; float: left; } .col-sm-3 { width: 25%; float: left; } .col-sm-9 { width: 75%; float: left; } .col-sm-11 { width: 91.66666667%; float: left; /* should be on the div not here but well.. */ padding-left: 30px; padding-bottom: 20px; } .row { clear: both; } .btn.btn-default { background-color: #e6940e; color: #FFF; line-height: 46px; height: 50px; font-size: 19px; cursor: pointer; text-align: center; border-radius: 5px; border: 3px solid #e6940e; vertical-align: middle; display: inline-block; padding: 0 30px; position: relative; outline: none !important; transition: color .3s ease,background .3s ease,border-color .3s ease,opacity .3s ease; box-shadow: none; }

Mastering PostgreSQL – Enterprise Edition

$179 $152
with coupon code PYTHON-LOVES-POSTGRESQL

The Enterprise Edition includes:

The book in PDF, EPUB and MOBI formats.

Interviews from industry veterans who began building web application in the previous century. They've been there and have opinions to share about how to approach SQL.

The PostgreSQL database dump that you need to run the queries against, with a script to restore it easily. The database includes all the 12 datasets used in the book.

A Docker container image of an already loaded PostgreSQL database with the whole 12 datasets in 56 tables, and the 265 SQL queries each in their own .sql file and a Web-based application for easily running and editing the SQL queries.

Licence for you to share the book and the Docker set-up with up to 50 people, including you. That's everything you need for your whole team to master SQL!

Buy the Enterprise Edition

Mastering PostgreSQL – Full Edition

$89 $75
with coupon code PYTHON-LOVES-POSTGRESQL

The Full Edition includes:

The book in PDF, EPUB and MOBI formats.

Interviews from industry veterans who began building web application in the previous century. They've been there and have opinions to share about how to approach SQL.

The PostgreSQL database dump that you need to run the queries against, with a script to restore it easily. The database includes all the 12 datasets used in the book.

Buy the Full Edition

Mastering PostgreSQL

$39 $33
with coupon code PYTHON-LOVES-POSTGRESQL

The Standard Edition includes:

The book in PDF, EPUB and MOBI formats.

Interviews from industry veterans who began building web application in the previous century. They’ve been there and have opinions to share about how to approach SQL.

Buy the book

If you have any question, feel free to reach Dimitri directly and he will be happy to reply. Or write in the comment section below!

And don't worry: if the book is not what you expect it to be and has no value to you, then just say so and Dimitri will refund you, no questions asked.

My interview with Cool Python Codes

Thu, 05 Oct 2017 00:00:00 GMT

A few days ago, I've recently been contacted by Godson Rapture from Cool Python codes to answer a few questions about what I work on in open source. Godson regularly interview developers and I invite you to check out his website!

Here's a copy of my original interview. Enjoy!

Good day, Julien Danjou, welcome to Cool Python Codes. Thanks for taking your precious time to be here.

You’re welcome!

Could you kindly tell us about yourself like your full name, hobbies, nationality, education, and experience in programming?

Sure. I’m Julien Danjou, I’m French and live in Paris, France. I studied Computer science for 5 years around 15 years ago, and continued my career in that field since then, specializing in open source projects.

Those last years, I’ve been working as a software engineer at Red Hat. I’ve spent the last 10 years working with the Python programming language. Now I work on the Gnocchi project which is a time series database.

When I’m not coding, I enjoy running half-marathon and playing FPS games.

Can you narrate your first programming experience and what got you to start learning to program?

I started programming around 2001, and my first serious programs were in Perl. I was contributing to a hosting platform for free software named VHFFS. It was a free software project itself, and I enjoyed being able to learn from other more experienced developers and being able to contribute back to it. That’s what got me stuck into that world of open source projects.

Which programming language do you know and which is your favorite?

I know quite a few, I’ve been doing serious programming in Perl, C, Lua, Common Lisp, Emacs Lisp and Python.

Obviously, my favorite is Common Lisp, but I was never able to use it for any serious project, for various reasons. So I spend most of my time hacking with Python, which I really enjoy as it is close to Lisp, in some ways. I see it as a small subset of Lisp.

What inspired you to venture into the world of programming and drove you to learn a handful of programming languages?

It was mostly scratching my own itches when I started. Each time I saw something I wanted to do or a feature I wanted in an existing software, I learned what I needed to get going and get it working.

I studied C and Lua while writing awesome- the window manager that I created 10 years ago and used for a while. I learned Emacs Lisp while writing extensions that I wanted to see in Emacs, etc. It’s the best way to start.

What is your blog about?

My blog is a platform where I write about what I work on most of the time. Nowadays, it’s mostly about Python and the main project I contribute to,
Gnocchi.

When writing about Gnocchi, I usually try to explain what part of the project I worked on, what new features we achieved, etc.

On Python, I try to share solutions to common problems I encountered or identified while doing e.g. code reviews. Or presenting a new library I created!

Tell us more about your book, The Hacker’s Guide to Python.

It’s a compilation of everything I learned those last years building large Python applications. I spent the last 6 years developing on a large code base with thousands of other developers.

I’ve reviewed tons of code and identified the biggest issues, mistakes, and bad practice that developers tend to have. I decided to compile that in a guide, helping developers that played a bit with Python to learn the stages to get really productive with Python.

OpenStack is the biggest open source project in Python, Can you tell us more about OpenStack?

OpenStack is a cloud computing platform, started 7 years ago now. Its goal is to provide a programmatic platform to manage your infrastructure while being open source and avoiding vendor lock-in.

Who uses OpenStack? Is it for programmers, website owners?

It’s used by a lot of different organizations – not really by individuals. It’s a big piece of software. You can find it in some famous public cloud providers (Dreamhost, Rackspace…), and also as a private cloud in a lot of different organizations, from Bloomberg to eBay or the CERN in Switzerland, a big OpenStack user. Tons of telecom providers also leverages OpenStack for their own internal infrastructure.

Have you participated in any OpenStack conference? What did you speak on if
you did?

I’ve attended the last 9 OpenStack summits and a few other OpenStack events around the world. I’ve been engaged in the upstream community for the last 6 years now.

My area of expertise is telemetry, the stack of software that is in charge of collecting and storing metrics from the various OpenStack components. This is what I regularly talk about during those events.

How can one join the OpenStack community?

There’s an entire documentation about that, called the Developer’s Guide. It explains how to setup your environment to send patches, how to join the community using the mailing-lists or IRC.

What makes your book, The Hacker’s Guide to Python stand out from other Python books? Also, who exactly did you write this book for?

I wrote the book that I always wanted to read about Python, but never found. It’s not a book for people that want to learn Python from scratch. It’s a great guide for those who know the language but don’t know the details that experienced developers know and that make the difference. The best practice, the elegant solutions to common problems, etc. That’s why it also includes interviews with prominent Python developers, so they can share their advice on different areas.

How can someone get your book?

I’ve decided to self-publish my book, so he does not have an editor like you can be used to see. The best place to get it is online at where you can pick the format you want, electronic or paper.

What do you mean when you say you hack with Python?

Unfortunately, most people refer to hacking as the activity of some bad guys trying to get access to whatever they’re not supposed to see. In the book title, I mean “hacking” as the elegant way of writing code and making things worse smoothly even when you were not expecting to make it.

You mentioned earlier that Gnocchi is a time series database. Can you please be more elaborate about Gnocchi? Is there also any documentation about Gnocchi?

So Gnocchi is a project I started a few years ago to store time series at large scale. Timeseries are basically a series of tuple composed of a timestamp and a value.

Imagine you wanted to store the temperature of all the rooms of the world at any point of time. You’d need a dedicated database for that with the right data structure. This is what Gnocchi does: it provides this data structure storage at very, very large scale.

The primary use case is infrastructure monitoring, so most people use it to store tons of metrics about their hardware, software, etc. It’s fully documented on its website.

How can a programmer without much experience contribute to open source projects?

The best way to start is to try to fix something that irritates you in some way. It might be a bug, it might be a missing feature. Start small. Don’t try big things first or you could be discouraged.

Never stop.

Also, don’t plunge right away in the community and start poking random people or spam them with questions. Do your homework, and listen to the community for a while to get a sense of how things are going. That can be joining IRC and lurking or following the mailing lists for example.

Big open source communities dedicate programs to help you become engaged. It might be worth a try. Generic programs like Outreachy or Google Summer of Code are a great way to start if you don’t feel confident enough to jump by your own means in a community.

Just out of curiosity, do you write code in French?

Never ever. I think it’s acceptable to write in your language if you are sure that your code will never be open sourced and that your whole team is talking in that language, no matter what – but it’s a ballsy assumption, clearly.

Truth is that if you do open source, English is the standard, so go with it. Be sad if you want, but please be pragmatic.

I’ve seen projects being open sourced by companies where all the code source comments were in Korean. It was impossible for any non-Korean people to get a glance of what the code and the project was doing, so it just failed and disappeared.

How does a team of programmers handle bugs in a large open source project?

I wish there was some magic recipe, but I don’t think it’s the case. What you want is to have a place where your users can feel safe reporting bugs. Include a template so they don’t forget any details: how to reproduce the bugs, what they expected, etc. The worst thing is to have users reporting “That does not work.” with no details. It’s a waste of time.

What tool to use to log all of that really depends on the team size and culture.

Once that works, the actual fixing of bug doesn’t follow any rule. Most developers fix the bug they encounter or the ones that are the most critical for users. Smaller problems might not be fixed for a long time.

Can you tell us about the new book you are working on and when do we expect
to get it?

That new book is entitled “Scaling Python” and it provides insight into how to build largely scalable and distributed applications using Python.

It is also based on my experience in building this kind of software during the past years. This book also includes interviews of great Python hackers who work on scalable system or know a thing or two about writing applications for performance – an important point to have scalable applications.

The book is in its final stage now, and it should be out at the beginning of 2018.

How can someone get in contact with you?

I’m reachable at julien@danjou.info by email or via Twitter, @juldanjou.

Scaling Python is on its way

Mon, 16 Jan 2017 00:00:00 GMT

My day-to-day activities are still evolving around the Python programming language, as I continue working on the OpenStack project as part of my job at Red Hat. OpenStack is still the biggest Python project out there, and attract a lot of Python hackers.

Those last few years, however, things have taken a different turn for me when I made the choice with my team to rework the telemetry stack architecture. We
decided to make a point of making it scale way beyond what has been done in the project so far.

I started to dig into a lot of different fields around Python. Topics you don't often look at when writing a simple and straight-forward application. It turns out that writing scalable applications in Python is not impossible, nor that difficult. There are a few hiccups to avoid, and various tools that can help, but it really is possible – without switching to another whole language, framework, or exotic tool set.

Working on those projects seemed to me like a good opportunity to share with the rest of the world what I learned. Therefore, I decided to share my most recent knowledge addition around distributed and scalable Python application in a new book, entitled The Hacker's Guide to Scaling Python (or Scaling Python, in short). The book should be released in a few months – fingers crossed.

And as the book is still a work-in-progress, I'll be happy to hear any remark, subject, interrogation or topic idea you might have or any particular angle you would like me to take in this book (reply in the comments section or shoot me an email). And if you'd like to get be kept updated on this book advancement, you can subscribe in the following form or from the book homepage.

The adventure of working on my previous book, The Hacker's Guide to Python, has been so tremendous and the feedback so great, that I'm looking forward releasing this new book later this year!

AsciiDoc book toolchain released

Tue, 20 Sep 2016 00:00:00 GMT

Writing a book is a big undertaking. You have to think about what you will actually write, the content, its organization, the examples you want to show, illustrations, etc.

When publishing with the help of a regular editor, your job stops there at writing – and that's already a big and hard enough task. Your editor will handle the publishing process, leaving you free of the printing task. Though they might have their own set of requirements, such as making you work with a word processing tool (think LibreOffice Writer or Microsoft Word).

When you self-publish like I did with The Hacker's Guide to Python, none of that happens. You have to deal yourself with getting your work out there, released and available in a viable format for your readership.

Most of the time, you need to render your book in different formats. You will have to make sure it works correctly on different devices and that the formatting and content disposition is correct.

I knew exactly what I wanted exactly when writing my book. I wanted to have the book published in at least PDF (for computer reading) and ePub (for e-readers). I also knew, as an Emacs user, that I did not want to spend hours writing a book in LibreOffice. It's not for me.

When I wrote about the making of The Hacker's Guide to Python, I briefly mentioned which tools I used to build the book and that I picked AsciiDoc as the input format. It makes it easy to write your book inside your favorite text editor, and AsciiDoc has plenty of output format. Customizing these formats to my liking and requirements was another challenge.

It took me hours and hours of work to have all the nitty-gritty details right. Today I am happy to announce that I can save you a few hours of work if you also want to publish a book.

I've published a new project on my GitHub called asciidoc-book-toolchain. It is the actual toolchain that I use to build The Hacker's Guide to Python. It should be easy to use and is able to render any book in HTML, PDF, PDF (printable 6"×9" format), ePub and MOBI.

So feel free to use it, hack it, pull-request it, or whatever. You don't have any good excuse to not write a book now! 😇 And if you want to self-publish a book and need some help getting started, let me know, I would be glad giving you a few hints!

The Hacker's Guide to Python 3rd edition is out

Wed, 04 May 2016 00:00:00 GMT

Exactly a year ago, I released the second edition of my book The Hacker's Guide to Python. One more time, it has been a wonderful release and I received a lot of amazing feedback from my readers all over this year.

Since then, the book has been translated into 2 languages: Korean and Chinese. A few thousands of copies has been distributed there, and I'm very glad the book has been such a success. I'm looking into getting it translated into more languages – don't hesitate to get in touch with me if you have any interesting connections in your country.

For those who still don't know about this guide, that I first released a couple of years ago, let me sum up by saying it's the Python book that I always wanted to read, never found, and finally wrote. It does not cover the basics of the language, but deals with concrete problems, best practice and some of the languages internals.

It includes content about unit testing, methods, decorators, AST, distribution, documentation, functional programming, scaling, Python 3, etc. All of that made it pretty successful! It comes with awesome 9 interviews that I realized with some of my fellow experienced Python hackers and developers!

In that 3rd edition, there is, like in each new edition, a few fixes on code, typos, etc. I guess books need a lot of time to become perfect! I also updated some of the content: things evolved a bit since I last revised the content a year ago. Finally, a new chapter about timestamps handling and timezone has made his appearance too.

If you didn't get the book yet, it's time to go check it out and use the coupon THGTP3LAUNCH to get 20 % off during the next 48 hours!

My interview in le Journal du Hacker

Thu, 17 Sep 2015 00:00:00 GMT

A few days ago, the French equivalent of Hacker News, called "Le Journal du Hacker", interviewed me about my work on OpenStack, my job at Red Hat and my self-published book The Hacker's Guide to Python. I've spent some time translating it into English so you can read it if you don't understand French! I hope you'll enjoy it.

Hi Julien, and thanks for participating in this interview for the Journal du Hacker. For our readers who don't know you, can you introduce you briefly?

You're welcome! My name is Julien, I'm 31 years old, and I live in Paris. I now have been developing free software for around fifteen years. I had the pleasure to work (among other things) on Debian, Emacs and awesome these last years, and more recently on OpenStack. Since a few months now, I work at Red Hat, as a Principal Software Engineer on OpenStack. I am in charge of doing upstream development for that cloud-computing platform, mainly around the Ceilometer, Aodh and Gnocchi projects.

Being myself a system architect, I follow your work in OpenStack since a while. It's uncommon to have the point of view of someone as implied as you are. Can you give us a summary of the state of the project, and then detail your activities in this project?

The OpenStack project has grown and changed a lot since I started 4 years ago. It started as a few projects providing the basics, like Nova (compute), Swift (object storage), Cinder (volume), Keystone (identity) or even Neutron (network) who are basis for a cloud-computing platform, and finally became composed of a lot more projects.

For a while, the inclusion of projects was the subject of a strict review from the technical committee. But since a few months, the rules have been relaxed, and we see a lot more projects connected to cloud-computing joining us.

As far as I'm concerned, I've started with a few others people the Ceilometer project in 2012, devoted to handling metrics of OpenStack platforms. Our goal is to be able to collect all the metrics and record them to analyze them later. We also have a module providing the ability to trigger actions on threshold crossing (alarm).

The project grew in a monolithic way, and in a linear way for the number of contributors, during the first two years. I was the PTL (Project Technical Leader) for a year. This leader position asks for a lot of time for bureaucratic things and people management, so I decided to leave my spot in order to be able to spend more time solving the technical challenges that Ceilometer offered.

I've started the Gnocchi project in 2014. The first stable version (1.0.0) was released a few months ago. It's a timeseries database offering a REST API and a strong ability to scale. It was a necessary development to solve the problems tied to the large amount of metrics created by a cloud-computing platform, where tens of thousands of virtual machines have to be metered as often as possible. This project works as a standalone deployment or with the rest of OpenStack.

More recently, I've started Aodh, the result of moving out the code and features of Ceilometer related to threshold action triggering (alarming). That's the logical suite to what we started with Gnocchi. It means Ceilometer is to be split into independent modules that can work together – with or without OpenStack. It seems to me that the features provided by Ceilometer, Aodh and Gnocchi can also be interesting for operators running more classical infrastructures. That's why I've pushed the projects into that direction, and also to have a more service-oriented architecture (SOA).

I'd like to stop for a moment on Ceilometer. I think that this solution was very expected, especially by the cloud-computing providers using OpenStack for billing resources sold to their customers. I remember reading a blog post where you were talking about the high-speed construction of this brick, and features that were not supposed to be there. Nowadays, with Gnocchi and Aodh, what is the quality of the brick Ceilometer and the programs it relies on?

Indeed, one of the first use-case for Ceilometer was tied to the ability to get metrics to feed a billing tool. That's now a reached goal since we have billing tools for OpenStack using Ceilometer, such as CloudKitty.

However, other use-cases appeared rapidly, such as the ability to trigger alarms. This feature was necessary, for example, to implement the auto scaling feature that Heat needed. At the time, for technical and political reasons, it was not possible to implement this feature in a new project, and the functionality ended up in Ceilometer, since it was using the metrics collected and stored by Ceilometer itself.

Though, like I said, this feature is now in its own project, Aodh. The alarm feature is used since a few cycles in production, and the Aodh project brings new features on the table. It allows to trigger threshold actions and is one of the few solutions able to work at high scale with several thousands of alarms.
It's impossible to make Nagios run with millions of instances to fetch metrics and triggers alarms. Ceilometer and Aodh can do that easily on a few tens of nodes automatically.

On the other side, Ceilometer has been for a long time painted as slow and complicated to use, because its metrics storage system was by default using MongoDB. Clearly, the data structure model picked was not optimal for what the users were doing with the data.

That's why I started Gnocchi last year, which is perfectly designed for this use case. It allows linear access time to metrics (O(1) complexity) and fast access time to the resources data via an index.

Today, with 3 projects having their own perimeter of features defined – and which can work together – Ceilometer, Aodh and Gnocchi finally erased the biggest problems and defects of the initial project.

To end with OpenStack, one last question. You're a Python developer for a long time and a fervent user of software testing and test-driven development. Several of your blogs posts point how important their usage are. Can you tell us more about the usage of tests in OpenStack, and the test prerequisites to contribute to OpenStack?

I don't know any project that is as tested on every layer as OpenStack is. At the start of the project, there was a vague test coverage, made of a few unit tests. For each release, a bunch of new features were provided, and you had to keep your fingers crossed to have them working. That's already almost unacceptable. But the big issue was that there was also a lot of regressions, et things that were working were not anymore. It was often corner cases that developers forgot about that stopped working.

Then the project decided to change its policy and started to refuse all patches – new features or bug fix – that would not implement a minimal set of unit tests, proving the patch would work. Quickly, regressions were history, and the number of bugs largely reduced months after months.

Then came the functional tests, with the Tempest project, which runs a test battery on a complete OpenStack deployment.

OpenStack now possesses a complete test infrastructure, with operators hired full-time to maintain them. The developers have to write the test, and the operators maintain an architecture based on Gerrit, Zuul, and Jenkins, which runs the test battery of each project for each patch sent.

Indeed, for each version of a patch sent, a full OpenStack is deployed into a virtual machine, and a battery of thousands of unit and functional tests is run to check that no regressions are possible.

To contribute to OpenStack, you need to know how to write a unit test – the policy on functional tests is laxer. The tools used are standard Python tools, unittest for the framework and tox to run a virtual environment (venv) and run them.

It's also possible to use DevStack to deploy an OpenStack platform on a virtual machine and run functional tests. However, since the project infrastructure also do that when a patch is submitted, it's not mandatory to do that yourself locally.

The tools and tests you write for OpenStack are written in Python, a language which is very popular today. You seem to like it more than you have to, since you wrote a book about it, The Hacker's Guide to Python, that I really enjoyed. Can you explain what brought you to Python, the main strong points you attribute to this language (quickly) and how you went from developer to author?

I stumbled upon Python by chance, around 2005. I don't remember how I hear about it, but I bought a first book to discover it and started toying with that language. At that time, I didn't find any project to contribute to or to start. My first project with Python was rebuildd for Debian in 2007, a bit later.

I like Python for its simplicity, its object orientation rather clean, its easiness to be deployed and its rich open source ecosystem. Once you get the basics, it's very easy to evolve and to use it for anything, because the ecosystem makes it easy to find libraries to solve any kind of problem.

I became an author by chance, writing blog posts from time to time about Python. I finally realized that after a few years studying Python internals (CPython), I learned a lot of things. While writing a post about
the differences between method types in Python – which is still one of the most read post on my blog – I realized that a lot of things that seemed obvious to me where not for other developers.

I wrote that initial post after thousands of hours spent doing code reviews on OpenStack. I, therefore, decided to note all the developers pain points and to write a book about that. A compilation of what years of experience taught me and taught to the other developers I decided to interview in the book.

I've been very interested by the publication of your book, for the subject itself, but also the process you chose. You self-published the book, which seems very relevant nowadays. Is that a choice from the start? Did you look for an editor? Can you tell use more about that?

I've been lucky to find out about others self-published authors, such as Nathan Barry – who even wrote a book on that subject, called Authority. That's what convinced me it was possible and gave me hints for that project.

I've started to write in August 2013, and I ran the firs interviews with other developers at that time. I started to write the table of contents and then filled the pages with what I knew and what I wanted to share. I manage to finish the book around January 2014. The proof-reading took more time than I expected, so the book was only released in March 2014. I wrote a complete report about that on my blog, where I explain the full process in detail, from writing to launching.

I did not look for editors though I've been proposed some. The idea of self-publishing really convince me, so I decided to go on my own, and I have no regret. It's true that you have to wear two hats at the same time and handle a lot more things, but with a minimal audience and some help from the Internet, anything's possible!

I've been reached by two editors since then, a Chinese and Korean one. I gave them rights to translate and publish the books in their countries, so you can buy the Chinese and Korean version of the first edition of the book out there.

Seeing how successful it was, I decided to launch a second edition in May 2015, and it's likely that a third edition will be released in 2016.

Nowadays, you work for Red Hat, a company that represents the success of using Free Software as a commercial business model. This company fascinates a lot in our community. What can you say about your employer from your point of view?

It only has been a year since I joined Red Hat (when they bought eNovance), so my experience is quite recent.

Though, Red Hat is really a special company on every level. It's hard to see from the outside how open it is, and how it works. It's really close to and it really looks like an open source project. For more details, you should read The Open Organization, a book wrote by Jim Whitehurst (CEO of Red Hat), which he just published. It describes perfectly how Red Hat works. To summarize, meritocracy and the lack of organization in silos is what makes Red Hat a strong organization and puts them as
one of the most innovative company.

In the end, I'm lucky enough to be autonomous for the project I work on with my team around OpenStack, and I can spend 100% working upstream and enhance the Python ecosystem.

My interview about software tests and Python

Mon, 11 May 2015 00:00:00 GMT

I've recently been contacted by Johannes Hubertz, who is writing a new book about Python in German called "Softwaretests mit Python" which will be published by Open Source Press, Munich this summer. His book will feature some interviews, and he was kind enough to let me write a bit about software testing. This is the interview that I gave for his book. Johannes translated to German and it will be included in Johannes' book, and I decided to publish it on my blog today. Following is the original version.

How did you come to Python?

I don't recall exactly, but around ten years ago, I saw more and more people using it and decided to take a look. Back then, I was more used to Perl. I didn't really like Perl and was not getting a good grip on its object system.

As soon as I found an idea to work on – if I remember correctly that was rebuildd – I started to code in Python, learning the language at the same time.

I liked how Python worked, and how fast I was to able to develop and learn it, so I decided to keep using it for my next projects. I ended up diving into Python core for some reasons, even doing things like briefly hacking on projects like Cython at some point, and finally ended up working on OpenStack.

OpenStack is a cloud computing platform entirely written in Python. So I've been writing Python every day since working on it.

That's what pushed me to write The Hacker's Guide to Python in 2013 and then self-publish it a year later in 2014, a book where I talk about doing smart and efficient Python.

It had a great success, has even been translated in Chinese and Korean, so I'm currently working on a second edition of the book. It has been an amazing adventure!

Zen of Python: Which line is the most important for you and why?

I like the "There should be one – and preferably only one – obvious way to do it". The opposite is probably something that scared me in languages like Perl. But having one obvious way to do it is and something I tend to like in functional languages like Lisp, which are in my humble opinion, even better at that.

For a python newbie, what are the most difficult subjects in Python?

I haven't been a newbie since a while, so it's hard for me to say. I don't think the language is hard to learn. There are some subtlety in the language itself when you deeply dive into the internals, but for beginners most of the concept are pretty straight-forward. If I had to pick, in the language basics, the most difficult thing would be around the generator objects (yield).

Nowadays I think the most difficult subject for new comers is what version of Python to use, which libraries to rely on, and how to package and distribute projects. Though things get better, fortunately.

When did you start using Test Driven Development and why?

I learned unit testing and TDD at school where teachers forced me to learn Java, and I hated it. The frameworks looked complicated, and I had the impression I was losing my time. Which I actually was, since I was writing disposable programs – that's the only thing you do at school.

Years later, when I started to write real and bigger programs (e.g. rebuildd), I quickly ended up fixing bugs… I already fixed. That recalled me about unit tests and that it may be a good idea to start using them to stop fixing the same things over and over again.

For a few years, I wrote less Python and more C code and Lua (for the awesome window manager), and I didn't use any testing. I probably lost hundreds of hours testing manually and fixing regressions – that was a good lesson. Though I had good excuses at that time – it is/was way harder to do testing in C/Lua than in Python.

Since that period, I have never stopped writing "tests". When I started to hack on OpenStack, the project was adopting a "no test? no merge!" policy due to the high number of regressions it had during the first releases.

I honestly don't think I could work on any project that does not have – at least a minimal – test coverage. It's impossible to hack efficiently on a code base that you're not able to test in just a simple command. It's also a real problem for new comers in the open source world. When there are no test, you can hack something and send a patch, and get a "you broke this" in response.

Nowadays, this kind of response sounds unacceptable to me: if there is no test, then I didn't break anything!

In the end, it's just too much frustration to work on non tested projects as I demonstrated in my study of whisper source code.

What do you think to be the most often seen pitfalls of TDD and how to avoid them best?

The biggest problems are when and at what rate writing tests.

On one hand, some people starts to write too precise tests way too soon. Doing that slows you down, especially when you are prototyping some idea or concept you just had. That does not mean that you should not do test at all, but you should probably start with a light coverage, until you are pretty sure that you're not going to rip every thing and start over. On the other hand, some people postpone writing tests for ever, and end up with no test all or a too thin layer of test. Which makes the project with a pretty low coverage.

Basically, your test coverage should reflect the state of your project. If it's just starting, you should build a thin layer of test so you can hack it on it easily and remodel it if needed. The more your project grow, the more you should make it sold and lay more tests.

Having too detailed tests is painful to make the project evolve at the start. Having not enough in a big project makes it painful to maintain it.

Do you think, TDD fits and scales well for the big projects like OpenStack?

Not only I think it fits and scales well, but I also think it's just impossible to not use TDD in such big projects.

When unit and functional tests coverage was weak in OpenStack – at its beginning – it was just impossible to fix a bug or write a new feature without breaking a lot of things without even noticing. We would release version N, and a ton of old bugs present in N-2 – but fixed in N-1 – were reopened.

For big projects, with a lot of different use cases, configuration options, etc, you need belt and braces. You cannot throw code in a repository thinking it's going to work ever, and you can't afford to test everything manually at each commit. That's just insane.

The Hacker's Guide to Python, 2nd edition!

Mon, 04 May 2015 00:00:00 GMT

A year passed since the first release of The Hacker's Guide to Python in March 2014. A few hundreds copies have been distributed so far, and the feedback is wonderful!

I already wrote extensively about the making of that book last year, and I cannot emphasize enough how this adventure has been amazing so far. That's why I decided a few months ago to update the guide and add some new content.

So let's talk about what's new in this second edition of the book!

First, I obviously fixed a few things. I had some reports about small mistakes and typos which I applied as I received them. Not a lot fortunately, but it's still better to have fewer errors in a book, right?

Then, I updated some of the content. Things changed since I wrote the first chapters of that guide 18 months ago. Therefore I had to rewrite some of the sections and take into account new software or libraries that were released.

At last, I decided to enhance the book with one more interview. I've requested my fellow OpenStack developer Joshua Harlow, who is leading a few interesting Python projects, to join the long list of interviewees in the book. I hope you'll enjoy it!

If you didn't get the book yet, go check it out and use the coupon THGTP2LAUNCH to get 20% off during the next 48 hours!

Making of The Hacker's Guide to Python

Wed, 07 May 2014 00:00:00 GMT

As promised, today I would like to write a bit about the making of The Hacker's Guide to Python. It has been a very interesting experimentation, and I think it is worth sharing it with you.

The inspiration

All started out at the beginning of August 2013. I was spending my summer, as the rest of the year, hacking on OpenStack.

As years passed, I got more and more deeply involved in the various tools that we either built or contributed to within the OpenStack community. And I somehow got the feeling that my experience with Python, the way we used it inside OpenStack and other applications during these last years was worth sharing. Worth writing something bigger than a few blog posts.

The OpenStack project is doing code reviews, and therefore so did I for almost two years. That inspired a lot of topics, like the definitive guide to method decorators that I wrote at the time I started the hacker's guide. Stumbling upon the same mistakes or misunderstanding over and over is, somehow, inspiring.

I also stumbled upon Nathan Barry's blog and book Authority which were very helpful to get started and some sort of guideline.

All of that brought me enough ideas to start writing a book about Python software development for people already familiar with the language.

The writing

The first thing I started to do is to list all the topics I wanted to write about. The list turned out to have subjects that had no direct interest for a practical guide. For example, on one hand, very few developers know in details how metaclasses work, but on the other hand, I never had to write a metaclass during these last years. That's the kind of subject I decided not to write about, dropped all subjects that I felt were not going to help my reader to be more productive. Even if they could be technically interesting.

Then, I gathered all problems I saw during the code reviews I did during these last two years. Some of them I only recalled in the days following the beginning of that project. But I kept adding them to the table of contents, reorganizing stuff as needed.

After a couple of weeks, I had a pretty good overview of the contents that there I will write about. All I had to do was to fill in the blank (that sounds so simple now).

The entire writing of the took hundred hours spread from August to November, during my spare time. I had to stop all my other side projects for that.

The interviews

While writing the book, I tried to parallelize every thing I could. That included asking people for interviews to be included in the book. I already had a pretty good list of the people I wanted to feature in the book, so I took some time as soon as possible to ask them, and send them detailed questions.

I discovered two categories of interviewees. Some of them were very fast to answer (≤ 1 week), and others were much, much slower. A couple of them even set up Git repositories to answer the questions, because that probably looked like an entire project to them. :-) So I had to not lose sight and kindly ask from time to time if everything was alright, and at some point started to kindly set some deadline.

In the end, the quality of the answers was awesome, and I like to think that was because I picked the right people!

The proof-reading

Once the book was finished, I somehow needed to have people proof-reading it. This was probably the hardest part of this experiment. I needed two different types of reviews: technical reviews, to check that the content was correct and interesting, and language review. That one is even more important since English is not my native language.

Finding technical reviewers seemed easy at first, as I had ton of contacts that I identified as being able to review the book. I started by asking a few people if they would be comfortable reading a simple chapter and giving me feedbacks. I started to do that in September: having the writing and the reviews done in parallel was important to me in order to minimize latency and the book's release delay.

All people I contacted answered positively that they would be interested in doing a technical review of a chapter. So I started to send chapters to them. But in the end, only 20% replied back. And even after that, a large portion stopped reviewing after a couple of chapters.

Don't get me wrong: you can't be mad at people not wanting to spend their spare time in book edition like you do.

However, from the few people that gave their time to review a few chapters, I got tremendous feedback, at all level. That's something that was very important and that helped a lot getting confident. Writing a book alone for months without having anyone taking a look upon your shoulder can make you doubt that you are creating something worth it.

As far as English proof-reading, I went ahead and used ODesk to recruit a professional proof-reader. I looked for people with the right skills: a good English level (being a native English speaker at least), be able to understand what the book was about, and being able to work with correct delays. I had mixed results from the people I hired, but I guess that's normal. The only error I made was not to parallelize those reviews enough, so I probably lost a couple of months on
that.

The toolchain

While writing the book, I did a few breaks to build a toolchain. What I call a toolchain is set of tools used to render the final PDF, EPUB and MOBI files of the guide.

After some research, I decided to settle on AsciiDoc, using the DocBook output, which is then being transformed to LaTeX, and then to PDF, or either to EPUB directly. I rely on Calibre to convert the EPUB file to MOBI. It took me a few hours to do what I wanted, using some magic LaTeX tricks to have a proper render, but it was worth it and I'm particularly happy with the result.

For the cover design, I asked my talented friend Nicolas to do something for me, and he designed the wonderful cover and its little snake!

The publishing

Publishing in an interesting topic people kept asking me about. This is what I had to answer a few dozens of time:

"Who is your editor?"
"Me."

I never had any plan for asking an editor to publish this book. Nowadays, asking an editor to publish a book feels to me like asking a major company to publish a CD. It feels awkward.

However, don't get me wrong: there can be a few upsides of having an editor. They will find reviewers and review your book for you. Having the book review handled for you is probably a very good thing, considering how it was hard to me to get that in place. It can be especially important for a technical book.

Also, your book may end up in brick and mortar stores and be part of a collection, both improving visibility. That may improve your book's selling, though the editor and all the intermediaries are going to keep the largest amount of the money anyway.

"Oh, you will publish it yourself, great. So you will print it and sell it to people?"
"Not really."

I've heard good stories about people using Gumroad to sell electronic contents, so after looking for competitors in that market, I picked them. I also had the idea to sell the book with Bitcoins, so I settled on Coinbase, because they have a nice API to do that.

Setting up everything was quite straight-forward, especially with Gumroad. It only took me a few hours to do so. Writing the Coinbase application took a few hours too.

"Oh, you will sell it only as an ebook? That's too bad. You need a paper version. Many people will want a paper version."

My initial plan was to only sell online an electronic version. On the other hand, since I kept hearing that a printed version should exist, I decided to give it a try. I chose to work with Lulu because I knew people using it, and it was pretty simple to set up.

The launch

Once I had everything ready, I built the selling page and connected everything between Mailchimp, Gumroad, Coinbase, Google Analytics, etc.

Writing the launch email was really exciting. I used Mailchimp feature to send the launch mail in several batches, just to have some margin in case of a sudden last minute problem. But everything went fine. Hurrah!

I distributed around 200 copies of the ebook in the first 48 hours, for about $5000. That covered all the cost I had from the writing the book, and even more, so I was already pretty happy with the launch.

Retrospective

In retrospect, something that I didn't do the best way possible is probably to build a solid mailing list of people interested, and to build an important anticipation and incentive to buy the book at launch date. My mailing list counted around 1500 people subscribed because they were interested in the launch of the book subscribed; in the end, probably only 10-15% of them bought the book during the launch, which is probably a bit lower than what I could expect.

But more than a month later, I distributed in total almost 500 copies of the book (including physical units) for more than $10000, so I tend to think that this was a success. I still sell a few copies of the book each weeks, but the number are small compared to the launch.

I sold less than 10 copies of the ebook using Bitcoins, and I admit I'm a bit disappointed and surprised about that.

Physical copies represent 10% of the book distribution. It's probably a lot lower than most people that pushed me to do it thought it would be. But it is still higher of what I thought it would be. So I still would advise to have a paperback version of your book. At least because it's nice to have it
in your library.

I only got positive feedbacks, a few typo notices, and absolutely no refund demand, which I really find amazing.

The good news is also that I've been contacted with a couple of Korean and Chinese editors to get the book translated and published in those countries. If everything goes well, the book should be translated in the upcoming months and be available on these markets in 2015!

If you didn't get a copy, it's still time to do so!

The Hacker's Guide to Python released!

Tue, 25 Mar 2014 00:00:00 GMT

And done! It took me just 8 months to do this entire book project around Python. From the first day I started writing to today, where I finally publish and sell – almost entirely – myself this book. I'm really proud of what I've achieved so far, as this was something totally new to me.

Doing all of that has been a great adventure, and I'll promise I'll write something about that later on. A making of.

For now, you can enjoy reading the book and learn a bit more about Python. I really hope it'll help you bring your Python-fu to a new level, and help you build great projects!

Go check it out, and since this is first day of sale, enjoy 20% off by using the offer code THGTP20.

Announcing The Hacker's Guide to Python

Tue, 03 Sep 2013 00:00:00 GMT

I've been hacking on Python for a lot of years now, on various project. For the last two years, I've been heavily involved in OpenStack, which makes an heavy usage of Python.

Once you start working with a hundred of hackers, on several software and libraries representing more than half a million source lines of Python, things change. The scalability, testing and deployment problems inherent to a cloud platform meddle with everything in designing components.

During these two years working on OpenStack development, I've learned a lot on Python from astounding Python hackers. From general architecture and design principles to various tips and tricks of the language.

It seemed to me like a good opportunity to share what I learnt doing so with others so you can benefit from it in other projects. I've started working a book, entitled "The Hacker's Guide to Python", where I will try to share what I learnt while working with Python.

The book is still a work in progress at this stage, but if you'd like to get in touch and keep updated on its advancement, you can subscribe in the following form or from the book homepage.