databases — jd:/dev/blog

The Art of PostgreSQL is out!

Wed, 28 Aug 2019 00:00:00 GMT

If you remember well, a couple of years ago, I wrote about Mastering PostgreSQL, a fantastic book written by my friend Dimitri Fontaine.

Dimitri is a long-time PostgreSQL core developer — for example, he wrote the extension support in PostgreSQL — no less. He is featured in my book Serious Python, where he advises on using databases and ORM in Python.

Today, Dimitri comes back with the new version of this book, named The Art of PostgreSQL.

I love the motto of this book: Turn Thousands of Lines of Code into Simple Queries. I have spent all my career working with code that talks to databases, and I can't count the number of times where I've seen people write lengthy, slow code in their pet language rather than a single well-thought SQL query which would do a better job.

This is exactly what this book is about.

That's why it's my favorite SQL book. I learned so many things from it. In many cases, I've been able to divide by 10 the size of the code I had to write in Python to implement a feature. All I had to do is to browse the book to discover the right PostgreSQL feature and write a single SQL query. The right query that does the job for me_._

Less code, fewer bugs, more happiness!

The book also features interviews with great PostgreSQL users and developers — hey, no wonder where Dimitri got this idea, right? ;-)

I loved those interviews. What's better than reading Kris Jenkins explaining how Clojure and PostgreSQL play nice together, or Markus Winand (from the famous use-the-index-luke.com) talking about the relationship developers have with their database. :-)

No need to say that you should get your hands on this right now. Dimitri just made a launch offer where he offers a 15% discount on the book until the end of this month! You can also read the free chapter to get an idea of what you'll get.

Last thing: it's DRM-free and money-back guaranteed. You can get this book with your eyes closed.

Why You Should Care That Your SQL DDL is Transactional

Mon, 07 Jan 2019 00:00:00 GMT

I don't write a lot about database management. How come? I'm a software engineer, and like many of my peers, I leverage databases to store data. I should talk more about this! What made me write this today is that I've discovered that many of my peers wouldn't be able to understand the title of this post.

However, I can tell you that once you've finished reading this, you'll thank me!

DDL?

To understand what this post is about, let's start with DDL. DDL is the abbreviation of Data Definition Language. In summary, a DDL is a language that allows defining your data structure. A famous one is the SQL DDL — and that's the one I talk about here.

I'm sure you already used it if you created a relational database with CREATE TABLE foo (id INTEGER). That is a DDL statement.

In SQL, there's a lot of DDL operations you can do, such as creating a table, renaming a table, creating or removing a column, converting a column to a new type, etc.

Those DDL statements are commonly used in two cases:

When creating your database' tables for the first time. You issue a bunch of CREATE TABLE statements, and your database is ready to be used.
When updating your database by adding, removing or modifying tables or columns. This is typically done when upgrading your application to a new version.

The fact that our DDL is transactional or not in option 1. has often little impact in practice. It's can still be useful, where for example you could get an error because the disk is full — having the ability to roll back in this case can be a life saviour.

In our case here, we'll talk about why you need a transactional DDL when upgrading your database.

Transactional You Said?

What transactional means here? It means that we can issue those DDL statements inside a transaction.

Wait, what's a transaction? To make it simple, in a database, a transaction is a group of operations that are treated as a single coherent operation, independently of other transactions. The final operation has to be atomic, consistent, isolated and durable — therefore that ACID property you keep reading about while always wondering what it meant. The operations composing the transaction are either entirely executed, or not at all.

In our case, having the DDL being transactional means one simple thing: the ability to execute several operations (e.g., several ALTER TABLE) in a single operation, that can be either committed or rolled back.

Let's use an example. Here's a table ingredients with a name column created with:

CREATE TABLE ingredients (
  name text NOT NULL
);

In this table, there is a list of ingredients in the form of water 20 mL, flour 300 g, etc.

Now, we're upgrading our application, and we want to handle the quantity of ingredients in their columns to make it easier to query the data. Let's say we're going to handle quantity and quantity units for our ingredients. We need to add two new columns to our table schema, quantity and unit:

ALTER TABLE ingredients ADD COLUMN quantity integer NOT NULL;
ALTER TABLE ingredients ADD COLUMN unit text NOT NULL;

We also need to convert the name by splitting it into <name> <quantity> <unit> and insert this into the new columns. We can do this like that:

UPDATE ingredients SET name=split_part(name, ' ', 1), quantity=split_part(name, ' ', 2)::int, unit=split_part(name, ' ', 3);

In this example, I'm using the split_part operator from PostgreSQL to split the string.

With the UPDATE statement, the name column containing flour 300 grams now contains flour, and the columns quantity and unit respectively stores 300 and grams.

When we run our upgrade procedure consisting of those two ALTER TABLE and one UPDATE, we got our final table like this:

## SELECT * FROM ingredients;
 name  │ quantity │ unit
───────┼──────────┼──────
 flour │      300 │ g
(1 row)

Exactly what we want.

Ok, So What?

In the previous example, everything worked fine. Our 300 grams of flour string is split, converted and stored into the three different columns. However, let's think about what happens if the conversion fails because our ingredient name is foobar:

## ALTER TABLE ingredients ADD COLUMN quantity integer;
ALTER TABLE
## ALTER TABLE ingredients ADD COLUMN unit text;
ALTER TABLE
## UPDATE ingredients SET name=split_part(name, ' ', 1), quantity=split_part(name, ' ', 2)::int, unit=split_part(name, ' ', 3);
ERROR:  invalid input syntax for integer: ""

Right, so in this case our update failed because it's impossible to convert an empty string to an integer.

We're going to fix this piece of data in our database (manually or automatically, whatever) to make it work, changing foobar to something like foobar 1 kg.

Then, when we rerun the upgrade script, this is what happens:

## ALTER TABLE ingredients ADD COLUMN quantity integer NOT NULL;
ERROR:  column "quantity" of relation "ingredients" already exists

The upgrade script failed earlier — not in the UPDATE statement. It has a good reason to fail: the column quantity already exists.

Why is that? Well, when we run the upgrade procedure the first time, we did not run it inside a transaction. Every DDL statement was committed right after its execution. Therefore, the current state of our database is half-migrated: we have the new schema installed, but not the data migrated.

This sucks. This should not happen. Ever.

Why?

Some database systems (e.g., MySQL) do not support DDL running in a transaction, so you have no choice than running the three operations (ALTER, ALTER and then UPDATE) as three distinct operations: if any of those fails, there's no way to recover and get back to the initial state.

If you're using a database that supports running DDL statements inside a transaction (e.g., PostgreSQL), we can run your upgrade script like this:

postgres=# BEGIN;
postgres=# ALTER TABLE ingredients ADD COLUMN quantity integer;
ALTER TABLE
postgres=# ALTER TABLE ingredients ADD COLUMN unit text;
ALTER TABLE
postgres=# UPDATE ingredients SET name=split_part(name, ' ', 1), quantity=split_part(name, ' ', 2)::int, unit=split_part(name, ' ', 3);
ERROR:  invalid input syntax for integer: ""
postgres=# ROLLBACK;
ROLLBACK

Since the transaction failed, we ended up doing a ROLLBACK. When checking the state of the database, we can see the state did not change:

## \d ingredients;
           Table "public.ingredients"
 Column │ Type │ Collation │ Nullable │ Default
────────┼──────┼───────────┼──────────┼─────────
 name   │ text │           │ not null │

Therefore, it's possible to fix our database content and rerun the migration procedure without being in a half-migrated state.

A Database That Lies

When you're giving data to a database, you're trusting it. It'd be awful if it were lying to you, right? Check this out:

mysql> CREATE TABLE ingredients (name text NOT NULL);
Query OK, 0 rows affected (0.03 sec)

mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER TABLE ingredients ADD COLUMN quantity integer;
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE ingredients ADD COLUMN unit text;
Query OK, 0 rows affected (0.01 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ROLLBACK;
Query OK, 0 rows affected (0.00 sec)

mysql> DESC ingredients;
+----------+---------+------+-----+---------+-------+
| Field    | Type    | Null | Key | Default | Extra |
+----------+---------+------+-----+---------+-------+
| name     | text    | NO   |     | NULL    |       |
| quantity | int(11) | YES  |     | NULL    |       |
| unit     | text    | YES  |     | NULL    |       |
+----------+---------+------+-----+---------+-------+
3 rows in set (0.00 sec)

In the output above, you can see that we issued two DDL statements inside a transaction and that we then rolled back that transaction. MySQL did not output any error at any time, making us think that it did not alter our table. However, when checking the schema of the database, we can see that nothing has been rolled back. Not only MySQL does not support transactional DDL, but it also fails you entirely and lie about what it's doing.

How Important is That?

Transactional DDL is a feature that is often ignored by software engineers, whereas it's a key feature for managing your database life cycle.

I'm writing this post today because I've been hit by this multiple times over the last years. OpenStack made the choice years ago to go with MySQL, and in consequences, every database upgrade script that fails in the middle of the procedure leave the database is an inconsistence state. In that case, it means that either:

The operator have to determine where the upgrade script stopped, roll back the upgrade by itself, fix the failure, and rerun the upgrade procedure.
The developer must anticipate every case of potential upgrade failure, write a roll back procedure for each of this case by and test every one of those cases.
Use a database system that handles transactional DDL.

No need to tell you that option 3. is the best, option 2. is barely possible to implement and option 1. is what reality looks like. In Gnocchi, we picked option 3. by recommending operators to use PostgreSQL.

Next time you use a database, think carefully about what your upgrade procedure will look like!

Mastering PostgreSQL

Tue, 07 Nov 2017 00:00:00 GMT

A few months ago, my friend Dimitri Fontaine and I discussed writing books and sharing our knowledge. If you do not know Dimitri yet, he is an old-time PostgreSQL Major Contributor – meaning he writes code for the PostgreSQL software itself!

I interviewed Dimitri a few years ago in The Hacker's Guide to Python, where he shared his insight about writing proper Python application code with relational database management system.

All of this gave Dimitri the idea of writing his own book about PostgreSQL. And he released his book this week! If like me, you can't wait to read it, just scroll down below and grab a package with a 15% discount.

To celebrate the event, I went ahead and decided to ask him a few questions.

Hey Dimitri! So what made you start writing this book in the first place?

Dimitri: As a PostgreSQL consultant, I've met with many developers for whom SQL just didn't click. They then tend to consider SQL much as they would consider HTML: some string you need to build dynamically then send over to an external part of the system, either the browser or the database server.

As soon as you start on this path, SQL is more and more of a problem in your daily life and developer workflow. It doesn't integrate well with the usual testing and continuous integration tools, not to mention it's hard to review (as in code review) and hard to maintain.

At the end of the day, when using a relational database system, you have to
know your SQL.

As a developer, you need to be fluent in SQL and master window functions, common table expressions, recursive queries, time zone handling and advanced string and regexp processing functions, transaction behaviors and also how to build a query result set in JSON. And so much more.

For most developers, it's a daunting task. Just too much to learn when they have so many other things to take care of. So many lines of code to write to implement that new product idea. Well, as Dijkstra put it, lines of code are “spent” on writing a new feature. When you master SQL, you spend much less of those lines of code.

So I wrote Mastering PostgreSQL in Application Development to teach SQL to developers. Focusing on real use cases and authentic data set, so that it's easier to grasp all those advanced features. The book also addresses the tooling you need to integrate SQL as another programming language with a decent worfklow, from code review to unit testing, including regression testing and production debugging.

Who should read this book? What are the prerequisites to get the most of it?

Dimitri: The pre-requisites are quite easy to reach.

If you ever deployed an application that embeds SQL queries and talks to a
database server, you're in the target audience, the book is for you.

If you've never used PostgreSQL before, reading Mastering PostgreSQL in Application Development may convince you to have a look at that awesome piece of technology. My bet is on you switching to PostgreSQL and finding it much better at helping you in your daily work and challenges.

Even if you've been using MySQL all your life, you will learn about standard SQL features and how to use them in a way that applies to more than just PostgreSQL, so the book is going to help you in your daily life.

What's your next adventure now that this book is out?

Dimitri: There are more things that I want to do that a lifetime allows, and I am in the process of choosing what is going to be my next adventure. I feel so lucky to be able to have that problem to solve… and it still isn't the easiest one for me.

What I can tell you is that I have much more PostgreSQL knowledge to share after having been using, promoting, and contributing to this database server technology for about 20 years now. So if that first book sells well, I will get back to filling empty pages and deliver more contents to help developers making the best of SQL, to help developers on their road to Mastering PostgreSQL!

Thanks Dimitri!

I've just read the book and found it fantastic. It contains tons of tips on how to use PostgreSQL correctly, and I discovered SQL features I had no clue about. The book uses real data that you can fetch and play with. It provides the data and a Docker container with everything included so you can edit the query yourself and try it out. There is no better way to learn things than to play with the examples that are included, in just a few clicks!

The book also features a few interviews with SQL experts from the PostgreSQL community and from the development community, which gives great insight about how to use the software.

Dimitri is offering 15% off for my readers during the next 48 hours for any of the edition of the book. Just use the PYTHON-LOVES-POSTGRESQL coupon code in any of the following package:

.product img { max-width: 100%; margin-top: 10px; } .col-sm-1 { width: 8.33333333%; float: left; } .col-sm-3 { width: 25%; float: left; } .col-sm-9 { width: 75%; float: left; } .col-sm-11 { width: 91.66666667%; float: left; /* should be on the div not here but well.. */ padding-left: 30px; padding-bottom: 20px; } .row { clear: both; } .btn.btn-default { background-color: #e6940e; color: #FFF; line-height: 46px; height: 50px; font-size: 19px; cursor: pointer; text-align: center; border-radius: 5px; border: 3px solid #e6940e; vertical-align: middle; display: inline-block; padding: 0 30px; position: relative; outline: none !important; transition: color .3s ease,background .3s ease,border-color .3s ease,opacity .3s ease; box-shadow: none; }

Mastering PostgreSQL – Enterprise Edition

$179 $152
with coupon code PYTHON-LOVES-POSTGRESQL

The Enterprise Edition includes:

The book in PDF, EPUB and MOBI formats.

Interviews from industry veterans who began building web application in the previous century. They've been there and have opinions to share about how to approach SQL.

The PostgreSQL database dump that you need to run the queries against, with a script to restore it easily. The database includes all the 12 datasets used in the book.

A Docker container image of an already loaded PostgreSQL database with the whole 12 datasets in 56 tables, and the 265 SQL queries each in their own .sql file and a Web-based application for easily running and editing the SQL queries.

Licence for you to share the book and the Docker set-up with up to 50 people, including you. That's everything you need for your whole team to master SQL!

Buy the Enterprise Edition

Mastering PostgreSQL – Full Edition

$89 $75
with coupon code PYTHON-LOVES-POSTGRESQL

The Full Edition includes:

The book in PDF, EPUB and MOBI formats.

Interviews from industry veterans who began building web application in the previous century. They've been there and have opinions to share about how to approach SQL.

The PostgreSQL database dump that you need to run the queries against, with a script to restore it easily. The database includes all the 12 datasets used in the book.

Buy the Full Edition

Mastering PostgreSQL

$39 $33
with coupon code PYTHON-LOVES-POSTGRESQL

The Standard Edition includes:

The book in PDF, EPUB and MOBI formats.

Interviews from industry veterans who began building web application in the previous century. They’ve been there and have opinions to share about how to approach SQL.

Buy the book

If you have any question, feel free to reach Dimitri directly and he will be happy to reply. Or write in the comment section below!

And don't worry: if the book is not what you expect it to be and has no value to you, then just say so and Dimitri will refund you, no questions asked.

From decimal to timestamp with MySQL

Thu, 08 Sep 2016 00:00:00 GMT

When working with timestamps, one question that often arises is the precision of those timestamps. Most software is good enough with a precision up to the second, and that's easy. But in some cases, like working on metering, a finer precision is required.

I don't know exactly why, and it makes me suffer every day, but OpenStack is really tied to MySQL (and its clones). It hurts because MySQL is a very poor solution if you want to leverage your database to actually solve problems. But that's how life is, unfair. And in the context of the projects I work on, that boils down to that we can't afford to not support MySQL.

So here we are, needing to work with MySQL and at the same time requiring timestamp with a finer precision than just seconds. And guess what: MySQL did not support that until 2011.

No microseconds in MySQL? No problem: DECIMAL!

MySQL 5.6.4 (released in 2011), a beta version of MySQL 5.6 (hello MySQL, ever heard of Semantic Versioning?), brought microsecond precision to timestamps. But the first stable version supporting that, MySQL 5.6.10, was only released in 2013. So for a long time, there was a problem without any solution.

The obvious workaround, in this case, is to reassess your choices in technologies, discover that PostgreSQL supports microsecond precision for at least a decade and problem solved.

This is not what happened in our case, and in order to support MySQL, one had to find a workaround. And so did they in our Ceilometer project, using a DECIMAL type instead of DATETIME.

The DECIMAL type takes 2 arguments: the total number of digits you need to store, and how many in that total will be used for the fractional part. Knowing that the internal storage of MySQL uses 1 byte for 2 digits, 2 bytes for 4 digits, 3 bytes for 6 digits and 4 bytes for 9 digits, and that each part is stored independently, in order to maximize your storage space, you want to pick a number of digits that fits that correctly.

This is why Ceilometer picked 14 for the integer part (9 digits on 4 bytes and 5 digits on 3 bytes) and 6 for the decimal part (3 bytes).

Wait. It's stupid because:

DECIMAL(20, 6) implies that you uses 14 digits for the integer part, which using epoch as a reference makes you able to encode timestamp (10^14) - 1 which is year 3170843. I am certain Ceilometer won't last that far.
14 digits is 9 + 5 digits in MySQL which is 7 bytes, the same size that is used for 9 + 6 digits. So if you could have DECIMAL(21, 6) for the same storage space (and go up to year 31690708 which is a nice bonus, right?)

Well, I guess the original author of the patch did not read the documentation entirely (DECIMAL(20, 6) being on the MySQL documentation page as an example, I imagine it just has been copy-pasted blindly?).

The best choice for this use case would have been DECIMAL(17, 6) which would allow storing 11 digits for integer (5 bytes), supporting timestamp up to (2^11)-1 (year 5138), and 6 digits for decimal part (3 bytes), using only 8 bytes in total per timestamp.

Nonetheless, this workaround has been implemented using a SQLAlchemy custom type and works as expected:

class PreciseTimestamp(sqlalchemy.types.TypeDecorator):
    """Represents a timestamp precise to the microsecond."""

    impl = sqlalchemy.DateTime

    def load_dialect_impl(self, dialect):
        if dialect.name == 'mysql':
            return sqlalchemy.dialect.type_descriptor(
                sqlalchemy.types.DECIMAL(precision=20,
                                         scale=6,
                                         asdecimal=True))
        return sqlalchemy.dialect.type_descriptor(self.impl)

Microseconds in MySQL? Damn, migration!

As I said, MySQL 5.6.4 brought microseconds precision to the table (pun intended). Therefore, it's a great time to migrate away from this hackish format to the brand new one.

First, be aware that the default DATETIME type has no microseconds precision: you have to specify how many digits you want as an argument.
To support microseconds, you should therefore use DATETIME(6).

If we were using a great RDBMS, let's say, hum, PostgreSQL, we could do that
very easily, see:

postgres=# CREATE TABLE foo (mytime decimal);
CREATE TABLE
postgres=# \d foo
      Table "public.foo"
 Column │  Type   │ Modifiers
────────┼─────────┼───────────
 mytime │ numeric │
postgres=# INSERT INTO foo (mytime) VALUES (1473254401.234);
INSERT 0 1
postgres=# ALTER TABLE foo ALTER COLUMN mytime SET DATA TYPE timestamp with time zone USING to_timestamp(mytime);
ALTER TABLE
postgres=# \d foo
              Table "public.foo"
 Column │           Type           │ Modifiers
────────┼──────────────────────────┼───────────
 mytime │ timestamp with time zone │

postgres=# select * from foo;
           mytime
────────────────────────────
 2016-09-07 13:20:01.234+00
(1 row)

And since this is a pretty common use case, it's even an example in the PostgreSQL documentation. The version from the documentation uses a calculation based on epoch, whereas my example here leverages the to_timestamp() function. That's my personal touch.

Obviously, doing this conversion in a single line is not possible with MySQL: it does not implement the USING keyword on ALTER TABLE … ALTER COLUMN. So what's the solution gonna be? Well, it's a 4 steps job:

Create a new column of type DATETIME(6)
Copy data from the old column to the new column, converting them to the new format
Delete the old column
Rename the new column to the old column name.

But I know what you're thinking: there are 4 steps, but that's not a problem, we'll just use a transaction and embed these operations inside.

MySQL does not support transactions on data definition language (DDL).
So if any of those steps fails, you'll be unable rollback steps 1, 3 and 4. Who knew that using MySQL was like living on the edge, right?

Doing this in Python with our friend Alembic

I like Alembic. It's a Python library based on SQLAlchemy that handles schema migration for your favorite RDBMS.

Once you created a new alembic migration script using alembic revision, it's time to edit it and write something along those lines:

from alembic import op
import sqlalchemy as sa
from sqlalchemy.sql import func

class Timestamp(sa.types.TypeDecorator):
    """Represents a timestamp precise to the microsecond."""

    impl = sqlalchemy.DateTime

    def load_dialect_impl(self, dialect):
        if dialect.name == 'mysql':
            return dialect.type_descriptor(mysql.DATETIME(fsp=6))
        return self.impl

def upgrade():
    bind = op.get_bind()
    if bind and bind.engine.name == "mysql":
        existing_type = sa.types.DECIMAL(
            precision=20, scale=6, asdecimal=True)
        existing_col = sa.Column("mytime", existing_type, nullable=False)
        temp_col = sa.Column("mytime_ts", Timestamp(), nullable=False)
        # Step 1: ALTER TABLE mytable ADD COLUMN mytime_ts DATETIME(6)
        op.add_column("mytable", temp_col)
        t = sa.sql.table("mytable", existing_col, temp_col)
        # Step 2: UPDATE mytable SET mytime_ts=from_unixtime(mytime)
        op.execute(t.update().values(mytime_ts=func.from_unixtime(existing_col)}))
        # Step 3: ALTER TABLE mytable DROP COLUMN mytime
        op.drop_column("mytable", "mytime")
        # Step 4: ALTER TABLE mytable CHANGE mytime_ts mytime
        # Note: MySQL needs to have all the old/new information to just rename a column…
        op.alter_column("mytable",
                        "mytime_ts",
                        nullable=False,
                        type_=Timestamp(),
                        existing_nullable=False,
                        existing_type=existing_type,
                        new_column_name="mytime")

In MySQL, the function to convert a float to a UNIX timestamp is from_unixtime(), so the script leverages it to convert the data. As said, you'll notice we don't bother using any kind of transaction, so if anything goes wrong, there's no rollback, and it won't be possible to re-run the migration without a manual intervention.

TimestampUTC is a custom class that implements sqlalchemy.DateTime using a DATETIME(6) type for MySQL, and a regular sqlalchemy.DateTime type for other back-ends. It is used by the rest of the code (e.g. ORM model) but I've pasted it in this example for a better understanding.

Once written, you can easily test your migration using pifpaf to run a temporary database:

$ pifpaf run mysql $SHELL
$ alembic -c alembic/alembic.ini upgrade 1c98ac614015 # upgrade to the initial revision
$ mysql -S $PIFPAF_MYSQL_SOCKET pifpaf
mysql> INSERT INTO mytable (mytime) VALUES (1325419200.213000);
Query OK, 1 row affected (0.00 sec)

mysql> SELECT * FROM mytable;
+-------------------+
| mytime            |
+-------------------+
| 1325419200.213000 |
+-------------------+
1 row in set (0.00 sec)

$ alembic -c alembic/alembic.ini upgrade head

$ mysql -S $PIFPAF_MYSQL_SOCKET pifpaf
mysql> SELECT * FROM mytable;
+----------------------------+
| mytime                     |
+----------------------------+
| 2012-01-01 13:00:00.213000 |
+----------------------------+
1 row in set (0.00 sec)

And voilà, we just migrated unsafely our data to a new fancy format. Thank you Alembic for solving a problem we would not have without MySQL. 😊

Overriding cl-json object encoding

Fri, 11 Jan 2013 00:00:00 GMT

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that.

Identifying the problem

CL-JSON & CLOS

CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:

(defclass kitten ()
  ((tail :initarg :tail)))

(json:encode-json-to-string (make-instance 'kitten :tail 'black))

will produce:

{"tail":"black"}

Still using CL-JSON, we can also decode the JSON object to a CLOS object:

(slot-value
 (json:with-decoder-simple-clos-semantics
   (json:decode-json-from-string "{\"tail\":\"black\"}"))
 :tail)

That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article.

Postmodern

Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects.

With this, we can easily store our kitten into a table.

(defclass kitten ()
  ((tail :initarg :tail))
  (:metaclass postmodern:dao-class))

If we try to encode this to JSON, it will produce the exact same result seen previously.

The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol.

So this code:

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))

(postmodern:deftable kitten
  (postmodern:!dao-def))

(postmodern:connect-toplevel …)

(postmodern:create-table 'kitten)

(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

will return:

"{"tail":"null"}"

Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string.

This is not at all what we want here!

Overriding encode-json

CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:

(defmethod encode-json ((o standard-object)
                        &optional (stream *json-output*))
  "Write the JSON representation (Object) of the CLOS object O to
STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (stream-object-member-encoder stream) o)))

All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))

(export 'kitten)

;; Switch package just to define the new method
(in-package :json)
(defmethod encode-json ((o cl-user:kitten)
                        &optional (stream json:*json-output*))
  "Write the JSON representation (Object) of the postmodern DAO CLOS object
O to STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (lambda (key value)
                 (as-object-member (key stream)
                   (encode-json (if (eq value :null) nil value) stream)))
               o)))

;; Go back into our package
(in-package :cl-user)

(postmodern:deftable kitten
  (postmodern:!dao-def))

(postmodern:connect-toplevel …)

(postmodern:create-table 'kitten)

(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil.

Now if we try to encode another kitten, we'll get:

{"tail":null}

which is far better for our JavaScript data consumers!

In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation.
The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp.

The ability to teach any library about how it should handle your class just by defining a new method is really handy!