lisp — jd:/dev/blog

FOSDEM 2016, recap

Sat, 06 Feb 2016 00:00:00 GMT

Last week-end, I was in Brussels, Belgium for the FOSDEM, one of the greatest open source developer conference. I was not sure to go there this year (I already skipped it in 2015), but it turned out I was requested to do a talk in the shared Lua & GNU Guile devroom.

As a long time Lua user and developer, and a follower of GNU Guile for several years, the organizer asked me to run a talk that would be a link between the two languages.

I've entitled my talk "How awesome ended up with Lua and not Guile" and gave it to a room full of interested users of the awesome window manager 🙂.

We continued with a panel discussion entitled "The future of small languages Experience of Lua and Guile" composed of Andy Wingo, Christopher Webber, Ludovic Courtès, Etiene Dalcol, Hisham Muhammaad and myself. It was a pretty interesting discussion, where both language shared their views on the state of their languages.

It was a bit awkward to talk about Lua & Guile whereas most of my knowledge was years old, but it turns out many things didn't change. I hope I was able to provide interesting hindsight to both community. Finally, it was a pretty interesting FOSDEM to me, and it was a long time I didn't give talk here, so I really enjoyed it. See you next year!

Python 3.4 single dispatch, a step into generic functions

Tue, 17 Sep 2013 00:00:00 GMT

I love to say that Python is a nice subset of Lisp, and I discover that it's getting even more true as time passes. Recently, I've stumbled upon the PEP 443 that describes a way to dispatch generic functions, in a way that looks like what CLOS, the Common Lisp Object System, provides.

What are generic functions

If you come from the Lisp world, this won't be something new to you. The Lisp object system provides a really good way to define and handle method dispatching. It's a base of the Common Lisp object system. For my own pleasure to see Lisp code in a Python post, I'll show you how generic methods work in Lisp first.

To begin, let's define a few very simple classes.

(defclass snare-drum ()
  ())

(defclass cymbal ()
  ())

(defclass stick ()
  ())

(defclass brushes ()
  ())

This defines a few classes: snare-drum, symbal, stick and brushes, without any parent class nor attribute. These classes compose a drum kit, and we can combine them to play sound. So we define a play method that takes two arguments, and returns a sound (as a string).

(defgeneric play (instrument accessory)
  (:documentation "Play sound with instrument and accessory."))

This only defines a generic method: it has no body, and cannot be called with any instance yet. At this stage, we only inform the object system that the method is generic and can be then implemented with various type of arguments. We'll start by implementing versions of this method that knows how to play with the snare-drum.

(defmethod play ((instrument snare-drum) (accessory stick))
  "POC!")

(defmethod play ((instrument snare-drum) (accessory brushes))
  "SHHHH!")

Now we just defined concrete methods with code. They also takes two arguments: instrument which is an instance of snare-drum and accessory that is an instance of stick or brushes.

At this stage, you should note the first difference with object system as built into language like Python: the method isn't tied to any class in particular. The methods are generic, and any class can implement them, or not.

Let's try it.

* (play (make-instance 'snare-drum) (make-instance 'stick))
"POC!"

* (play (make-instance 'snare-drum) (make-instance 'brushes))
"SHHHH!"

* (play (make-instance 'cymbal) (make-instance 'stick))
debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {1002ADAF23}>:
  There is no applicable method for the generic function
    #<STANDARD-GENERIC-FUNCTION PLAY (2)>
  when called with arguments
    (#<CYMBAL {1002B801D3}> #<STICK {1002B82763}>).

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY] Retry calling the generic function.
  1: [ABORT] Exit debugger, returning to top level.

((:METHOD NO-APPLICABLE-METHOD (T)) #<STANDARD-GENERIC-FUNCTION PLAY (2)> #<CYMBAL {1002B801D3}> #<STICK {1002B82763}>) [fast-method]

As you see, the function called depends on the class of the arguments. The object systems dispatch the function calls to the right function for us, depending on the arguments classes. If we call play with instances that are not know to the object system, an error will be thrown.

Inheritance is also supported and the equivalent (but more powerful and less error prone) equivalent of Python's super() is available via (call-next-method).

(defclass snare-drum () ())
(defclass cymbal () ())

(defclass accessory () ())
(defclass stick (accessory) ())
(defclass brushes (accessory) ())

(defmethod play ((c cymbal) (a accessory))
  "BIIING!")

(defmethod play ((c cymbal) (b brushes))
  (concatenate 'string "SSHHHH!" (call-next-method)))

In this example, we define the stick and brushes classes as subclass of the accessory class. The play method defined will return the sound BIIING! regardless of the accessory instance that is used to play the cymbal. Except in the case where it's a brushes instance; only the most precise method is always called. The (call-next-method) function is used to call the closest parent method, in this case that would be the method returning _"BIIING!".

* (play (make-instance 'cymbal) (make-instance 'stick))
"BIIING!"

* (play (make-instance 'cymbal) (make-instance 'brushes))
"SSHHHH!BIIING!"

Note that CLOS is also able to dispatch on object instances themself by using the eql specializer.

But if you're really curious about all features CLOS provides, I suggest you read the brief guide to CLOS by Jeff Dalton as a starter.

Python implementation

Python implements a simpler equivalence of this workflow with the singledispatch function. It will be provided with Python 3.4 as part of the functools module. Here's a rough equivalence of the above Lisp program.

import functools

class SnareDrum(object): pass
class Cymbal(object): pass
class Stick(object): pass
class Brushes(object): pass

@functools.singledispatch
def play(instrument, accessory):
    raise NotImplementedError("Cannot play these")

@play.register(SnareDrum)
def _(instrument, accessory):
    if isinstance(accessory, Stick):
        return "POC!"
    if isinstance(accessory, Brushes):
        return "SHHHH!"
    raise NotImplementedError("Cannot play these")

We define our four classes, and a base play function that raises NotImplementedError, indicating that by default we don't know what to do. We can then write specialized version of this function with a first instrument, the SnareDrum. We then check for the accessory type that we get, and return the appropriate sound or raise NotImplementedError again if we don't know what to do with it.

If we run it, it works as expected:

>>> play(SnareDrum(), Stick())
'POC!'
>>> play(SnareDrum(), Brushes())
'SHHHH!'
>>> play(Cymbal(), Brushes())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jd/Source/cpython/Lib/functools.py", line 562, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/home/jd/sd.py", line 10, in play
    raise NotImplementedError("Cannot play these")
NotImplementedError: Cannot play these
>>> play(SnareDrum(), Cymbal())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jd/Source/cpython/Lib/functools.py", line 562, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/home/jd/sd.py", line 18, in _
    raise NotImplementedError("Cannot play these")
NotImplementedError: Cannot play these

The singledispatch module looks through the classes of the first argument passed to the play function, and calls the right version of it. The first defined version of the play function is always run for the object class, so if our instrument is a class that we did not register for, this base function will be called.

For whose eager to try and use it, the singledispatch function is provided Python 2.6 to 3.3 through the Python Package Index.

Limitations

First, as you noticed in the Lisp version, CLOS provides a multiple dispatcher that can dispatch on the type of any of the argument defined in the method prototype, not only the first one. Unfortunately, Python dispatcher is named singledispatch for this good reason: it only knows to dispatch on the first argument. Guido van Rossum wrote a short article about the subject that he called multimethod a few years ago.

Then, there's no way to call the parent function directly. There's no equivalent of the (call-next-method) from Lisp nor the super() function that allows to do that in Python class system. This means you will have to use various trick to bypass this limitation.

So while I am really glad that Python is going toward that direction, as it's a really powerful way to enhance an object system, it really lacks a lot of more advanced features that CLOS provides out of the box.

Though, improving this could be an interesting challenge. Especially to bring more CLOS power to Hy. :-)

OpenStack meets Lisp: cl-openstack-client

Thu, 04 Jul 2013 00:00:00 GMT

A month ago, a mail hit the OpenStack mailing list entitled "The OpenStack Community Welcomes Developers in All Programming Languages". You may know that OpenStack is essentially built using Python, and therefore it is the reference language for the client libraries implementations. As a Lisp and OpenStack practitioner, I used this excuse to build a challenge for myself: let's prove this point by bringing Lisp into OpenStack!

Welcome cl-openstack-client, the OpenStack client library for Common Lisp!

The project is hosted on the classic OpenStack infrastructure for third party project, StackForge. It provides the continuous integration system based on Jenkins and the Gerrit infrastructure used to review contributions.

How the tests works

OpenStack projects ran a fabulous contribution workflow, which I already talked about, based on tools like Gerrit and Jenkins.

OpenStack Python projects are used to run tox, to build a virtual environment and run test inside. We don't have such thing in Common Lisp as far as I know, so I had to build it myself.

Fortunately, using Quicklisp, the fabulous equivalent of Python's PyPI, it has been a breeze to set this up. cl-openstack-client just includes a basic shell script that does the following:

Download quicklisp.lisp
Run a Lisp program to install the dependencies using Quicklisp
Run a Lisp program running the test suite using FiveAM, that exit with 0 or 1 based on the tests results.

I just run the test using SBCL, though adding more compiler on the table would be a really good plan in the future, and should be straightforward. You can admire a log from a successful test run done when I proposed a patch via Gerrit, to check what it looks like.

Implementation status

For the curious, here's an example of how it works:

* (require 'cl-openstack-client)
* (use-package 'cl-keystone-client)
* (defvar k (make-instance 'connection-v2 :username "demo" :password "somepassword" :tenant-name "demo" :url "http://devstack:5000"))

K

* (authenticate k)

((:ISSUED--AT . "2013-07-04T05:59:55.454226")
 (:EXPIRES . "2013-07-05T05:59:55Z")
 (:ID
  . "wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM")
 (:TENANT (:DESCRIPTION) (:ENABLED . T)
  (:ID . "1774fd545df4400380eb2b4f4985b3be") (:NAME . "demo")))

* (connection-token-id k)

"wNFQwNzo1OTo1NS40NTQyMthisisaverylongtokenwNFQwNzo1OTo1NS40NTQyM"

Unfortunately, the implementation is far from being complete. It only implements for now the Keystone token retrieval.

I've actually started this project to build an already working starting point. With this, future potential contributors will be able to spend efforts on writing code, and not on setting up the basic continuous integration system or module infrastructure.

If you wish to help me and contribute, just follow the OpenStack Gerrit workflow howto or feel free to come by me and ask any question (I'm hanging out on #lisp on Freenode too).

See you soon, hopping to bring more Lisp into OpenStack!

Hy, Lisp in Python

Wed, 03 Apr 2013 00:00:00 GMT

I've meant to look at Hy since Paul Tagliamonte started to talk to me about it, but never took a chance until now. Yesterday, Paul indicated it was a good time for me to start looking at it, so I spent a few hours playing.

But what's Hy?

Python is very nice: it has a great community and a wide range of useful libraries. But let's face it, it misses a great language.

Hy is an implementation of a Lisp on top of Python.

Technically, Hy is built directly with a custom made parser (for now) which then translates expressions using the Python AST module to generate code, which is then run by Python. Therefore, it shares the same properties as Python, and is a Lisp-1 (i.e. with a single namespace for symbols and functions).

If you're interested to listen Paul talking about Hy during last PyCon US, I recommend watching his lightning talk. As the name implies, it's only a few minutes long.

Does it work?

I've been cloning the code and played around a bit with Hy. And to my greatest surprise and pleasure, it works quite well. You can imagine writing Python from there easily. Part of the syntax smells like Clojure's, which looks like a good thing since they're playing in the same area.

You can try a Hy REPL in your Web browser if you want.

Here's what some code look like:

(import requests)

(setv req (requests.get "http://hy.pault.ag"))
(if (= req.status_code 200)
  (for (kv (.iteritems req.headers))
    (print kv))
  (throw (Exception "Wrong status code")))

This code would ouput:

('date', 'Wed, 03 Apr 2013 12:09:23 GMT')
('connection', 'keep-alive')
('content-encoding', 'gzip')
('transfer-encoding', 'chunked')
('content-type', 'text/html; charset=utf-8')
('server', 'nginx/1.2.6')

As you can see, it's really simple to write Lispy code that really uses Python idioms.

There's obviously still a lots of missing features in Hy. The language if far from complete and many parts are moving, but it's really promising, and Paul's doing a great job implementing every idea.

I actually started to hack a bit on Hy, and will try to continue to do so, since I'm really eager to learn a bit more about both Lisp and Python internals in the process. I've already send a few patches on small bugs I've encountered, and proposed a few ideas. It's really exciting to be able to influence early a language design that I'll love to use! Being a recent fan of Common Lisp, I tend to grab the good stuff from it to add them into Hy.

Overriding cl-json object encoding

Fri, 11 Jan 2013 00:00:00 GMT

CL-JSON provides an encoder for Lisp data structures and objects to JSON format. Unfortunately, in some case, its default encoding mechanism for CLOS objects isn't exactly doing the right thing. I'll show you how Common Lisp makes it easy to change that.

Identifying the problem

CL-JSON & CLOS

CL-JSON mechanism encoding CLOS object is really neat. Let's see how it works for a simple case:

(defclass kitten ()
  ((tail :initarg :tail)))

(json:encode-json-to-string (make-instance 'kitten :tail 'black))

will produce:

{"tail":"black"}

Still using CL-JSON, we can also decode the JSON object to a CLOS object:

(slot-value
 (json:with-decoder-simple-clos-semantics
   (json:decode-json-from-string "{\"tail\":\"black\"}"))
 :tail)

That code will return "black". Note that it's also possible to specify which class should be used when decoding objects, but that's beyond the purpose of this article.

Postmodern

Now, let's introduce Postmodern, a wonderful Common Lisp system providing access to the wonderful PostgreSQL database. It also provides a simple system to map rows in a database to CLOS classes, called DAO for Database access objects.

With this, we can easily store our kitten into a table.

(defclass kitten ()
  ((tail :initarg :tail))
  (:metaclass postmodern:dao-class))

If we try to encode this to JSON, it will produce the exact same result seen previously.

The problem is what happens when one of our column has a NULL value. Postmodern encodes this using the :null symbol.

So this code:

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))

(postmodern:deftable kitten
  (postmodern:!dao-def))

(postmodern:connect-toplevel …)

(postmodern:create-table 'kitten)

(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

will return:

"{"tail":"null"}"

Fail! The fact that the column is NULL is represented by the :null symbol. And CL-JSON encodes all symbols as string.

This is not at all what we want here!

Overriding encode-json

CL-JSON provides and uses the encode-json method to encode all kind of object. It is defined as a generic function, and a lot of different methods are implemented to handle the different standard Common Lisp types. The one used for standard-object is defined liked that:

(defmethod encode-json ((o standard-object)
                        &optional (stream *json-output*))
  "Write the JSON representation (Object) of the CLOS object O to
STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (stream-object-member-encoder stream) o)))

All we need to do here, is to create a new method for our kitten objects, that handles correctly the :null case.

(defclass kitten ()
  ((tail :initarg :tail :col-type (or s-sql:db-null text)))
  (:metaclass postmodern:dao-class))

(export 'kitten)

;; Switch package just to define the new method
(in-package :json)
(defmethod encode-json ((o cl-user:kitten)
                        &optional (stream json:*json-output*))
  "Write the JSON representation (Object) of the postmodern DAO CLOS object
O to STREAM (or to *JSON-OUTPUT*)."
  (with-object (stream)
    (map-slots (lambda (key value)
                 (as-object-member (key stream)
                   (encode-json (if (eq value :null) nil value) stream)))
               o)))

;; Go back into our package
(in-package :cl-user)

(postmodern:deftable kitten
  (postmodern:!dao-def))

(postmodern:connect-toplevel …)

(postmodern:create-table 'kitten)

(json:encode-json-to-string
  (postmodern:make-dao 'kitten))

With that new method, as soon as we encounter a :null symbol as a value for an object's slot, we replace it by nil.

Now if we try to encode another kitten, we'll get:

{"tail":null}

which is far better for our JavaScript data consumers!

In the end, I think that this kind of trick is feasible that easily because of the way CLOS provides its generic method implementation.
The fact that methods don't belong to any class makes the extension of every program, library and class so much easier. Doing this in another language like Java would likely by impossible, and in Python it would unlikely be as clean as it is done in Common Lisp.

The ability to teach any library about how it should handle your class just by defining a new method is really handy!

Integrating cl-irc and cl-async

Fri, 04 Jan 2013 00:00:00 GMT

Recently, I've started programming in Common Lisp.

My idea here is to use cl-irc, an IRC library into an event loop. This can be really useful, for example to trigger action based on time, using timers.

Creating a connection

The first step is to create a basic cl-irc:connection object on our own. This can be achieved easily with this:

(require :cl-irc)

(defun connect (server)
  (cl-irc:make-connection :connection-type 'cl-irc:connection
                                              :client-stream t
                                              :network-stream ?
                                              :server-name server))

This will return a cl-irc:connection object, logging to stdout (:client-stream t) and having the server name server. Note that the server name could be any string.

You probably noticed the ? I used as :network-stream value. This is not a real and working value: this should be a stream established to the IRC server you want to chat with. This is where we'll need to use cl-async:tcp connect to establish a TCP connection.

As you can read in this function's documentation, all we need to pass is the server address, two callbacks for read and general events, and the :stream option to get a stream rather than a socket.

So you would do something like:

(require :cl-irc)
(require :cl-async)

(defun connection-socket-read (socket stream)
  (format t "We should read the IRC message from ~a ~%" stream))

(defun connection-socket-event (ev)
  (format t "Socket event: ~a~%" ev))

(defun connect (server &optional (port 6667))
  (cl-irc:make-connection :connection-type 'cl-irc:connection
                          :client-stream t
                          :network-stream (as:tcp-connect server port
                                                          #'connection-socket-read
                                                          #'connection-socket-event
                                                          :stream t)
                          :server-name server))

(as:start-event-loop (lambda () (connect "irc.oftc.net")))

If you run this program, it will connect to the OFTC IRC server, and then notice you each time the server is sending you a message.

Therefore our problem here is how we you treat the message read from the stream in connection-socket-read and handle them in the name of our connection object you used? We can't link both together at this point.

We can't build a closure, because as the time we use as:tcp-connect we don't have the cl-irc:connection instance. Also we can't change easily the read-cb parameter of our network-stream established by as:tcp-connect, simply because cl-async doesn't use to do allow that.

Building a closure

So one solution here is to hack cl-irc:make-connection so we can build an cl-irc:connection instance without providing in advance the network-stream, allowing us to build a closure including the cl-irc:connection to read event for. This is what we're going to do in the connect function.

(require :cl-irc)
(require :cl-async)
(require :flexi-streams)

(defun connection-socket-read (connection)
  (loop for message = (cl-irc::read-irc-message connection)
        while message
        do (cl-irc:irc-message-event connection message)))

(defun connection-socket-event (ev)
  (format t "Socket event: ~a~%" ev))

(defun connect (server port nickname
                &key
                  (username nil)
                  (realname nil)
                  (password nil))
  ;; Build an instance of cl-irc:connection, without any network/output stream
  (let* ((connection (make-instance 'cl-irc:connection
                                    :user username
                                    :password password
                                    :server-name server
                                    :server-port port
                                    :client-stream t))
         ;; Use as:tcp-connect to build our network stream, and build a
         ;; closure calling `connection-socket-read' with our `connection'
         ;; as arguments
         (network-stream (as:tcp-connect server port
                                         (lambda (socket stream)
                                           (declare (ignore socket stream))
                                           (connection-socket-read connection))
                                         #'connection-socket-event
                                         :stream t)))
    ;; Set the network stream on the connection
    (setf (cl-irc:network-stream connection) network-stream)
    ;; Set the output stream on the connection
    (setf (cl-irc:output-stream connection)
         ;; This is grabbed from cl-irc:make-connection
          (flexi-streams:make-flexi-stream
           network-stream
           :element-type 'character
           :external-format '(:utf8 :eol-style :crlf)))

    ;; Now handle the IRC protocol authentication pass
    (unless (null password)
      (cl-irc:pass connection password))
    (cl-irc:nick connection nickname)
    (cl-irc:user- connection (or username nickname) 0 (or realname nickname))
    connection))

(as:start-event-loop (lambda () (connect "irc.oftc.net" 6667 "jd-blog")))

And here we are! If we run this, we're now using an event loop to run cl-irc. Each time the socket has something to read, the function connection-socket-read will be called on the non-blocking mode socket. If there's no message to be read, then the function will exit and the loop will continue to run.

Using timers

You can now modify the last line with this:

(defun say-hello (connection)
  (cl-irc:privmsg connection "#jd-blog" "Hey I read your blog!")
  (as:delay (lambda () (say-hello connection)) :time 60))

(as:start-event-loop (lambda ()
                       (let ((connection (connect "irc.oftc.net" 6667 "jd-blog")))
                         (cl-irc:join connection "#jd-blog")
                         (say-hello connection))))

This will connect to the IRC server, join a channel and then say the same sentence every minute.

Challenge accomplished!

And I'd like to thank Andrew Lyon, the author of cl-async, who has been incredibly helpful with my recent experimentations in this area.