lua — jd:/dev/blog

FOSDEM 2016, recap

Sat, 06 Feb 2016 00:00:00 GMT

Last week-end, I was in Brussels, Belgium for the FOSDEM, one of the greatest open source developer conference. I was not sure to go there this year (I already skipped it in 2015), but it turned out I was requested to do a talk in the shared Lua & GNU Guile devroom.

As a long time Lua user and developer, and a follower of GNU Guile for several years, the organizer asked me to run a talk that would be a link between the two languages.

I've entitled my talk "How awesome ended up with Lua and not Guile" and gave it to a room full of interested users of the awesome window manager 🙂.

We continued with a panel discussion entitled "The future of small languages Experience of Lua and Guile" composed of Andy Wingo, Christopher Webber, Ludovic Courtès, Etiene Dalcol, Hisham Muhammaad and myself. It was a pretty interesting discussion, where both language shared their views on the state of their languages.

It was a bit awkward to talk about Lua & Guile whereas most of my knowledge was years old, but it turns out many things didn't change. I hope I was able to provide interesting hindsight to both community. Finally, it was a pretty interesting FOSDEM to me, and it was a long time I didn't give talk here, so I really enjoyed it. See you next year!

Why not Lua

Tue, 26 Apr 2011 00:00:00 GMT

Since my latest announcement of the Lua workshop, I received a couple of emails asking why I discourage the use of Lua.

Actually, I already wrote out many of the things I dislike about Lua. I won't come back on this technical issues here, but since Lua 5.2 is not yet released (it's still at alpha stage), they are still relevant nowadays.

Stack based API is harder

The ease of integration of Lua into a C program is one of the point of Lua. They claim it's very easy to integrate Lua into your C application, because it does not use pointer, nor reference counting, nor anything that requires a minimum amount of skills to be used.

It uses a virtual stack based approach. You push or pop things on a stack, and refers to them using a relative or absolute index.

In order to people who never wrote Lua code to understand, here's a quick example on how this work. The L pointer is a Lua environment.

/* Create a table on the stack: index 1 */
lua_newtable(L);
/* Push a string on the stack: index 2 */
lua_pushstring(L, "hello");
/* Push a number on the stack: index 3 */
lua_pushnumber(L, 123);
/* Set newtable["hello"] = 123 */
lua_settable(L, -3);

You first push a table (in Lua, a table is almost equivalent to what you'd call a hash table in other language), then push the key, the value, and do the assignment operation. In the settable, we use -3 as index, meaning the "3rd item on the stack counting from top". We could also have written lua_settable(L, 1), since the table is also the first item on the stack from the bottom.

So far, so good.

Problems arise when you do more complicated stuff. My previous example is what you would typically find in a tutorial, but of course, real life is different, and usually more complex. If you cut the things in different parts, it can start to be more complicated.

Let's take a look at the following:

/* Create a table on the stack: index 1 */
lua_newtable(L);
/* Push a string on the stack: index 2 */
lua_pushstring(L, "hello");
/* Push a number on the stack: index 3 */
lua_pushnumbe(L, mycomputingfunction());
/* Set newtable["hello"] = 123 */
lua_settable(L, -3);

Here, we do exactly the same thing, but we do not push 123 directly: we compute it.

And here's the trick: if your computing function is also using the Lua stack, things can become very messy. As long as your computing function use the stack cleanly by pushing and poping all its item, and returning the stack in the same state it was before, you're safe. The problem is that in a complex program, you also write bugs. You do not chose to, but you do. And sometimes, you forget to pop one of the item you fetched from a table.

Imagine that mycomputingfunction is:

int
mycomputingfunctiong(void)
{
  /* Just push the table we want to fetch
     the number from on the stack */
  pushatableonstack(L);
  lua_pushstring(L, "mykey");
  lua_gettable(L, -2);
  return lua_tonumber(L, -1);
}

This function works perfectly. It pushes a table, then a key ("mykey"), then fetches mytable["mykey"] and pops the key (lua_gettable does push value/pop key itself), and then returns the numeric value of the last item (the fetched one) of the stack.

However, this function has a bug: it does not pop the table! This does not prevent the function to work. It does not raise a segmentation fault. It does not show any problem under gdb. It does not show any leak under Valgrind. It does now show any problem under any standard C debugging tool.

But when you'll start using it, your program will start to do weird things, and you'll have to spend a huge amount of time debugging it manually, dumping the stack content at each step of your program to watch out what's wrong.

Another bad thing, that can happen, is some code poping accidentally an item from the stack, or worst, from an empty stack. This does not raise any error on the Lua side, but will break your program in very unfunny way.

Even if I've been very meticulous writing awesome, but we hit that problem regularly.

The easiest workaround is to use lua_settop(L, 0) to reset the stack to 0 element. Doing this regularly (like after each program event or treatment) can remove left-over items and avoid the never ending stack grow you may experience if your left-over items continue to pile up. Did I tell you I dislike work-around?

You could also use lua_call(), which would avoid such an error, but this would require a huge amount of indirection, and would make write more (useless) code.

This kind of problem does not exists with pointer based API. If you screw things up, the problem will cause a segmentation fault or leak memory, or cause things you can (easily) debug with standard tools like gdb or Valgrind.

No reference counting is a pain

Userdata objects are variable Lua size objects embedding a C struct you define. It's the equivalent of an object in object oriented language.

Lua does not provide any reference counting for the userdata objects. That means you can push this objects on the stack, use them, but they cannot directly reference each others. If you have a "car" userdata and a "wheel" one, the car cannot hold directly a reference to the wheel. This is not possible because userdata are allocated and garbage collected by Lua, and there's no way to increase the reference counting yourself.

So the common hack is to store the wheel into a table as a value, and store
the table index as an integer into the car data structure.

This obviously makes memory leaks tracking harder, add huge level of reference indirection in usage (still more code), and does not make the whole process less error prone (at least in my opinion).

No paradigm makes you lose time

Lua is proud to come with no paradigm and to provide metatables. I already showed 3 years ago that it has big flaws.

To me, this ain't no good. Lua is not functional, nor it is object oriented.

Most people, including me, want one of this paradigm, or any else. Plain old imperative is not enough.

So you'll start to build more, or to use something like LOOP, which implements an object model. You'll implement your paradigm. I say life is too short to (re)write a paradigm.

In awesome we wanted to have an object oriented approach (this is kind of typical in such a graphical application context), so we tried to build one.
To me, this started to be a show stopper when I realized that I've ended writing Python object model into Lua while developing awesome (which aims to be a window manager, not a language). This is one of the reason I stopped hacking on Lua things.

I liked Python object model and wanted to have it in Lua, and spending time rewriting Python is just not worth it. I probably should have chose Python, not Lua. YMMV.

Embedding may not be a good choice

This is not Lua related, but I want to mention it. Googling for "embedding vs extending" will probably tell you more about why you should double check that you really need to embed Lua rather than to extend it.

Being small is not an excuse

One common argument to choose Lua is that it has a small footprint. Yeah, that's true, but that's useless. Bummer! When I program, I don't have any resource usage pressure. People who have such pressure are either paranoids or playing in the world of embedded computers. This is also a no more existing conception since quad core processors equiped phones are coming into the market. I'm rather confident that what we used to call embedded devices are just dead and are now plain computers. But as usual, YMMV.

So start to forget about it, run in your underpants and yelling "yay we killed that shit!", and then use real computers stuff. :-)

Even if benchmarks show how Lua is damn fast, remember what a benchmark proves: that you can do useless things very fast.

Too few extension modules

This is not directly Lua's fault, but there's too few extension modules for Lua. The community is quite small compared to other big languages' ones.

So think twice

before you choose Lua (or any other language). My recommendations these days would be not to embed, but to extend. If you really have no choice and need to embed a language into your application, GNU Guile is probably worth considering, because it's a Scheme and therefore a functional language :-), and because it can provides also different languages.

Including Lua.

Lua workshop at Fabelier/tmplab

Thu, 14 Apr 2011 00:00:00 GMT

It seems I'll be at the Lua workshop at Fabelier/tmplab on April 28th 2011, where I'll try to present and talk about Lua, how to use it, and why you should probably not use it. ;-)

Various news: what happend during summer

Tue, 22 Sep 2009 00:00:00 GMT

It's been a while since I blogged about something. So here's a bunch of things I've done the last month.

Holidays

Well, I've been in holidays one week. :-P

awesome

There have been a huge number of changes between 3.3 (released in June) and 3.4 (almost relesed). I wrote a small but very useful object layer on top of Lua, which adds a class/object system a bit like gobject. I've also replaced all the hooks by per-class/object signals. Finally, the awesome Lua basement are cleaner than they were before, and the extendability is improved. How nice.

We're trying to release 3.4 (rc2 should be out soon), but the development pace is a bit slower than a year before. We're basically almost 2 months late on what was our previous release rate. Not a big deal however.

I've started working on 3.5 slowly. It gonna get amazing new features too. :-)

Google Summer Of Code 2009

I've mentored Mariusz Ceier on XCB GSoC. He worked on adding Xinput2 and XKB extensions. And he managed to do this. His work should be imported ASAP, the discussion has started on XCB maling list last week.

In exchange, Google offered me (and to every mentor) an awful blue t-shirt! Thanks Google! :-P

Rants about Lua

Tue, 30 Dec 2008 00:00:00 GMT

I've started using Lua some months ago, while looking for a more powerful way to configure awesome. At this time, around March 2008, Lua seemed to be the best language to integrate inside the core system of awesome.

I still think that Lua was a good choice, but after 8 months, it shows some important drawbacks.

I'll try to keep my explanation simple and to make you understand everything, even if you do not know Lua.

I refer here to Lua version 5.1.

Design flaws

Length operator

Lua has a length operator on its objects, known as #. It can be used to get the size of various objects.

> return #"lol"
> 3

This operator works for table, string, etc… It's possible to define this operator by setting a __len meta-method on a userdata value.

The problem is that you cannot redefine it on string or table objects, see:

> a = { "hello", "world" }
> return #a
2
> setmetatable(a, { __len = function () return 18 end })
> return #a
2

Indeed, looking at the Lua core code:

      case OP_LEN: {
        const TValue *rb = RB(i);
        switch (ttype(rb)) {
          case LUA_TTABLE: {
            setnvalue(ra, cast_num(luaH_getn(hvalue(rb))));
            break;
          }
          case LUA_TSTRING: {
            setnvalue(ra, cast_num(tsvalue(rb)->len));
            break;
          }
          default: {  /* try metamethod */
            Protect(
              if (!call_binTM(L, rb, luaO_nilobject, ra, TM_LEN))
                luaG_typeerror(L, rb, "get length of");
            )
          }
        }
        continue;
      }

You clearly see that tables and strings always use the internal length operator, never the __len meta-method.

That's for me a design problem, which will cause more trouble. We'll see later.

index and newindex metamethods

Lua defines two useful meta-methods, which are __index and __newindex. Both can be set on a table or any other object. __index will be called upon each read access to an undefined key on an object, and __newindex upon each write access.

> a = {}
> -- function are not defined, this is just an example
> setmetatable(a, { __index = myindexfunction, __newindex = mynewindexfunction })
> a[1] = "hello" -- This will call __newindex metamethod
> return a[2] -- This will call __index metamethods
> return a[1] -- This will NOT call __index

The last line does not call __index meta-method because a[1] does exists. This is a problem when you want to use table as object, because sometimes you want to monitor access to the table elements.

This can be easily worked around using a proxy system: you don't store
things in the table you manipulate, but in another table.

> a = {}
> realtable = {}
> -- function are not defined, this is just an example:
> setmetatable(a, { __index = myindexfunction, __newindex = mynewindexfunction })

Where meta-methods are something like:

function myindexfunction(table, key)
   return realtable[key]
end

function mynewindexfunction(table, key, value)
   realtable[key] = value
end

This way, our a table will always be empty, and realtable will have the data. At every read or write access to a, the meta-methods will be called. This is very convenient and widely used hack.

But this has serious drawbacks: as we saw before, the length operator (#) cannot be redefined on a table. That means #a will always be 0, and you cannot get the table length anymore, except by defining another method or a special attribute.

Also, Lua has several functions in the table library that are used to manipulate table in a easy way. The problem is that standard functions like table.insert or table.remove do raw accesses to the table. Meaning that if you do table.insert(a, 1, 1) it will insert the value 1 at the key 1 into a, without calling the __newindex meta-methods, breaking all your beautiful object-oriented model.

Another solution is to use a userdata object, like done in the Lua newproxy function (which is under-documented).

The problem is that it breaks all the other functions that are waiting for a table as argument, because they see a userdata, not a table. So this time table.insert is now more usable, which somehow fixes the problem, but not in the right way IMHO. However, this allows to use the __len meta-method.

Development model

The development model of Lua is, from my point of view, non-existent.

There is no public version control system repository available, so there's no chance to really contribute to Lua. It seems only a defined set of people work on it and therefore, the development system is very closed to my eyes, in comparison of usual projects.

Conclusion

I still think Lua is a good choice, because it is very easy to integrate into any C program, and to expand to fulfill your needs. However, some bad design choices were made, and the poor and closed development model chosen does not allow to have a good overview of the future of Lua.

This has been well stated by the authors themselves.