ANN: Grammar/Parser for Lua Available

revusky

I'm pleased to announce that there is a quite robust (it seems that way...) grammar/parser available for the Lua programming language. See here.

Lua is apparently entering its 4th decade of existence. It is actually about contemporaneous with Python. It was created in a university in Brazil in the early nineties. (Lua is the Portuguese word for moon.) I think Lua was first on my own radar something like 20 years ago. I do remember it came to my attention back then, it looked interesting, but I never really got into it.

Now, to be clear, it was never on my mental TODO list to implement a Lua parser. In fact, the idea had not really occurred to me. This really came about in a sort of random fashion. Basically, at some point just yesterday, I was looking at Lua again. I think the reason was that I was intrigued by the fact that Lua was the scripting language for Neovim, which is an increasingly popular fork of the old Vim editor. I think there were people raving about how much better the Lua scripting engine was than the old Vimscript. I was then googling around a bit and it really seemed that Lua was (is) much more popular and widely use than I ever would have guessed. In particular, that kind of use, embedding it in another app to provide scripting abilities, seems to be Lua's forte. Blizzard/Activision uses Lua as a scripting engine inside their gaming engine, and that actually provided me with most of the test suite I was using to converge on a correct grammar. They have a codebase in Lua that is over 400 thousand lines of code. (Plot spoiler. The current Lua parser now parses all of that, building the parse tree and so on in about 4 seconds, on rather unimpressive hardware, by today's standards.) I also have a couple of hundred thousand lines of Lua code from the Neovim project.

Well, I was looking at this, and started wondering whether there was a grammar available. Well, of course, there is. One from the Lua project itself, and also a grammar for ANTLR. Of course. I was intrigued by how relatively short the grammar was, and so I just started poking around and writing a Congo grammar for Lua. After what seemed like just an hour or so (though it turns out it was more like 4 hours!) I had a first pass of the Lua grammar written. I built it and tried it and it couldn't parse a single Lua file. (At that point, I just had something like 35 Lua source files to try it on.) Of course, I expecte that I would have some things wrong, but I was optimistic and thought my first pass implementation would parse maybe half of the Lua files I had. But it didn't parse a single one! Well, then I just got very iterative and obsessive. After a bit of puttering around, I did get it to parse some files and soon it was parsing over half the files I was using as my test suite.

It's a funny thing how obsessive this can get. This sort of task has the same obsessive possibilities that a crossword puzzle can exert and things like this. Like I say, one just loses one's notions of time. So I finally ended up putting together a test base of over 2000 source files and I just kept tweaking and iterating until I could parse every last one. And you can see the result here. I think I started sketching out the Lua grammar maybe around noon yesterday and by late afternoon today, I had it parsing the entire test suite.

I'm pretty happy with this not just because it's nice to have another example, and reinforce our testing, but it's also that it provides an example of a certain intermeiate scale. There is a JSON grammar which is really just a toy example. To go from there to the Python example, more like 1000 lines in CongoCC, is quite a jump. But the Lua grammar, maybe 400 lines of code total for both the syntactic and lexical grammars, provides an example that is not such a toy example, but still substantially smaller than the grammar for Python.

One point about all this that I can't resist making is that it is striking (I think so...) how fast I could go from just initial prototyping to having something that really is pretty rock solid. Every once in a while, I happen on some conversation among developers, on Reddit, for example, and there is always somebody there saying that parser generators are pointless, that you might as well write any parser by hand. Well, I have to say that is surely ridiculous. It is just about inconceivable that I could have something this robust and maintainable this quickly hand-coding a parser in Java.

Well, 'nuff said for now. As of now, the Lua.ccc grammar is rolled into the CongoCC jarfile. In fact, you can use it as a building block in your own work with:

  INCLUDE LUA

in your grammar. I now have to figure out where to announce this. (Not just here!)

vsajip

That's a nice bit of news. Lua is used in quite a lot of diverse scripting environments (Neovim and gaming, as you've mentioned, the well-known key-value store Redis, the nginx web server, and in the Windows command-line enhancer cmder - just a handful of examples among many out there). It doesn't have a huge library/ecosystem but is perfect for self-contained minimal-resource environments.

revusky

vsajip

I just tried to build the parsers for Python and C#. Python built fine, i.e.

java -jar congocc-full.jar -lang python Lua.ccc

I haven't tested that it actually works! But I would bet a substantial sum that it does. (Ever the optimist!)

With C#, on the other hand, it doesn't work. We have some failure in the translateLexerInjections method, looks like this:

freemarker.template.TemplateModelException: Method public java.lang.String 
org.congocc.codegen.TemplateGlobals.translateLexerInjections(boolean) threw an exception when invoked 
...snip...
Caused by: java.lang.UnsupportedOperationException
at org.congocc.codegen.csharp.CSharpTranslator.internalTranslateStatement(CSharpTranslator.java:819)
at org.congocc.codegen.csharp.CSharpTranslator.internalTranslateStatement(CSharpTranslator.java:480)
at org.congocc.codegen.csharp.CSharpTranslator.internalTranslateStatement(CSharpTranslator.java:706)
at org.congocc.codegen.Translator.translateStatement(Translator.java:1573)
at org.congocc.codegen.TemplateGlobals.translateInjections(TemplateGlobals.java:343)
at org.congocc.codegen.TemplateGlobals.translateLexerInjections(TemplateGlobals.java:442)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at freemarker.ext.beans.BeansWrapper.invokeMethod(BeansWrapper.java:741)
at freemarker.ext.beans.SimpleMethodModel.exec(SimpleMethodModel.java:55)
... 15 more

I've noticed that the failure to translate injections can make the tool quite fragile in terms of generating parsers in either Python or C#. I think transpilation is a good thing to have, since a lot of simple stuff can actually be transpiled pretty reliably. However, we also really do need injection in the target languages working. The big refactoring that I reported on a little while ago actually should get us most of the way there. That should be coming very soon.

I really feel that, aside from getting INJECT working for non-Java targets, we also to have at least some skeletal reference manual. I think that's going to be coming pretty soon. Generally speaking, I feel quite energized/motivated to get the Congo branding fullly done and have something that looks more polished.

As of the policy on "official releases", well, I'm thinking that with Congo, we will just give the people what they want. So my idea is that we'll just roll up a release every month, a snapshot of the thing at that point in time, label it and put it up there... where??? Well, I guess in the various typical places. Maybe we just roll up a release with the code, the binary, the manual... And we just do it at the beginning of every month, like paying the rent. We should have an automated script that makes the whole thing basically painless. Heck, it could be a chron job at the beginning of every month maybe. Well, I guess it can't be that automated because we do need to hand-write some release notes of some sort, like we did this, that or the other thing.

But, generally speaking, we really need to get the word out much more.

By the way, I installed this wiki app called Bookstack that some people definitely like very much. I sent you an invite to join. That is at https://bookstack.congocc.org/ but actually the top-level also redirects there. https://congocc.org/

By the way, if anybody wants to try it out, you don't even have to create an account. There is a "public playground" that is editable. So anybody can get on there and write a page. Though I think if one intends to do much editing on there, one should have an account and I'll give anybody an account who asks for one.

I'm not at all committed to Bookstack, mind you, but it does seem like a pretty slick app. Also, you can take a "book" and generate a PDF from it. That feature got me thinking that this could be a solution to our documentation problems/needs. (Not sure...) I started using the wiki built into Github, thinking I was just being silly installing my own web apps. I had a Dokuwiki instance set up earlier. (Still do.) But I think that Gollum thing that the built-in github wiki uses is kind of limited. It seems to have just about no concept of hierarchy, while Bookstack has "shelves", "books", and "chapters". Well, the Gollum thing does have its strong points, like the ability to edit the content in all sorts of formats -- Markdown, Asciidoc, rst I think as well. Also, its integration with git. I think Bookstack basically just supports Markdown and that's it, though there may be some plugins that allow other formats. But you can convert from Markdown to other formats pretty easily using pandoc.

Of course, there are a lot of possibilities for webapps for collaborative editing and so on, but the hard truth is that whatever tool stack you use, you still have to buckle down and write the material! And actually, it's not that there is no material. I mean, the wordpress blog (that, by the way, I moved from javacc.com to parsers.org) does have 49 articles on it (I just counted) some of which are quite in-depth. And the dokuwiki instance has maybe 20 pages on it. Between the two, there is quite a bit of information, but it's not well organized like a reference manual, no. It was just because I did things, like the streamlined syntax and so on, and felt I had to document them somewhere, so I wrote some blog article.

So, those are the two main things -- getting INJECT working, and really starting to have some reference docs finally. In writing the Lua grammar, I became (painfully) aware of certain glitches that do need to be addressed, but I'll write about that later.

But, anyway, it would be nice if you could get the Python and CSharp parsers for Lua working, probably add that as a testcase, and then I will try to scream about it on the appropriate venues. I did just subscribe to the lua-users mailing list, for example.