Seeking Feedback on Radical Change to Template Syntax

revusky

Anybody who hacks congo templates (CTL) will know that at some point not so long ago, I introduced the double-colon :: operator which is used specifically when you are explicitly calling into the Java layer with reflection.

So, if you have a java.lang.String and you want to call toUpperCase on it, you would write string::toUpperCase(). The motivation behind all this is explained here: https://github.com/congo-cc/congo-template-engine/wiki/Double-Colon-Operator

The essence of it is that, in FreeMarker, the dot operator was (and is) overloaded for two different purposes:

properties (or maybe one could call them mappings)
the regular use in Java, which is for referring to an object's methods and field. (Except we only reference methods from the template layer.)

This dual use of the dot operator led to some rather annoying ambiguities and I finally wanted to get this straightened out. So, what we have is that foo.bar is taken to mean foo.get("bar") if foo is a java.util.Map, and otherwise it is taken to mean foo.getBar() otherwise (or possibly foo.isBar() in a boolean context).

But if we really want to refer to a method called bar, then we use foo::bar. Thus, ${s::toUpperCase()} because we are explicitly calling into the Java layer using reflection. (It is true that foo.bar aliasing a call to foo.getBar() is also mapping into a Java method call by reflection, but it is kind of hidden. foo.bar could also be just that foo is a java.util.Map and we're calling foo.get("bar").

Okay, all this is done already, so what am I seeking feedback on? This: I think (though I waver, but I am tending to think) it would be better to invert the above -- I mean the roles of the double-colon and dot. I am tending to think it would be better if the foo.bar notation was used specifically for Java methods and the foo::bar was used specifically for properties (or mappings). I think it just makes more sense to keep using the dot for what it is used for in Java, and then use :: for properties/mappings. I think that using the . to call Java methods just adheres better to the "principle of least surprise".

Well, I think that John Bradley might be the only person using CTL outside of the CongoCC project. Though maybe somebody else or other is, I really don't know. But I intend to write some tutorial material and try to do some various outreach or advocacy so I want to get this squared away ASAP.

By the way, this is already implemented. In the branch swap-delimiters here: https://github.com/revusky/congocc/tree/swap-delimiters (In fact, I decided to do things according to Hoyle and put in a pull request here: https://github.com/congo-cc/congo-parser-generator/pull/252 rather than just committing it directly.

You can see what the converted templates look like, for example: https://github.com/revusky/congocc/blob/swap-delimiters/src/templates/java/ParserProductions.java.ctl

Well, I thought to give people a chance to express an opinion. I've been looking over this stuff and I want to get certain things about the semantics squared away.

rcdev58

revusky I don't directly use FreeMarker, so I don't have an intuition about whether . or :: should be used to call Java methods. My only comment is that the congo-docs repository might need changes to be kept up to date.

adMartem

revusky
I kind of like it. The "." for properties has always evoked a random double-take when I'm looking at templates, and I do agree that from the Java perspective it is more natural to expect a method name rather than a property. So I, for one, would not be upset if you made this change. It actually fits well with my hopes to use CTL in the near future outside of (but in conjunction with, of course) CongoCC.

revusky

rcdev58 Well, at the moment, the material in congo-docs only relates to using the CongoCC parser generator. How the templates work only affects people who are working on the internals of the parser generator. It seems that Claude still gets confused about this. But, be that as it may, sometimes Monsieur Claude says some insightful things -- or at least things that get me thinking. One such thing was that, in a recent conversation, Claude casually commented that there were far more people out there interested in a template engine than a parser generator. And that is almost certainly true. In fact, for me, it was sort of a duhh bang my head sort of moment. It's that since putting so much energy into getting this CongoCC beaten into shape, I have been fixated on that, and...

So, yeah, this situation of having this very robust usable template library sort of hiding there inside a parser generator -- which most people consider some very specialized thing. Also, they consider it to be this very specialized thing that they don't understand. And never will... Somehow the whole parser generator space has this mystique built up around it... (I've said that before, but I suppose it bears repeating...)

But, anyway, that casual comment from Monsieur Claude somehow did get me thinking... about how much stuff there is kind of hidden away in CongoCC. I mean, aside from the templating library, there are all these very robust parsers for the various languages. I mean, even if you're not interested in writing a grammar yourself, i.e. using a parser generator, you can just use the Java parser and it's extremely robust and performant really. For example, there is this quite popular project called Javaparser (how imaginative of them!) that has as its basic goal to parse Java code. https://github.com/javaparser/javaparser That has over 6000 stars on Github, 1200 people have forked it. Quite a few people are, or have been, involved with it. Meanwhile, that project's raison d'etre is basically to write/maintain a Java parser -- which is really a relatively minor subset of the functionality of this project. Frankly, I doubt that, of the time I've spent on CongoCC, that any more than about 3% of it was devoted to working on that Java grammar/parser. (Possibly less.) False modesty aside, that does imply that I am much more effective than them. I mean, at programming, I am more effective. They are somehow much more effective at self-promotion!

Well, anyway, I'm going on a bit, but one thing that all of this train of thought led me to is that these docs we're working up (and don't get me wrong, I'm very happy about the way things are going) probably should address the standalone use of the various parsers. Particularly the Java parser. I mean, some people might just want to roll a Java parser into their app and to have a HOWTO that explains how to do that. BUT... then again... I'm not sure that should be part of the main congocc documentation. It could live in its own separate little documentation universe. Somebody who just wants a Java parser most likely doesn't care about all the various features of CongoCC. And certainly, somebody who wants a template engine doesn't. That, by the way, is why I asked you, when moving the repository over, to call it congo-docs, not congocc-docs. Not that it makes that much difference, but that is related to this train of thought. Funny enough, the "organization" that this is part of was named "congo-cc" because, amazingly, somebody had already taken the congocc name on github. See: https://github.com/congocc Somebody created that and has not added any repositories. I have no idea who that is. (Maybe somebody who doesn't like me very much.)

But, finally, it doesn't matter much and it could well be better to rename congo-cc to congo-tools or something like that and really brainstorm about how to get the word out about what is on offer, that there is a set of tools available that is not just the parser generator.

But, I mean, it doesn't seem that concentrating almost exclusively on the fact that we have a parser generator is a winning strategy in terms of gaining mindshare. But, anyway, yes, there is a need for CTL docs as well. And, in fact, I was taking some baby steps in that direction on my local machine, just getting another Sphinx project going, but I haven't committed any of that. But the docs for the template functionality will just be completely separate. Well, aside from pointing out that CTL is a very good integrated functional example of using CongoCC.

rcdev58

revusky I wonder what a bake-off with javaparser vs congocc's java language parser would reveal in terms of comparative performance? If congocc does well, and since congocc can generate parsers in 4 languages, it would be a great way to promote congocc. As a matter of fact, submitting a technical paper on congocc that includes that comparison would be a reasonable way to make people aware of congocc. Putting congocc's java parser in a JAR file in Maven might also make it easier to use that parser. Crucial to any get-the-word-out effort, however, is getting congo-docs pages served live on the internet.

revusky

adMartem I'm glad you agree. I think that the whole idea of using :: for calling the Java methods, as opposed to using it for properties, was somehow ass-backwards. It was just a few days ago that I was thinking that we could pretty easily still invert it.

Another funny aspect of the legacy syntax is the whole thing with the square brackets. The "normal" use of square brackets would be with lists, as in myList[i] and i is, of course an integer (a positive one) but the use of square brackets was overloaded as the key for a map or property. So you could write: foo["bar"] instead of foo.bar, but of course, the real purpose is when the key is not a Java identifier, so you can write foo["bar-baz"] while foo.bar-baz obviously doesn't work. With the double-colon property syntax, I guess the normal thing could be just to write:

  foo::"bar-baz"

the quotes being necessary for the key because bar-baz is not a valid identifier. Of course, FreeMarker syntax never allowed foo."bar-baz" but I guess that's just because it didn't. I suppose there is no inherent problem with it. Still, I don't like it somehow. I find foo::"bar-baz" more palatable for some reason.

Another thing I'm trying to nail down is the precise null/missing value semantics. I'll write about that separately. But the thing is that now that I'm seeing my way (our way) to getting some serious docs together, I'm thinking about documenting all these nooks and crannies and then find myself looking at how it works and not being 100% satisfied with it. But the thing is that changing any of this stuff is not backward compatible (though sometimes the incompatibilities would only be for relatively rare cases, but still...) Anyway, I think I'm just going to commit the swap of the . and :: pretty much immediately. It's all done. It was surprisingly easy.

rcdev58

revusky It's not always easy to know what current trends are or what new software package will eventually win out over the competition. Here's a list existing template languages, some of which are not portable template languages like FreeMarker, but whole frameworks. I've used jinga2, which seems to have a big following even outside the python world.