Open Question: How desirable is this feature?

revusky

Every once in a while, I take a look at the legacy JavaCC project's issue tracker. I've been aware of this feature request for some time: https://github.com/javacc/javacc/issues/199

As it's been there for a year and a half, I must have come across that over a year ago and made a mental note to revisit the question. It's actually not very hard to implement against the JavaCC 21 codebase, maybe just an hour or two of effort to get it working. (And I'm hardly the fastest coder in the world...) Probably if the person had shown up on the JavaCC21 project page and requested this, it would be done by now.

But there is this rather weird situation where people show up in the legacy JavaCC project, requesting whatever feature, or reporting a bug or whatever, as if there is some even infinitesimal chance that these people will address the issue! Well, if you don't know, you don't know, I guess... The thing is that the only place to request a new feature is here because the JavaCC21 codebase has been cleaned up to a point that a lot of nice-to-have things can be implemented quite easily.

But there is a countervailing consideration, which is that we are so behind in terms of documentation, so adding new features that basically are not (or hardly) documented... Quite arguably, it would be better to exercise more self-discipline and not implement new features willy-nilly, at least not until we are more caught-up in terms of documentation.

I think what I'm going to do in this case is see if somebody (whoever is around or whatever lurkers there come out of the woodwork) saying that they very much want this. In that case, I'll implement it. I'm also going to reach out to the person who requested it (a year and a half ago) and see if they're willing to show up here and request the feature. (The implication being that they will use it, I guess...)

There are some very strange things that happen. I had some contact with that JSQLParser project back in mid-2020. I mean, I didn't know that project existed before the project lead made contact.. I eventually, though it was about a year later, did implement the requested feature.. I mean, that Tobias Warnecke guy had requested the feature a year earlier and, after a round of code cleanup, it was quite low-hanging fruit and I implemented it -- though, admittedly, not for him specifically. I make use of token activation/deactivation quite a bit now internally. A couple of other people use it too. Probably more, but I only know of two.

Well, really, I wish people would maintain a higher standard of behavior. The guy did request the feature quite explicitly and I did implement it. To simply not respond when I tell him that the feature he himself requested is implemented.... I've been on the receiving end of quite a bit of casual rudeness (and worse) from people since trying to resuscitate this JavaCC thing. I guess I'm not supposed to mention it, just turn the other cheek, but I choose to mention it. (Though now that I have mentioned it, I won't mention it again.)

Anyway, anybody who thinks that the requested feature would be very helpful to him (or her) on seeing this, has the opportunity to pipe up and if I get the feedback that this is worth doing, I'll buckle down and implement it. Probably same-day.

And if nobody says anything, I'll probably still implement it eventually, but it won't have very high priority. I think that's reasonable of me. (But I can hardly be objective about that!)

vsajip

I can't say anything based on a personal need for this feature, but it seems not an unreasonable thing. I would just ask, if you do implement it, it's easy enough to do in all generation languages ...

revusky

vsajip I can't say anything based on a personal need for this feature, but it seems not an unreasonable thing. I would just ask, if you do implement it, it's easy enough to do in all generation languages ...

Well, once the ability to inject code in other languages is squared away, the implementation (the one that occurred to me anyway) would be about as easy in non-Java languages as in Java. Basically, we need to be able to specify some kind of accept method for the various TokenTypes -- the default one being just maybe:

    boolean accept(String input) {return true;}

But if that is overridden for a given TokenType, and the method (based on whatever user-defined logic) returns false, then you wind back to where the token started, deactivate that token type and re-tokenize. (And then you turn the token type back on.)

But then, I was thinking about this further and I find myself wondering whether there are really very many (or any) use cases where you couldn't handle whatever it is equally well using the TOKEN_HOOK mechanism anyway.

So, this does kind of lead into the question of whether one is going to implement things willy-nilly just because they can be done. But I sort of throw the question out there just to see if anybody out there pops up and says "oh, wow, that's exactly what I need"....

And then there is this other more general question about people -- I guess, even people who are not even currently using the tool -- requesting something... I'm thinking in particular about this whole Maven Central question... There is somebody (I'm not even going to say his name) who was saying somewhere that I had adamantly refused to put JavaCC 21 on Maven Central.

I don't feel that is really an accurate characterization of what is going on with that. In fact, I earlier intended to put up some versions on Maven Central, but only because this was so important to certain people -- according to them! But then I ran into the problem that their system (I mean Sonatype) was so baroque and user unfriendly that I always had better things to do than to figure out, basically.

In other matters (though similar, I guess) recently, on the command-line in Ubuntu, I idly typed:

  sudo apt install javacc

And, yep, it installed it. So then I saw that there was now a javacc script in my /usr/bin that looks like:

 #!/bin/sh
 JAR=/usr/share/java/javacc.jar

 java -classpath "$JAR" javacc "$@"

(There is also a jjtree script that looks like the above one, except with jjtree insteas of javacc.

So it drops the javacc.jar into /usr/share/java (a place where a lot of jarfiles get dropped, it seems.) But here is an interesting thing. The ubuntu package manager installs JavaCC 5.0, which was released in September of 2009, about 13 years ago. I mean, it would be nice if you could type:

   sudo apt install javacc21

in Ubuntu (and all its offspring) and it would just install the thing, but then that does, in principle, leave one with the obligation of keeping it up to date. (Though JavaCC 21 actually has a separate solution to that, that it actually checks the server for a newer version, unless you ask it not to... so maybe that's not even a big deal...)

But getting back to the Maven Central question, I guess I can say that if a high-profile project that uses the legacy JavaCC were to approach us and say: "Well, look, it's very important for us that the thing be on Maven Central. That prevents us from using it.", well, I guess I would finally go through the steps and put it up there. However, what I anticipate is that they are perfectly capable of saying how important this is to them, you put it up there, and then they still don't use it!

And that would be a bit of a crisis for me because I would find that VERY VERY irritating.

adMartem

revusky But then, I was thinking about this further and I find myself wondering whether there are really very many (or any) use cases where you couldn't handle whatever it is equally well using the TOKEN_HOOK mechanism anyway.

This is, in fact, how I do this right now (and how I did it with javacc). Essentially, I detect a token that has been "deactivated" by the user's configuration (usually due to the dialect being parsed) and convert it into a different token. If the issue 199 capability exists, I'm not sure I would use it, as it would add the try...catch to essentially every token, instead of in a single location.