Richard posed the following question: "I wonder what a bake-off with javaparser vs congocc's java language parser would reveal in terms of comparative performance?" in a different thread and I started writing a response but finally figured I ought to start a new thread.
Yeah, well, generating some benchmarks wrt something like Javaparser is a good idea obviously and finally I downloaded the thing and did some benchmarking. Here is a test harness:
import java.io.IOException;
import java.util.ArrayList;
import com.github.javaparser.StaticJavaParser;
import com.github.javaparser.ParserConfiguration.LanguageLevel;
ArrayList<String> failures = new ArrayList<String>();
public void main(String[] args) {
long startTime = System.currentTimeMillis();
var filenames = new ArrayList<String>();
for (var arg : args) filenames.add(arg);
filenames.stream().forEach(this::parseFile);
// Comment out the preceding line and
// uncomment the following line to parse in multiple threads.
//filenames.parallelStream().forEach(this::parseFile);
IO.println("Successfully parsed " + (filenames.size() - failures.size()) + " files.");
IO.println("Failed on: " + failures.size() + " files.");
for (var f : failures) {
System.out.println("Failed on: " +f);
}
IO.println("Duration: " + (System.currentTimeMillis()-startTime) + " milliseconds.");
}
void parseFile(String filename) {
IO.println("Parsing file: " + filename);
var config = StaticJavaParser.getParserConfiguration();
config.setLanguageLevel(LanguageLevel.JAVA_26);
var path = FileSystems.getDefault().getPath(filename);
try {
StaticJavaParser.parse(path);
} catch (Exception e) {
failures.add(filename);
e.printStackTrace();
}
}
That can be compiled with javac -cp javaparser.jar:. JPTest.java
and run with:
java -cp javaparser.jar:. JPTest <files>
Well, after running it on various sets of files, I drew the conclusion that Javaparser is about 50% to 60% as fast as the Java parser generated by CongoCC. I suppose that is worth pointing out, but in response to Richard' comment:
If congocc does well, and since congocc can generate parsers in 4 languages, it would be a great way to promote congocc
Well... it's worth pointing out that our parser is faster, sure, but actually, I don't think it's a major selling point. Or really, it kind of misses the point. The fact is that CongoCC's java parser parses something around 100K lines per second, something like that. So, Javaparser parses 50k to 60k LOC a second. The fact is that, as a practical real-world question, it's very had to believe that anybody (or hardly anybody anyway) really needs to parse Java source code any faster than that. Even though Javaparser is slower, it's plenty fast for practical use.
Oh, actually (and I nearly forgot to mention it) there is possibly a more significant difference in memory usage. To run the main test suite, which is to parse all the files in the JDK's src.zip, Javaparser needs just about 400 megs of heap. (Just about, like 380 I think.) I get that from running with -Xmx400M and such, experimenting with different values. The CongoCC java parser only needs about 80 megs. A bit less, so it's about a 5 to 1 difference in memory usage. So, on those grounds, our implementation is pretty clearly more efficient overall.
I think maybe a more fundamental issue is just correctness. At some point, when looking at the Java parsers available from the Antlr community I started noticing just how sloppy they are, in terms of the input that they accept uncomplainingly. For example, consider:
public class Foobar {
void foo() {
foobar()++;
++this;
++(this);
(x()++);
(x) = 7++;
7++;
x = 7++;
foobar() = 7;
x = (7+=1);
(this) = 7;
this = 7;
x?y:z = t;
x + 3;
-8;
}
}
Every single line in the foo() method above is invalid! Javaparser parses the above up to and including the x=7++ line, and then doesn't accept the lines after that. The funny thing is that if you eyeball their Java grammar you end up seeing that there is nothing in the grammar that precludes it accepting foobar() = 7; as a valid statement. I think what happens is that it rejects that because it does a post-parsing tree walk to identify these things.
The reason that it accepts ++this; is that, for some reason, the logic of their post-parse tree-walk doesn't include the ++ and -- prefixes and suffixes. If you replace ++this with this = this + 1, which is, in principle, the same thing, and invalid for the same reason, it does complain. You get:
com.github.javaparser.ParseProblemException: (line 6,col 8) Illegal left hand side of an assignment.
at com.github.javaparser.JavaParserAdapter.handleResult(JavaParserAdapter.java:80)
at com.github.javaparser.JavaParserAdapter.parse(JavaParserAdapter.java:96)
at com.github.javaparser.StaticJavaParser.parse(StaticJavaParser.java:174)
at JPTest.parseFile(JPTest.java:31)
So, I assume that the post-parse checks are what that handleResult method does and it doesn't have the smarts incorporated into it to realize that ++this; is the same thing essentially as this = this+1; and thus, invalid for the same reason!
Well, there's a level of sloppiness in this that is somewhat shocking, but it is fixable. So, I mean, they parse in a loose way but then rely on a post-parse tree walk to address certain issues -- which is, in principle, okay, I suppose. The problem is that the actual implementation is obviously deficient!
With the main Java parser that comes from ANTLR (the only one of the four that seems to be usable) there just doesn't seem to be any awareness on the part of the author (and the author is the famous Terence Parr!) that this is a problem. You can see where this is right here. Surely TP knows that any expression followed by a semicolon is not a valid statement in Java! (Or does he?)
Actually, the most glaring, offensive thing in that grammar is here. This is part of the lexical grammar and it defines what is a valid character in an identifier in Java. Any unicode symbol over 0x7F is a letter and can be part of an identifier.
Maybe I shouldn't say what I am about to say, but... I look at that and it is impossible for me to take Terence Parr (or his acolytes) seriously. I mean, it's one thing if you say that everything beyond 7-bit ASCII can be in a Java identifier, and it's an initial kludge to get going and you know you're going to revisit this later, but that's not what's going on here. If you look at the history of that JavaLexer.g4 file, it goes back to July 2017, amost exactly 9 years ago, but the file was first committed as a result of splitting a previous Java.g4 into two files, JavaParser.g4 andJavaLexer.g4`, so this thing of everything beyond 0x7F being a "letter" has been there probably for far longer.
Now, at least Javaparser has a pretty good handle on what a Java identifier is. See here. Well, except that they don't specify all the various characters that are in extended unicode (beyond 0xFFFF) that are valid Java identifiers, but that is actually not their fault, because the tool they're using (legacy JavaCC) incorporates no understanding of extended unicode, so the 32-bit characters can't be specified. I grant that the vast majority of users, even of non-English scripts (since all the widely used symbols are below 0xFFFF) probably don't have much of an issue with this. But, regardless, there is a basic correctness issue here, though it's not as glaring as the ANTLR Java grammar, which just accepts all characters beyond 7-bit ascii as "letters" in an identifier. (I dunno... maybe I shouldn't say it. I find that really offensive. What a bunch of ****ing ass-clowns!)
So, what to say? I think one can draw the conclusion that Javaparser is more or less usable, while the Java parser on offer from the Antlr people obviously is not. Well, some people use it surely, but the thing reflects such a sloppy mentality that it is IMHO not something a serious person should opt for.
The biggest single problem, when doing the "bake-off" of Javaparser vs CongoCC is actually elsewhere. They use legacy JavaCC, but not JJTree (which is understandable because JJTree is borderline unusable!) But, as a result, their entire AST is made up of hand-coded classes and they build up the AST during parsing using hand-coded code actions. And then, for example, they have an entire hand-coded Node traversal API that their Java AST is part of. I mean, this entire package hierarchy here. There is something like 50,000 lines of code in there. All these classes like MethodDeclaration.java and so on and so forth. And. AFAICS, none of that API is reusable outside their project. At least, it's not designed to be AFAICS. It's just this hugely elaborate node API just for their internal use. With CongoCC, all of the equivalent code is just generated from templates. If you edit the grammar so that the tree has a different shape, all the java source code that makes up your AST just gets regenerated. Or, another way of looking at it is that if you use the Java parser, and then you decide to use the CSharp or Rust parser, the core API for traversing the tree is all the same! Or fundamentally it is. And, in fact, if you use the ROOT_API_PACKAGE option, the various AST nodes from different parser projects can share the same base API. But, again, the really fundamental point is that it's all generated! I mean to say, once you understand the implications of this, the question of the performance benchmarking and so on just becomes kind of moot. That's why keeping the Java parser up to date in the evolution from JDK 8 to JDK 26 was such a small time commitment for me. (Not because I'm some superman coder. Really, I'm not! It's just that I'm using a vastly more powerful tool.) If I had to maintain 50K LOC in 250-odd hand-maintained Java source files, it would be another story. And then I had to do the same thing for Python, CSharp, and now Rust... I mean to say, if I took their approach, this whole project would be totally untenable.
So, what it boils down to is that somehow, we're just not getting the real message across, and if our message was just "well, our parser is nearly 2x as fast as theirs", that's true and all, but it's soooo much less than where the real issues are. Well, I'll close this here now. I guess I'll write this up as blog post on parsers.org.
Well, I'd just add that one practical implication of all this situation IMHO is that there is surely a need for separate documentation, like a HOWTO on how to simply reuse the parsers, like the Java parser etcetera -- without hardly mentioning the parser generator itself. "There's a free Java parser here and you can use it and it's pretty easy to do so and..."
And I think what we need on the Rust front is to have a Rust parser in Rust that has injection and the lot. And we have to aggressively promote it in the Rust community. But, as regards the situation with the Java parser, there's not much excuse for not doing much more outreach/advocacy.
P.S. Oh, what I'd really like to know is what the performance of our Java parser is compared to the Java parser that is inside of javac in the JDK. That's all open source, but I never put in the energy to figure out how to run the Java parser inside of javac on its own -- without compilation. A similar question arises in terms of comparing our Rust parser with the Rust parser that is (must be!) inside of rustc. But, in particular, that one will be interesting once we have a solid Rust parser in Rust!