I managed to benchmark the Java parser that is inside of javac -- i.e. measure how fast it parses without compilation. I can say one thing. The people who put that together were not thinking about how to make it easy for any third party to use it. Or at least it was not any sort of priority for them! In fact, I kept running against a wall until it finally occurred to me to ask Claude for help.
The first impedance I hit was just trying to compile anything against the jdk.compiler code. Initially, I kept running into:
StandaloneParse.java:1: error: package com.sun.tools.javac.api is not visible
import com.sun.tools.javac.api.JavacTool;
Eventually, after a bit of googling, I figured out that I needed to use --add-exports but even that I couldn't get working.
And that was only the beginning of the problems, like NPE's that I couldn't get to the bottom of. The first question I asked Claude was:
How do I use the package com.sun.tools.javac.parser when the module does not export the package?
It turned out that the magical command-line incantation I needed to run my little test harness was:
javac --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED \
StandaloneParse.java
Naturally, I asked Claude whether there was any way to do a wild-card import of all the packages in the jdk.compiler module and Claude said no, there wasn't.
There was various give-and-take and finally I managed to communicate to Claude what I wanted. I asked:
Do you know of any examples of using the Java parser inside of javac on its own -- without the compilation part, just parsing?
And pretty much immediately, Claude gave me the following boilerplate:
import com.sun.tools.javac.api.JavacTool;
import com.sun.tools.javac.file.JavacFileManager;
import com.sun.tools.javac.parser.JavacParser;
import com.sun.tools.javac.parser.ParserFactory;
import com.sun.tools.javac.tree.JCTree.JCCompilationUnit;
import com.sun.tools.javac.util.Context;
import com.sun.tools.javac.util.Log;
import javax.tools.JavaFileObject;
import java.io.*;
import java.nio.file.Path;
public class StandaloneParse {
public static void main(String[] args) throws Exception {
String source = """
package foo;
class Bar {
void baz() { System.out.println("hi"); }
}
""";
Context context = new Context();
// Sets up diagnostic reporting so parse errors don't just vanish
JavacFileManager.preRegister(context);
JavaFileObject fileObject = new SimpleSourceFileObject(source);
Log log = Log.instance(context);
log.useSource(fileObject);
ParserFactory parserFactory = ParserFactory.instance(context);
JavacParser parser = parserFactory.newParser(
source,
/* keepDocComments */ true,
/* keepEndPos */ true,
/* keepLineMap */ true
);
JCCompilationUnit tree = parser.parseCompilationUnit();
System.out.println(tree); // pretty-printed AST
}
// A minimal in-memory JavaFileObject wrapping a String
static class SimpleSourceFileObject extends javax.tools.SimpleJavaFileObject {
private final String code;
SimpleSourceFileObject(String code) {
super(java.net.URI.create("string:///Bar.java"), Kind.SOURCE);
this.code = code;
}
@Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return code;
}
}
}
and the magic incantation to compile it. I look at this and really wonder whether I could have gotten to this point without Claude. You know, sometimes, whether intentionally or unintentionally, they just make things so difficult that you're bound to give up!
But finally, using the above code from Claude as a starting point, I ended with this test harness.
import com.sun.tools.javac.api.JavacTool;
import com.sun.tools.javac.file.JavacFileManager;
import com.sun.tools.javac.parser.JavacParser;
import com.sun.tools.javac.parser.ParserFactory;
import com.sun.tools.javac.tree.JCTree.JCCompilationUnit;
import com.sun.tools.javac.util.Context;
import com.sun.tools.javac.util.Log;
import javax.tools.JavaFileObject;
import java.io.*;
import java.nio.file.FileSystems;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.stream.Stream;
import java.io.IOException;
/**
* A test harness to benchmark the Java parser inside of javac.
* The magic incantation to compile is:
javac --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED \
StandaloneParse.java
* and the magic incantation to run it is:
java --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED \
StandaloneParse files
* You can add the -p flag to have it run in multiple threads
* or the -q to have it run more quietly
* or the -r to have it retain the ASTs in memory, if you want to
* test memory usage.
*/
public class StandaloneParse {
int successes, failures;
boolean parallel, quiet, retainTrees;
ArrayList<Object> roots = new ArrayList<>();
public void main(String[] args) throws IOException {
long start = System.currentTimeMillis();
ArrayList<String> filenames = new ArrayList<>();
for (var arg : args) {
if (arg.charAt(0) == '-') {
if (arg.substring(1).startsWith("p")) parallel = true;
else if (arg.substring(1).startsWith("q")) quiet = true;
else if (arg.substring(1).startsWith("r")) retainTrees = true;
}
else filenames.add(arg);
}
if (parallel) IO.println("Parsing in multiple threads");
Stream<String> stream = parallel ? filenames.parallelStream() : filenames.stream();
stream.forEach(f->parse(f));
IO.println("Parsed " + successes + " files successfully.");
IO.println("Failed on " + failures + " files.");
IO.println("Duration: " + (System.currentTimeMillis() - start) + " milliseconds.");
}
public void parse(String filename) {
if (!quiet) IO.println("Parsing " + filename);
var path = FileSystems.getDefault().getPath(filename);
byte[] bb = null;
try {
bb = Files.readAllBytes(path);
} catch (IOException ioe) {
}
var source = new String(bb);
Context context = new Context();
// Sets up diagnostic reporting so parse errors don't just vanish
JavacFileManager.preRegister(context);
JavaFileObject fileObject = new SimpleSourceFileObject(source);
Log log = Log.instance(context);
log.useSource(fileObject);
ParserFactory parserFactory = ParserFactory.instance(context);
JavacParser parser = parserFactory.newParser(
source,
/* keepDocComments */ true,
/* keepEndPos */ true,
/* keepLineMap */ true
);
try {
JCCompilationUnit tree = parser.parseCompilationUnit();
++successes;
if (retainTrees) roots.add(tree);
} catch (Exception e) {
e.printStackTrace();
++failures;
}
}
// A minimal in-memory JavaFileObject wrapping a String
static class SimpleSourceFileObject extends javax.tools.SimpleJavaFileObject {
private final String code;
SimpleSourceFileObject(String code) {
super(java.net.URI.create("string:///Bar.java"), Kind.SOURCE);
this.code = code;
}
@Override
public CharSequence getCharContent(boolean ignoreEncodingErrors) {
return code;
}
}
}
You can compile this as above and run it with:
java --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED \
StandaloneParse files...
Well, it's easy enough to alias that and call it on the command line. Don't ask me why you have to repeat all the add-exports boilerplate when running it, when you presumably compiled with that. There could be a reason. Or that's just how it is...
Well, I'll leave you in suspense no longer. The hand-written Java parser inside of javac is approximately 3 times as fast as the Java parser that is inside of CongoCC. Though it depends on the exact code and so forth. One thing is that my test harness for this allows you to parse in multiple threads (or not) -- both when running the Java parser inside of CongoCC and the one inside of javac. As I said earlier, javac can compile the jdk.compiler module in a bit over 5 seconds. If you simply run the parsing test harness over that code, it takes less than a second. So, on the whole, it seems that javac spends at most about 20%, maybe a bit less, of its time parsing.
I had had the fairly consistent result that CongoCC's parser could parse Java source code in about half the time it took javac to compile. So, if (contrary to fact) javac was spending 50% of its time parsing, then the two parsers would have been at par. I tended to assume that the hand-coded parser was faster, but not by that much. I thought that the Java parser generated by CongoCC would be at least 50% as fast as the one inside javac. The truth is that it's probably in the 30% to 40% range.
Now, one thing to be clear about is that, given that javac only spends at most 20% of its time parsing, even a 3x slow-down in the parsing component would not slow down compilation by that much. The 5-second compilation time I mentioned above would become maybe 7 seconds.
My basic benchmark for a big parsing job is to parse all of the code in $JAVA_HOME/lib/src.zip. In fact, that is the main java test that is run when you run ant test. So, you could run:
java JParse ~/jdksrc/26/**/*.java
and that takes about 33 seconds on the machine on which I'm writing these lines. But if you run it multithreaded, i.e.
time java JParse -q -p ~/jdksrc/26/**/*.java
(It's useful to run the command with the UNIX time utility.) The latter command ends with:
Parsed 15457 files successfully
Failed on 0 files
Duration: 15226 milliseconds
java -classpath ~/projects/congo/examples/java JParse -p -q -s 104,88s user 1,06s system 686% cpu 15,432 total
It manages to use (nearly) 7 cores, i.e. 686% CPU use. The time elapsed is a bit over 15 seconds.
When I run the javac java parser test harness over the same code, as in:
time java --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED \
--add-exports jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED \
StandaloneParse -q -p ~/jdksrc/26/**/*.java
it outputs:
Parsing in multiple threads
Parsed 15457 files successfully.
Failed on 0 files.
Duration: 4644 milliseconds.
java --add-exports jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED 34,24s user 0,97s system 711% cpu 4,948 total
So it's also managing to employ 7 cores (711% CPU usage) and it takes a bit under 5 seconds. Well, the JDK 26 src.zip is a bit over 5 million lines of code. So in a multithreaded mode, it's hitting over a million lines a second. CongoCC's Java parser is more like 300K lines per second. (I previously mentioned a figure of about 100K lines a second, but that is running without the multithreading.)
Well, that's the result of my benchmark. Now, I would make certain points about all this:
- The CongoCC generated Java parser (and CongoCC generally) has not been optimized for speed. We've never put any energy into profiling and it may well be there is a fair bit of low-hanging fruit in terms of speeding it up. The truth is that the parsers that the tool generates are sufficiently fast that it has never been much of a priority.
- I have absolutely no idea what tricks the hand-coded parser may be doing in terms of memo-ization and such. The CongoCC parser is a very bloody-minded implementation of a recursive descent algorithm. If you know a lot about the input you are parsing (Java source code in this case) it may be possible to use that knowledge to optimize at key points, like you know that 90% of the time that a certain code structure occurs, it is followed by another one. Well, I'd have to look more closely.
- This is all kind of for fun. Almost nobody needs to parse Java source code any faster than this. But it might be fun to see how much extra speed can be squeezed out.
Of course, the big thing is maintainability. The Java parser from the CongoCC project -- along with the entire AST -- is all generated from a Java grammar that is less than 2K lines of code. (CongoCC code obviously.) The parser in javac comprises tens of thousands of lines of hand-written Java code. For example, if you eyeball this file or this package all of the equivalent code in the CongoCC parser/AST is just generated. Actually, to tell the truth, just looking at the javac source code makes me feel tired! Well, the Sun/Oracle insiders who wrote all that code are likely more skilled coders than I am, but can one seriously doubt, when browsing through all that code, that one is looking at the product of some man-years of work!?
Meanwhile, I recently worked up a grammar/parser for Rust and it took me about a month. And I was slowed down by the fact that I didn't even really know the language.
By the way, I forgot to mention that memory usage of the two parsers seems to be on about a par 1-1. Well, anyway, I was a bit disappointed that the Java parser from this project is about 3x slower than the one inside javac. I thought it would be a fair bit closer. Oh well. But still, the pragmatic case for the tool-generated parser looks pretty solid. The cost in time (and money!) in writing and maintaining a hand-written parser comparable to the one in com.sun.tools.* is probably such that anything less than a Fortune 500 company would balk at undertaking something like that, and even then... And I recently worked up that Rust grammar in a month, so...
Anyway, where all this is going is I need to put these various benchmarks and observations together into an article and submit it to some onlines e-zines. I think it's a good idea. It originates with Richard Cardone actually.