ATTEMPT/RECOVER and Bang

adMartem

This might be something for Congo, not JavaCC21, but I thought I would ask now...
I'm investigating how to eliminate my legacy way of recovering from syntax errors while issuing error messages and such. In general the way it was done (and still is in the JavaCC21 version of the grammar) is with a try/catch block that catches the ParseException and then indicates the error with a meaningful message, followed by a "manual" re-sync hopefully to a known point in the source (consuming everything along the way). At that point another message is issued to declare that scanning has resumed, and the try/catch expansion ends successfully. It looks something like this:

UsingArgs :
    ATTEMPT
        ( <USING> Usings [ ReturningArg ] | ReturningArg [ <USING> Usings ] )     
    RECOVER {
        indicateError(ErrorCode.BAD_USING_PHRASE);
        skipToBefore(
            DECLARATIVES,                       /* try beginning of DECLARATIVES section */
            Tokens.sequence(COBOL_WORD,SECTION),/* try a section header */
            Tokens.sequence(LEVEL_66,DOT),      /* try a paragraph header */
            Tokens.sequence(LEVEL_77,DOT),
            Tokens.sequence(LEVEL_78,DOT),
            Tokens.sequence(LEVEL_88,DOT),
            Tokens.sequence(LEVEL_01,DOT),
            Tokens.sequence(CHILD_LEVEL_NUMBER,DOT),
            Tokens.sequence(INTEGER,DOT),
            Tokens.sequence(COBOL_WORD,DOT),
            ACCEPT,                             /* try a statement */
            ADD,
            ALTER,
            CALL,
            CANCEL,
            CLOSE,
            COMPUTE,
            CONTINUE,
            DELETE,
            DISPLAY,
            DIVIDE,
            EVALUATE,
            EXEC_SQL,
            EXIT,
            GOBACK,
            Tokens.sequence(GO,TO),
            GO,
            IF,
            INITIALIZE,
            INSPECT,
            MERGE,
            MOVE,
            MULTIPLY,
            OPEN,
            PERFORM,
            READ,
            RELEASE,
            RETURN,
            REWRITE,
            SEARCH,
            SET,
            SORT,
            START,
            STOP,
            STRING,
            SUBTRACT,
            UNLOCK,
            UNSTRING,
            WRITE,  
            Tokens.sequence(END,PROGRAM),       /* try the end of program */
            EOF);                               /* punt */
        printScanResumed();
    }
;

Note the fragile, labor-intensive way it has to "guess" at the follow set (at least approximately) in order to position itself correctly to resume parsing. Clearly it is desirable to replace all this with something using the Javacc21 (or Congo) feature of re-syncing in fault-tolerant mode. Something intuitively like:

UsingArgs :
    ATTEMPT
        <USING> Usings [  ReturningArg  ]  | ReturningArg  [ <USING> Usings  ]
    RECOVER (
        { indicateError(ErrorCode.BAD_USING_PHRASE); }
        ( <USING> Usings! [  ReturningArg  ]! |  ReturningArg!  [ <USING> Usings  ]! )
        { printScanResumed(); }
    )
;

This way, the error associated with the failing expansion can be issued, the expansion can be retried with fault-tolerant re-sync in effect, and the point at which normal parsing resumes can be identified.
[ I realize this thought experiment has issues right now, among which are the ones that the comments in the INJECT AttemptBlock call attention to regarding whether or not the first and follow sets should include the RECOVER expansion(s), if any. This is just to illustrate what I am thinking.

So I began by investigating this to see what was actually generated:

Sentence# :
  ATTEMPT    
    ( ( Statement )* SomeDots )  // MV quirk (empty sentence)    
  RECOVER (
    ( Statement )*! SomeDots  // Re-sync on Statement or dots
    { printScanResumed(); }
  ) 
;

Well, it seems to cause an exception generating the parser (lookahead I think). A simpler case that also causes the same error is:

X :
    FooOrBar
;

FooOrBar :
    ATTEMPT 
        "foo"
    RECOVER 
       ( "bar" )
;

It seems that an ExpansionSequence is expected, but an AttemptBlock is an Expansion.
So I think failure of the simple case above is a bug, and now I've revealed where I am headed with this, so any other guidance regarding that would be welcome.
BTW, this is the error:

Jan 13, 2023 4:30:39 PM freemarker.log.JDK14LoggerFactory$JDK14Logger error
SEVERE: java.lang.ClassCastException: com.javacc.parser.tree.AttemptBlock cannot be cast to com.javacc.core.ExpansionSequence

java.lang.ClassCastException: com.javacc.parser.tree.AttemptBlock cannot be cast to com.javacc.core.ExpansionSequence
----------
==> if grammar.choicePointExpansions?size !=0 [on line 39, column 5 in LookaheadRoutines.java.ftl]
 in user-directive LookaheadCode.Generate [on line 326, column 1 in Parser.java.ftl]
----------
```  

 ...snip ...

revusky

adMartem It seems that an ExpansionSequence is expected, but an AttemptBlock is an Expansion.
So I think failure of the simple case above is a bug,

Yes, it definitely is. (Or was.) There was a problem in how that method isAtChoicePoint was defined. It didn't take into account the possibility that the expansion was in an AttemptBlock.

That should be fixed now.

As for the rest of it, regarding whether the approach you are taking is right, well, I'd have to think about that a bit more.

The history of all this is that the ATTEMPT/RECOVER construct was the first bit of work on fault tolerance. And it was never really used or tested or anything. Practically any new feature that I introduced was typically used sooner or later (typically sooner) in internal development, so things like up-to-here, could never be really broken. Well, maybe in subtle ways or corner cases, but not really major league broken, because it is used everywhere throughout the codebase. However, that is not the case for any of this fault-tolerant machinery.

In general, whatever fault-tolerant machinery is in place is not really used/tested very much and I've also been thinking that after the Congo transition, fault-tolerant will be the default mode and used for the tool itself, so it should become much more robust and well-defined. But, anyway, the ATTEMPT/RECOVER emerges from a mind-set in which one is going to try to manually recover from errors, and the other stuff described here with the Bang to indicate resynch points, is really about trying to have automatic recovery. And that is really pretty much orthogonal to the ATTEMPT/RECOVER stuff.

Of course, the two things are not necessarily mutually exclusive either. The legacy try-catch, which is essentially the Java try-catch, is pretty close to hopeless, since the exception, in the general case anyway, is bubbling up from some deeply nested construct, so basically, you're trying to get back on the rails when you don't really even know where you are! At least with ATTEMPT/RECOVER, you start the "recovery" in what should be a well defined state, since it rewinds to where you were before the ATTEMPTed block.

But... you still have to manually recover somehow. The resynch point stuff with the BANG, I think that's more the way to go. But still, any attempt to use it will surely run into some glitches, since it isn't really very tested. But it's amazing to realize that I wrote that description of the thing just about 2 years ago and it's just been sitting there. But I think that getting this really ready for prime time is going to be one of the first things when we have the Congo rebranding done. The other thing about that is that I'm not promising that Congo is 100% backward compatible with JavaCC 21, just mostly.

But again, all that fault-tolerant stuff (despite the fact that I wrote that description 2 years ago) is really still very much a work-in-progress. So, most likely, when you try to use it, you'll find that there are things not quite right and so forth, but we'll just iteratively get it in better shape.

adMartem

Sounds like a plan. Thanks.