The only explanation I can think of is that your parser now matches at a point where it previously did not match.
In my example:
(<A>)? (TempB | TempC)
It used to fail because it did not generate a scan for tempB, and the input contained nothing that could be matched by the lexical state of A. But what if in my example, TempC would have had the same lexical state as A, and would have matched the input? Then it would have gone into TempC even though it should have preferred TempB.
Here is a (very contrived) example:
<DEFAULT,LS_A,LS_B> SKIP: " ";
<DEFAULT> TOKEN:
<UNUSED: "unused">
;
<LS_A> TOKEN:
<A: "a" >
;
<LS_B> TOKEN:
<B: "b" >
;
Start : LS_A :
FindABC <EOF>
;
FindABC :
(<A>)? (TempBB )? FindB
;
FindA :
<A>
;
TempBB :
FindBB
;
FindBB : LS_B :
<B><B>
;
TempB :
FindB
;
FindB : LS_B :
<B>(<B>)*
;
The intention is that the first two b's are matched separately by FindBB, while the rest of the b's are matched by FindB.
The old version would never match the first two b's together:
Dumping the AST...
<Start (1, 1)-(1, 8)>
<FindB (1, 1)-(1, 8)>
b
b
b
b
b
b
b
EOF
The new one does:
Dumping the AST...
<Start (1, 1)-(1, 8)>
<FindABC (1, 1)-(1, 8)>
<FindBB (1, 1)-(1, 2)>
b
b
<FindB (1, 3)-(1, 8)>
b
b
b
b
b
EOF