An update on this. I take back the part about no lexical state change dependency. I think they all do, but sometimes it is way down in the parser expansion tree below where the former simple token "contains" turned into a scan. There are around 50 more non-terminals in my parser that return a positive indication of starting with a lexical state change than formerly were there. That leads to a lot more scans, obviously. I note that of the example parsers, this change only produces 1 Python non-terminal that differs in that way, and about this many for CSharp:
non-terminals that return differing "startsWithLexicalChange" result: [NonNullableType, InListPattern, SwitchExpression, StatementExpression, NonTupleType, ConstantDeclaration, UnaryExpression, RangeExpression, WithExpression, AndPattern, ArrayType, ElementInitializer, AndExpression, AssignmentExpression, ExplicitAnonymousFunctionSignature, CompilationUnitBody, ConditionalAndExpression, MultiNonInterpolatedText, NonArrayType, EqualityExpression, MultiplicativeExpression, FormalParameters, AdditiveExpression, RelationalExpression, InclusiveOrExpression, ConditionalExpression, ExclusiveOrExpression, NullConditionalExpression, NullCoalescingExpression, ShiftExpression, ExplicitAnonymousFunctionParameter, SubPatterns, ArgumentList, MemberDeclarator, InterfaceMemberDeclaration, EmbeddedStatement, MemberInitializer, Expression, ExpressionStatement, OrPattern, ClassMemberDeclaration, NotPattern, InterfaceIndexerDeclaration, VariableInitializer, ConditionalOrExpression, PrimaryPattern]
I still don't know for sure why my parser is broken, presumably as a result of a predicate replacing a simple next token check. More as I delve further into this.