Document unlexer limitations?

[stevenjohnstone]

Awesome tool, really useful. Thanks!

I read the README and accompanying paper but failed to realise that random outputs may be chosen which would be rejected by a corresponding ANTLR generated lexer. The unlexer doesn't capture the rules implied by the ordering in the ANTLR lexer file.

Here's a simple example:

parser grammar ExampleParser;
options {
	tokenVocab = ExampleLexer;
}

session: command ARG EOF;

command: A | B;

lexer grammar ExampleLexer;

A: 'a';
B: 'b';

ARG: [a-z];

WS: [ \t\u000C\r\n]+ -> channel(HIDDEN);

Inputs such as a a and a b would not be accepted by the ANTLR generated lexer. The matching rules are such that the first rule (reading the lexer file from start to end) which matches is selected so an "ARG" can be any letter a-z as long as it doesn't match "A" or "B" i.e. [c-z]. "ExampleGenerator" has the following code showing that it'll generate an "ARG" in the range [a-z]:

    @depthcontrol
    def ARG(self, parent=None):
        current = UnlexerRule(name='ARG', parent=parent)
        self.enter_rule(current)
        UnlexerRule(src=self.model.charset(current, 0, self._charsets[1]), parent=current)
        self.exit_rule(current)
        return current
    ARG.min_depth = 0

...

 _charsets = {
        0: list(chain.from_iterable([range(32, 127)])),
        1: list(chain.from_iterable([range(97, 123)])),
        2: list(chain.from_iterable([range(9, 10), range(10, 11), range(12, 13), range(13, 14), range(32, 33)])),
    }

In a more realistic setting, fuzzing languages like Lua where variable names cannot be keywords like "or", "and" etc (and this is captured in the order of rules in the lexer) will require a few tweaks to avoid wasting time fuzzing uninteresting parts of the language runtime. I opted to override unlexer methods so that names are chosen from a pool which won't collide with reserved words.

The README and paper cover fixing the random outputs to meet semantic requirements really well. I noticed the above when I saw coverage in lexer error paths so maybe it's subtle enough to document? Could be that I'm being dense: happy either way.