JIT/non-JIT match difference with anchored regexes using (*SKIP)

[addisoncrump]

Discovered by #322.

The pattern A(*SKIP)A| (number of As on either side may be replaced by any non-zero amount of literals) inconsistently matches itself (and I think any string) in combination with the anchored flag being set:

sh-5.2$ xxd skip_crash 
00000000: 0000 0000 0000 0080 4128 2a53 4b49 5029  ........A(*SKIP)
00000010: 417c                                     A|
sh-5.2$ ./pcre2_fuzzer skip_crash 
Encountered failure while performing match errorcode comparison; context:
Pattern/sample string (hex encoded): 41282a534b495029417c
Compile options 80100000 never_backslash_c,anchored
Match options 00002000
Non-JIT'd operation emitted an error: no match (-1)
JIT'd operation did not emit an error.
1 matches discovered by JIT'd regex:
Match 0 (hex encoded): 

I suspect that in this case, the JIT code emitted for SKIP does not consider anchoring, but I haven't investigated further.

[PhilipHazel]

I think this might be a JIT bug. There is definitely something wrong, as shown by this test:

$ ./pcre2test -jit zz
PCRE2 version 10.43-DEV 2023-04-14 (8-bit)
/A(*SKIP)A|/anchored
A(*SKIP)A|=aftertext
0:
0+ (*SKIP)A|

The "aftertext" option causes pcre2test to output the text that follows the match and the output shows that it is matching an empty string following the first A, which must be wrong because the pattern is anchored.

[addisoncrump]

Potentially related case that the fuzzer spit out just now:

  re> /(*ACCEPT)*j/endanchored
data> j\=notempty
No match
data> j\=no_jit,notempty
 0: j

[zherczeg]

For me (*ACCEPT)* looks like a parse error. Maybe some perl mess.

  re> /(*ACCEPT)*j/B
------------------------------------------------------------------
        Bra
        Brazero
        SBra
        *ACCEPT
        KetRmax
        j
        Ket
        End
------------------------------------------------------------------

It looks like it is /(?:(*ACCEPT))*j/, and that should be valid.

[zherczeg]

Anyway, j cannot be matched since (*ACCEPT) stops matching the pattern, and the result is always an empty string here. With notempty the pattern never matches.

[PhilipHazel]

I think (ACCEPT) should provoke an error "qualifier does not follow a repeatable item" just like (SKIP) does, though oddly Perl does not give an error for these.

[zherczeg]

I agree. So there are two issues here: one is parser related, the other is execution (since /(?:(*ACCEPT))*j/ is valid, and generates the same code as /(*ACCEPT)*j/)

[addisoncrump]

Another case with THEN for the repertoire:

$ ./pcre2test -jit
PCRE2 version 10.43-DEV 2023-04-14 (8-bit)
  re> /(?<!(*THEN)a|)/
data> b
 0: 
data> b\=no_jit
No match

[addisoncrump]

Finally with PRUNE:

$ ~/git/pcre2/pcre2test -jit
PCRE2 version 10.43-DEV 2023-04-14 (8-bit)
  re> /.*(*PRUNE)a|/  
data> a
 0: 
data> a\=no_jit
No match

[PhilipHazel]

On checking the history, I found that quantifying (*ACCEPT) is explicitly allowed. The ChangeLog entry for 10.34 says this: "Allow (*ACCEPT) to be quantified, because an ungreedy quantifier with a zero minimum is potentially useful." Also, it seems that recent changes have brought JIT and the interpreter into agreement for the (*ACCEPT) example:

/(*ACCEPT)*j/endanchored
j=notempty
No match

/(*ACCEPT)*j/endanchored,jitverify
j=notempty
No match (JIT)

The (*PRUNE) example is also OK. (*THEN) is still an issue that seems to be an interpreter bug; I'm looking into it.

[PhilipHazel]

I think the interpreter is correct here:

/(?<!(*THEN)a|)/
b
No match

/(?<!(*THEN)a|)/jitverify
b
0: (JIT)

(*THEN) at the start of a branch is effectively a no-op. The lookbehind should match an empty string, but it's a negative lookbehind so it should therefore fail. And I can confirm that Perl agrees with the interpreter. So I think this is a JIT issue.

[zherczeg]

Fixed in e76eae6

Another victim of vreverse. I knew it will be painful.

[addisoncrump]

Fix is incomplete:

$ ./pcre2test -jit
PCRE2 version 10.43-DEV 2023-04-14 (8-bit)
  re> /A(*SKIP)\n|/anchored
data> AB\n
 0: 
data> AB\n\=no_jit
No match

[zherczeg]

I just fixed the last test, this is unrelated. Btw why do you combine skip with anchored? They do the opposite things.

[addisoncrump]

I am automatically looking for these cases with a fuzzer. Handling these edge cases now means that any future changes which cause JIT/interpreter to be inconsistent will automatically be flagged by OSS-Fuzz, making it effectively an automated correctness testing tool.