Repository navigation
lexbor: Shrink the initial tokenizer pools and mraw cache - #23949
nicolas-grekas wants to merge 1 commit into
Conversation
8f41929 to
7a758c4
Compare
846fe04 to
ad352ea
Compare
ad352ea to
d58660f
Compare
|
I moved the lexbor/lexbor#427 is still open. The maintainer likes the @kocsismate @ndossche WDYT? Parsing a small string needs about 200KB instead of 1.1MB. |
d58660f to
63cb639
Compare
|
I agree that the unmerged PR warrants a custom fix on our side. @alexandre-daubois An FYI for you that this would be another patch which shouldn't be removed after a full lexbor upgrade. That said, if no objection arrives by tomorrow, I'm going to merge this PR. |
|
@nicolas-grekas Another patch has just been added in #25320, so please rebase your MR |
63cb639 to
480553a
Compare
|
Unfortunately, some test started to fail on 32 bit: https://github.com/php/php-src/actions/runs/43944344571/job/133570009208?pr=23949#step:16:145 |
480553a to
5ed2284
Compare
|
Thanks, fixed. gh25305.phpt relied on the 1.1MB that lexbor preallocated to run out of memory. The test now sizes its string from the memory already in use, so that lxb_dom_document_init() fails whatever the build already uses. The test still crashes when the fix for GH-25305 is reverted. |
The HTML tokenizer preallocated 4096 tokens and 4096 token attributes, and each lexbor_mraw_t preallocated 512 entries for its cache of freed blocks, 7 of them per parser. This made Dom\HTMLDocument::createFromString() and setting innerHTML allocate about 1.1MB even for a few bytes of input. The tree builder uses a single token and these pools grow on demand, so start with 16 tokens, 256 attributes and 64 cache entries. This is patch 0027, the pending lexbor/lexbor#427. It allows setting the sizes at build time, which PHP does in config.m4 and config.w32. gh25305.phpt relied on these preallocations to run out of memory in lxb_dom_document_init(). It now sizes its string from the memory already in use, so that every build has the same amount left.
5ed2284 to
c780416
Compare
Dom\HTMLDocument::createFromString()and settinginnerHTMLallocate about 1.1MB on every call, even for a few bytes of input, because lexbor preallocates 4096 tokens, 4096 token attributes and, in each of the 7lexbor_mraw_tof a parser, 512 entries for its cache of freed blocks. It's freed after the call, but it's enough to push a small request into a second 2MB memory chunk.The tree builder uses a single token and these pools grow on demand, so this makes them start with 16 tokens, 256 attributes and 64 cache entries. The change is patch 0027. It's lexbor/lexbor#427, which isn't merged yet. The patch allows setting the sizes at build time, and PHP sets them in config.m4 and config.w32. So php-src doesn't depend on the defaults that upstream picks, and the patch can be replaced by the upstream commit once it's merged.
Parsing a small string then needs about 200KB instead of 1.1MB, and large documents parse as fast as before.