Skip to content

lexbor: Shrink the initial tokenizer pools and mraw cache - #23949

Open
nicolas-grekas wants to merge 1 commit into
php:masterfrom
nicolas-grekas:lexbor-small-pools
Open

nicolas-grekas wants to merge 1 commit into
php:masterfrom
nicolas-grekas:lexbor-small-pools

Conversation

@nicolas-grekas

@nicolas-grekas nicolas-grekas commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Dom\HTMLDocument::createFromString() and setting innerHTML allocate about 1.1MB on every call, even for a few bytes of input, because lexbor preallocates 4096 tokens, 4096 token attributes and, in each of the 7 lexbor_mraw_t of a parser, 512 entries for its cache of freed blocks. It's freed after the call, but it's enough to push a small request into a second 2MB memory chunk.

The tree builder uses a single token and these pools grow on demand, so this makes them start with 16 tokens, 256 attributes and 64 cache entries. The change is patch 0027. It's lexbor/lexbor#427, which isn't merged yet. The patch allows setting the sizes at build time, and PHP sets them in config.m4 and config.w32. So php-src doesn't depend on the defaults that upstream picks, and the patch can be replaced by the upstream commit once it's merged.

Parsing a small string then needs about 200KB instead of 1.1MB, and large documents parse as fast as before.

@nicolas-grekas

Copy link
Copy Markdown
Contributor Author

I moved the lexbor_array_obj_t fix to #25431. lexbor/lexbor#428 is merged upstream, and it's the bigger win. This PR is now only about the pool sizes, on top of #25431.

lexbor/lexbor#427 is still open. The maintainer likes the #ifndef form, and the default values are what's left to decide. The patch here now matches that form, and PHP sets the three sizes in its build. So php-src doesn't depend on the defaults that upstream picks, and the patch can be replaced once #427 is merged.

@kocsismate @ndossche WDYT? Parsing a small string needs about 200KB instead of 1.1MB.

Comment thread NEWS Resolved
@kocsismate

Copy link
Copy Markdown
Member

I agree that the unmerged PR warrants a custom fix on our side. @alexandre-daubois An FYI for you that this would be another patch which shouldn't be removed after a full lexbor upgrade.

That said, if no objection arrives by tomorrow, I'm going to merge this PR.

@kocsismate

Copy link
Copy Markdown
Member

@nicolas-grekas Another patch has just been added in #25320, so please rebase your MR

@nicolas-grekas

Copy link
Copy Markdown
Contributor Author

Rebased, together with #25431. The patch of #25431 is now 0026, and the one of this PR is 0027.

@kocsismate

Copy link
Copy Markdown
Member

Unfortunately, some test started to fail on 32 bit: https://github.com/php/php-src/actions/runs/43944344571/job/133570009208?pr=23949#step:16:145

@nicolas-grekas

Copy link
Copy Markdown
Contributor Author

Thanks, fixed. gh25305.phpt relied on the 1.1MB that lexbor preallocated to run out of memory. The test now sizes its string from the memory already in use, so that lxb_dom_document_init() fails whatever the build already uses. The test still crashes when the fix for GH-25305 is reverted.

The HTML tokenizer preallocated 4096 tokens and 4096 token attributes,
and each lexbor_mraw_t preallocated 512 entries for its cache of freed
blocks, 7 of them per parser. This made Dom\HTMLDocument::createFromString()
and setting innerHTML allocate about 1.1MB even for a few bytes of input.

The tree builder uses a single token and these pools grow on demand, so
start with 16 tokens, 256 attributes and 64 cache entries.

This is patch 0027, the pending lexbor/lexbor#427. It allows setting
the sizes at build time, which PHP does in config.m4 and config.w32.

gh25305.phpt relied on these preallocations to run out of memory in
lxb_dom_document_init(). It now sizes its string from the memory already
in use, so that every build has the same amount left.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants