You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Crawling 150 pages/minute: is this expected speed? #882
The test Wordpress site is very lightweight and 100/100 lighthouse even when server-rendered.
This system crawls only 150 pages/minute with the mysql daemon using ~1-2MB/s disk I/O (HDD).
For comparison, on the exact same system, the TinaCMS Wordpress plugin exports ~3,000 pages/minute into Markdown, ie 20x the speed with the same mysqld disk I/O.
I appreciate that composing a full page over the wordpress API has overhead compared to a text-only export into Markdown, but is 20x slower expected performance? If true, a 10,000-page site would take more than 1 hour to export every time global templates or pagination are changed. :((
As described, the system passes all WP2Static health checks and PHP limits have been removed. The 1-2 MB/s mysqld disk speed feels slow, even for a HDD and especially compared to node.js read/write, but not sure if this is relevant or if it can be improved.
Is there some kind of a known bottleneck during the crawling stage that I am missing?
eg the MySQL config or the API/plugin?
Activity
john-shaffer commentedon Oct 29, 2022
developbranch has a Crawl Concurrency option that speeds up crawling significantly.wp2static does most of its writes directly to files on disk, and not to mysql. There is a lot of room for optimization, though.