Skip to content

fix(site): serve real robots.txt, sitemap.xml and 404 page - #559

Merged
jamesconwaydev merged 1 commit into
mainfrom
fix/seo-robots-sitemap-404
Nov 18, 2026
Merged

jamesconwaydev merged 1 commit into
mainfrom
fix/seo-robots-sitemap-404

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Cloudflare Pages answered /robots.txt, /sitemap.xml and every unknown
path with the home page and a 200, so Google found no sitemap and saw
soft 404s. Search Console lists /, /how-it-works/ and /archived/ as
"Crawled - currently not indexed".

robots.txt points at a sitemap.xml built from the category list, and a
404.astro gives Pages a real 404.html to serve.

Cloudflare Pages answered /robots.txt, /sitemap.xml and every unknown
path with the home page and a 200, so Google found no sitemap and saw
soft 404s. Search Console lists /, /how-it-works/ and /archived/ as
"Crawled - currently not indexed".

robots.txt points at a sitemap.xml built from the category list, and a
404.astro gives Pages a real 404.html to serve.
@jamesconwaydev
jamesconwaydev merged commit 64c5c0b into main Nov 18, 2026
9 checks passed
@jamesconwaydev
jamesconwaydev deleted the fix/seo-robots-sitemap-404 branch November 18, 2026 04:32
@justtakingover-sudo

justtakingover-sudo commented Nov 18, 2026 via email

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants