Repository navigation
fix(seo): canonical tags, apex sitemap and valid robots.txt (sysadmins-infra#654) - #561
Merged
Neophytis merged 4 commits intoOct 8, 2026
Merged
Conversation
…e/sysadmins-infra#654) Since the move to bewelcome.org (www now redirects to the apex) Google treats the apex as canonical, but the site still pointed at www: - sitemap.xml listed only www URLs (all redirect), 8 of them dead (/stats, /about/faq/0-6). Now 23 apex URLs, all 200, no stale lastmod. - robots.txt had a relative Sitemap line (invalid) and Crawl-delay 10. Now an absolute sitemap URL and no crawl delay. - No page had a canonical tag, so ?gclid= and other query strings were indexed as duplicates. Every full-page layout now includes _canonical.html.twig, built from the route so defaults normalise (/faq/1 and /faq give one URL), with a path-only fallback.
…ysadmins-infra#654) /signup and other legacy pages render through roxpage/page.php, which had no canonical. It now gets one that keeps the query string (legacy pages may need it) and only drops tracking parameters (utm_*, gclid, fbclid...). The Twig partial drops the path fallback: error pages render without a route, so they no longer get a canonical.
…/sysadmins-infra#654) templates/shared/roxpage/page.php is not loaded anywhere; legacy pages such as /signup render their head through tools/page/html.page.php. Move the canonical there and build it from baseuri, which already carries the right scheme and host.
…Welcome/sysadmins-infra#654) /about/statistics, /about/commentguidelines, /press-information and /feedback are listed in config/sitemap.yaml on develop (#476) and return 200; the static sitemap now has them too.
Contributor
Author
|
Compared with the sitemap generator on Why this PR still uses a static file for prod now
What develop gets right, and what this PR takes from it
Suggested follow-up (after the develop/docker-master consolidation, rox#518) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Target branch:
feature/docker-master(prod build source). Part of BeWelcome/sysadmins-infra#654.Why
Since the Leopard cutover on 19 Sep,
www.bewelcome.org301-redirects tobewelcome.org. Google switched its canonical to the apex on 24 Sep (URL Inspection confirms this for/,/signupand/about), but the site still pointed search engines at www:sitemap.xmllisted 26 www URLs, all redirecting. 8 were dead (/stats,/about/faq/0to/about/faq/6). Search Console: 26 submitted, 0 indexed.robots.txthad a relativeSitemap:line, which is invalid, andCrawl-delay: 10. Google ignores the crawl delay; Bing honours it.?gclid=and other tracking URLs were indexed as duplicates. One homepage gclid URL had 34k impressions.What changes
public/sitemap.xml: 23 apex URLs, all returning 200. Uses/faqplus the live FAQ categories. No stalelastmodorpriority.public/robots.txt:Sitemap: https://bewelcome.org/sitemap.xml, crawl delay removed.templates/_canonical.html.twig: a self-referencing canonical built from the route without the query string, so route defaults normalise (/faq/1and/faqgive one URL). Included in every full-page Twig layout:base,base-lite,home,login,searchmap,profile.base. Error pages have no route, so they get no canonical. The personal data export (mydata) is an offline file and is left out.tools/page/html.page.php: legacy pages such as/signupget a canonical fromPageWithHTML::head(). It keeps the query string, which legacy pages may need, and only drops tracking parameters (utm_*,gclid,gbraid,wbraid,fbclid,msclkid).Tested on stage (images
sha-180e94d)/and/?gclid=abc&utm_source=x//faqand/faq/1/faq/faq/7,/about,/login,/safety/tips,/terms/signupand/signup?gclid=1&utm_medium=x/signup/does-not-exist,/faq/3(404)/,/about,/login,/faq,/signup,/safety,/search/mapand/terms.bewelcome.org, because www and bewelcome.com are redirected at Traefik.Note on CI
Scan imagefails on every build today because of a new Alpine base-image CVE (libexpat2.8.5, CVE-2026-102633, MEDIUM, fixed in 2.9.0). It is unrelated to this change and will be tracked separately.After release
Submit
https://bewelcome.org/sitemap.xmlon thesc-domain:bewelcome.orgSearch Console property and remove the stale www sitemap entry (tracked in sysadmins-infra#654).