Skip to content

fix(seo): canonical tags, apex sitemap and valid robots.txt (sysadmins-infra#654) - #561

Merged
Neophytis merged 4 commits into
feature/docker-masterfrom
neophytis/654-seo-canonical-sitemap
Oct 8, 2026
Merged

Neophytis merged 4 commits into
feature/docker-masterfrom
neophytis/654-seo-canonical-sitemap

Conversation

@Neophytis

Copy link
Copy Markdown
Contributor

Target branch: feature/docker-master (prod build source). Part of BeWelcome/sysadmins-infra#654.

Why

Since the Leopard cutover on 19 Sep, www.bewelcome.org 301-redirects to bewelcome.org. Google switched its canonical to the apex on 24 Sep (URL Inspection confirms this for /, /signup and /about), but the site still pointed search engines at www:

  • sitemap.xml listed 26 www URLs, all redirecting. 8 were dead (/stats, /about/faq/0 to /about/faq/6). Search Console: 26 submitted, 0 indexed.
  • robots.txt had a relative Sitemap: line, which is invalid, and Crawl-delay: 10. Google ignores the crawl delay; Bing honours it.
  • No page had a canonical tag, so ?gclid= and other tracking URLs were indexed as duplicates. One homepage gclid URL had 34k impressions.

What changes

  • public/sitemap.xml: 23 apex URLs, all returning 200. Uses /faq plus the live FAQ categories. No stale lastmod or priority.
  • public/robots.txt: Sitemap: https://bewelcome.org/sitemap.xml, crawl delay removed.
  • templates/_canonical.html.twig: a self-referencing canonical built from the route without the query string, so route defaults normalise (/faq/1 and /faq give one URL). Included in every full-page Twig layout: base, base-lite, home, login, searchmap, profile.base. Error pages have no route, so they get no canonical. The personal data export (mydata) is an offline file and is left out.
  • tools/page/html.page.php: legacy pages such as /signup get a canonical from PageWithHTML::head(). It keeps the query string, which legacy pages may need, and only drops tracking parameters (utm_*, gclid, gbraid, wbraid, fbclid, msclkid).

Tested on stage (images sha-180e94d)

URL Canonical
/ and /?gclid=abc&utm_source=x /
/faq and /faq/1 /faq
/faq/7, /about, /login, /safety/tips, /terms self
/signup and /signup?gclid=1&utm_medium=x /signup
/does-not-exist, /faq/3 (404) none
  • Exactly one canonical on /, /about, /login, /faq, /signup, /safety, /search/map and /terms.
  • The stage deploy smoke suite passed.
  • On prod, the host is bewelcome.org, because www and bewelcome.com are redirected at Traefik.

Note on CI

Scan image fails on every build today because of a new Alpine base-image CVE (libexpat 2.8.5, CVE-2026-102633, MEDIUM, fixed in 2.9.0). It is unrelated to this change and will be tracked separately.

After release

Submit https://bewelcome.org/sitemap.xml on the sc-domain:bewelcome.org Search Console property and remove the stale www sitemap entry (tracked in sysadmins-infra#654).

…e/sysadmins-infra#654)

Since the move to bewelcome.org (www now redirects to the apex) Google
treats the apex as canonical, but the site still pointed at www:

- sitemap.xml listed only www URLs (all redirect), 8 of them dead
  (/stats, /about/faq/0-6). Now 23 apex URLs, all 200, no stale lastmod.
- robots.txt had a relative Sitemap line (invalid) and Crawl-delay 10.
  Now an absolute sitemap URL and no crawl delay.
- No page had a canonical tag, so ?gclid= and other query strings were
  indexed as duplicates. Every full-page layout now includes
  _canonical.html.twig, built from the route so defaults normalise
  (/faq/1 and /faq give one URL), with a path-only fallback.
…ysadmins-infra#654)

/signup and other legacy pages render through roxpage/page.php, which had
no canonical. It now gets one that keeps the query string (legacy pages
may need it) and only drops tracking parameters (utm_*, gclid, fbclid...).
The Twig partial drops the path fallback: error pages render without a
route, so they no longer get a canonical.
…/sysadmins-infra#654)

templates/shared/roxpage/page.php is not loaded anywhere; legacy pages
such as /signup render their head through tools/page/html.page.php. Move
the canonical there and build it from baseuri, which already carries the
right scheme and host.
…Welcome/sysadmins-infra#654)

/about/statistics, /about/commentguidelines, /press-information and
/feedback are listed in config/sitemap.yaml on develop (#476) and return
200; the static sitemap now has them too.
@Neophytis

Copy link
Copy Markdown
Contributor Author

Compared with the sitemap generator on develop (#476, bin/console sitemap:generate):

Why this PR still uses a static file for prod now

  • The generator is not on feature/docker-master, so prod doesn't have it.
  • Nothing runs it: no cron entry and no deploy step.
  • It can't work in the Docker setup as written. It writes public/sitemap.xml inside the PHP container at runtime, but the nginx image copies public/ from the PHP image at build time (COPY --from=bewelcome_php /srv/bewelcome/public), so nginx would keep serving the committed file. That file is the 2020 www one on develop too.
  • Its static list carries the same dead URLs: /stats (404) and /about/faq (302).
  • It sets lastmod to the generation time for every static page, so all pages look changed on every run. Google learns to ignore lastmod when it's always new.
  • robots.txt on develop still has the relative Sitemap: line.

What develop gets right, and what this PR takes from it

  • Four extra public pages (/about/statistics, /about/commentguidelines, /press-information, /feedback). All return 200 and are added here (27 URLs, all 200).
  • FAQ categories from the database, and newsletters per language (/newsletters/{id}/{lang}, which return 200). These should be dynamic, so they belong in the follow-up below rather than in a static list.

Suggested follow-up (after the develop/docker-master consolidation, rox#518)
Serve /sitemap.xml from a Symfony route that reuses the generator's collection logic: FAQ categories and newsletters, built with the request host, cached, with no fake lastmod. Then delete public/sitemap.xml and the command. That keeps it current without a cron or a shared volume between PHP and nginx.

@Neophytis
Neophytis merged commit 5368d83 into feature/docker-master Oct 8, 2026
3 of 11 checks passed
@Neophytis
Neophytis deleted the neophytis/654-seo-canonical-sitemap branch October 8, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant