diff --git a/CHANGELOG.md b/CHANGELOG.md index 2f49d0d..17521e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,16 @@ All notable changes to this project are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- **link-health: stdlib Python link checker as the default.** `LINK_CHECKER=python` (the new default) uses the vendored `link-check.py`, so link-health needs only `python3` >= 3.8, `gh`, `jq`, and `bash` — no lychee install — and runs on a basic agent image. Egress-proxy refusals are classified as `unchecked`: they never open or auto-close issues, and the run summary, `latest.json`, and the automation-health dashboard report the unchecked count and top hosts. `LINK_CHECKER=lychee` remains an opt-in fallback, and `LINK_HEALTH_SHADOW_OUT` runs a checker side-by-side in shadow mode. (#3) + +### Changed + +- Bundled engine: link-health scanner, extractor, and automation-health dashboard updates from `rossoctl/automation`; `link-check.py` joins the vendored set (`scripts/vendor-engine.sh` extras). + ## [0.1.1] - 2026-11-05 ### Added diff --git a/scripts/vendor-engine.sh b/scripts/vendor-engine.sh index f22b82f..13b93fd 100644 --- a/scripts/vendor-engine.sh +++ b/scripts/vendor-engine.sh @@ -48,7 +48,7 @@ pr-review-scanner program-lib repo-sync repoman-check repoman-index \ repoman-requirements repoman-run repoman-setup weekly-report" # Non-shell engine files copied alongside the manifest (explicit, no glob). -EXTRA_FILES="doc-drift-check.py" +EXTRA_FILES="doc-drift-check.py link-check.py" # Pre-validate the WHOLE manifest before copying anything, so a source missing # late in the list cannot leave a half-vendored dest with a stale _vendor.json. diff --git a/skills/automation-health-dashboard/SKILL.md b/skills/automation-health-dashboard/SKILL.md index 52729b4..a4f2efb 100644 --- a/skills/automation-health-dashboard/SKILL.md +++ b/skills/automation-health-dashboard/SKILL.md @@ -86,7 +86,7 @@ bash scripts/automation-health-dashboard.sh --live --main-repo-dir /path/to/auto ## Dashboard Sections - **Executive Summary** — total issues created/resolved, PRs opened, estimated hours saved -- **Link Health** — broken links by type, trend table, cumulative issues +- **Link Health** — broken links by type, trend table, cumulative issues, unchecked (egress-refusal) count and top hosts - **Dependency Bumps** — stale PRs by tier, SLA compliance, median TTM, coverage - **Cross-Program Coverage** — which repos are under which programs - **Cron Health** — job schedules and last run status diff --git a/skills/link-health-scanner/SKILL.md b/skills/link-health-scanner/SKILL.md index 7cb1537..428a8ce 100644 --- a/skills/link-health-scanner/SKILL.md +++ b/skills/link-health-scanner/SKILL.md @@ -1,10 +1,12 @@ --- name: link-health-scanner description: >- - Scan all repos in a GitHub org for broken links using lychee, diff against - previous results, create GitHub issues for new findings, close issues for - fixed links, and write structured JSON reports. Use when you need to audit - link health across an organization's repositories. + Scan all repos in a GitHub org for broken links using the stdlib Python + link checker (no extra install), diff against previous results, create + GitHub issues for new findings, close issues for fixed links, and write + structured JSON reports. Egress-proxy refusals are reported as unchecked + and never open or auto-close issues. Use when you need to audit link + health across an organization's repositories. license: Complete terms in LICENSE.txt --- @@ -17,11 +19,29 @@ Scan all repositories in a GitHub organization for broken links. Creates GitHub ## Prerequisites - `bash` 3.2+ (works with the macOS system default; no bash-4 features used) -- `lychee` >= 0.23.0 (link checker: `brew install lychee`) — the scanner relies on - lychee's default exclusion of URLs inside code blocks; do not pass `--include-verbatim` +- `python3` >= 3.8 (the default link checker, `scripts/link-check.py`, is stdlib-only: + `urllib`, `concurrent.futures`, `html.parser`, `re` — nothing to install) - `gh` (GitHub CLI, authenticated with org access) - `jq` (JSON processor) +### Link checker + +- Default: `LINK_CHECKER=python` — the stdlib checker, so the program runs on a + basic agent image with no lychee install. +- Opt-in fallback: `LINK_CHECKER=lychee` (requires `lychee` >= 0.23.0 on PATH; + do not pass `--include-verbatim` — the scanner relies on excluding URLs + inside code blocks, which the python checker does by default too). +- **Unchecked (egress refusals):** links the egress proxy refused to fetch are + reported with status `unchecked`, not broken. No issue is opened for them, + and a previously broken link that comes back unchecked is never auto-closed. + Widen the egress allowlist to turn them into real results. +- **Shadow mode:** set `LINK_HEALTH_SHADOW_OUT=` to write a side report + without touching the production baseline (it forces dry-run), e.g. to run a + candidate checker side-by-side with the production one. +- Per-repo `.lychee.toml` is honored when present; the python checker implements + the `exclude`, `exclude_path`, and `accept` keys (including ranges like + `"200..=204"`) and warns about and ignores any other key. + ### Requirements (machine-readable) - pat_scopes: [repo] @@ -87,11 +107,13 @@ Use `--issue-limit` to cap issue creation, especially on first runs. The scanner prints: - Repos scanned and any failures - Total links checked and broken link count (internal vs external) +- Unchecked count (egress refusals) and, when non-zero, the top unchecked hosts - Delta: new broken links since last scan, fixed links, recurring - Issues created/closed Reports written: -- `$REPORTS_DIR/latest.json` -- full scan results +- `$REPORTS_DIR/latest.json` -- full scan results (including `unchecked_count` + and `unchecked_top_hosts`) - `$REPORTS_DIR/history.json` -- append-only trend data ## Safety diff --git a/skills/repoman/scripts/_vendor.json b/skills/repoman/scripts/_vendor.json index 7808014..2ce43e6 100644 --- a/skills/repoman/scripts/_vendor.json +++ b/skills/repoman/scripts/_vendor.json @@ -1,7 +1,7 @@ { "source_repo": "rossoctl/automation", - "source_sha": "fa61909029207b08d17eff5625bd0aa4772c90b2", - "vendored_at": "2026-11-03T22:56:54Z", + "source_sha": "4f284d0d65712fce2309b6f5911d481d5bc45351", + "vendored_at": "2026-11-11T19:04:00Z", "files": [ "automation-health-dashboard.sh", "core.sh", @@ -29,6 +29,7 @@ "repoman-run.sh", "repoman-setup.sh", "weekly-report.sh", - "doc-drift-check.py" + "doc-drift-check.py", + "link-check.py" ] } diff --git a/skills/repoman/scripts/automation-health-dashboard.sh b/skills/repoman/scripts/automation-health-dashboard.sh index 5385b8d..8d0659e 100644 --- a/skills/repoman/scripts/automation-health-dashboard.sh +++ b/skills/repoman/scripts/automation-health-dashboard.sh @@ -291,6 +291,8 @@ LH_REPOS_SCANNED=0 LH_TOTAL_LINKS=0 LH_BROKEN_INTERNAL=0 LH_BROKEN_EXTERNAL=0 +LH_UNCHECKED_COUNT=0 +LH_UNCHECKED_TOP_HOSTS="(none)" LH_FIRST_INTERNAL=0 LH_FIRST_EXTERNAL=0 LH_TREND_TABLE="| - | - | - | - |" @@ -300,6 +302,13 @@ if [ "$HAS_LINK_HEALTH" = true ]; then LH_TOTAL_LINKS=$(jq '.total_links_checked // 0' "$LINK_SCAN_DIR/latest.json") LH_BROKEN_INTERNAL=$(jq '[.broken[] | select(.category == "internal")] | length' "$LINK_SCAN_DIR/latest.json" 2>/dev/null || echo "0") LH_BROKEN_EXTERNAL=$(jq '[.broken[] | select(.category == "external")] | length' "$LINK_SCAN_DIR/latest.json" 2>/dev/null || echo "0") + LH_UNCHECKED_COUNT=$(jq '.unchecked_count // 0' "$LINK_SCAN_DIR/latest.json" 2>/dev/null || echo "0") + LH_UNCHECKED_TOP_HOSTS=$(jq -r ' + (.unchecked_top_hosts // []) as $h | + if ($h | length) == 0 then "(none)" + else ($h | map("\(.host) (\(.count))") | join(", ")) + end + ' "$LINK_SCAN_DIR/latest.json" 2>/dev/null || echo "(none)") # First scan values for trend comparison LH_FIRST_INTERNAL=$(jq '.[0].broken_internal // 0' "$LINK_SCAN_DIR/history.json") @@ -526,6 +535,8 @@ if [ "$HAS_LINK_HEALTH" = true ]; then | Total links checked | $LH_TOTAL_LINKS | | | Broken (internal) | $LH_BROKEN_INTERNAL | $lh_int_trend | | Broken (external) | $LH_BROKEN_EXTERNAL | $lh_ext_trend | +| Unchecked (egress refusals) | $LH_UNCHECKED_COUNT | | +| Unchecked top hosts | $LH_UNCHECKED_TOP_HOSTS | | | Issues created (cumulative) | $LH_ISSUES_CREATED | | | Issues resolved (cumulative) | $LH_ISSUES_CLOSED | | diff --git a/skills/repoman/scripts/dep-bump-fixer.sh b/skills/repoman/scripts/dep-bump-fixer.sh index 606aabe..9bb80df 100644 --- a/skills/repoman/scripts/dep-bump-fixer.sh +++ b/skills/repoman/scripts/dep-bump-fixer.sh @@ -70,16 +70,18 @@ SCAN_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ") MAX_HISTORY_ROWS=500 # Provenance footer appended to every fixer comment/issue. Resolved once at # startup: the helper lazily writes _meta..json beside this skill and returns the -# pinned-SHA RepoMan attribution (blob/main fallback when unresolved). +# pinned-SHA attribution (linkless fallback when unresolved). SKILL_FOOTER=$(skill_attribution "dep-bump-fixer") -# Stable dedup key: the substring every footer form shares, regardless of the -# per-run SHA (both "...RepoMan dep-bump-fixer@..." and the blob/main -# fallback contain it). The dedup search (below) looks for THIS, not the whole -# footer, so a changing SHA never defeats dedup. Comments carrying the former -# marker value are no longer recognized, so an already-analyzed PR may receive -# one duplicate comment on the next run after the marker changed — acceptable, -# one-time. -FIXER_SIGNATURE="RepoMan dep-bump-fixer" +# Stable dedup key: a regex (jq `test`, below) anchored to the footer so it +# matches a fixer comment but not a human comment that merely mentions the tool. +# It matches all three footer forms, regardless of the per-run SHA: the resolved +# footer ("...by repoman [dep-bump-fixer@](...)"), the linkless fallback +# ("...by repoman dep-bump-fixer."), and the legacy "...by RepoMan dep-bump-fixer@" +# form (so comments posted before the format flip keep deduping). A bare mention +# like "dep-bump-fixer skipped this one" or a link to skills/dep-bump-fixer/ is +# NOT anchored to "by (repoman|RepoMan)", so it never false-positives into +# suppressing analysis. +FIXER_SIGNATURE="by (repoman \[?|RepoMan )dep-bump-fixer" # --- Workspace setup --- setup_workspace "dep-bump-fixer" @@ -257,16 +259,19 @@ while IFS= read -r issue_record; do continue fi - # Check if fixer already commented on this PR - # $sig below is a jq variable passed via --arg, not a shell expansion - # shellcheck disable=SC2016 - already_commented=$(gh_with_backoff pr view "$pr_number" --repo "$issue_repo" \ - --json comments --jq --arg sig "$FIXER_SIGNATURE" \ - '[.comments[] | select(.body | contains($sig))] | length' 2>/dev/null || echo "0") + # Check if fixer already commented on this PR. The signature is a regex bound + # via a real jq --arg inside the helper — `gh pr view --jq` cannot take --arg. + # The helper returns -1 (UNKNOWN) when the fetch fails; skip on anything but a + # hard 0 so a transient error never re-posts a duplicate comment. + already_commented=$(fixer_comment_count "$issue_repo" "$pr_number" "$FIXER_SIGNATURE") - if [ "$already_commented" -gt 0 ]; then + if [ "$already_commented" != "0" ]; then if [ "$VERBOSE" = true ]; then - echo " Already commented, skipping" + if [ "$already_commented" = "-1" ]; then + echo " Comment-status check failed, skipping to avoid a duplicate" + else + echo " Already commented, skipping" + fi fi SKIPPED_ALREADY_COMMENTED=$((SKIPPED_ALREADY_COMMENTED + 1)) continue diff --git a/skills/repoman/scripts/dep-bump-scanner.sh b/skills/repoman/scripts/dep-bump-scanner.sh index ab12542..cc8c4fc 100644 --- a/skills/repoman/scripts/dep-bump-scanner.sh +++ b/skills/repoman/scripts/dep-bump-scanner.sh @@ -74,7 +74,7 @@ ESCALATION_THRESHOLD=5 # Provenance footer for the issue bodies this scanner creates. Resolved once at # startup: the helper lazily writes _meta..json beside this skill and returns the -# pinned-SHA RepoMan attribution (blob/main fallback when the SHA is unresolved). +# pinned-SHA attribution (linkless fallback when the SHA is unresolved). SKILL_FOOTER=$(skill_attribution "dep-bump-scanner") # SLA thresholds (days) diff --git a/skills/repoman/scripts/doc-drift-precheck.sh b/skills/repoman/scripts/doc-drift-precheck.sh index e529801..af356da 100755 --- a/skills/repoman/scripts/doc-drift-precheck.sh +++ b/skills/repoman/scripts/doc-drift-precheck.sh @@ -22,7 +22,9 @@ while [ $# -gt 0 ]; do Usage: doc-drift-precheck.sh [--repo owner/name] [--worklist /path] Run the deterministic doc-drift checker. Exit 1 when the worklist is empty; -exit 0 and print "worklist: " when it is not. Fetch failures are +exit 0 and print "worklist: " when it is not. The printed path persists +after the run. --worklist overrides the default of +${REPORTS_DIR:-$HOME/reports/doc-drift}/worklist.json. Fetch failures are fail-open (exit 2) so a broken gate does not block the run. EOF exit 0 @@ -39,8 +41,13 @@ if [ ! -f "$CHECK_PY" ]; then fi setup_workspace "doc-drift-precheck" +# Default the worklist to a PERSISTENT path, not the per-run temp dir: setup_workspace +# registers an EXIT trap that deletes $PROGRAM_TMPDIR, so a temp default would hand the +# caller a path to a file that is already gone (#130). Mirror the scanner's REPORTS_DIR. if [ -z "$WORKLIST" ]; then - WORKLIST="$PROGRAM_TMPDIR/worklist.json" + REPORTS_DIR="${REPORTS_DIR:-$HOME/reports/doc-drift}" + mkdir -p "$REPORTS_DIR" + WORKLIST="$REPORTS_DIR/worklist.json" fi ENROLLED=$(repoman_load_enrolled) || exit 2 diff --git a/skills/repoman/scripts/extract-broken-links.sh b/skills/repoman/scripts/extract-broken-links.sh index ab1cbb0..94cdf02 100755 --- a/skills/repoman/scripts/extract-broken-links.sh +++ b/skills/repoman/scripts/extract-broken-links.sh @@ -8,7 +8,7 @@ set -euo pipefail # hostnames (Kubernetes cluster-local .svc names, *.local, RFC1918 ranges) are # suppressed, since an external scanner can never reach them. lychee status is # normalized to enum tokens: numeric HTTP codes stay as-is; text statuses map -# to timeout / dns / unreachable / error / unknown. +# to timeout / dns / unreachable / unchecked / error / unknown. # # This logic was previously inline in link-health-scanner.sh; it is extracted # here so it can be unit-tested against synthetic lychee fixtures. @@ -53,9 +53,11 @@ if [ "$LYCHEE_INPUT" != "-" ] && [ ! -s "$LYCHEE_INPUT" ]; then fi # --- Extract broken links from error_map (skip non-URL entries like "Error building URL") --- -# Normalize lychee status to enum tokens: numeric codes stay as-is, -# text statuses map to: timeout, dns, unreachable, error, unknown. +# Normalize lychee/link-check status to enum tokens: numeric codes stay as-is, +# text statuses map to: timeout, dns, unreachable, unchecked, error, unknown. # Suppress URLs with unreachable-by-design hostnames (cluster-local, .local, RFC1918). +# "unchecked" (egress-proxy refusal) must match before the error catch-all so +# the scanner can keep those out of the issue open/close path. jq -r --arg repo "$REPO_FULL" --arg repos_dir "$REPOS_PREFIX" ' .error_map // {} | to_entries[] | .key as $filepath | @@ -73,6 +75,7 @@ jq -r --arg repo "$REPO_FULL" --arg repos_dir "$REPOS_PREFIX" ' if $raw_status == null then "unknown" elif ($raw_status | type) == "number" then ($raw_status | tostring) elif ($raw_status | test("^[0-9]{3}$")) then $raw_status + elif ($raw_status | ascii_downcase) == "unchecked" then "unchecked" elif ($raw_status | ascii_downcase | test("timeout")) then "timeout" elif ($raw_status | ascii_downcase | test("resolve|dns")) then "dns" elif ($raw_status | ascii_downcase | test("refused|reset|closed|unreachable|connect")) then "unreachable" diff --git a/skills/repoman/scripts/github-api.sh b/skills/repoman/scripts/github-api.sh index fc855eb..712d01d 100644 --- a/skills/repoman/scripts/github-api.sh +++ b/skills/repoman/scripts/github-api.sh @@ -50,6 +50,41 @@ gh_with_backoff() { return 1 } +# Count the PR comments whose body matches a signature regex — the fixer's +# dedup check, so it does not re-comment on a PR it already commented on. +# +# `gh pr view --jq` accepts a single jq expression and has no `--arg`, so the +# signature must be bound with a REAL jq on a pipe, not inside gh's --jq. We +# fetch the comments JSON (no --jq) and pipe to `jq --arg sig … test($sig)`. +# +# A fetch/parse failure returns -1 (UNKNOWN), kept distinct from 0 (no matching +# comment) so a transient gh error does not read as "not commented yet" and make +# the caller re-post a duplicate. The caller treats any non-zero result as +# "skip": only a hard 0 proves it is safe to comment. +# +# Usage: +# n=$(fixer_comment_count "kagenti/adk" 42 "$FIXER_SIGNATURE") +# [ "$n" = "0" ] || echo "skip (already commented, or status unknown)" +# +# Args: +# $1 - full repo name (e.g., "kagenti/adk") +# $2 - PR number +# $3 - jq regex matched against each comment body (test()) +# Prints: the match count (>=0), or -1 if the fetch/parse failed +fixer_comment_count() { + local repo="$1" pr_number="$2" sig="$3" comments count + if ! comments=$(gh_with_backoff pr view "$pr_number" --repo "$repo" --json comments 2>/dev/null); then + echo "-1" + return 0 + fi + if ! count=$(printf '%s' "$comments" \ + | jq -r --arg sig "$sig" '[.comments[] | select(.body | test($sig))] | length' 2>/dev/null); then + echo "-1" + return 0 + fi + echo "$count" +} + # ============================================================================= # GITHUB ISSUES # ============================================================================= @@ -201,7 +236,7 @@ issue_has_open_pr() { # Canonical skill source repo. Every attribution footer points here (the repo # where skills are canonical), NOT at the automation repo that happens to host # this library copy. -SKILL_SOURCE_REPO="${SKILL_SOURCE_REPO:-rossoctl/agent-skills}" +SKILL_SOURCE_REPO="${SKILL_SOURCE_REPO:-rossoctl/repoman}" # Resolve a skill's pinned commit SHA, writing a per-skill _meta file beside the # skill on first success (create-only, self-healing). @@ -217,7 +252,7 @@ SKILL_SOURCE_REPO="${SKILL_SOURCE_REPO:-rossoctl/agent-skills}" # canonical source repo's commit history for that skill's path. On success the # meta file is written and the SHA echoed; on failure (offline, unauthorized, # rate limited) NOTHING is written -- the function warns to stderr and echoes -# empty, so the caller falls back to the blob/main attribution and the next run +# empty, so the caller falls back to the linkless attribution and the next run # retries. It must never write a sentinel such as {"version":"unknown"}: a # present-but-bogus file would stop the retry and pin a wrong version forever. # @@ -237,16 +272,28 @@ resolve_skill_meta() { # authoritative, never overwritten (refreshing on skill update is the deploy # layer's job). But a present-yet-corrupt file (truncated write, hand-edit, # not JSON) yields no .version -- treating that as authoritative would pin the - # skill to the blob/main fallback forever with no self-heal. So we only short- + # skill to the linkless fallback forever with no self-heal. So we only short- # circuit on a readable version; a corrupt file falls through to re-resolution, # which overwrites it on success, honoring the spec's self-heal invariant. if [ -f "$meta_file" ]; then sha=$(jq -r '.version // empty' "$meta_file" 2>/dev/null) - if [ -n "$sha" ]; then + local stored_source + stored_source=$(jq -r '.source // empty' "$meta_file" 2>/dev/null) + if [ -n "$sha" ] && [ "$stored_source" = "$SKILL_SOURCE_REPO" ]; then printf '%s' "$sha" return 0 fi - echo "WARN: $meta_file exists but carries no usable .version (corrupt?); re-resolving provenance." >&2 + # A readable .version whose .source no longer matches the current + # SKILL_SOURCE_REPO is STALE (the default flipped underneath it): the pinned + # SHA belongs to the old source and would 404 under the new one. Treat it like + # a corrupt file -- fall through to re-resolution, which rewrites both version + # and source on success (automation#131 — the default flip must not strand + # existing installs on a dead link). + if [ -n "$sha" ]; then + echo "WARN: $meta_file source '$stored_source' != '$SKILL_SOURCE_REPO'; re-resolving provenance." >&2 + else + echo "WARN: $meta_file exists but carries no usable .version (corrupt?); re-resolving provenance." >&2 + fi fi # Resolve the pinned SHA from the canonical source repo's history for this @@ -256,7 +303,7 @@ resolve_skill_meta() { --jq '.[0].sha' 2>/dev/null) if [ -z "$sha" ] || [ "$sha" = "null" ]; then - echo "WARN: could not resolve provenance SHA for skill '$skill_name'; using blob/main attribution (will retry next run)." >&2 + echo "WARN: could not resolve provenance SHA for skill '$skill_name'; using linkless attribution (will retry next run)." >&2 printf '' return 0 fi @@ -287,14 +334,16 @@ resolve_skill_meta() { # # Resolves the skill's pinned SHA (via resolve_skill_meta, using the caller's # own SCRIPT_DIR as the skill directory so it works on every deploy layout) and -# returns the pinned-SHA RepoMan footer. On resolution failure it returns the -# blob/main fallback that pins to the live default branch instead -- additive, -# so a working scan never depends on a reachable GitHub API. +# returns a lowercase markdown footer: 'Generated by repoman [@](blob/)'. +# On resolution failure it returns a LINKLESS footer ('Generated by repoman ') +# rather than an unverified link -- additive, so a working scan never depends on a +# reachable GitHub API. # # Usage: footer=$(skill_attribution "link-health-scanner") # Args: # $1 - skill name (also the path segment under skills/) -# Prints: a one-line "Generated by RepoMan ..." attribution footer. +# Prints: a one-line "Generated by repoman ..." attribution footer (markdown link +# when a SHA resolves, linkless otherwise). skill_attribution() { local skill_name="$1" # REPOMAN_SKILL_DIR lets tests pin _meta outside the shared scripts dir. @@ -303,10 +352,12 @@ skill_attribution() { sha=$(resolve_skill_meta "$skill_dir" "$skill_name") if [ -n "$sha" ]; then - printf 'Generated by RepoMan %s@%s (https://github.com/%s/blob/%s/skills/%s/SKILL.md).' \ + printf 'Generated by repoman [%s@%s](https://github.com/%s/blob/%s/skills/%s/SKILL.md).' \ "$skill_name" "${sha:0:7}" "$SKILL_SOURCE_REPO" "$sha" "$skill_name" else - printf 'Generated by RepoMan %s (https://github.com/%s/blob/main/skills/%s/SKILL.md).' \ - "$skill_name" "$SKILL_SOURCE_REPO" "$skill_name" + # No SHA resolved: emit a LINKLESS footer rather than an unverified blob/main + # link that may 404 (a skill absent from the source repo — the doc-drift case + # that opened automation#131, item 4). Attribution without a dead link. + printf 'Generated by repoman %s.' "$skill_name" fi } diff --git a/skills/repoman/scripts/link-check.py b/skills/repoman/scripts/link-check.py new file mode 100755 index 0000000..07ee09c --- /dev/null +++ b/skills/repoman/scripts/link-check.py @@ -0,0 +1,1017 @@ +#!/usr/bin/env python3 +"""Stdlib link checker that emits lychee-compatible JSON. + +Used by link-health-scanner.sh when LINK_CHECKER=python. Accepts the same +scanner argv previously passed to lychee so the call site stays one branch. + +Python 3.8+; stdlib only (urllib, concurrent.futures, html.parser, re, +ipaddress, tomllib when available). +""" +from __future__ import annotations + +import argparse +import concurrent.futures +import datetime +import email.utils +import functools +import html.parser +import ipaddress +import json +import os +import re +import socket +import ssl +import sys +import time +import urllib.error +import urllib.parse +import urllib.request +from collections import defaultdict +from pathlib import Path +from typing import Dict, Iterable, List, Optional, Set, Tuple + +USER_AGENT = "RepoManLinkChecker/1.0 (+https://github.com/rossoctl/repoman)" +MAX_REDIRECTS = 10 +PER_HOST_CAP = 2 +RETRY_AFTER_CAP_SECS = 30 +SCAN_EXTENSIONS = {".md", ".markdown", ".html", ".htm"} + +# --- TOML subset (exclude / exclude_path / accept) --------------------------------- + +def _strip_toml_comment(line: str) -> str: + """Cut the line at the first '#' outside a quoted string. + + A '#' inside an exclude regex (e.g. "...page#section") is a literal + character, not a comment. Escaped quotes (\\") inside basic strings do + not close the string. + """ + quote = None + i = 0 + while i < len(line): + ch = line[i] + if quote == '"': + if ch == "\\": + i += 2 + continue + if ch == '"': + quote = None + elif quote == "'": + if ch == "'": + quote = None + elif ch in ('"', "'"): + quote = ch + elif ch == "#": + return line[:i] + i += 1 + return line + + +def _bracket_depth(line: str) -> int: + """Net '[' / ']' depth of a line, ignoring brackets inside quotes. + + Escaped quotes (\\") inside basic strings do not close the string, so a + ']' after one is not counted as closing the array. + """ + depth = 0 + quote = None + i = 0 + while i < len(line): + ch = line[i] + if quote == '"': + if ch == "\\": + i += 2 + continue + if ch == '"': + quote = None + elif quote == "'": + if ch == "'": + quote = None + else: + if ch in ('"', "'"): + quote = ch + elif ch == "[": + depth += 1 + elif ch == "]": + depth -= 1 + i += 1 + return depth + + +_TOML_SIMPLE_ESCAPES = { + "b": "\b", "t": "\t", "n": "\n", "f": "\f", "r": "\r", + '"': '"', "\\": "\\", "/": "/", +} + + +def _decode_toml_basic(s: str) -> str: + """Decode TOML basic-string escapes ("..." strings; '...' are literal). + + tomllib decodes these, so the fallback must too or exclusion regexes + like "foo\\\\.net" silently stop matching on Python < 3.11. Unknown + escapes are kept literally (best effort for regex content). + """ + out: List[str] = [] + i = 0 + while i < len(s): + ch = s[i] + if ch != "\\" or i + 1 >= len(s): + out.append(ch) + i += 1 + continue + nxt = s[i + 1] + if nxt in _TOML_SIMPLE_ESCAPES: + out.append(_TOML_SIMPLE_ESCAPES[nxt]) + i += 2 + elif nxt in ("u", "U"): + width = 4 if nxt == "u" else 8 + hexdigits = s[i + 2 : i + 2 + width] + try: + out.append(chr(int(hexdigits, 16))) + i += 2 + width + except ValueError: + out.append(ch) + i += 1 + else: + out.append(ch) + i += 1 + return "".join(out) + + +# Array items: double-quoted basic strings (escape-aware), single-quoted +# literal strings, or integers. Group index — not capture content — decides +# the token type, so an empty string ("") is never mistaken for a number. +_TOML_ITEM_RE = re.compile(r'"((?:[^"\\]|\\.)*)"|\'([^\']*)\'|(-?\d+)') + + +def _parse_toml_subset(text: str) -> Dict[str, object]: + """Minimal parser for .lychee.toml keys we honor, when tomllib is absent.""" + result: Dict[str, object] = {} + # Join multi-line arrays into one logical line (the common .lychee.toml + # formatting) before parsing. + logical: List[str] = [] + buf = "" + for raw in text.splitlines(): + line = _strip_toml_comment(raw).strip() + if not line: + continue + buf = f"{buf} {line}" if buf else line + if _bracket_depth(buf) > 0: + continue # array continues on the next line + logical.append(buf) + buf = "" + if buf: + logical.append(buf) + for line in logical: + if "=" not in line: + continue + key, _, rest = line.partition("=") + key = key.strip() + rest = rest.strip() + if key not in ("exclude", "exclude_path", "accept"): + continue + if rest.startswith("[") and rest.endswith("]"): + inner = rest[1:-1] + items: List[object] = [] + for m in _TOML_ITEM_RE.finditer(inner): + if m.group(1) is not None: + items.append(_decode_toml_basic(m.group(1))) + elif m.group(2) is not None: + items.append(m.group(2)) + else: + items.append(int(m.group(3))) + result[key] = items + elif rest.startswith('"'): + result[key] = _decode_toml_basic(rest[1:-1]) + elif rest.startswith("'"): + result[key] = rest[1:-1] + else: + try: + result[key] = int(rest) + except ValueError: + result[key] = rest + return result + + +def load_lychee_toml(path: Path) -> Dict[str, object]: + text = path.read_text(encoding="utf-8", errors="replace") + try: + import tomllib # type: ignore[attr-defined] + + data = tomllib.loads(text) + except ImportError: + data = _parse_toml_subset(text) + except Exception as exc: # noqa: BLE001 — fall back to subset parser + print(f"WARN: tomllib failed on {path}: {exc}; using subset parser", file=sys.stderr) + data = _parse_toml_subset(text) + + known = {"exclude", "exclude_path", "accept"} + for key in list(data.keys()): + if key not in known: + print(f"WARN: ignoring unsupported .lychee.toml key: {key}", file=sys.stderr) + return {k: data[k] for k in known if k in data} + + +# --- Markdown / HTML extraction --------------------------------------------------- + +FENCE_RE = re.compile(r"^(`{3,}|~{3,})") +INLINE_CODE_RE = re.compile(r"`+[^`]*`+") +# One nesting level of balanced parens in the URL (CommonMark-approximate): +# keeps ".../Foo_(bar)" intact while still ending at the link's closing paren. +MD_INLINE_RE = re.compile( + r"!?\[[^\]]*\]\(((?:[^()\s]+|\([^()\s]*\))+)(?:\s+\"[^\"]*\")?\)" +) +MD_REF_DEF_RE = re.compile(r"^\s*\[([^\]]+)\]:\s*(\S+)", re.MULTILINE) +MD_REF_USE_RE = re.compile(r"\[([^\]]*)\]\[([^\]]*)\]") +# Shortcut reference "[label]" alone: not an inline link ("[t](…)"), not a +# full/collapsed reference ("[t][…]"), not a definition ("[label]: …"). +MD_REF_SHORTCUT_RE = re.compile(r"\[([^\]]+)\](?![:(\[])") +MD_AUTOLINK_RE = re.compile(r"<(https?://[^>\s]+)>", re.IGNORECASE) +BARE_URL_RE = re.compile( + # Block only "](" (an inline-markdown link opener; MD_INLINE_RE owns those) + # rather than any "(" — prose like "(https://example.com/x)" must extract. + r"(?\(\)\]\"']|\([^()\s]*\))+)", + re.IGNORECASE, +) + + +class _HrefSrcParser(html.parser.HTMLParser): + def __init__(self) -> None: + super().__init__(convert_charrefs=True) + self.urls: List[str] = [] + self._skip_depth = 0 + + def handle_starttag(self, tag: str, attrs) -> None: # noqa: ANN001 + tag_l = tag.lower() + if tag_l in ("code", "pre"): + self._skip_depth += 1 + return + if self._skip_depth: + return + attr_map = {k.lower(): v for k, v in attrs if v} + for key in ("href", "src"): + if key in attr_map: + self.urls.append(attr_map[key]) + + def handle_endtag(self, tag: str) -> None: + if tag.lower() in ("code", "pre") and self._skip_depth: + self._skip_depth -= 1 + + +LIST_MARKER_RE = re.compile(r"^(?: {0,3})(?:[-*+]|\d{1,9}[.)])(?: +)") + + +def _strip_code_regions(text: str) -> str: + """Remove fenced, indented, and inline code so their URLs are not extracted. + + Indented-code detection is CommonMark-approximate: an indented line is + code only when it cannot be a paragraph continuation, and inside a list + item only when indented past the item's content column. Links in nested + lists and indented prose (common in READMEs) must stay extractable. + """ + lines = text.splitlines(keepends=True) + out: List[str] = [] + in_fence = False + fence_marker = "" + para_open = False + list_content_indent = 0 + for line in lines: + if in_fence: + if line.lstrip().startswith(fence_marker): + in_fence = False + fence_marker = "" + out.append("\n" if line.endswith("\n") else "") + continue + m = FENCE_RE.match(line.lstrip()) + if m: + in_fence = True + fence_marker = m.group(1)[0] * len(m.group(1)) + out.append("\n" if line.endswith("\n") else "") + continue + if not line.strip(): + # Blank line: closes the paragraph, keeps the list context (loose lists). + para_open = False + out.append("\n" if line.endswith("\n") else "") + continue + expanded = line.replace("\t", " ") + ind = len(expanded) - len(expanded.lstrip(" ")) + if LIST_MARKER_RE.match(expanded): + # A list item: content of the item starts after the marker, and + # the marker line itself opens the item's paragraph (indented + # code cannot interrupt a paragraph). + list_content_indent = LIST_MARKER_RE.match(expanded).end() + para_open = True + elif list_content_indent and ind < list_content_indent: + # Dedented past the item content: the list context ends. + list_content_indent = 0 + para_open = True + elif ind >= 4 and not para_open: + # Indented code block: 4+ columns past the enclosing list item's + # content (or past column 0), and never a paragraph continuation. + if ind >= list_content_indent + 4: + out.append("\n" if line.endswith("\n") else "") + continue + para_open = True + else: + para_open = True + # Blank out inline code spans. + cleaned = INLINE_CODE_RE.sub(lambda mm: " " * len(mm.group(0)), line) + out.append(cleaned) + return "".join(out) + + +def _drop_unused_ref_defs(text: str) -> str: + """Remove `[label]: url` lines whose label is never referenced. + + Lychee does not extract those unused definitions, and a later bare-URL + pass would otherwise treat the definition target as a link. + """ + used = set() + for text_label, ref_label in MD_REF_USE_RE.findall(text): + label = (ref_label or text_label).strip().lower() + if label: + used.add(label) + # Shortcut references ("see [docs].") also keep their definition alive; + # without this the definition is dropped as unused and its URL never + # checked, so existing findings could be falsely treated as fixed. + for label in MD_REF_SHORTCUT_RE.findall(text): + label = label.strip().lower() + if label: + used.add(label) + + def repl(match: re.Match) -> str: + if match.group(1).strip().lower() in used: + return match.group(0) + return "" + + return MD_REF_DEF_RE.sub(repl, text) + + +HTML_PRE_CODE_RE = re.compile(r"<(pre|code)\b[^>]*>.*?", re.IGNORECASE | re.DOTALL) + + +def _strip_html_code(text: str) -> str: + """Blank
/ regions so no extraction pass sees their URLs.
+
+    The HTML parser already skips these tags for href/src, but the markdown
+    and bare-URL passes ran on the same text and picked up example URLs.
+    """
+    return HTML_PRE_CODE_RE.sub(lambda m: " " * len(m.group(0)), text)
+
+
+def extract_urls_from_text(text: str, is_html: bool = False) -> List[str]:
+    if is_html:
+        # HTML has no markdown code blocks: four-space indents and backticks
+        # are ordinary formatting, and only 
/ hides URLs.
+        cleaned = _strip_html_code(text)
+    else:
+        cleaned = _drop_unused_ref_defs(_strip_code_regions(text))
+        cleaned = _strip_html_code(cleaned)
+    found: List[str] = []
+    found.extend(MD_INLINE_RE.findall(cleaned))
+    found.extend(url for _label, url in MD_REF_DEF_RE.findall(cleaned))
+    found.extend(MD_AUTOLINK_RE.findall(cleaned))
+    found.extend(BARE_URL_RE.findall(cleaned))
+
+    parser = _HrefSrcParser()
+    try:
+        parser.feed(cleaned)
+        parser.close()
+    except Exception:  # noqa: BLE001 — malformed HTML should not abort the scan
+        pass
+    found.extend(parser.urls)
+
+    # Normalize: drop angle-bracket leftovers / trailing punctuation common in prose.
+    normalized: List[str] = []
+    for url in found:
+        u = url.strip().rstrip(".,;:!\"'")
+        u = _trim_trailing_parens(u)
+        if u:
+            normalized.append(u)
+    return normalized
+
+
+def _trim_trailing_parens(url: str) -> str:
+    """Drop trailing ')' characters not balanced by a '(' (prose closers).
+
+    Keeps ".../Foo_(bar)" intact while trimming a URL that swallowed a
+    sentence's closing parenthesis.
+    """
+    while url.endswith(")") and url.count(")") > url.count("("):
+        url = url[:-1]
+    return url
+
+
+def path_matches_exclude(rel: str, patterns: List[str]) -> bool:
+    for pat in patterns:
+        try:
+            if re.search(pat, rel):
+                return True
+        except re.error:
+            if pat in rel:
+                return True
+    return False
+
+
+def collect_links(
+    root: Path,
+    schemes: Set[str],
+    exclude_paths: List[str],
+) -> List[Tuple[str, str]]:
+    """Return deduped (relative_filepath, url) pairs under root."""
+    pairs: List[Tuple[str, str]] = []
+    seen: Set[Tuple[str, str]] = set()
+    root = root.resolve()
+
+    for dirpath, dirnames, filenames in os.walk(root):
+        # Prune .git and exclude-path matches early.
+        rel_dir = os.path.relpath(dirpath, root)
+        if rel_dir == ".":
+            rel_dir = ""
+        pruned = []
+        for d in list(dirnames):
+            if d == ".git":
+                continue
+            candidate = f"{rel_dir}/{d}" if rel_dir else d
+            if path_matches_exclude(candidate, exclude_paths) or path_matches_exclude(
+                d, exclude_paths
+            ):
+                continue
+            pruned.append(d)
+        dirnames[:] = pruned
+
+        for name in filenames:
+            ext = Path(name).suffix.lower()
+            if ext not in SCAN_EXTENSIONS:
+                continue
+            is_html = ext in (".html", ".htm")
+            abs_path = Path(dirpath) / name
+            rel = str(abs_path.relative_to(root)).replace(os.sep, "/")
+            if path_matches_exclude(rel, exclude_paths):
+                continue
+            try:
+                text = abs_path.read_text(encoding="utf-8", errors="replace")
+            except OSError as exc:
+                print(f"WARN: cannot read {rel}: {exc}", file=sys.stderr)
+                continue
+            for url in extract_urls_from_text(text, is_html=is_html):
+                parsed = _safe_split(url)
+                if parsed is None:
+                    # Malformed URL (e.g. "http://[::1"): when it claims to be a
+                    # web link it is a real doc bug — keep it so the checker
+                    # reports it; never let it crash the scan.
+                    if not url.lower().startswith(("http://", "https://")):
+                        continue
+                elif parsed.scheme.lower() not in schemes:
+                    continue
+                key = (rel, url)
+                if key in seen:
+                    continue
+                seen.add(key)
+                pairs.append(key)
+    return pairs
+
+
+# --- Checking ---------------------------------------------------------------------
+
+def proxy_env_set() -> bool:
+    for key in os.environ:
+        if key.lower() in ("http_proxy", "https_proxy", "all_proxy") and os.environ[key]:
+            return True
+    return False
+
+
+@functools.lru_cache(maxsize=1024)
+def is_private_host(host: str) -> Optional[bool]:
+    """Return True if host is private, False if public, None on DNS failure.
+
+    Cached: --exclude-all-private is always passed by the scanner, so every
+    URL paid an uncached getaddrinfo in the main thread before the pool
+    started — N lookups even when the links share a host.
+    """
+    if not host:
+        return False
+    # Strip brackets from IPv6 literals.
+    h = host.strip("[]")
+    try:
+        ip = ipaddress.ip_address(h)
+        return bool(
+            ip.is_private
+            or ip.is_loopback
+            or ip.is_link_local
+            or ip.is_reserved
+            or (isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped and (
+                ip.ipv4_mapped.is_private
+                or ip.ipv4_mapped.is_loopback
+                or ip.ipv4_mapped.is_link_local
+            ))
+        )
+    except ValueError:
+        pass
+    try:
+        infos = socket.getaddrinfo(h, None)
+    except socket.gaierror:
+        return None
+    if not infos:
+        return None
+    all_private = True
+    for info in infos:
+        addr = info[4][0]
+        try:
+            ip = ipaddress.ip_address(addr)
+        except ValueError:
+            all_private = False
+            continue
+        if not (
+            ip.is_private
+            or ip.is_loopback
+            or ip.is_link_local
+            or ip.is_reserved
+        ):
+            all_private = False
+    return all_private
+
+
+def url_excluded(url: str, exclude_res: List[re.Pattern]) -> bool:
+    for cre in exclude_res:
+        if cre.search(url):
+            return True
+    return False
+
+
+def _safe_split(url: str) -> Optional[urllib.parse.SplitResult]:
+    """urlsplit that returns None instead of raising on malformed URLs.
+
+    urlsplit raises ValueError on netlocs like "http://[::1" (unclosed IPv6
+    bracket); one such string in any scanned file must not kill the run.
+    """
+    try:
+        return urllib.parse.urlsplit(url)
+    except ValueError:
+        return None
+
+
+def _host_of(url: str) -> str:
+    parts = _safe_split(url)
+    return (parts.hostname or "") if parts is not None else ""
+
+
+def parse_retry_after(value: Optional[str]) -> Optional[float]:
+    if not value:
+        return None
+    value = value.strip()
+    if re.fullmatch(r"\d+", value):
+        return float(value)
+    try:
+        dt = email.utils.parsedate_to_datetime(value)
+        if dt.tzinfo is None:
+            # parsedate_to_datetime returns naive datetimes when the date has
+            # no explicit offset; HTTP dates are GMT (RFC 7231), not local time.
+            dt = dt.replace(tzinfo=datetime.timezone.utc)
+        return max(0.0, dt.timestamp() - time.time())
+    except (TypeError, ValueError, OverflowError):
+        return None
+
+
+def _is_egress_refusal(exc: BaseException) -> bool:
+    """True when a proxy gateway refused CONNECT (only meaningful with a proxy)."""
+    if not proxy_env_set():
+        return False
+    msg = str(exc)
+    lower = msg.lower()
+    if "tunnel connection failed" in lower and "403" in msg:
+        return True
+    # A reset after the CONNECT tunnel is up is the origin, not an egress refusal.
+    if "tunnel connection failed" in lower and any(
+        tok in lower for tok in ("reset", "closed", "refused", "abort")
+    ):
+        return True
+    # urllib wraps proxy CONNECT failures as URLError/OSError with the tunnel text.
+    if isinstance(exc, urllib.error.URLError):
+        reason = getattr(exc, "reason", None)
+        if reason is not None and reason is not exc:
+            return _is_egress_refusal(reason if isinstance(reason, BaseException) else Exception(str(reason)))
+    return False
+
+
+def _classify_exception(exc: BaseException) -> Dict[str, object]:
+    if _is_egress_refusal(exc):
+        return {"text": "unchecked", "details": str(exc)}
+    if isinstance(exc, socket.timeout) or isinstance(exc, TimeoutError):
+        return {"text": "Timeout", "details": str(exc)}
+    if isinstance(exc, socket.gaierror):
+        return {"text": "Failed to resolve host", "details": str(exc)}
+    msg = str(exc).lower()
+    if "timed out" in msg or "timeout" in msg:
+        return {"text": "Timeout", "details": str(exc)}
+    if "name or service not known" in msg or "nodename nor servname" in msg or "getaddrinfo" in msg:
+        return {"text": "Failed to resolve host", "details": str(exc)}
+    if any(tok in msg for tok in ("refused", "reset", "closed", "unreachable", "connect")):
+        # Prefer unchecked over unreachable when proxy refusal signatures match.
+        return {"text": "Connection refused", "details": str(exc)}
+    return {"text": "Network error", "details": str(exc)}
+
+
+def _open(req: urllib.request.Request, timeout: float, exclude_private: bool = False):
+    ctx = ssl.create_default_context()
+
+    class _LimitedRedirect(urllib.request.HTTPRedirectHandler):
+        def redirect_request(self, req, fp, code, msg, headers, newurl):  # noqa: ANN001
+            if exclude_private:
+                host = _host_of(newurl)
+                if host and is_private_host(host) is True:
+                    raise urllib.error.URLError(f"redirect to private host {host}")
+            count = getattr(req, "_redirect_count", 0) + 1
+            if count > MAX_REDIRECTS:
+                raise urllib.error.HTTPError(
+                    req.full_url, code, f"too many redirects (>{MAX_REDIRECTS})", headers, fp
+                )
+            new_req = super().redirect_request(req, fp, code, msg, headers, newurl)
+            if new_req is not None:
+                new_req._redirect_count = count  # type: ignore[attr-defined]
+            return new_req
+
+    opener = urllib.request.build_opener(
+        _LimitedRedirect,
+        urllib.request.HTTPSHandler(context=ctx),
+    )
+    return opener.open(req, timeout=timeout)
+
+
+def check_one(
+    url: str,
+    timeout: float,
+    max_retries: int,
+    accept: Set[int],
+    exclude_private: bool = False,
+) -> Optional[Dict[str, object]]:
+    """Return a status dict on failure/unchecked, or None on success."""
+    attempts = max_retries + 1
+    last_status: Optional[Dict[str, object]] = None
+
+    for attempt in range(attempts):
+        try:
+            status = _check_once(url, timeout, accept, exclude_private)
+            if status is None:
+                return None
+            # Retry on 429 — even when 429 is in the accept list, the
+            # Retry-After backoff must run before the link is called OK.
+            if status.get("code") == 429 and attempt < attempts - 1:
+                wait = parse_retry_after(status.pop("_retry_after", None))
+                if wait is None:
+                    wait = 1.0
+                if wait > RETRY_AFTER_CAP_SECS:
+                    # Cap exceeded: stop retrying; treat as success only when
+                    # the accept list includes 429 (the scanner's does).
+                    if 429 in accept:
+                        return None
+                    return {"code": 429, "text": "429"}
+                time.sleep(wait)
+                last_status = status
+                continue
+            # Retries exhausted with 429: accepted codes are not broken.
+            if status.get("code") == 429 and 429 in accept:
+                return None
+            # Egress refusals are never retried — they are not "broken".
+            if status.get("text") == "unchecked":
+                return status
+            # Retry on transient network errors represented as text statuses.
+            if "code" not in status and status.get("text") in (
+                "Timeout",
+                "Connection refused",
+                "Network error",
+            ):
+                if attempt < attempts - 1:
+                    last_status = status
+                    time.sleep(0.5 * (attempt + 1))
+                    continue
+            return status
+        except Exception as exc:  # noqa: BLE001
+            classified = _classify_exception(exc)
+            if classified.get("text") == "unchecked":
+                return classified
+            last_status = classified
+            if attempt < attempts - 1 and classified.get("text") in (
+                "Timeout",
+                "Connection refused",
+                "Network error",
+            ):
+                time.sleep(0.5 * (attempt + 1))
+                continue
+            return classified
+    return last_status
+
+
+def _check_once(
+    url: str, timeout: float, accept: Set[int], exclude_private: bool = False
+) -> Optional[Dict[str, object]]:
+    headers = {"User-Agent": USER_AGENT}
+    # HEAD first.
+    try:
+        req = urllib.request.Request(url, method="HEAD", headers=headers)
+        with _open(req, timeout, exclude_private) as resp:
+            code = getattr(resp, "status", None) or resp.getcode()
+            if code == 429:
+                return {
+                    "code": 429,
+                    "text": "429",
+                    "_retry_after": resp.headers.get("Retry-After"),
+                }
+            # HEAD is regularly denied (405/403/501) while GET works: consult
+            # the GET answer before letting the accept list decide.
+            if code in (405, 403, 501):
+                return _get_fallback(url, timeout, accept, headers, exclude_private)
+            if code in accept:
+                return None
+            return {"code": int(code), "text": str(code)}
+    except urllib.error.HTTPError as exc:
+        code = exc.code
+        if code == 429:
+            return {
+                "code": 429,
+                "text": "429",
+                "_retry_after": exc.headers.get("Retry-After") if exc.headers else None,
+            }
+        if code in (405, 403, 501):
+            return _get_fallback(url, timeout, accept, headers, exclude_private)
+        if code in accept:
+            return None
+        return {"code": int(code), "text": str(code)}
+    except Exception as exc:  # noqa: BLE001
+        return _classify_exception(exc)
+
+
+def _get_fallback(
+    url: str,
+    timeout: float,
+    accept: Set[int],
+    headers: Dict[str, str],
+    exclude_private: bool = False,
+) -> Optional[Dict[str, object]]:
+    try:
+        req = urllib.request.Request(url, method="GET", headers=headers)
+        with _open(req, timeout, exclude_private) as resp:
+            code = getattr(resp, "status", None) or resp.getcode()
+            # Drain a little so some servers don't reset.
+            try:
+                resp.read(1024)
+            except Exception:  # noqa: BLE001
+                pass
+            if code == 429:
+                return {
+                    "code": 429,
+                    "text": "429",
+                    "_retry_after": resp.headers.get("Retry-After"),
+                }
+            if code in accept:
+                return None
+            return {"code": int(code), "text": str(code)}
+    except urllib.error.HTTPError as exc:
+        code = exc.code
+        if code == 429:
+            return {
+                "code": 429,
+                "text": "429",
+                "_retry_after": exc.headers.get("Retry-After") if exc.headers else None,
+            }
+        if code in accept:
+            return None
+        return {"code": int(code), "text": str(code)}
+    except Exception as exc:  # noqa: BLE001
+        return _classify_exception(exc)
+
+
+class HostSemaphorePool:
+    """Limit concurrent in-flight checks per hostname."""
+
+    def __init__(self, per_host: int) -> None:
+        import threading
+
+        self._per_host = per_host
+        self._lock = threading.Lock()
+        self._sems: Dict[str, "threading.Semaphore"] = {}
+
+    def acquire(self, host: str) -> None:
+        import threading
+
+        with self._lock:
+            sem = self._sems.get(host)
+            if sem is None:
+                sem = threading.Semaphore(self._per_host)
+                self._sems[host] = sem
+        sem.acquire()
+
+    def release(self, host: str) -> None:
+        with self._lock:
+            sem = self._sems.get(host)
+        if sem is not None:
+            sem.release()
+
+
+def check_all(
+    pairs: List[Tuple[str, str]],
+    timeout: float,
+    max_retries: int,
+    max_concurrency: int,
+    accept: Set[int],
+    exclude_res: List[re.Pattern],
+    exclude_private: bool,
+) -> Tuple[int, int, int, Dict[str, List[Dict[str, object]]]]:
+    """Return (total, errors, unchecked, error_map)."""
+    # Filter excludes / private before counting toward total (lychee excludes
+    # are not part of total checked in our report; we count only checked URLs).
+    to_check: List[Tuple[str, str]] = []
+    for filepath, url in pairs:
+        if url_excluded(url, exclude_res):
+            continue
+        host = _host_of(url)
+        if exclude_private:
+            priv = is_private_host(host)
+            if priv is True:
+                continue
+            # DNS failure during private check: still attempt the URL; the
+            # checker will report dns if it cannot resolve.
+        to_check.append((filepath, url))
+
+    total = len(to_check)
+    error_map: Dict[str, List[Dict[str, object]]] = defaultdict(list)
+    errors = 0
+    unchecked = 0
+    host_pool = HostSemaphorePool(PER_HOST_CAP)
+
+    def worker(item: Tuple[str, str]) -> Tuple[str, str, Optional[Dict[str, object]]]:
+        filepath, url = item
+        host = _host_of(url)
+        host_pool.acquire(host)
+        try:
+            status = check_one(url, timeout, max_retries, accept, exclude_private)
+        finally:
+            host_pool.release(host)
+        return filepath, url, status
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=max(1, max_concurrency)) as pool:
+        futures = [pool.submit(worker, item) for item in to_check]
+        for fut in concurrent.futures.as_completed(futures):
+            filepath, url, status = fut.result()
+            if status is None:
+                continue
+            # Strip internal retry metadata.
+            status = {k: v for k, v in status.items() if not k.startswith("_")}
+            entry = {"url": url, "status": status}
+            error_map[filepath].append(entry)
+            if status.get("text") == "unchecked":
+                unchecked += 1
+            else:
+                errors += 1
+
+    return total, errors, unchecked, dict(error_map)
+
+
+# --- CLI --------------------------------------------------------------------------
+
+def build_arg_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(description="Stdlib lychee-compatible link checker")
+    p.add_argument("root", help="Directory to scan")
+    p.add_argument("--format", dest="fmt", default="json", choices=["json"])
+    p.add_argument("--scheme", action="append", default=[])
+    p.add_argument("--exclude", action="append", default=[])
+    p.add_argument("--exclude-path", action="append", default=[])
+    p.add_argument("--exclude-all-private", action="store_true")
+    p.add_argument("--accept", default="")
+    p.add_argument("--timeout", type=float, default=10.0)
+    p.add_argument("--max-retries", type=int, default=2)
+    p.add_argument("--max-concurrency", type=int, default=8)
+    p.add_argument("--config", default=None, help="Path to .lychee.toml")
+    return p
+
+
+def _accept_codes_from_text(raw: str) -> Set[int]:
+    """Expand a comma-separated accept string: ints or lychee ranges.
+
+    lychee's documented config examples include accept = ["200..=204", "429"]
+    and accept = "200, 429"; a range or list that collapses to nothing would
+    make every unlisted code (including 200) report as broken, so both forms
+    must expand.
+    """
+    codes: Set[int] = set()
+    for part in raw.split(","):
+        part = part.strip()
+        if not part:
+            continue
+        m = re.fullmatch(r"(\d+)\.\.=(\d+)", part)
+        if m:
+            codes.update(range(int(m.group(1)), int(m.group(2)) + 1))
+            continue
+        m = re.fullmatch(r"(\d+)\.\.(\d+)", part)
+        if m:
+            # Rust range semantics: end-exclusive.
+            codes.update(range(int(m.group(1)), int(m.group(2))))
+            continue
+        try:
+            codes.add(int(part))
+        except ValueError:
+            print(f"WARN: ignoring non-integer accept entry: {part!r}", file=sys.stderr)
+    return codes
+
+
+def parse_accept(raw: str) -> Set[int]:
+    return _accept_codes_from_text(raw)
+
+
+def normalize_accept(val: object) -> Set[int]:
+    """Coerce a config accept value (list or scalar) to a set of int codes.
+
+    Handles ints, range strings ("200..=204"), and comma strings
+    ("200, 429"). A malformed entry is warned about and skipped rather than
+    crashing the scan.
+    """
+    entries: List[object] = val if isinstance(val, list) else [val]
+    codes: Set[int] = set()
+    for entry in entries:
+        if isinstance(entry, str):
+            codes |= _accept_codes_from_text(entry)
+        else:
+            try:
+                codes.add(int(entry))
+            except (TypeError, ValueError):
+                print(f"WARN: ignoring non-integer accept entry: {entry!r}", file=sys.stderr)
+    return codes
+
+
+def main(argv: Optional[List[str]] = None) -> int:
+    args = build_arg_parser().parse_args(argv)
+    schemes = {s.lower() for s in (args.scheme or ["http", "https"])}
+    excludes = list(args.exclude)
+    exclude_paths = list(args.exclude_path)
+    accept = parse_accept(args.accept) if args.accept else {200, 204, 206}
+    exclude_private = bool(args.exclude_all_private)
+
+    if args.config:
+        cfg_path = Path(args.config)
+        if cfg_path.is_file():
+            cfg = load_lychee_toml(cfg_path)
+            if "exclude" in cfg:
+                val = cfg["exclude"]
+                if isinstance(val, list):
+                    excludes.extend(str(x) for x in val)
+                else:
+                    excludes.append(str(val))
+            if "exclude_path" in cfg:
+                val = cfg["exclude_path"]
+                if isinstance(val, list):
+                    exclude_paths.extend(str(x) for x in val)
+                else:
+                    exclude_paths.append(str(val))
+            if "accept" in cfg:
+                cfg_accept = normalize_accept(cfg["accept"])
+                if cfg_accept:
+                    accept = cfg_accept
+                else:
+                    # An accept config that expands to nothing would flag
+                    # every link as broken; keep the scanner-level list.
+                    print(
+                        "WARN: config accept expands to no codes; "
+                        "keeping the scanner accept list",
+                        file=sys.stderr,
+                    )
+        else:
+            print(f"WARN: config not found: {cfg_path}", file=sys.stderr)
+
+    exclude_res: List[re.Pattern] = []
+    for pat in excludes:
+        try:
+            exclude_res.append(re.compile(pat))
+        except re.error as exc:
+            print(f"WARN: bad exclude regex {pat!r}: {exc}", file=sys.stderr)
+
+    root = Path(args.root)
+    if not root.is_dir():
+        print(f"ERROR: not a directory: {root}", file=sys.stderr)
+        # Still emit empty JSON so the scanner does not treat this as "no output".
+        json.dump({"total": 0, "errors": 0, "unchecked": 0, "error_map": {}}, sys.stdout)
+        print()
+        return 0
+
+    pairs = collect_links(root, schemes, exclude_paths)
+    total, errors, unchecked, error_map = check_all(
+        pairs,
+        timeout=args.timeout,
+        max_retries=args.max_retries,
+        max_concurrency=args.max_concurrency,
+        accept=accept,
+        exclude_res=exclude_res,
+        exclude_private=exclude_private,
+    )
+
+    report = {
+        "total": total,
+        "errors": errors,
+        "unchecked": unchecked,
+        "error_map": error_map,
+    }
+    json.dump(report, sys.stdout)
+    print()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/skills/repoman/scripts/link-health-fixer.sh b/skills/repoman/scripts/link-health-fixer.sh
index 3c1a4bf..ec381e2 100755
--- a/skills/repoman/scripts/link-health-fixer.sh
+++ b/skills/repoman/scripts/link-health-fixer.sh
@@ -14,8 +14,8 @@ source "$SCRIPT_DIR/program-lib.sh"
 REPORTS_DIR="${REPORTS_DIR:-$HOME/reports/link-health}"
 
 # Provenance footer for fix-PR bodies. Resolved once at startup: the helper
-# lazily writes _meta..json beside this skill and returns the pinned-SHA RepoMan
-# attribution (blob/main fallback when the SHA cannot be resolved).
+# lazily writes _meta..json beside this skill and returns the pinned-SHA
+# attribution (linkless fallback when the SHA cannot be resolved).
 SKILL_FOOTER=$(skill_attribution "link-health-fixer")
 
 FIX_DATE=$(date +%Y-%m-%d)
diff --git a/skills/repoman/scripts/link-health-scanner.sh b/skills/repoman/scripts/link-health-scanner.sh
index 9715d40..f3b54dd 100755
--- a/skills/repoman/scripts/link-health-scanner.sh
+++ b/skills/repoman/scripts/link-health-scanner.sh
@@ -29,6 +29,25 @@ while [[ $# -gt 0 ]]; do
   esac
 done
 
+# LINK_CHECKER selects the link checker. Default python ([repoman#3](https://github.com/rossoctl/repoman/issues/3)): the
+# stdlib checker needs only python3, so the program runs on a basic agent
+# image; lychee stays available as an opt-in fallback.
+LINK_CHECKER="${LINK_CHECKER:-python}"
+case "$LINK_CHECKER" in
+  python|lychee) ;;
+  *)
+    echo "ERROR: LINK_CHECKER must be 'python' or 'lychee' (got: $LINK_CHECKER)" >&2
+    exit 2
+    ;;
+esac
+
+# Shadow mode: write the same report document to LINK_HEALTH_SHADOW_OUT and
+# force dry-run mutation behavior so a python side-by-side run cannot poison
+# the production baseline or file issues.
+if [ -n "${LINK_HEALTH_SHADOW_OUT:-}" ]; then
+  DRY_RUN=true
+fi
+
 # Resolve deployment constants (repos_dir, fork_owner) from ~/.repoman/config.json.
 repoman_config
 
@@ -49,7 +68,7 @@ REPORT_TARGET_PATH="automation-health/link-health.md"
 
 # Provenance footer for the report-PR body. Resolved once at startup: the
 # helper lazily writes _meta..json beside this skill and returns the pinned-SHA
-# RepoMan attribution (blob/main fallback when the SHA cannot be resolved).
+# RepoMan attribution (linkless footer when the SHA cannot be resolved).
 SKILL_FOOTER=$(skill_attribution "link-health-scanner")
 
 # Clone dir for the report target: honor an explicit MAIN_REPO_DIR override,
@@ -75,7 +94,9 @@ SCAN_ID=$(generate_scan_id "$REPORTS_DIR" "$SCAN_DATE")
 echo "=== Link Health Scan $SCAN_ID ==="
 echo "Repos dir: $REPOS_DIR"
 echo "Reports dir: $REPORTS_DIR"
+echo "Link checker: $LINK_CHECKER"
 if [ "$DRY_RUN" = true ]; then echo "Mode: DRY RUN (no issues, no PRs)"; fi
+if [ -n "${LINK_HEALTH_SHADOW_OUT:-}" ]; then echo "Shadow out: $LINK_HEALTH_SHADOW_OUT"; fi
 if [ "$ISSUE_LIMIT" -gt 0 ]; then echo "Issue limit: $ISSUE_LIMIT"; fi
 
 # --- Scan all repos ---
@@ -84,8 +105,15 @@ TOTAL_ERRORS=0
 REPOS_SCANNED=0
 REPOS_FAILED=0
 
-# Collect all broken links into a single JSONL file
+# Collect all findings (broken + unchecked) then split. Unchecked (egress
+# refusals) never open or auto-close issues.
+: > "$TMPDIR/all_findings.jsonl"
 : > "$TMPDIR/broken.jsonl"
+: > "$TMPDIR/unchecked.jsonl"
+# Repos whose checker run failed this scan (crash, no output, bad JSON).
+# They contribute no findings, which is not evidence their links were
+# fixed; their prior findings are re-injected before the diff.
+: > "$TMPDIR/failed_repos.txt"
 
 # Drive the loop from the enrolled set (authoritative), statting each clone,
 # rather than globbing "$REPOS_DIR"/*/ and filtering -- a "*/" glob silently
@@ -107,10 +135,10 @@ while IFS= read -r full_repo; do
   # cannot catch it -- the file is non-empty). That same-name-different-owner
   # case is exactly what the owner-namespaced model exists to support.
   safe_repo="${full_repo//\//_}"
-  LYCHEE_OUTPUT="$TMPDIR/lychee_${safe_repo}.json"
+  CHECK_OUTPUT="$TMPDIR/linkcheck_${safe_repo}.json"
 
-  # Run lychee -- scanner-level args applied to all repos
-  LYCHEE_SCANNER_ARGS=(
+  # Scanner-level args applied to all repos. Shared by lychee and link-check.py.
+  LINK_CHECKER_ARGS=(
     --format json
     --scheme http --scheme https
     --exclude 'localhost' --exclude '127\.0\.0\.1' --exclude 'localtest\.me'
@@ -124,39 +152,110 @@ while IFS= read -r full_repo; do
     --max-concurrency 8
   )
 
+  CONFIG_ARGS=()
   if [ -f "$repo_dir/.lychee.toml" ]; then
-    lychee "${LYCHEE_SCANNER_ARGS[@]}" --config "$repo_dir/.lychee.toml" "$repo_dir" > "$LYCHEE_OUTPUT" 2>/dev/null || true
+    CONFIG_ARGS=(--config "$repo_dir/.lychee.toml")
+  fi
+
+  # Keep the checker's stderr (tracebacks, WARN lines) so failures are
+  # diagnosable; it was previously discarded with 2>/dev/null.
+  CHECK_STDERR="$TMPDIR/linkcheck_${safe_repo}.stderr"
+  CHECKER_RC=0
+  if [ "$LINK_CHECKER" = "python" ]; then
+    python3 "$SCRIPT_DIR/link-check.py" "${LINK_CHECKER_ARGS[@]}" \
+      ${CONFIG_ARGS[@]+"${CONFIG_ARGS[@]}"} "$repo_dir" \
+      > "$CHECK_OUTPUT" 2> "$CHECK_STDERR" || CHECKER_RC=$?
   else
-    lychee "${LYCHEE_SCANNER_ARGS[@]}" "$repo_dir" > "$LYCHEE_OUTPUT" 2>/dev/null || true
+    # lychee exits 1 when it merely finds broken links; only the python
+    # checker's exit code is fatal (it exits 0 unless it crashed).
+    lychee "${LINK_CHECKER_ARGS[@]}" \
+      ${CONFIG_ARGS[@]+"${CONFIG_ARGS[@]}"} "$repo_dir" \
+      > "$CHECK_OUTPUT" 2> "$CHECK_STDERR" || CHECKER_RC=$?
   fi
 
-  if [ ! -s "$LYCHEE_OUTPUT" ]; then
-    echo "  WARN: lychee produced no output for $full_repo"
+  if [ "$LINK_CHECKER" = "python" ] && [ "$CHECKER_RC" -ne 0 ]; then
+    echo "  WARN: link-check.py exited $CHECKER_RC for $full_repo; stderr:"
+    head -50 "$CHECK_STDERR" | sed 's/^/    /' >&2
     REPOS_FAILED=$((REPOS_FAILED + 1))
+    printf '%s\n' "$full_repo" >> "$TMPDIR/failed_repos.txt"
+    continue
+  fi
+
+  if [ ! -s "$CHECK_OUTPUT" ]; then
+    echo "  WARN: $LINK_CHECKER produced no output for $full_repo; stderr:"
+    head -50 "$CHECK_STDERR" | sed 's/^/    /' >&2
+    REPOS_FAILED=$((REPOS_FAILED + 1))
+    printf '%s\n' "$full_repo" >> "$TMPDIR/failed_repos.txt"
+    continue
+  fi
+
+  # Truncated output (a crash mid-write) must not kill the whole scanner via
+  # set -e on the jq reads below; treat it as a failed repo instead.
+  if ! jq empty "$CHECK_OUTPUT" >/dev/null 2>&1; then
+    echo "  WARN: $LINK_CHECKER emitted invalid JSON for $full_repo; stderr:"
+    head -50 "$CHECK_STDERR" | sed 's/^/    /' >&2
+    REPOS_FAILED=$((REPOS_FAILED + 1))
+    printf '%s\n' "$full_repo" >> "$TMPDIR/failed_repos.txt"
     continue
   fi
 
   # Parse results
-  repo_total=$(jq '.total // 0' "$LYCHEE_OUTPUT")
-  repo_errors=$(jq '.errors // 0' "$LYCHEE_OUTPUT")
+  repo_total=$(jq '.total // 0' "$CHECK_OUTPUT")
+  repo_errors=$(jq '.errors // 0' "$CHECK_OUTPUT")
   TOTAL_LINKS=$((TOTAL_LINKS + repo_total))
   TOTAL_ERRORS=$((TOTAL_ERRORS + repo_errors))
   REPOS_SCANNED=$((REPOS_SCANNED + 1))
 
-  # Extract broken links from the lychee report. The parsing/suppression/status
-  # normalization logic lives in extract-broken-links.sh so it can be unit-tested
-  # (see tests/test-extract-broken-links.sh).
+  # Surface checker warnings (unsupported .lychee.toml keys, bad regexes,
+  # non-integer accepts) on a healthy run too; CHECK_STDERR was previously
+  # only printed on failure, so config problems were invisible.
+  if [ -s "$CHECK_STDERR" ]; then
+    grep '^WARN' "$CHECK_STDERR" | head -10 | sed 's/^/    /' >&2 || true
+  fi
+
+  # Extract findings from the checker report. Parsing/suppression/status
+  # normalization lives in extract-broken-links.sh (unit-tested).
   "$SCRIPT_DIR/extract-broken-links.sh" \
-    "$LYCHEE_OUTPUT" "$full_repo" "$repo_dir" \
-    >> "$TMPDIR/broken.jsonl" 2>/dev/null || true
+    "$CHECK_OUTPUT" "$full_repo" "$repo_dir" \
+    >> "$TMPDIR/all_findings.jsonl" 2>/dev/null || true
 
   echo "  Links: $repo_total, Errors: $repo_errors"
 done <<< "$CLONES"
 
+# Split unchecked (egress refusals) out of the broken set. Unchecked never
+# opens or auto-closes an issue; it is reported for the owner to widen egress.
+if [ -s "$TMPDIR/all_findings.jsonl" ]; then
+  jq -c 'select(.status != "unchecked")' "$TMPDIR/all_findings.jsonl" \
+    > "$TMPDIR/broken.jsonl" 2>/dev/null || : > "$TMPDIR/broken.jsonl"
+  jq -c 'select(.status == "unchecked")' "$TMPDIR/all_findings.jsonl" \
+    > "$TMPDIR/unchecked.jsonl" 2>/dev/null || : > "$TMPDIR/unchecked.jsonl"
+else
+  : > "$TMPDIR/broken.jsonl"
+  : > "$TMPDIR/unchecked.jsonl"
+fi
+
+UNCHECKED_COUNT=$(jq -s 'length' "$TMPDIR/unchecked.jsonl" 2>/dev/null || echo 0)
+UNCHECKED_TOP_HOSTS_JSON=$(jq -s -c '
+  [ .[] | .url | capture("//(?[^/]+)") | .h ]
+  | group_by(.)
+  | map({host: .[0], count: length})
+  | sort_by(-.count)
+  | .[0:10]
+' "$TMPDIR/unchecked.jsonl" 2>/dev/null || echo '[]')
+UNCHECKED_TOP_HOSTS_TEXT=$(echo "$UNCHECKED_TOP_HOSTS_JSON" | jq -r '
+  if length == 0 then "(none)"
+  else map("\(.host) (\(.count))") | join(", ")
+  end
+' 2>/dev/null || echo "(none)")
+
 echo ""
 echo "=== Scan complete ==="
 echo "Repos scanned: $REPOS_SCANNED (failed: $REPOS_FAILED)"
 echo "Total links: $TOTAL_LINKS, Total errors: $TOTAL_ERRORS"
+echo "Unchecked (egress refusals): $UNCHECKED_COUNT"
+if [ "$UNCHECKED_COUNT" -gt 0 ]; then
+  echo "Unchecked top hosts: $UNCHECKED_TOP_HOSTS_TEXT"
+fi
 
 # --- Load previous scan for diffing ---
 # Subset-run carry-forward (see docs/specs run-path, "Subset-aware stateful
@@ -190,6 +289,33 @@ else
   cp "$ALL_PREV_BROKEN" "$PREV_BROKEN"
 fi
 
+# Do not auto-close a previously broken link that is unchecked this run
+# (egress refusal is not evidence the link was fixed). Re-inject the prior
+# broken record (old status) into the current broken set so the diff treats
+# it as recurring. A later real HTTP success still closes it; a later real
+# 404 stays open.
+if [ -s "$TMPDIR/unchecked.jsonl" ] && [ -s "$PREV_BROKEN" ]; then
+  UNCHECKED_ARR=$(jq -s '.' "$TMPDIR/unchecked.jsonl" 2>/dev/null || echo '[]')
+  PREV_ARR=$(jq -s '.' "$PREV_BROKEN" 2>/dev/null || echo '[]')
+  jq -n -c --argjson unchecked "$UNCHECKED_ARR" --argjson prev "$PREV_ARR" '
+    ($unchecked | map([.repo, .file, .url] | join("|"))) as $keys |
+    $prev[] |
+    select(([.repo, .file, .url] | join("|")) as $k | ($keys | index($k)) != null)
+  ' >> "$TMPDIR/broken.jsonl" 2>/dev/null || true
+fi
+
+# Same rule for repos whose checker run failed: they produced no findings,
+# which is not evidence anything was fixed. Without this, every previously
+# broken link in a failed repo diffs as "fixed" and its issue auto-closes
+# while repos_failed merely ticks up.
+if [ -s "$TMPDIR/failed_repos.txt" ] && [ -s "$PREV_BROKEN" ]; then
+  FAILED_ARR=$(jq -Rsc 'split("\n") | map(select(length > 0))' "$TMPDIR/failed_repos.txt" 2>/dev/null || echo '[]')
+  PREV_F_ARR=$(jq -s '.' "$PREV_BROKEN" 2>/dev/null || echo '[]')
+  jq -n -c --argjson failed "$FAILED_ARR" --argjson prev "$PREV_F_ARR" '
+    $prev[] | select(.repo as $r | ($failed | index($r)) != null)
+  ' >> "$TMPDIR/broken.jsonl" 2>/dev/null || true
+fi
+
 # --- Compute diff using shared library ---
 DIFF_KEY='[.repo, .file, .url] | join("|")'
 read -r NEW_LINKS FIXED_LINKS RECURRING_LINKS < <(
@@ -373,6 +499,8 @@ LATEST_JSON=$(cat << LATEST_EOF
   "repos_failed": $REPOS_FAILED,
   "total_links_checked": $TOTAL_LINKS,
   "broken": $BROKEN_ARRAY,
+  "unchecked_count": $UNCHECKED_COUNT,
+  "unchecked_top_hosts": $UNCHECKED_TOP_HOSTS_JSON,
   "delta": {
     "new": $NEW_LINKS,
     "fixed": $FIXED_LINKS,
@@ -382,6 +510,13 @@ LATEST_JSON=$(cat << LATEST_EOF
 LATEST_EOF
 )
 
+# Shadow side-report: write the same document without touching the baseline.
+if [ -n "${LINK_HEALTH_SHADOW_OUT:-}" ]; then
+  mkdir -p "$(dirname "$LINK_HEALTH_SHADOW_OUT")"
+  printf '%s\n' "$LATEST_JSON" > "$LINK_HEALTH_SHADOW_OUT"
+  echo "Wrote shadow report: $LINK_HEALTH_SHADOW_OUT"
+fi
+
 # A dry run must NOT write the baseline: overwriting latest.json would make the
 # next real run see these findings as "recurring" (not "new") and skip filing
 # them, and skip closing anything shown as fixed here. Preview only.
@@ -401,6 +536,7 @@ HISTORY_ROW=$(cat << HIST_EOF
   "total_links_checked": $TOTAL_LINKS,
   "broken_internal": $BROKEN_INTERNAL,
   "broken_external": $BROKEN_EXTERNAL,
+  "unchecked_count": $UNCHECKED_COUNT,
   "new": $NEW_LINKS,
   "fixed": $FIXED_LINKS,
   "issues_created": $ISSUES_CREATED,
@@ -510,6 +646,8 @@ cat > "$TMPDIR/link-health.md" << DASHBOARD_EOF
 | Total links checked | $TOTAL_LINKS |
 | Broken (internal) | $BROKEN_INTERNAL |
 | Broken (external) | $BROKEN_EXTERNAL |
+| Unchecked (egress refusals) | $UNCHECKED_COUNT |
+| Unchecked top hosts | $UNCHECKED_TOP_HOSTS_TEXT |
 | New since last scan | +$NEW_LINKS |
 | Fixed since last scan | -$FIXED_LINKS |
 
diff --git a/skills/repoman/scripts/pr-review-fixer.sh b/skills/repoman/scripts/pr-review-fixer.sh
index c0a248d..7f2c79f 100755
--- a/skills/repoman/scripts/pr-review-fixer.sh
+++ b/skills/repoman/scripts/pr-review-fixer.sh
@@ -73,7 +73,7 @@ if [ -z "$COMMAND" ]; then
 fi
 
 # Pin provenance: write _meta only after the SHA resolves, never overwrite a
-# valid file, and fall back to blob/main when resolution fails. `footer` is the
+# valid file, and fall back to a linkless footer when resolution fails. `footer` is the
 # line for the review body and its Slack post. Inline comments do not get it.
 SKILL_FOOTER=$(skill_attribution "github-pr-review")
 if [ "$COMMAND" = "footer" ]; then
@@ -142,24 +142,31 @@ if [ "$COMMAND" = "begin" ]; then
     [ "$VERBOSE" = true ] && echo "    Fetching diff..." >&2
     gh pr diff "$number" --repo "$repo" > "$TMPDIR/pr-${number}.diff" 2>/dev/null || true
 
-    # Parse diff into structured map using the standalone parser
-    DIFF_MAP="[]"
+    # Parse diff into structured map using the standalone parser.
+    # The map is a PR-controlled blob and can run to hundreds of KB, so it is
+    # written straight to a file and never handed to jq on the command line: a
+    # single argv string is capped at 128 KiB (MAX_ARG_STRLEN) no matter how
+    # large ARG_MAX is, and a big PR's diff_map crosses that with E2BIG.
+    diff_map_file="$TMPDIR/diff-map-${number}.json"
+    echo "[]" > "$diff_map_file"
     if [ -s "$TMPDIR/pr-${number}.diff" ]; then
-      DIFF_MAP=$("$SCRIPT_DIR/parse-diff-map.sh" "$TMPDIR/pr-${number}.diff")
-      line_count=$(echo "$DIFF_MAP" | jq '[.[].hunks[].lines[]] | length')
-      [ "$VERBOSE" = true ] && echo "    Parsed diff_map: $line_count lines across $(echo "$DIFF_MAP" | jq 'length') files" >&2
+      "$SCRIPT_DIR/parse-diff-map.sh" "$TMPDIR/pr-${number}.diff" > "$diff_map_file"
+      line_count=$(jq '[.[].hunks[].lines[]] | length' "$diff_map_file")
+      [ "$VERBOSE" = true ] && echo "    Parsed diff_map: $line_count lines across $(jq 'length' "$diff_map_file") files" >&2
     fi
 
-    # Build enriched PR entry
-    enriched=$(jq -n \
+    # Build enriched PR entry (diff_map read from file, not argv)
+    enriched_file="$TMPDIR/entry-${number}.json"
+    jq -n \
       --arg repo "$repo" \
       --argjson number "$number" \
       --arg head_sha "$head_sha" \
-      --argjson diff_map "$DIFF_MAP" \
-      '{repo: $repo, number: $number, head_sha: $head_sha, diff_map: $diff_map}')
+      --slurpfile diff_map "$diff_map_file" \
+      '{repo: $repo, number: $number, head_sha: $head_sha, diff_map: $diff_map[0]}' \
+      > "$enriched_file"
 
-    # Append to output queue
-    jq --argjson entry "$enriched" '. += [$entry]' "$TMPDIR/output_queue.json" > "$TMPDIR/output_tmp.json"
+    # Append to output queue (entry read from file, not argv)
+    jq --slurpfile entry "$enriched_file" '. += [$entry[0]]' "$TMPDIR/output_queue.json" > "$TMPDIR/output_tmp.json"
     mv "$TMPDIR/output_tmp.json" "$TMPDIR/output_queue.json"
 
   done < "$TMPDIR/eligible_prs.jsonl"
diff --git a/skills/repoman/scripts/repoman-setup.sh b/skills/repoman/scripts/repoman-setup.sh
index 00ddec3..e45610a 100755
--- a/skills/repoman/scripts/repoman-setup.sh
+++ b/skills/repoman/scripts/repoman-setup.sh
@@ -2,9 +2,10 @@
 # Pure writer/validator for the three ~/.repoman file kinds RepoMan reads.
 #
 # Subcommands: init-config, add-repo, enable-program, set-output,
-# set-requirements. Each writes exactly one thing, validates it, and persists
-# atomically (temp file in the target directory, then `mv` into place, so a
-# crash mid-write never leaves a half-written JSON file).
+# set-requirements. Each writes
+# exactly one thing, validates it, and persists atomically (temp file in the
+# target directory, then `mv` into place, so a crash mid-write never leaves a
+# half-written JSON file).
 #
 # This script never calls gh, never prompts, never clones, never forks. It
 # receives already-resolved inputs (from the repoman-setup skill) and writes
diff --git a/tests/test-vendor-engine.sh b/tests/test-vendor-engine.sh
index 0e9b68d..6f2d182 100644
--- a/tests/test-vendor-engine.sh
+++ b/tests/test-vendor-engine.sh
@@ -14,9 +14,9 @@ extract-broken-links fork github-api link-health-fixer link-health-scanner \
 org parse-diff-map pr-review-fixer pr-review-impact pr-review-lib \
 pr-review-scanner program-lib repo-sync repoman-check repoman-index \
 repoman-requirements repoman-run repoman-setup weekly-report"
-EXTRA_FILES="doc-drift-check.py"
+EXTRA_FILES="doc-drift-check.py link-check.py"
 EXPECTED_SH=26
-EXPECTED_FILES=27
+EXPECTED_FILES=28
 
 # Fake automation source repo holding the engine scripts, committed so it has a HEAD.
 SRC="$TMP/auto"; mkdir -p "$SRC/scripts"
@@ -36,6 +36,7 @@ fi
 n=$(find "$DEST" -maxdepth 1 -name '*.sh' 2>/dev/null | wc -l | tr -d ' ')
 if [ "$n" = "$EXPECTED_SH" ]; then echo "PASS: $EXPECTED_SH scripts copied"; else echo "FAIL: copied $n, expected $EXPECTED_SH"; fail=1; fi
 if [ -f "$DEST/doc-drift-check.py" ]; then echo "PASS: doc-drift-check.py copied"; else echo "FAIL: doc-drift-check.py missing"; fail=1; fi
+if [ -f "$DEST/link-check.py" ]; then echo "PASS: link-check.py copied"; else echo "FAIL: link-check.py missing"; fail=1; fi
 
 # --- Provenance stamp: correct SOURCE repo HEAD sha + file list ---
 want=$(git -C "$SRC" rev-parse HEAD)
@@ -47,6 +48,8 @@ if jq -e '.source_repo == "rossoctl/automation"' "$DEST/_vendor.json" >/dev/null
   echo "PASS: stamp names source repo"; else echo "FAIL: stamp source_repo wrong"; fail=1; fi
 if jq -e '.files | index("doc-drift-check.py")' "$DEST/_vendor.json" >/dev/null 2>&1; then
   echo "PASS: stamp lists doc-drift-check.py"; else echo "FAIL: stamp missing doc-drift-check.py"; fail=1; fi
+if jq -e '.files | index("link-check.py")' "$DEST/_vendor.json" >/dev/null 2>&1; then
+  echo "PASS: stamp lists link-check.py"; else echo "FAIL: stamp missing link-check.py"; fail=1; fi
 
 # --- Idempotency: second run, same source => identical stamp sha ---
 bash "$VENDOR" --source "$SRC" --dest "$DEST" >/dev/null 2>&1