Skip to content

fix!: preserve encoded slashes in request unique keys - #2459

Merged
vdusek merged 3 commits into
apify:v2from
peterolkhov:codex/preserve-escaped-slashes-in-request-keys
Nov 18, 2026
Merged

vdusek merged 3 commits into
apify:v2from
peterolkhov:codex/preserve-escaped-slashes-in-request-keys

Conversation

@peterolkhov

@peterolkhov peterolkhov commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Request.from_url("https://example.com/items/a%2Fb") currently gets the same unique key as /items/a/b. The same issue affects %2B: /items/a%2Bb collides with /items/a+b.

Use yarl's canonical raw path when removing a literal trailing slash, then pass it back with encoded=True. This preserves escaped slashes and plus signs in request keys. Existing query sorting, tracking-parameter removal, fragment options and branch-specific casing behavior stay intact. The casing change in #2371/#2008 remains separate.

Compatibility: default unique keys change for affected %2F and %2B paths. Existing persisted request queues keep their stored keys, so adding those URLs after an upgrade can crawl them again. This patch does not migrate stored keys.

Added 18 unit-test cases, including exact compute_unique_key values for upper/lowercase escapes and literal-path controls. The _utils tests no longer import Request.

Validation: the revised cases fail against the old implementation. Affected/adjacent native tests pass on master (90) and the local v2 port (98), with the upstream public-suffix fixture and OS network denial. Built-wheel Request/helper tests pass (37 master / 41 v2); scoped source and wheel probes pass (34 per branch/mode, seven pre-existing yarl canonicalization limits explicitly excluded). Ruff lint/format and sdist/wheel builds pass on both branches; affected-path types pass on master. V2's type command reports the same renamed-rule warning on baseline and candidate.

The configured full suite, browser integrations and full Python/OS/dependency matrix were not run. The landing branch remains under maintainer discussion.

BREAKING CHANGE: normalize_url and compute_unique_key keep percent-encoded / and + in the URL path. Before, /a%2Fb got the same unique key as /a/b, and /a%2Bb the same as /a+b. Requests stored in a persisted RQ keep their old keys, so the same URL added after the upgrade is enqueued again.

✍️ Drafted by Claude Code

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 17, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing unique keys change for URLs with %2F or %2B in the path. A persisted RQ that is resumed across the upgrade can crawl these URLs one more time. IMO this should go rather to v2, like #2371.

WDYT @Pijukatel?

Comment thread src/crawlee/_utils/requests.py Resolved
Comment thread tests/unit/_utils/test_requests.py Resolved
@vdusek
vdusek requested a review from Pijukatel November 17, 2026 08:22
@peterolkhov

Copy link
Copy Markdown
Contributor Author

Confirmed: affected %2F and %2B paths get different default keys after this change. Persisted queues retain their old keys, so adding those URLs after an upgrade can crawl them again. The description now calls that out.

A local v2 port passes the focused checks and keeps #2371's casing behavior. Changing only this PR's base would pull in unrelated master commits and conflicts, so moving it to v2 would need a clean port/rebase of the head. The exact-key test revision is now added, and the base remains master pending the landing-branch decision. The clean two-file v2 port is prepared locally if that is the preferred target.

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. In the end, I'm more towards merging this to master as a fix, so approving. This also means you don't need the v2 port, because v2 gets rebased on master.

@Pijukatel

Copy link
Copy Markdown
Collaborator

I would prefer to merge it as fix! to v2 branch for consistency. Seems to me like a same kind of fix as this #2371

@vdusek
vdusek force-pushed the codex/preserve-escaped-slashes-in-request-keys branch from 7bd6706 to 1933db8 Compare November 18, 2026 10:55
@vdusek
vdusek changed the base branch from master to v2 November 18, 2026 10:55
@vdusek vdusek changed the title fix: preserve encoded slashes in request unique keys fix!: preserve encoded slashes in request unique keys Nov 18, 2026

@vdusek vdusek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

retargetted to v2, and merging

@vdusek
vdusek merged commit 9ea3fcd into apify:v2 Nov 18, 2026
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants