Repository navigation
fix!: preserve encoded slashes in request unique keys - #2459
Conversation
vdusek
left a comment
There was a problem hiding this comment.
Existing unique keys change for URLs with %2F or %2B in the path. A persisted RQ that is resumed across the upgrade can crawl these URLs one more time. IMO this should go rather to v2, like #2371.
WDYT @Pijukatel?
|
Confirmed: affected A local v2 port passes the focused checks and keeps #2371's casing behavior. Changing only this PR's base would pull in unrelated master commits and conflicts, so moving it to v2 would need a clean port/rebase of the head. The exact-key test revision is now added, and the base remains master pending the landing-branch decision. The clean two-file v2 port is prepared locally if that is the preferred target. |
vdusek
left a comment
There was a problem hiding this comment.
Thanks. In the end, I'm more towards merging this to master as a fix, so approving. This also means you don't need the v2 port, because v2 gets rebased on master.
|
I would prefer to merge it as |
7bd6706 to
1933db8
Compare
vdusek
left a comment
There was a problem hiding this comment.
retargetted to v2, and merging
Request.from_url("https://example.com/items/a%2Fb")currently gets the same unique key as/items/a/b. The same issue affects%2B:/items/a%2Bbcollides with/items/a+b.Use yarl's canonical raw path when removing a literal trailing slash, then pass it back with
encoded=True. This preserves escaped slashes and plus signs in request keys. Existing query sorting, tracking-parameter removal, fragment options and branch-specific casing behavior stay intact. The casing change in #2371/#2008 remains separate.Compatibility: default unique keys change for affected
%2Fand%2Bpaths. Existing persisted request queues keep their stored keys, so adding those URLs after an upgrade can crawl them again. This patch does not migrate stored keys.Added 18 unit-test cases, including exact
compute_unique_keyvalues for upper/lowercase escapes and literal-path controls. The_utilstests no longer importRequest.Validation: the revised cases fail against the old implementation. Affected/adjacent native tests pass on master (90) and the local v2 port (98), with the upstream public-suffix fixture and OS network denial. Built-wheel Request/helper tests pass (37 master / 41 v2); scoped source and wheel probes pass (34 per branch/mode, seven pre-existing yarl canonicalization limits explicitly excluded). Ruff lint/format and sdist/wheel builds pass on both branches; affected-path types pass on master. V2's type command reports the same renamed-rule warning on baseline and candidate.
The configured full suite, browser integrations and full Python/OS/dependency matrix were not run. The landing branch remains under maintainer discussion.
BREAKING CHANGE:
normalize_urlandcompute_unique_keykeep percent-encoded/and+in the URL path. Before,/a%2Fbgot the same unique key as/a/b, and/a%2Bbthe same as/a+b. Requests stored in a persisted RQ keep their old keys, so the same URL added after the upgrade is enqueued again.✍️ Drafted by Claude Code