Repository navigation
fix: fall back to GET <uid>.ics and a full scan when the UID search fails - #826
Conversation
|
I had Claude Opus 5.5 to do a review, and it did have something to complain about. One of the questions here is how much workaround-logic we want to throw on "innocent" servers, versus to require users to configure their server with I'm a bit overloaded now, but I'll try to get back to this one soon. |
|
Thanks, no rush. On profiles: Home Assistant creates the client without |
|
It's a general problem - I don't want to force regular users to set |
|
I've thought a bit and made up my mind. I've already burnt myself a bit on uid searches scanning the whole directory and very slow performance when doing mass-lookups on big calendars. We should also not forget the I'm now implementing new "features" and expanding old ones a bit:
So the logic will be something like this:
Does this sound good? I'm in a bit of a rush, so I will amend the PR with those changes myself. If you're available for a review in an hour or two it would be super. |
|
Mostly sounds good. I can only get to the review tomorrow morning, I'll also run my iCloud scripts against your branch then. Some notes:
|
0a99516 to
0ae61be
Compare
A server may fail a calendar-query carrying a UID filter although it handles other filters: iCloud answers 412 (see python-caldav#783). The new feature search.text.uid describes this, and the icloud profile marks it ungraceful. Nothing in the library uses it yet. Prompt: It should be search.text.uid. Fix that one first, as a new pull request in caldav and caldav-server-tester [= add a feature for UID search failing, as discussed in the review of python-caldav#826, which needs feature definitions and signature-based guessing] Followup-Prompt: I didn't want any extra search logic in this commit, only the feature-definition and the check for it. Then we should work on the python-caldav#826 PR and use the fallback logic there, if the feature is either not set or the 412 is observed. [python-caldav#826 = the pull request python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails] Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
Unless search.text.uid is configured as supported, a failing UID search falls back to GET <uid>.ics. The full scan is only done after a 412, or when the feature is unsupported/broken/ungraceful, in which case the failing search is skipped altogether. The logic is now one generator with a sync and an async driver instead of two copies. Prompt: Please continue building on python-caldav#826. The commit in this branch should be included in the PR. The first fallback (GET $uid.ics) should apply if search.text.uid is not explicitly supported (that means, unsupported, broken, fragile, unknown, etc). The second fallback should only apply if search.text.uid is unsupported/broken or if the iCloud special 412 is observed. Refactor to avoid the duplication in get_object_by_uid/_async_get_object_by_uid. [python-caldav#826 = python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails] Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
…ails iCloud answers any UID filter in a calendar-query with 412, so get_object_by_uid() and everything built on it failed there. When the search fails, fetch <uid>.ics, the name this library gives an object. If that is not the object, list all objects of the type and match the UID on the client. 404, 401/403 and rate limits are still raised, and an empty search result is still "not found" with no extra requests. Fixes python-caldav#783 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A server may fail a calendar-query carrying a UID filter although it handles other filters: iCloud answers 412 (see python-caldav#783). The new feature search.text.uid describes this, and the icloud profile marks it ungraceful. Nothing in the library uses it yet. Prompt: It should be search.text.uid. Fix that one first, as a new pull request in caldav and caldav-server-tester [= add a feature for UID search failing, as discussed in the review of python-caldav#826, which needs feature definitions and signature-based guessing] Followup-Prompt: I didn't want any extra search logic in this commit, only the feature-definition and the check for it. Then we should work on the python-caldav#826 PR and use the fallback logic there, if the feature is either not set or the 412 is observed. [python-caldav#826 = the pull request python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails] Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
Unless search.text.uid is configured as supported, a failing UID search falls back to GET <uid>.ics. The full scan is only done after a 412, or when the feature is unsupported/broken/ungraceful, in which case the failing search is skipped altogether. The logic is now one generator with a sync and an async driver instead of two copies. Prompt: Please continue building on python-caldav#826. The commit in this branch should be included in the PR. The first fallback (GET $uid.ics) should apply if search.text.uid is not explicitly supported (that means, unsupported, broken, fragile, unknown, etc). The second fallback should only apply if search.text.uid is unsupported/broken or if the iCloud special 412 is observed. Refactor to avoid the duplication in get_object_by_uid/_async_get_object_by_uid. [python-caldav#826 = python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails] Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A server may fail a calendar-query carrying a UID filter although it handles other filters: iCloud answers 412 (see python-caldav#783). The new feature search.text.uid describes this, and the icloud profile marks it ungraceful. Nothing in the library uses it yet. Prompt: It should be search.text.uid. Fix that one first, as a new pull request in caldav and caldav-server-tester [= add a feature for UID search failing, as discussed in the review of python-caldav#826, which needs feature definitions and signature-based guessing] Followup-Prompt: I didn't want any extra search logic in this commit, only the feature-definition and the check for it. Then we should work on the python-caldav#826 PR and use the fallback logic there, if the feature is either not set or the 412 is observed. [python-caldav#826 = the pull request python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails] Assisted-By: Claude Sonnet 5.5
0ae61be to
308402f
Compare
... but many servers would not allow another object to be created with the same UID and a different filename. It seems like the checker only checks Edit: That's |
|
What would be very useful: a run of the caldav-server-tester towards iCloud, branch 'bleeding-edge' towards the latest push in this PR |
308402f to
702a452
Compare
If-None-Match: * only guards the URL, so a UID stored under another filename slips through where the UID search is skipped. The search in save(no_overwrite=True) is now left out only where the server refuses a UID conflict itself (save.no-uid-conflict), as before the UID-lookup fallbacks, and no longer where search.text.uid fails. Robur would otherwise accept duplicate UIDs. Prompt: check my comment and consider if it makes sense to spend time on Yet Another Commit here [= the last comment of python-caldav#826: gate the UID search in save(no_overwrite=True) on save.no-uid-conflict] Followup-Prompt: do the new commit Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
…flict If-None-Match: * only guards the URL, so a UID stored under another filename slips through where the UID search is skipped. The search in save(no_overwrite=True) is left out only where the server refuses a UID conflict itself (save.no-uid-conflict), whatever the state of search.text.uid. Robur would otherwise accept duplicate UIDs. Prompts: some of them seems to be lost, but basically my comments on python-caldav#826 sumarizes my request Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A server that names every object <uid>.ics whatever name it was PUT to can say so with the extra key forces_uid_ics on save-load.stable-url. When its GET of <uid>.ics then answers 404 or 410, the UID is not there and get_object_by_uid() skips the full scan. No server profile sets it yet. Prompt: We'll use save-load.stable-url and add information via the extra_keys framework to describe the exact behaviour. `'forces_uid_ics': True` means the uid.ics convension (sic) is enforced (I may be hallucinating - I was quite sure I observed a server with this behaviour, but the caldav-server-tester found none) Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
When the UID search failed (not with 412) and the GET of <uid>.ics missed, get_object_by_uid() raised NotFoundError and lost the real error, which callers such as save() read as "the object is absent". It now raises the search error, unless the server is declared to name every object <uid>.ics, where the GET miss is conclusive. Also: say "ungraceful" where the docs list the values that skip the search, say that search.text.uid is inherited from search.text when not set, and test the async error paths and profiles that inherit the feature. Prompt: [fix issues found during the review process] Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
search.text.uid is inherited from search.text where a profile does not set it, which made Zimbra and both Bedework profiles look like servers whose UID filter fails. caldav-server-tester measured it: Zimbra and Bedework 3 and 5 search by UID although they do not search text, so they say full and keep the UID search. Robur ignores the filter and says unsupported. Prompts (comments made inline in the document while reviewing it): > (on the finding that save(no_overwrite=True) drops the UID search on servers inheriting search.text.uid) Please run the caldav-server-tester and update the server definitions Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
702a452 to
6cadd58
Compare
|
Ran caldav-server-tester
My own test scripts: all objects are found, also the one not named LGTM. caldav-server-tester output (iCloud)caldav_version: 3.6.1.dev39+g6cadd5866
features:
auth.www-authenticate:
behaviour: offers Basic realm="MMCalDav", X-MobileMe-AuthToken realm="MMCalDav"
support: full
auth.www-authenticate.usable-scheme:
behaviour: offers basic, x-mobileme-authtoken
support: full
calendar-color:
support: full
calendar-color.name:
behaviour: read-only (set 'blue'/'green', both return '#CB30E0FF')
support: unsupported
calendar-order:
support: full
create-calendar.with-supported-component-types.multiple:
behaviour: asked for ['VEVENT', 'VTODO'], the calendar reports ['VEVENT']
support: broken
freebusy-query:
behaviour: 'freebusy query failed: ReportError at ''400 Bad Request
'', reason no reason'
support: ungraceful
multiget.limit:
behaviour: 'not probed: set CALDAV_SERVER_TESTER_LIMIT_OBJECTS to a number of
objects'
support: unknown
principal-search:
support: ungraceful
propfind.allprop.resourcetype:
support: unsupported
propfind.limit:
behaviour: 'not probed: set CALDAV_SERVER_TESTER_LIMIT_OBJECTS to a number of
objects'
support: unknown
propfind.principal-depth-1:
behaviour: HTTP 400
support: unsupported
save:
support: ungraceful
save-load.event.no-dtend:
behaviour: a VEVENT with a start time and neither DTEND nor DURATION is refused
(404), though RFC 5545 section 3.6.1 allows it; a VEVENT on a date and neither
DTEND nor DURATION is refused (404), though RFC 5545 section 3.6.1 allows it
support: ungraceful
save-load.icalendar.unknown-property:
behaviour: cannot test, tasks cannot be stored
support: unknown
save-load.icalendar.x-property:
behaviour: probed in an event only, tasks cannot be stored
support: full
save-load.journal:
support: unsupported
save-load.mutable.if-match-wildcard:
behaviour: 'If-Match: * holds backwards: refused with 412 on an existing object,
and creates a missing object'
support: broken
save-load.todo:
support: ungraceful
save.no-uid-conflict:
behaviour: refused with 412, but without the CALDAV:no-uid-conflict precondition
support: ungraceful
search.combined-is-logical-and:
support: unknown
search.is-not-defined:
support: unsupported
search.is-not-defined.category:
support: unsupported
search.is-not-defined.class:
support: unsupported
search.is-not-defined.dtend:
support: unsupported
search.limit:
behaviour: 'not probed: set CALDAV_SERVER_TESTER_LIMIT_OBJECTS to a number of
objects'
support: unknown
search.recurrences.expanded.todo:
support: unknown
search.recurrences.includes-implicit.todo:
support: unknown
search.recurrences.includes-implicit.todo.pending:
support: unknown
search.text.case-insensitive:
support: unsupported
search.text.case-sensitive:
support: unsupported
search.text.category:
support: unsupported
search.text.comp-type-optional:
description: cannot test - category/text search not supported by this server
support: unknown
search.text.substring:
support: unknown
search.text.uid:
support: ungraceful
search.time-range.alarm:
support: unsupported
search.time-range.comp-type-optional:
description: server rejects a time-range filter in a query without a component
type (RFC4791 section 9.7 compliant)
support: unsupported
search.time-range.open:
support: unknown
search.time-range.open.start:
support: unknown
search.time-range.open.start.duration:
support: unknown
search.time-range.todo:
support: unsupported
search.time-range.todo.no-dtstart:
support: unsupported
search.time-range.todo.no-dtstart.no-due:
support: unsupported
search.time-range.todo.strict:
support: unknown
sync-token.initial-sync:
behaviour: lists the collection itself
support: quirk
sync-token.limit:
behaviour: DAV:limit is ignored; lists the collection itself as a member
support: unsupported
synchronous-write:
save-load-delay: 0
support: full
url.encode-at.encoded:
behaviour: 'object paths: both spellings resolve and reach the same resource -
lenient rather than conformant, but nothing a client has to work around; calendar
paths: an object stored through the literal ''@'' spelling of the calendar path
is served back through ''%40'' - one collection under two names'
support: full
url.encode-at.identity:
behaviour: 'object paths: both spellings resolve and reach the same resource -
lenient rather than conformant, but nothing a client has to work around; calendar
paths: an object stored through the literal ''@'' spelling of the calendar path
is served back through ''%40'' - one collection under two names'
support: unsupported
url.encode-at.literal.collection:
behaviour: an object stored through the literal '@' spelling of the calendar path
is served back through '%40' - one collection under two names
support: full
url.encode-at.literal.object:
behaviour: both spellings resolve and reach the same resource - lenient rather
than conformant, but nothing a client has to work around
support: full
url.encode-at.literal.principal:
support: unknown
well-known:
behaviour: unexpected status 400
support: unknown
name: (noname)
ts: 1793164680.215826
url: https://p179-caldav.icloud.com:443/***/calendars/
|
|
I think the solution (for not having The test breakages seems to be due to some external change of how the CI jobs are done, I'll throw in a fix for that as well. |
pytest 9.2.0 applies filterwarnings = error to warnings raised while configuring, so pytest-asyncio's "asyncio_default_fixture_loop_scope is unset" deprecation warning aborted the run before any test was collected. Set it to "function", which is what the unset option already resolves to for our fixtures. Prompt: Why does the tests in python-caldav#826 fail? Prompt: I just merged python-caldav#833, so tests should be running for the master branch now. Please have a look (master CI has the same failure) Prompt: verify, and fix it in PR python-caldav#826 Assisted-By: Claude Sonnet 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
This one matters for calendaring-sync too: there a |
|
I tried to let Sonnet fix this, but it was a disaster. I'm now letting Opus have a go, here is the latest spec (and Opus has already found some weaknesses in it): What I want to achieve in the end:
Here are some thoughts:
There is a slightly related thing here which should probably be filed - what if an object exists on a filename with an UID and the client attempts changing the UID? Is that a breach of the RFC? Can we add probes for it in the caldav-server-tester? |
A duplicate UID refused otherwise than the RFC's 403/409 with CALDAV:no-uid-conflict is now a 'quirk' (keys 'status', 'precondition'). On a bare status, save() raises UidConflictError when a GET finds the URL empty - not for 403, nor where a missing object gives no 404. The .if-none-match child grades no_overwrite's PUT apart (Bedework, iCloud: bare 412), and no_overwrite skips the UID search wherever duplicates are refused. Profiles re-measured. Prompt: Please consider the `save.no-uid-conflict` "feature". The last two commits was (sic) done by Sonnet, but I think they should be rolled back or amended. I've written some thoughts in [a notes file], please read through. It it (sic) makes sense, then see if you can do a better job on implementing it. If something is unclear or not well enough thought through, get back to me. Prompt: (the notes: raise UidConflictError also on non-compliant servers; no calendar download for no_overwrite on iCloud; rewrite the description; 'full' only for 403/409 with the XML element; a bare refusal is a 'quirk' with keys for status and element, after the tester checks that GET gives 404 and the same PUT with another UID succeeds; a href-given child; and whether changing the UID of an existing object breaches the RFC) Prompts (comments made inline in the document while reviewing it): > Fair point - leave `.no-uid-conflict` ungraceful for servers reporting > 403 with no XML error. A `.save()` should not raise `UidConflictError` > in those cases as it can be anything. However, `ungraceful` is a > better grade than `unsupported`. The `save(no_overwrite)` does not > need to do an UID-search if `.no-uid-conflict` is `ungraceful`. If > the URL is declared to be stable, a simple GET suffices. > Unknown being the default is acceptable if and only if "full" would > cause harmful side-effects on servers that does (sic) save duplicated > UIDs easily. In this case we do have harmful side-effects, so the > default should be `unknown`, but this needs to be properly explained > I think this is a feature (it obeys "SHOULD" in the RFC) and not a > behaviour. > That nothing in the caldav library reads it (as for now) is not an > argument against testing it (this may be added as a memory note). > I agree with most of the proposals/recommendations [above], > so go ahead and implement. > Let's avoid throwing an uncertain UidConflict if the `non-existing-not-found`-feature > is not supported. > (on the copy-pasted PUT wrappers) I hate duplicated code. Consolidate > wherever it's possible, if it's not too difficult. Assisted-By: Claude Opus 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
|
So I've worked a bit more on this and pushed more commits to your branch, @Rusich90 (perhaps I should have been asking first, sorry for messing up your work). I believe the behaviour of both So for iCloud, this should now apply:
Since I don't have an iCloud account I haven't tested those claims. I'd be happy if you could have a go at it. Thanks for all the help so far - though, I'd like to ask for yet another run of the caldav-server-tester, current bleeding-edge branch on github together with the code in this pull request. |
Stalwart's profile states save.no-uid-conflict as 'quirk', and an unset .href-given child inherits that instead of its default 'full', though Stalwart does name the holder; GMX names none. testCheckCompatibility caught both. SOGo and Purelymail refuse without the precondition, so their href-given is stated 'unknown'. The description now says to state the child alongside the parent. Prompt: Please run the compatbility (sic) integration test over all the servers Followup-Prompt: how are the tests going? if there are any failures, start looking into it, and if things are passing, let me know. The purelymail test can take very long time. Assisted-By: Claude Opus 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
save(no_overwrite=True) on a server that searches without a wildcard (Zimbra) raised a plain UidConflictError when the server refused a duplicate the search had missed, and so did save(no_create=True) where it searches; both now raise UidConflictConsistencyError, as the wildcard paths do. testNoOverwriteSameUidOtherUrl expects that wherever the refusal is not 'ungraceful', and the search.text.uid fragment says it inherits search.text. Prompts (comments made inline in the document while reviewing it): > (on how the review findings would be fixed) I read many of the > recommendations as "we'll first merge the PR and then make another > commit directly on master". Wouldn't it be better to ship yet another > bugfixing commit to the PR-branch? (At the other hand, things that > are not relevant for this PR should probably be fixed after the merge > - like the purelymail etag.honoured fix). Assisted-By: Claude Opus 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
|
No problem with the commits on my branch. It's your project, so feel free to change anything you need =) I tested de2515b on iCloud, with
caldav-server-tester save.no-uid-conflict:
behaviour: refused with a bare 412
precondition: false
status: 412
support: quirk
save.no-uid-conflict.href-given:
support: unknown
save.no-uid-conflict.if-none-match:
behaviour: refused with a bare 412
precondition: false
status: 412
support: quirkThe rest of the output is the same as in my previous run, except for Two small things:
Looks fine. |
... and in the meanwhile, my Claude Opus did yet another review and found even more nits and corner cases to complain about. But I don't think the last commit should affect anything.
... so it seems like there will be yet another commit here ... never-ending story :-) |
The probe stored <uid>.ics, which a server that names every object <uid>.ics passes as stable. It now also stores an object under another name and looks it up by UID. A different name is unsupported, and the extra key forces_uid_ics is set when the name is <uid>.ics, which lets get_object_by_uid() in caldav skip its full scan. Prompt: the caldav-server-tester also has to probe save-load.stable-url properly [= a probe for the forces_uid_ics key added to save-load.stable-url in the caldav library, in the context of python-caldav/caldav#826] Assisted-By: Claude Sonnet 5.5
Rusich90's caldav-server-tester run shows that iCloud answers a plain PUT of a taken UID with the same bare 412 as one carrying If-None-Match, so the profile states the parent instead of the .if-none-match child, and href-given as 'unknown' (no precondition to read it from). A plain save() of a taken UID now raises UidConflictError there rather than PutError. Prompt: check the last comment in the PR [Rusich90's comment python-caldav#826 (comment) asking for the iCloud profile to state save.no-uid-conflict before the merge] Assisted-By: Claude Opus 5.5 Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
|
There. The |
|
I'll just merge this now. Thanks @Rusich90! For the reference of anyone visiting this PR in the future, the PR description above was written before the later commits, so it's not completely up-to-date, here are two highlights:
The merge commit's message has the summary. |
Fixes #783 (plan B there).
When
search(uid=...)fails,get_object_by_uid()now triesGET <uid>.ics, the namesave()gives an object. If that isn't the object, it lists all objects of the type and matches the UID on the client. Any search error leads here (e.g. 412 or 500) except 404, 401/403 and rate limits, which are raised as before. The deprecatedcomp_filterdoesn't fall back. An empty search result is stillNotFoundErrorwith no extra requests.The cost: there's no feature flag, so on a server where the UID search fails, every lookup repeats the fallback. An object not stored as
<uid>.ics, or a missing one, means a full download each time. That includessave(no_overwrite=True)of a new object andload()of a deleted one. A hint in theicloudprofile to skip the failing REPORT could be a follow-up.On iCloud, lookups that got 412 on master now work, about 2 s at worst for 1000 events. The same goes for
save(no_overwrite=True),save()of one occurrence, the async client, andsave(no_create=True)withfeatures="icloud"(without it, it sendsIf-Match: *and doesn't search). Plan A is now just an optimisation, so I'd leave it unless you want it.The new unit tests fail on master, apart from five that check unchanged behaviour. The functional tests pass on Radicale and Xandikos.