Skip to content

fix: fall back to GET <uid>.ics and a full scan when the UID search fails - #826

Merged
tobixen merged 12 commits into
python-caldav:masterfrom
Rusich90:uid-lookup-fallback
Oct 28, 2026
Merged

tobixen merged 12 commits into
python-caldav:masterfrom
Rusich90:uid-lookup-fallback

Conversation

@Rusich90

Copy link
Copy Markdown
Contributor

Fixes #783 (plan B there).

When search(uid=...) fails, get_object_by_uid() now tries GET <uid>.ics, the name save() gives an object. If that isn't the object, it lists all objects of the type and matches the UID on the client. Any search error leads here (e.g. 412 or 500) except 404, 401/403 and rate limits, which are raised as before. The deprecated comp_filter doesn't fall back. An empty search result is still NotFoundError with no extra requests.

The cost: there's no feature flag, so on a server where the UID search fails, every lookup repeats the fallback. An object not stored as <uid>.ics, or a missing one, means a full download each time. That includes save(no_overwrite=True) of a new object and load() of a deleted one. A hint in the icloud profile to skip the failing REPORT could be a follow-up.

On iCloud, lookups that got 412 on master now work, about 2 s at worst for 1000 events. The same goes for save(no_overwrite=True), save() of one occurrence, the async client, and save(no_create=True) with features="icloud" (without it, it sends If-Match: * and doesn't search). Plan A is now just an optimisation, so I'd leave it unless you want it.

The new unit tests fail on master, apart from five that check unchanged behaviour. The functional tests pass on Radicale and Xandikos.

@tobixen

tobixen commented Oct 25, 2026

Copy link
Copy Markdown
Member

I had Claude Opus 5.5 to do a review, and it did have something to complain about. One of the questions here is how much workaround-logic we want to throw on "innocent" servers, versus to require users to configure their server with features=....

I'm a bit overloaded now, but I'll try to get back to this one soon.

@Rusich90

Copy link
Copy Markdown
Contributor Author

Thanks, no rush. On profiles: Home Assistant creates the client without features, so a fix gated on the iCloud profile wouldn't reach its users. I'm happy to narrow it, e.g. GET <uid>.ics on other search errors, but the full scan only on 412.

@tobixen

tobixen commented Oct 25, 2026

Copy link
Copy Markdown
Member

It's a general problem - I don't want to force regular users to set features. It would be useful with some auto-detection-logic - but that's another issue. I don't want to fall into that rabbit hole this week, at least.

@tobixen

tobixen commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

I've thought a bit and made up my mind.

I've already burnt myself a bit on uid searches scanning the whole directory and very slow performance when doing mass-lookups on big calendars. We should also not forget the save(no_overwrite=True)-method - if exporting a big calendar doing that, it's rather suboptimal if every save would have to scan the entire calendar (creds to @SashankBhamidi for pointing that one out).

I'm now implementing new "features" and expanding old ones a bit:

  • search.text.uid to indicate if a server supports uid lookups.
  • save-load.stable-url exists, we expand it with some extra keys so it's possible to flag that a server enforces $uid.ics.
  • save.if-none-match already exists.

So the logic will be something like this:

  • (It's tempting me a bit to say ... save-load.stable-url unsupported with uid-dot-ics-enforcement => use the GET-workaround and nothing else - but I'm not sure we can rely that strongly on the feature declarations)
  • search.text.uid is explicitly declared to be supported => no fallbacks should be applied
  • otherwise, the GET-fallback should be applied
  • The "fetch all calendar"-fallback should only be applied for servers:
    • declared to not support search.text.uid
    • not declared to enforce $uid.ics
    • ... or undeclared servers, if the 412 iCloud signature is found
  • in calendar.save, if the server is declared to support if-none-matches: * and not support text.search.uid, then it should not do a search by uid.

Does this sound good? I'm in a bit of a rush, so I will amend the PR with those changes myself. If you're available for a review in an hour or two it would be super.

@Rusich90

Copy link
Copy Markdown
Contributor Author

Mostly sounds good. I can only get to the review tomorrow morning, I'll also run my iCloud scripts against your branch then.

Some notes:

  • iCloud doesn't enforce $uid.ics. Most of my objects are named like that, but one I imported through Calendar.app on the Mac is stored as ha-test%2Fslash plus%20uid.ics (UID ha-test/slash+plus uid). So GET alone won't find everything there.
  • If-None-Match only checks the URL, so if save(no_overwrite=True) skips the uid search, a duplicate UID under another filename can slip through. Currently save() only skips it when save.no-uid-conflict is supported too. Not known for iCloud yet, I can check it tomorrow.
  • I'd only do the GET when the search fails, not when it returns nothing (that's how the PR does it now). Otherwise every new object saved with no_overwrite costs an extra GET.

@tobixen
tobixen force-pushed the uid-lookup-fallback branch from 0a99516 to 0ae61be Compare October 27, 2026 19:49
tobixen added a commit to Rusich90/caldav that referenced this pull request Oct 27, 2026
A server may fail a calendar-query carrying a UID filter although it
handles other filters: iCloud answers 412 (see
python-caldav#783).  The new feature
search.text.uid describes this, and the icloud profile marks it
ungraceful.  Nothing in the library uses it yet.

Prompt: It should be search.text.uid.  Fix that one first, as a new pull request in caldav and caldav-server-tester [= add a feature for UID search failing, as discussed in the review of python-caldav#826, which needs feature definitions and signature-based guessing]
Followup-Prompt: I didn't want any extra search logic in this commit, only the feature-definition and the check for it.  Then we should work on the python-caldav#826 PR and use the fallback logic there, if the feature is either not set or the 412 is observed. [python-caldav#826 = the pull request python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails]
Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
tobixen added a commit to Rusich90/caldav that referenced this pull request Oct 27, 2026
Unless search.text.uid is configured as supported, a failing UID search
falls back to GET <uid>.ics.  The full scan is only done after a 412, or
when the feature is unsupported/broken/ungraceful, in which case the
failing search is skipped altogether.  The logic is now one generator
with a sync and an async driver instead of two copies.

Prompt: Please continue building on python-caldav#826.  The commit in this branch should be included in the PR.  The first fallback (GET $uid.ics) should apply if search.text.uid is not explicitly supported (that means, unsupported, broken, fragile, unknown, etc).  The second fallback should only apply if search.text.uid is unsupported/broken or if the iCloud special 412 is observed.  Refactor to avoid the duplication in get_object_by_uid/_async_get_object_by_uid. [python-caldav#826 = python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails]
Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
Rusich90 and others added 3 commits October 27, 2026 20:53
…ails

iCloud answers any UID filter in a calendar-query with 412, so
get_object_by_uid() and everything built on it failed there.  When the
search fails, fetch <uid>.ics, the name this library gives an object.
If that is not the object, list all objects of the type and match the
UID on the client.  404, 401/403 and rate limits are still raised, and
an empty search result is still "not found" with no extra requests.

Fixes python-caldav#783

Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A server may fail a calendar-query carrying a UID filter although it
handles other filters: iCloud answers 412 (see
python-caldav#783).  The new feature
search.text.uid describes this, and the icloud profile marks it
ungraceful.  Nothing in the library uses it yet.

Prompt: It should be search.text.uid.  Fix that one first, as a new pull request in caldav and caldav-server-tester [= add a feature for UID search failing, as discussed in the review of python-caldav#826, which needs feature definitions and signature-based guessing]
Followup-Prompt: I didn't want any extra search logic in this commit, only the feature-definition and the check for it.  Then we should work on the python-caldav#826 PR and use the fallback logic there, if the feature is either not set or the 412 is observed. [python-caldav#826 = the pull request python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails]
Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
Unless search.text.uid is configured as supported, a failing UID search
falls back to GET <uid>.ics.  The full scan is only done after a 412, or
when the feature is unsupported/broken/ungraceful, in which case the
failing search is skipped altogether.  The logic is now one generator
with a sync and an async driver instead of two copies.

Prompt: Please continue building on python-caldav#826.  The commit in this branch should be included in the PR.  The first fallback (GET $uid.ics) should apply if search.text.uid is not explicitly supported (that means, unsupported, broken, fragile, unknown, etc).  The second fallback should only apply if search.text.uid is unsupported/broken or if the iCloud special 412 is observed.  Refactor to avoid the duplication in get_object_by_uid/_async_get_object_by_uid. [python-caldav#826 = python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails]
Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
tobixen added a commit to Rusich90/caldav that referenced this pull request Oct 27, 2026
A server may fail a calendar-query carrying a UID filter although it
handles other filters: iCloud answers 412 (see
python-caldav#783).  The new feature
search.text.uid describes this, and the icloud profile marks it
ungraceful.  Nothing in the library uses it yet.

Prompt: It should be search.text.uid.  Fix that one first, as a new pull request in caldav and caldav-server-tester [= add a feature for UID search failing, as discussed in the review of python-caldav#826, which needs feature definitions and signature-based guessing]
Followup-Prompt: I didn't want any extra search logic in this commit, only the feature-definition and the check for it.  Then we should work on the python-caldav#826 PR and use the fallback logic there, if the feature is either not set or the 412 is observed. [python-caldav#826 = the pull request python-caldav#826, fall back to GET <uid>.ics and a full scan when the UID search fails]
Assisted-By: Claude Sonnet 5.5
@tobixen
tobixen force-pushed the uid-lookup-fallback branch from 0ae61be to 308402f Compare October 27, 2026 19:56
@tobixen

tobixen commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Right ... that's a problem indeed.

... but many servers would not allow another object to be created with the same UID and a different filename. It seems like the checker only checks save.duplicate-uid.cross-calendar and not save.duplicate-uid.

Edit: That's save.no-uid-conflict and it is checked. So maybe the uid-search in save(overwrite=False) should be gated by this. if-none-matches: * explicitly supported and save.no-uid-conflict explicitly supported => no need to search for uid.

@tobixen

tobixen commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

What would be very useful: a run of the caldav-server-tester towards iCloud, branch 'bleeding-edge' towards the latest push in this PR

@tobixen
tobixen force-pushed the uid-lookup-fallback branch from 308402f to 702a452 Compare October 27, 2026 20:29
tobixen added a commit to Rusich90/caldav that referenced this pull request Oct 27, 2026
If-None-Match: * only guards the URL, so a UID stored under another
filename slips through where the UID search is skipped.  The search in
save(no_overwrite=True) is now left out only where the server refuses a
UID conflict itself (save.no-uid-conflict), as before the UID-lookup
fallbacks, and no longer where search.text.uid fails.  Robur would
otherwise accept duplicate UIDs.

Prompt: check my comment and consider if it makes sense to spend time on Yet Another Commit here [= the last comment of python-caldav#826: gate the UID search in save(no_overwrite=True) on save.no-uid-conflict]
Followup-Prompt: do the new commit

Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
…flict

If-None-Match: * only guards the URL, so a UID stored under another
filename slips through where the UID search is skipped.  The search in
save(no_overwrite=True) is left out only where the server refuses a UID
conflict itself (save.no-uid-conflict), whatever the state of
search.text.uid.  Robur would otherwise accept duplicate UIDs.

Prompts: some of them seems to be lost, but basically my comments on python-caldav#826 sumarizes my request

Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
A server that names every object <uid>.ics whatever name it was PUT to
can say so with the extra key forces_uid_ics on save-load.stable-url.
When its GET of <uid>.ics then answers 404 or 410, the UID is not there
and get_object_by_uid() skips the full scan.  No server profile sets
it yet.

Prompt: We'll use save-load.stable-url and add information via the extra_keys framework to describe the exact behaviour.  `'forces_uid_ics': True` means the uid.ics convension (sic) is enforced

(I may be hallucinating - I was quite sure I observed a server with this
behaviour, but the caldav-server-tester found none)

Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
When the UID search failed (not with 412) and the GET of <uid>.ics
missed, get_object_by_uid() raised NotFoundError and lost the real
error, which callers such as save() read as "the object is absent".
It now raises the search error, unless the server is declared to name
every object <uid>.ics, where the GET miss is conclusive.  Also: say
"ungraceful" where the docs list the values that skip the search, say
that search.text.uid is inherited from search.text when not set, and
test the async error paths and profiles that inherit the feature.

Prompt: [fix issues found during the review process]
Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
search.text.uid is inherited from search.text where a profile does not
set it, which made Zimbra and both Bedework profiles look like servers
whose UID filter fails.  caldav-server-tester measured it: Zimbra and
Bedework 3 and 5 search by UID although they do not search text, so
they say full and keep the UID search.  Robur ignores the filter and
says unsupported.

Prompts (comments made inline in the document while reviewing it):
> (on the finding that save(no_overwrite=True) drops the UID search on servers inheriting search.text.uid) Please run the caldav-server-tester and update the server definitions

Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
@tobixen
tobixen force-pushed the uid-lookup-fallback branch from 702a452 to 6cadd58 Compare October 27, 2026 22:03
@Rusich90

Copy link
Copy Markdown
Contributor Author

Ran caldav-server-tester bleeding-edge (ed45104) against iCloud with caldav at 6cadd58 (without the attendee and scheduling checks, so no invites go out). Output is below.

  • search.text.uid: ungraceful, same as in the profile.
  • save.no-uid-conflict: ungraceful, iCloud refuses a duplicate UID with a bare 412 (the PUT has If-None-Match: *). Not in the profile yet.
  • save: ungraceful is just save.duplicate-uid.cross-calendar: ungraceful (already in the profile), folded up by the compact output. Don't copy it as is, it would also turn off save.if-none-match and the other save.* features.

My own test scripts: all objects are found, also the one not named <uid>.ics, with and without features="icloud". One thing: with features="icloud", save(no_overwrite=True) of a new object still downloads the whole calendar, because only save.no-uid-conflict: full skips the search. But iCloud refuses the duplicate itself, so maybe a bare 412 could be enough to skip the search? The save would still fail, but with a plain ConsistencyError instead of UidConflictConsistencyError. I can try it in a separate PR.

LGTM.

caldav-server-tester output (iCloud)
caldav_version: 3.6.1.dev39+g6cadd5866
features:
  auth.www-authenticate:
    behaviour: offers Basic realm="MMCalDav", X-MobileMe-AuthToken realm="MMCalDav"
    support: full
  auth.www-authenticate.usable-scheme:
    behaviour: offers basic, x-mobileme-authtoken
    support: full
  calendar-color:
    support: full
  calendar-color.name:
    behaviour: read-only (set 'blue'/'green', both return '#CB30E0FF')
    support: unsupported
  calendar-order:
    support: full
  create-calendar.with-supported-component-types.multiple:
    behaviour: asked for ['VEVENT', 'VTODO'], the calendar reports ['VEVENT']
    support: broken
  freebusy-query:
    behaviour: 'freebusy query failed: ReportError at ''400 Bad Request


      '', reason no reason'
    support: ungraceful
  multiget.limit:
    behaviour: 'not probed: set CALDAV_SERVER_TESTER_LIMIT_OBJECTS to a number of
      objects'
    support: unknown
  principal-search:
    support: ungraceful
  propfind.allprop.resourcetype:
    support: unsupported
  propfind.limit:
    behaviour: 'not probed: set CALDAV_SERVER_TESTER_LIMIT_OBJECTS to a number of
      objects'
    support: unknown
  propfind.principal-depth-1:
    behaviour: HTTP 400
    support: unsupported
  save:
    support: ungraceful
  save-load.event.no-dtend:
    behaviour: a VEVENT with a start time and neither DTEND nor DURATION is refused
      (404), though RFC 5545 section 3.6.1 allows it; a VEVENT on a date and neither
      DTEND nor DURATION is refused (404), though RFC 5545 section 3.6.1 allows it
    support: ungraceful
  save-load.icalendar.unknown-property:
    behaviour: cannot test, tasks cannot be stored
    support: unknown
  save-load.icalendar.x-property:
    behaviour: probed in an event only, tasks cannot be stored
    support: full
  save-load.journal:
    support: unsupported
  save-load.mutable.if-match-wildcard:
    behaviour: 'If-Match: * holds backwards: refused with 412 on an existing object,
      and creates a missing object'
    support: broken
  save-load.todo:
    support: ungraceful
  save.no-uid-conflict:
    behaviour: refused with 412, but without the CALDAV:no-uid-conflict precondition
    support: ungraceful
  search.combined-is-logical-and:
    support: unknown
  search.is-not-defined:
    support: unsupported
  search.is-not-defined.category:
    support: unsupported
  search.is-not-defined.class:
    support: unsupported
  search.is-not-defined.dtend:
    support: unsupported
  search.limit:
    behaviour: 'not probed: set CALDAV_SERVER_TESTER_LIMIT_OBJECTS to a number of
      objects'
    support: unknown
  search.recurrences.expanded.todo:
    support: unknown
  search.recurrences.includes-implicit.todo:
    support: unknown
  search.recurrences.includes-implicit.todo.pending:
    support: unknown
  search.text.case-insensitive:
    support: unsupported
  search.text.case-sensitive:
    support: unsupported
  search.text.category:
    support: unsupported
  search.text.comp-type-optional:
    description: cannot test - category/text search not supported by this server
    support: unknown
  search.text.substring:
    support: unknown
  search.text.uid:
    support: ungraceful
  search.time-range.alarm:
    support: unsupported
  search.time-range.comp-type-optional:
    description: server rejects a time-range filter in a query without a component
      type (RFC4791 section 9.7 compliant)
    support: unsupported
  search.time-range.open:
    support: unknown
  search.time-range.open.start:
    support: unknown
  search.time-range.open.start.duration:
    support: unknown
  search.time-range.todo:
    support: unsupported
  search.time-range.todo.no-dtstart:
    support: unsupported
  search.time-range.todo.no-dtstart.no-due:
    support: unsupported
  search.time-range.todo.strict:
    support: unknown
  sync-token.initial-sync:
    behaviour: lists the collection itself
    support: quirk
  sync-token.limit:
    behaviour: DAV:limit is ignored; lists the collection itself as a member
    support: unsupported
  synchronous-write:
    save-load-delay: 0
    support: full
  url.encode-at.encoded:
    behaviour: 'object paths: both spellings resolve and reach the same resource -
      lenient rather than conformant, but nothing a client has to work around; calendar
      paths: an object stored through the literal ''@'' spelling of the calendar path
      is served back through ''%40'' - one collection under two names'
    support: full
  url.encode-at.identity:
    behaviour: 'object paths: both spellings resolve and reach the same resource -
      lenient rather than conformant, but nothing a client has to work around; calendar
      paths: an object stored through the literal ''@'' spelling of the calendar path
      is served back through ''%40'' - one collection under two names'
    support: unsupported
  url.encode-at.literal.collection:
    behaviour: an object stored through the literal '@' spelling of the calendar path
      is served back through '%40' - one collection under two names
    support: full
  url.encode-at.literal.object:
    behaviour: both spellings resolve and reach the same resource - lenient rather
      than conformant, but nothing a client has to work around
    support: full
  url.encode-at.literal.principal:
    support: unknown
  well-known:
    behaviour: unexpected status 400
    support: unknown
name: (noname)
ts: 1793164680.215826
url: https://p179-caldav.icloud.com:443/***/calendars/

@tobixen

tobixen commented Oct 28, 2026

Copy link
Copy Markdown
Member

I think the solution (for not having save(no_overwrite=True) scan the whole calendar) is to redefine the save.no-uid-conflict from ungraceful to quirk. It should be detected by the caldav-server-tester. All other servers graded to ungraceful with this behaviour should be regraded to quirk. The 412-behaviour should be recored in extra_keys. The definition of quirk is that the caldav server does not behave as it should according to the RFC, but that the caldav library is able to work around it. So the caldav library has to throw UidConflictConsistencyError when the quirk-mode is defined and a 412 is raised.

The test breakages seems to be due to some external change of how the CI jobs are done, I'll throw in a fix for that as well.

pytest 9.2.0 applies filterwarnings = error to warnings raised while
configuring, so pytest-asyncio's "asyncio_default_fixture_loop_scope
is unset" deprecation warning aborted the run before any test was
collected.  Set it to "function", which is what the unset option
already resolves to for our fixtures.

Prompt: Why does the tests in python-caldav#826 fail?
Prompt: I just merged python-caldav#833, so tests should be running for the master branch now.  Please have a look (master CI has the same failure)
Prompt: verify, and fix it in PR python-caldav#826

Assisted-By: Claude Sonnet 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
@SashankBhamidi

Copy link
Copy Markdown
Collaborator

So the caldav library has to throw UidConflictConsistencyError when the quirk-mode is defined and a 412 is raised.

This one matters for calendaring-sync too: there a UidConflictError becomes a conflict the app resolves, but a plain ConsistencyError on a create means "try again next sync", so on iCloud a taken UID would otherwise be sent again forever.

@tobixen

tobixen commented Oct 28, 2026

Copy link
Copy Markdown
Member

I tried to let Sonnet fix this, but it was a disaster. I'm now letting Opus have a go, here is the latest spec (and Opus has already found some weaknesses in it):


What I want to achieve in the end:

  • event.save() (with or without no_overwrite) should raise UidConflictConsistencyError or UidConflictError also on correctly configured servers that does not adhere to the RFC.
  • save(no_overwrite=True) should not result in the whole calendar being loaded to check for UID-conflicts when a server is configured as icloud.

Here are some thoughts:

  • unknown is the default, but why? For most other features, the default is "RFC-complient". Unknown being the default is acceptable if and only if

  • save.no-uid-conflict has a description that is hard to read, so it should be rewritten.

  • The description says ... "A PUT of an object whose UID another object resource in the same calendar already holds is refused". Then I think it's wrong to grade it ungraceful for servers that rejects duplicated UIDs.

  • To get "full" on this one, the RFC needs to be honored. That means returning 403 or 409, and a correct XML-error.

  • If the server allows to objects with same UID on same calendar, then that's a fail. If the server disallows it in a non-RFC-compliant way then it should most likely be graded quirk, and extra keys should be used to tell what kind of failure it is; one boolean on weather the xml element was given, and one int for the status code. If no xml element was found, the caldav-server-tester needs to do some double checking before grading it "quirk":

    • Check that the object was not saved and that a GET on the URL gives 404.
    • Retry the same PUT, with the same icalendar data and the same filename, just with a different UID. It should go through with 200 OK. If it also gives an error, then the test is inconclusive.
  • The RFC also states that the error SHOULD yield a href. This behaviour could go as a child save.no-uid-conflict.href-given.

There is a slightly related thing here which should probably be filed - what if an object exists on a filename with an UID and the client attempts changing the UID? Is that a breach of the RFC? Can we add probes for it in the caldav-server-tester?

A duplicate UID refused otherwise than the RFC's 403/409 with
CALDAV:no-uid-conflict is now a 'quirk' (keys 'status', 'precondition').
On a bare status, save() raises UidConflictError when a GET finds the
URL empty - not for 403, nor where a missing object gives no 404.  The
.if-none-match child grades no_overwrite's PUT apart (Bedework, iCloud:
bare 412), and no_overwrite skips the UID search wherever duplicates are
refused.  Profiles re-measured.

Prompt: Please consider the `save.no-uid-conflict` "feature".  The last
two commits was (sic) done by Sonnet, but I think they should be rolled
back or amended.  I've written some thoughts in [a notes file], please
read through.  It it (sic) makes sense, then see if you can do a better
job on implementing it.  If something is unclear or not well enough
thought through, get back to me.
Prompt: (the notes: raise UidConflictError also on non-compliant
servers; no calendar download for no_overwrite on iCloud; rewrite the
description; 'full' only for 403/409 with the XML element; a bare
refusal is a 'quirk' with keys for status and element, after the tester
checks that GET gives 404 and the same PUT with another UID succeeds;
a href-given child; and whether changing the UID of an existing object
breaches the RFC)
Prompts (comments made inline in the document while reviewing it):
> Fair point - leave `.no-uid-conflict` ungraceful for servers reporting
> 403 with no XML error.  A `.save()` should not raise `UidConflictError`
> in those cases as it can be anything.  However, `ungraceful` is a
> better grade than `unsupported`.  The `save(no_overwrite)` does not
> need to do an UID-search if `.no-uid-conflict` is `ungraceful`.  If
> the URL is declared to be stable, a simple GET suffices.
> Unknown being the default is acceptable if and only if "full" would
> cause harmful side-effects on servers that does (sic) save duplicated
> UIDs easily.  In this case we do have harmful side-effects, so the
> default should be `unknown`, but this needs to be properly explained
> I think this is a feature (it obeys "SHOULD" in the RFC) and not a
> behaviour.
> That nothing in the caldav library reads it (as for now) is not an
> argument against testing it (this may be added as a memory note).
> I agree with most of the proposals/recommendations [above],
> so go ahead and implement.
> Let's avoid throwing an uncertain UidConflict if the `non-existing-not-found`-feature
> is not supported.
> (on the copy-pasted PUT wrappers) I hate duplicated code.  Consolidate
> wherever it's possible, if it's not too difficult.

Assisted-By: Claude Opus 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
@tobixen

tobixen commented Oct 28, 2026

Copy link
Copy Markdown
Member

So I've worked a bit more on this and pushed more commits to your branch, @Rusich90 (perhaps I should have been asking first, sorry for messing up your work). I believe the behaviour of both save(no_overwrite=True), save(), get_object_by_uid() and the definition of save.no-uid-conflict is as good as it can be. I will merge this in some few hours unless anyone has further comments.

So for iCloud, this should now apply:

  • With features="icloud", save(no_overwrite=True) of a new object no longer downloads the calendar.

  • save(no_overwrite=True) now raises UidConflictConsistencyError on iCloud if the UID already exists (with or without features configured).

Since I don't have an iCloud account I haven't tested those claims. I'd be happy if you could have a go at it.

Thanks for all the help so far - though, I'd like to ask for yet another run of the caldav-server-tester, current bleeding-edge branch on github together with the code in this pull request.

Stalwart's profile states save.no-uid-conflict as 'quirk', and an unset
.href-given child inherits that instead of its default 'full', though
Stalwart does name the holder; GMX names none.  testCheckCompatibility
caught both.  SOGo and Purelymail refuse without the precondition, so
their href-given is stated 'unknown'.  The description now says to state
the child alongside the parent.

Prompt: Please run the compatbility (sic) integration test over all the
servers
Followup-Prompt: how are the tests going?  if there are any failures,
start looking into it, and if things are passing, let me know.  The
purelymail test can take very long time.

Assisted-By: Claude Opus 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
save(no_overwrite=True) on a server that searches without a wildcard
(Zimbra) raised a plain UidConflictError when the server refused a
duplicate the search had missed, and so did save(no_create=True) where
it searches; both now raise UidConflictConsistencyError, as the wildcard
paths do.  testNoOverwriteSameUidOtherUrl expects that wherever the
refusal is not 'ungraceful', and the search.text.uid fragment says it
inherits search.text.

Prompts (comments made inline in the document while reviewing it):
> (on how the review findings would be fixed) I read many of the
> recommendations as "we'll first merge the PR and then make another
> commit directly on master".  Wouldn't it be better to ship yet another
> bugfixing commit to the PR-branch?  (At the other hand, things that
> are not relevant for this PR should probably be fixed after the merge
> - like the purelymail etag.honoured fix).

Assisted-By: Claude Opus 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
@Rusich90

Copy link
Copy Markdown
Contributor Author

No problem with the commits on my branch. It's your project, so feel free to change anything you need =)

I tested de2515b on iCloud, with features=None and features="icloud", sync and async client. Both claims are correct:

  • features="icloud", save(no_overwrite=True) of a new object: one request, PUT 201. With the PR at 6cadd58 it was GET 404, a full REPORT, then the PUT.
  • save(no_overwrite=True) with a UID that another object already uses (different URL): UidConflictConsistencyError.
    • With features="icloud": PUT 412, GET 404.
    • Without features: REPORT 412, GET 404, full REPORT. This is expected, as save.no-uid-conflict.if-none-match is unknown by default.

caldav-server-tester bleeding-edge (f5906ea) with this PR:

save.no-uid-conflict:
  behaviour: refused with a bare 412
  precondition: false
  status: 412
  support: quirk
save.no-uid-conflict.href-given:
  support: unknown
save.no-uid-conflict.if-none-match:
  behaviour: refused with a bare 412
  precondition: false
  status: 412
  support: quirk

The rest of the output is the same as in my previous run, except for search.text.uid (see 2).

Two small things:

  1. The iCloud profile has only save.no-uid-conflict.if-none-match ("a plain PUT was not probed"). The tester now reports the parent too, so the profile could have:
    "save.no-uid-conflict": {"support": "quirk", "status": 412, "precondition": False, "behaviour": "refused with a bare 412"},
    "save.no-uid-conflict.href-given": {"support": "unknown"},
    Then the .if-none-match line and its comment are not needed. I tried it with the first line added to the profile: a plain save() with a UID that another object already uses raises UidConflictError (PUT 412, GET 404). With the current profile it is a PutError. Could you add these lines before merging?
  2. feat/search-text-uid (f2b74a0) is not in bleeding-edge anymore, so search.text.uid was not checked in this run. The profile already has search.text.uid, so this does not block the PR.

Looks fine.

@tobixen

tobixen commented Oct 28, 2026

Copy link
Copy Markdown
Member

I tested de2515b on iCloud, with features=None and features="icloud", sync and async client.

... and in the meanwhile, my Claude Opus did yet another review and found even more nits and corner cases to complain about. But I don't think the last commit should affect anything.

Two small things:

... so it seems like there will be yet another commit here ... never-ending story :-)

tobixen added a commit to python-caldav/caldav-server-tester that referenced this pull request Oct 28, 2026
The probe stored <uid>.ics, which a server that names every object
<uid>.ics passes as stable.  It now also stores an object under another
name and looks it up by UID.  A different name is unsupported, and the
extra key forces_uid_ics is set when the name is <uid>.ics, which lets
get_object_by_uid() in caldav skip its full scan.

Prompt: the caldav-server-tester also has to probe save-load.stable-url properly [= a probe for the forces_uid_ics key added to save-load.stable-url in the caldav library, in the context of python-caldav/caldav#826]
Assisted-By: Claude Sonnet 5.5
Rusich90's caldav-server-tester run shows that iCloud answers a plain
PUT of a taken UID with the same bare 412 as one carrying If-None-Match,
so the profile states the parent instead of the .if-none-match child,
and href-given as 'unknown' (no precondition to read it from).  A plain
save() of a taken UID now raises UidConflictError there rather than
PutError.

Prompt: check the last comment in the PR [Rusich90's comment
python-caldav#826 (comment)
asking for the iCloud profile to state save.no-uid-conflict before
the merge]

Assisted-By: Claude Opus 5.5
Reviewed-by: Tobias Brox <tobias@redpill-linpro.com>
@tobixen

tobixen commented Oct 28, 2026

Copy link
Copy Markdown
Member

There. The bleeding-edge branch at the caldav-server-tester has also been updated

@tobixen

tobixen commented Oct 28, 2026

Copy link
Copy Markdown
Member

I'll just merge this now.

Thanks @Rusich90!

For the reference of anyone visiting this PR in the future, the PR description above was written before the later commits, so it's not completely up-to-date, here are two highlights:

  • There is a feature flag now: search.text.uid. If configuring the server with the icloud-profile, the server-side search is skipped.
  • The workaround of last resort is to download the full calendar and do client-side filtering. This is considered expensive, so the latest version has some extra checks trying to ensure it will only be run when needed, rather than on all errors.

The merge commit's message has the summary.

@tobixen
tobixen merged commit 9b26954 into python-caldav:master Oct 28, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iCloud: UID lookups fail with 412

3 participants