diff --git a/AI-POLICY.md b/AI-POLICY.md index 5a1d94f3..f9b9813c 100644 --- a/AI-POLICY.md +++ b/AI-POLICY.md @@ -1,20 +1,20 @@ -# Policy on usage of Artifical Intelligence and other tools +# Policy on usage of Artificial Intelligence and other tools ## Read this first -The most important rule: be honest and inform about it! +The most important rule: be honest and inform about it! The +second-most important rule: YOU should add value to the project. That +value may be to find and reproduce a bug and verify that an +AI-generated fix solves the issue. -Tools should generally be used for improving the quality of the -project, not for rapidly adding new features. - -Avoid asking your AI agent to "find some low-hanging fruit in the -issue tracker and generate a PR". The maintainer can probably do that -himself, with a better overview of what needs to be prioritized. -Reviewing an AI-generated PR often takes the maintainer more effort than -doing the work directly. +If you ask the AI to "find some low-hanging fruit in the issue tracker +and generate a PR", then you're not adding value to the project. The +maintainer can probably do that himself, with a better overview of +what needs to be prioritized. Reviewing an AI-generated PR often +takes the maintainer more effort than doing the work directly. Keep a log of the prompts and/or commands used - this should be -included verbatimely in the git commits as long as it's possible +included verbatim in the git commits as long as it's possible without making the messages too messy. When relevant, chat-output may also need to be included or paraphrased. The `docs/design`-folder can be used for dumping AI-generated design documents, code reviews, @@ -22,9 +22,31 @@ prompts that are too large for being included in the commit message, etc. If you think the rest of the document is too long to read, at least -inform the AI that the policy should be followed. "YOU" in this +ask the AI to ensure the policy is followed. "YOU" in this document means the human, not the AI. +## NLnet funding and its GenAI policy + +This project receives funding from [NLnet](https://nlnet.nl/), and has +done so since before NLnet introduced a GenAI policy. The project is +currently committed to [version +1.1](https://nlnet.nl/foundation/policies/generativeAI/1.1/) of that +policy, which allows GenAI-assisted work as long as it is disclosed and +logged, and this document is written to fit it. [Version +2.0](https://nlnet.nl/foundation/policies/generativeAI/2.0/) of +2026-10-20 only allows GenAI to be used in a read-only manner (code +review, for instance), and NLnet wants it adopted. This document will +have to be revised if the project continues to receive NLnet funding. +Until that is settled, the rules below apply to all contributions. + +The priority is to deliver a good, feature-complete, easy-to-use and +bug-free library that stays compatible with as many servers as possible. +NLnet funding makes it possible for me to prioritize this, and spend +some hours a week rather than some hours a month on this project. +However, as the AI-models keep improving it may be questioned whether a +few hours a month of unfunded "vibe engineering" is of higher +value than a few hours a week of funded manual human labour. + ## Transparency matters If you've spent hours, perhaps a full day of your time writing up a @@ -86,7 +108,7 @@ the AI did to fix it - **just remember to be honest and transparent about it**. the pull request was generated by tools, then it's relevant. Do not write "I found this issue and here is a fix", but rather "I ran the (ruff tool, code review by Claude Mythos, etc) on the code, it found - this issue, and here is the fix". If the AI was fixing a bug for + this issue, and here is the fix it made". If the AI was fixing a bug for you, then write in the pull request that "this bugfix was vibed up, I have no idea what the AI has done here". End the commit message with "Assisted-By: ..." ("Co-Authored-By: ..." is deprecated). @@ -111,29 +133,29 @@ the AI did to fix it - **just remember to be honest and transparent about it**. the Code of Conduct, but at the end of the day, it's **YOUR** responsibility that the contribution follows those guidelines. -## Disclosure of GenAI usage, 2.2.6 - 3.3.1 +## Disclosure of GenAI usage The maintainer started playing with Claude Code in the end of 2025 - and [blogged about it](https://www.redpill-linpro.com/techblog/2026/03/20/from-luddite-to-vibe-coder.html) -Releases 2.2.6 - 3.3.1 has been heavily assisted by Claude - which is pretty obvious when looking into the commit messages. My experiences has been mixed - sometimes it seems to be doing a better and faster job than me, other times it seems to be making a mess a lot faster than what I can do it. Despiteof (or because of?) using Claude extensively, I spent much more time on the 3.0.0-release than estimated. +Since 2.2.6, development has been heavily assisted by Claude - which is pretty obvious when looking into the commit messages. My experiences have been mixed - sometimes it seems to be doing a better and faster job than me, other times it seems to be making a mess a lot faster than what I can do it. Despite (or because of?) using Claude extensively, I spent much more time on the 3.0.0-release than estimated. However, the models are getting better, and my understanding of how to make the best out of it is also growing. -Lots of time and efforts have been spent on doing QA on the changes, fixing up things and/or asking Claude to do a better job. The surge of issues reported after the 3.0-release is probably unrelated to the AI usage - it's a result of trying to shoehorn both async and API changes into it without breaking backward compatbility and without duplicating too much code. The CHANGELOG.md entry for 3.0 explicitly declared a caveat: "there are massive code changes in version 3.0, so if you're using the Python CalDAV client library in some sharp production environment, I would recommend to wait for two months before upgrading". +As for the 3.0.0-release, lots of time and efforts have been spent on doing QA on the changes, fixing up things and/or asking Claude to do a better job. The surge of issues reported after the 3.0-release is probably unrelated to the AI usage - it's a result of trying to shoehorn both async and API changes into it without breaking backward compatibility and without duplicating too much code. The CHANGELOG.md entry for 3.0 explicitly declared a caveat: "there are massive code changes in version 3.0, so if you're using the Python CalDAV client library in some sharp production environment, I would recommend to wait for two months before upgrading". -Generated changes and human-made changes are often mixed up. I prefer "logical" commits containing self-sustained changesets, one of the things I'm often asking Claude to do is to do a rebase of a branch and organize the commits in one or few logical commits with descriptive commit messages. +Before I started to use the AI, I was aiming for "logical" commits containing independent, self-sustained changesets. I've more or less given up squashing commits and keeping a linear history, now a "change" (pull request on GitHub) more often than not contains multiple commits tied together by a merge-commit. ## Future plans of GenAI-usage -I do want to go back to the old ways for doing the "core development tasks" - new features and complex refactoring. If nothing else, it's important for maintaining my brain cells, coding skills and making sure all the changes sticks to my memory. The new policy is that GenAI-tools should be used mainly for improving quality, not speeding up the development. +I do want to go back to the old ways for doing the "core development tasks" - new features and complex refactoring. If nothing else, it's important for maintaining my brain cells, coding skills and making sure all the changes stick to my memory. The new policy is that GenAI-tools should be used mainly for improving quality, not speeding up the development. However, as my brain apparently has continued rotting due to extensive AI-usage, combined with improved models, I haven't been able to get back to "the old ways". -I still intend to use GenAI heavily for certain tasks, particularly anything that is either "mundane and tedious" or unrelated to the "the working end" of the library. Examples: +GenAI is very well suited for certain tasks, particularly anything that is either "mundane and tedious" or unrelated to "the working end" of the library. Examples: -* Code reviews! I now have a policy that all changesets and releases should go through AI-driven code review. By itself it sounds like a good idea, though one should be aware of the risk is that this comes *instead* of human reviews rather than as an addition. (Perhaps I need to tweak my prompts a bit - it seems that if looping through "do code review", "fix code review issues", eventually it will rather hallucinate up issues than coming up with the conclusion "no issues found"). +* Code reviews! I now have a policy that all changesets and releases should go through AI-driven code review. By itself it sounds like a good idea, though one should be aware of the risk that this comes *instead* of human reviews rather than as an addition. (Perhaps I need to tweak my prompts/workflows a bit - it seems that if looping through "do code review", "fix code review issues", eventually it will rather hallucinate up issues than coming up with the conclusion "no issues found"). * Writing up test code. I do believe "test driven development" is a good idea (write test code first, then the logic), but writing tests may be both tedious and difficult. Claude can make them really fast, though it still needs some QA, care should be taken to ensure it's testing the right thing. -* Debugging. It's easy to get stuck and spend tons of time on debugging - sometimes (but not always) Claude can find them easily. (Best approach is sometimes to do manual debugging in parallell with AI-driven debugging ... sometimes I "win", othertimes the AI "wins"). -* Minor bugfixes ... the bugfix itself may be a one line changeset, but tests and debugging takes time. -* Maintaining the integration test framework. I've consistently failed setting up and maintaining a "battery" of caldav servers from 2013 to 2025, thanks to Claude we now have it in place. It's important, a majority of issues reported are about compatibility problems, the more servers I have for testing every release, the less troubles will be discovered downstream. +* Debugging. It's easy to get stuck and spend tons of time on debugging - sometimes (but not always) Claude can find them easily. (Best approach may sometimes be to do manual debugging in parallel with AI-driven debugging ... sometimes I "win", other times [Claude Sonnet 4.5] "wins". Edit: Opus 5.5 almost always wins). +* Minor bugfixes ... the bugfix itself may be a one line changeset, but tests and debugging take time. +* Maintaining the integration test framework. I've consistently failed setting up and maintaining a "battery" of caldav servers from 2013 to 2025, thanks to Claude we now have it in place. It's important, a majority of issues reported are about compatibility problems, the more servers I have for testing every release, the fewer troubles will be discovered downstream. * Setting up CI-related automated QA tests, pipelines etc -* The companion caldav-server-checker tool is quite suitable for GenAI-work - it's a bit like test code, writing up the checks to discover various server issues is rather tedious and time-consuming. Without AI-help I would probably have covered less than half of the "features" that are now tested for. -* Investigations of different architectural choices - like with the async work I had claude develop different design approaches and chose the one that I felt most comfortable with (though I'm still not sure that I did the right choice). -* Reading RFCs and quickly give a pointer to the relevant sections, or verifying that the code is according to the standards or not (but care should be taken - I've seen Claude hallucinating completely wrong RFC references). -* Various other mundane and tedious work (i.e. "I left a TODO-note in the code over there, could you have look into it and eliminate it?"). +* The companion caldav-server-tester tool is quite suitable for GenAI-work - it's a bit like test code, writing up the checks to discover various server issues is rather tedious and time-consuming. This tool has grown a lot in 2026 thanks to Claude. +* Investigations of different architectural choices - like with the async work I had Claude develop different design approaches and chose the one that I felt most comfortable with (though I'm still not sure that I did the right choice). +* Reading RFCs and quickly giving a pointer to the relevant sections, or verifying that the code is according to the standards or not (but care should be taken - I've seen Claude hallucinating completely wrong RFC references). +* Various other mundane and tedious work (e.g. "I left a TODO-note in the code over there, could you have a look at it and eliminate it?"). diff --git a/caldav/calendarobjectresource.py b/caldav/calendarobjectresource.py index 138cd55c..ce591b38 100644 --- a/caldav/calendarobjectresource.py +++ b/caldav/calendarobjectresource.py @@ -1225,11 +1225,19 @@ def _put( ) -> "None | Coroutine[Any, Any, None]": ## TODO: this is low-level http-communication - shouldn't it be in the davclient file rather than in calendarobjectresource.py? headers = self._put_headers(precondition, wildcard) + body = self.data + features = getattr(self.client, "features", None) + if ( + features is not None + and features.is_supported("save-load.icalendar.tzid-link", str) == "ungraceful" + ): + ## The server refuses a TZID that is a tz database link (Bedework 5) + body = vcal.canonical_tzids(str(body)) if self.is_async_client: return self._async_send_put( - str(self.data), headers, retry_on_failure, precondition, wildcard + str(body), headers, retry_on_failure, precondition, wildcard ) - return self._send_put(self.data, headers, retry_on_failure, precondition, wildcard) + return self._send_put(body, headers, retry_on_failure, precondition, wildcard) def _send_put( self, body, headers, retry_on_failure, precondition: Precondition, wildcard: Wildcard | None diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 84174688..fcb9b4bd 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -494,6 +494,7 @@ class FeatureSet: "save-load.todo.recurrences.count": {"description": "The server will receive and store a recurring task with a count set in the RRULE", "default": {"support": "full"}}, "save-load.todo.recurrences.thisandfuture": {"description": "Completing a recurring task with rrule_mode='thisandfuture' works (modifies RRULE and saves back to server)", "default": {"support": "full"}}, "save-load.todo.mixed-calendar": {"description": "The same calendar may contain both events and tasks (Zimbra only allows tasks to be placed on special task lists)", "default": {"support": "full"}}, + "save-load.todo.dtstart-without-due": {"description": "A task with a DTSTART and no DUE (RFC 5545 section 3.6.2 allows either alone) is stored with its DTSTART. 'unsupported' when the DTSTART is dropped, the behaviour saying whether it is kept beside a DUE: Zimbra drops it, but keeps it where the task also has a DUE.", "default": {"support": "full"}, "links": ["https://datatracker.ietf.org/doc/html/rfc5545#section-3.6.2"]}, "save-load.todo.uncomplete": {"description": "A completed task can be made pending again (Todo.uncomplete()): after saving it with STATUS:NEEDS-ACTION and without COMPLETED, it loads back that way.", "default": {"support": "full"}}, "save-load.todo.attendee": {"description": "A task with an ATTENDEE (RFC 5545 section 3.6.2 allows it in a VTODO) is stored with it. The probe sends no ORGANIZER, so no scheduling applies. 'unsupported' when the ATTENDEE is dropped. 'broken' when the task with it is gone although the PUT succeeded: the PUT is answered with 201 and the GET with 404, while the same task without the ATTENDEE is kept. 'ungraceful' when it is refused (Ox, with 409).", "default": {"support": "full"}}, "save-load.todo.alarm": {"description": "A task's VALARM is stored as sent: its TRIGGER, relative to the task's DTSTART, and its DESCRIPTION. A property the server adds to the VALARM, such as a UID, is fine. 'unsupported' when the VALARM is dropped. 'broken' when it is kept but rewritten (Ox makes the TRIGGER absolute, so the alarm no longer follows the task when it is moved), the behaviour saying what was changed: the alarm is still there, but not the one sent, which is worse than a missing one.", "default": {"support": "full"}, "links": ["https://datatracker.ietf.org/doc/html/rfc5545#section-3.6.6", "https://datatracker.ietf.org/doc/html/rfc5545#section-3.8.6.3"]}, @@ -620,6 +621,11 @@ class FeatureSet: "default": {"support": "full"}, "links": ["https://datatracker.ietf.org/doc/html/rfc5545#section-3.6.5", "https://datatracker.ietf.org/doc/html/rfc7809"], }, + "save-load.icalendar.tzid-link": { + "description": "An object whose TZID is a link name of the tz database (Europe/Oslo, a link to Europe/Berlin since tzdata 2022b; Asia/Calcutta, to Asia/Kolkata), sent with its VTIMEZONE, is stored with that TZID. 'ungraceful' when it is refused: Bedework 5 answers 403 with valid-calendar-object-resource and unknownTimezone, as its time zone service knows each zone by the TZID inside its definition only, and the definition of a link carries its target's. save() then sends the zone the link points to instead, which keeps every time since 1970 where it was (before 1970 a link may have had rules of its own); save(raw_data=...) does not. 'quirk' when the TZID is stored as the zone it points to, the behaviour saying so. 'unsupported' when it comes back as another TZID while an event in the zone it points to keeps its own.", + "default": {"support": "full"}, + "links": ["https://data.iana.org/time-zones/tz-link.html"], + }, "save-load.icalendar.olson-tzid": { "description": "An object whose TZID names a time zone of the tz database (\"Olson\" names, such as Europe/Berlin) is placed in that time zone without a VTIMEZONE for it. RFC 5545 section 3.6.5 requires the VTIMEZONE, so the default is 'unsupported', and a client should send one; where 'full', it may leave the VTIMEZONE out and keep the object small. The probe stores an event with TZID=Asia/Tokyo and no VTIMEZONE, and compares where time-range searches in UTC find it with where they find the same event carrying its VTIMEZONE and one at 12:00 UTC. 'full' when it is placed as the event with the VTIMEZONE is, so the verdict says whether the bare TZID is read as the zoned one; whether a zoned time is placed right is search.time-range.event.tzid. 'unsupported' when it is placed as UTC or elsewhere, 'ungraceful' when the event is refused. 'fragile' where the reading depends on circumstances, such as the client (Synology's web UI). The behaviour notes whether the server adds a VTIMEZONE of its own. A client of the server may still misread a bare TZID where the server does not: Synology's web UI was reported to read it as UTC (https://github.com/python-caldav/caldav/issues/849). See also save-load.icalendar.vtimezone, whether a VTIMEZONE sent is kept.", "default": {"support": "unsupported"}, @@ -2017,6 +2023,8 @@ def compare(self, observed): "scheduling.auto-schedule": {"support": "full"}, ## Probed 2026-11-06 by caldav-server-tester "save-load.icalendar.olson-tzid": {"support": "full", "behaviour": "the server adds a VTIMEZONE for Asia/Tokyo"}, + ## Probed 2026-11-15 by caldav-server-tester, on 8.8 and on 10.1 + "save-load.todo.dtstart-without-due": {"support": "unsupported", "behaviour": "the DTSTART is dropped; with a DUE, both are kept"}, } ## Zimbra 10.1.21, measured with caldav-server-tester 2026-10-29 against an @@ -2314,6 +2322,10 @@ def __getattr__(name: str) -> Any: "save.etag.matches-stored": {"support": "broken", "behaviour": "stored with CREATED and LAST-MODIFIED added, PRODID and DTSTAMP changed, yet the PUT answered with a strong ETag"}, ## Probed 2026-11-06 by caldav-server-tester "save-load.icalendar.olson-tzid": {"support": "full", "behaviour": "the server adds a VTIMEZONE for Asia/Tokyo"}, + ## Probed 2026-11-08 by hand: Europe/Oslo, Stockholm, Copenhagen, Amsterdam, Zagreb, Kiev, Belfast, + ## Arctic/Longyearbyen, America/Montreal and Asia/Calcutta refused, VTIMEZONE or not; Europe/Berlin, + ## Europe/Kyiv and the other canonical zones tried stored. Bedework 3.10 stores them all. + "save-load.icalendar.tzid-link": {"support": "ungraceful", "behaviour": "403 valid-calendar-object-resource: unknownTimezone Europe/Oslo"}, } baikal = { ## version 0.10.1 diff --git a/caldav/datastate.py b/caldav/datastate.py index c5fd5d72..6620ddb9 100644 --- a/caldav/datastate.py +++ b/caldav/datastate.py @@ -156,7 +156,7 @@ def get_uid(self) -> str | None: self._data[: match.start()], re.IGNORECASE | re.MULTILINE, ): - token, name = header.group(1).upper(), header.group(2).upper() + token, name = header.group(1).upper(), header.group(2).strip().upper() if token == "BEGIN": stack.append(name) elif token == "END" and stack and stack[-1] == name: diff --git a/caldav/lib/vcal.py b/caldav/lib/vcal.py index d0dd02a7..acdd5aab 100644 --- a/caldav/lib/vcal.py +++ b/caldav/lib/vcal.py @@ -1,8 +1,12 @@ #!/usr/bin/env python import datetime +import functools +import importlib.resources import logging +import pathlib import re import uuid +import zoneinfo from collections.abc import Iterator import icalendar @@ -14,6 +18,83 @@ fixup_error_loggings = 0 +@functools.cache +def tzdata_links() -> dict[str, str]: + """The tz database's link names, each to the zone it points to (Europe/Oslo to Europe/Berlin). + + Read from the ``tzdata`` package's ``tzdata.zi`` where it is installed, else from the + system's (``zoneinfo.TZPATH``); empty where neither is found. + """ + sources = [] + try: + sources.append(importlib.resources.files("tzdata").joinpath("zoneinfo", "tzdata.zi")) + except ModuleNotFoundError: + pass # tzdata is optional; fall back to the system's tz database below + sources += [pathlib.Path(path, "tzdata.zi") for path in zoneinfo.TZPATH] + for source in sources: + try: + text = source.read_text(encoding="utf-8") + except (OSError, ValueError): + continue + return _parse_links(text) + return {} + + +def _parse_links(text: str) -> dict[str, str]: + """The ``L`` lines of a ``tzdata.zi``, each link to the zone it ends up at.""" + links = {} + for line in text.splitlines(): + if line.startswith("L "): + _, target, link = line.split() + links[link] = target + for link, target in links.items(): # a link to a link points on to the zone + seen = {link} + while target in links and target not in seen: + seen.add(target) + target = links[target] + links[link] = target + return links + + +def canonical_tzids(data: str) -> str: + """``data`` with every TZID that is a tz database link given as the zone it points to. + + A VTIMEZONE for a link is renamed, or dropped where one for the zone is there already. + ``data`` comes back as it is where it names no link. + """ + ## UTC and GMT are links to Etc/UTC and Etc/GMT, but icalendar reads a + ## TZID=UTC time as a UTC one and would write it with both a TZID and a Z, + ## which RFC 5545 forbids. + links = { + link: zone for link, zone in tzdata_links().items() if zone not in ("Etc/UTC", "Etc/GMT") + } + cal = icalendar.Calendar.from_ical(data) + changed = False + named = {str(tz["TZID"]) for tz in cal.walk("VTIMEZONE") if "TZID" in tz} + for component in cal.walk(): + if component.name == "VTIMEZONE": + continue + for value in component.values(): + for prop in value if isinstance(value, list) else [value]: + params = getattr(prop, "params", None) + if params and params.get("TZID") in links: + params["TZID"] = links[params["TZID"]] + changed = True + for tz in list(cal.walk("VTIMEZONE")): + target = links.get(str(tz.get("TZID"))) + if target is None: + continue + changed = True + if target in named: + cal.subcomponents.remove(tz) + else: + if str(tz.get("X-LIC-LOCATION")) == str(tz["TZID"]): + tz["X-LIC-LOCATION"] = target + tz["TZID"] = target + named.add(target) + return cal.to_ical().decode() if changed else data + + def parse_ical(data, context: str | None = None) -> icalendar.Calendar: """icalendar.Calendar.from_ical(), with a usable error for a body that holds no iCalendar at all. diff --git a/caldav/response.py b/caldav/response.py index fbd7670d..254449f2 100644 --- a/caldav/response.py +++ b/caldav/response.py @@ -106,12 +106,16 @@ def _only_xml_declaration(raw: bytes | str) -> bool: def _validate_status(status: str | None) -> None: """Validate a status string like "HTTP/1.1 404 Not Found". - 200, 201, 207 and 404 are considered acceptable statuses. + 200, 201, 207 and 404 are considered acceptable statuses. Only the code + counts: Bedework 5 sends a bare "HTTP/1.1 404", without the reason + phrase and without the space RFC 9112 section 4 keeps before an empty + one. """ if status is None: return - if not any(code in status for code in (" 200 ", " 201 ", " 207 ", " 404 ")): - raise error.ResponseError(status, status=_parse_status_code(status)) + code = _parse_status_code(status) + if code not in (200, 201, 207, 404): + raise error.ResponseError(status, status=code) def _parse_status_code(status: str) -> int | None: @@ -188,7 +192,7 @@ def _collect_prop_elements(propstats: "list[_Element]") -> "dict[str, _Element]" status_elem = propstat.find(dav.Status.tag) if status_elem is not None and status_elem.text: _validate_status(status_elem.text) - if " 404 " in status_elem.text: + if _parse_status_code(status_elem.text) == 404: continue for prop in propstat.iterfind(dav.Prop.tag): for child in prop: @@ -660,7 +664,7 @@ def _parse_response(self, response: _Element) -> tuple[str, list[_Element], Any ## 404s; flag it in debug mode if a server pairs them with some ## other status so we notice and revisit this handling. error.assert_("404" in status) - if status and " 404 " in status: + if status and _parse_status_code(status) == 404: ## The resource is gone, so whatever its propstats say is moot. ## Infomaniak adds an empty propstat with 418 to a removed member ## of a sync-collection reply, which would otherwise raise. @@ -806,10 +810,8 @@ def all_statuses_ok(self) -> bool: for status in response.iter(dav.Status.tag): ## _status_to_code() falls back to 200 for anything it cannot ## parse, which would turn a garbled status into a success - parts = (status.text or "").split() - if len(parts) < 2 or not parts[1].isdigit(): - return False - if not 200 <= int(parts[1]) < 300: + code = _parse_status_code(status.text or "") + if code is None or not 200 <= code < 300: return False return True diff --git a/changes/+status-without-reason.fixed.md b/changes/+status-without-reason.fixed.md new file mode 100644 index 00000000..d5e94a17 --- /dev/null +++ b/changes/+status-without-reason.fixed.md @@ -0,0 +1 @@ +A status line without a reason phrase, such as Bedework 5's bare `HTTP/1.1 404` for a missing object in a calendar-multiget, is no longer an error: the status code alone decides. diff --git a/changes/+todo-dtstart-without-due.added.md b/changes/+todo-dtstart-without-due.added.md new file mode 100644 index 00000000..23cd96cf --- /dev/null +++ b/changes/+todo-dtstart-without-due.added.md @@ -0,0 +1 @@ +New server hint `save-load.todo.dtstart-without-due`: whether a task with a DTSTART and no DUE is stored with its DTSTART. Zimbra drops it. diff --git a/changes/+tzid-link.added.md b/changes/+tzid-link.added.md new file mode 100644 index 00000000..051dc820 --- /dev/null +++ b/changes/+tzid-link.added.md @@ -0,0 +1 @@ +New server hint `save-load.icalendar.tzid-link`: whether a server stores a TZID that is a link name of the tz database, such as Europe/Oslo (a link to Europe/Berlin). Bedework 5 refuses every such object with 403, VTIMEZONE or not, so on a server where it is `ungraceful`, `save()` sends the zone the link points to instead; a link and its zone agree on every time since 1970, so the times stay where they were. `save(raw_data=...)` still sends its bytes unchanged. diff --git a/tests/test_caldav_unit.py b/tests/test_caldav_unit.py index a28521c2..09ca388a 100755 --- a/tests/test_caldav_unit.py +++ b/tests/test_caldav_unit.py @@ -1080,7 +1080,7 @@ def test_get_calendars(self): xml = """ - /dav/tobias%40redpill-linpro.com/ + /dav/testuser%40example.com/ HTTP/1.1 200 OK @@ -1092,7 +1092,7 @@ def test_get_calendars(self): - /dav/tobias%40redpill-linpro.com/Inbox/ + /dav/testuser%40example.com/Inbox/ HTTP/1.1 200 OK @@ -1105,7 +1105,7 @@ def test_get_calendars(self): - /dav/tobias%40redpill-linpro.com/Emailed%20Contacts/ + /dav/testuser%40example.com/Emailed%20Contacts/ HTTP/1.1 200 OK @@ -1118,7 +1118,7 @@ def test_get_calendars(self): - /dav/tobias%40redpill-linpro.com/Calendarc5f1a47c-2d92-11e3-b654-0016eab36bf4.ics + /dav/testuser%40example.com/Calendarc5f1a47c-2d92-11e3-b654-0016eab36bf4.ics HTTP/1.1 200 OK @@ -1128,7 +1128,7 @@ def test_get_calendars(self): - /dav/tobias%40redpill-linpro.com/Yep/ + /dav/testuser%40example.com/Yep/ HTTP/1.1 200 OK @@ -1143,7 +1143,7 @@ def test_get_calendars(self): """ client = MockedDAVClient(xml) - calendar_home_set = CalendarSet(client, url="/dav/tobias%40redpill-linpro.com/") + calendar_home_set = CalendarSet(client, url="/dav/testuser%40example.com/") assert len(calendar_home_set.get_calendars()) == 1 def test_get_calendars_client_url_is_calendar_without_home_set(self): @@ -6241,6 +6241,44 @@ def test_404_propstat_is_still_a_missing_property(self): result = MockedDAVResponse(xml).expand_simple_props(props=[dav.DisplayName()]) assert result == {"/dav/calendars/user/broken/": {"{DAV:}displayname": None}} + def test_a_404_without_a_reason_phrase_is_a_404(self): + """Bedework 5 answers a calendar-multiget naming a missing href with a + bare "HTTP/1.1 404": no reason phrase, nor the space RFC 9112 section + 4 keeps before an empty one.""" + xml = self.XML.replace("500 Internal Server Error", "404") + result = MockedDAVResponse(xml).expand_simple_props(props=[dav.DisplayName()]) + assert result == {"/dav/calendars/user/broken/": {"{DAV:}displayname": None}} + gone = """ + + + /dav/calendars/user/cal/missing.ics + HTTP/1.1 404 + + +""" + [result] = MockedDAVResponse(gone).parse_propfind() + assert result.status == 404 + + def test_a_multiget_of_a_missing_object_answered_without_a_reason_phrase(self): + """The Bedework 5 case: multiget(raise_notfound=True) reports it missing.""" + gone = """ + + + /calendar/bedework/missing.ics + HTTP/1.1 404 + + +""" + calendar = Calendar(MockedDAVClient(gone), url="/calendar/bedework/") + with pytest.raises(error.NotFoundError): + calendar.multiget([URL("/calendar/bedework/missing.ics")], raise_notfound=True) + + def test_a_failure_without_a_reason_phrase_still_raises(self): + xml = self.XML.replace("500 Internal Server Error", "500") + with pytest.raises(error.ResponseError) as excinfo: + MockedDAVResponse(xml).expand_simple_props(props=[dav.DisplayName()]) + assert excinfo.value.status == 500 + @pytest.mark.parametrize( "status", ["500 Internal Server Error", "HTTP/1.1 Internal Server Error"] ) @@ -7092,6 +7130,24 @@ def test_folded_uid_needs_no_full_parse(self) -> None: with mock.patch("caldav.datastate.parse_ical", side_effect=AssertionError): assert RawDataState(self._ical("\r\n ")).get_uid() == self.UID + def test_trailing_space_after_component_name_needs_no_full_parse(self) -> None: + from caldav.datastate import RawDataState + + data = "BEGIN:VCALENDAR\r\nBEGIN:VEVENT \r\nUID:event-1\r\nEND:VEVENT \r\nEND:VCALENDAR\r\n" + with mock.patch("caldav.datastate.parse_ical", side_effect=AssertionError): + assert RawDataState(data).get_uid() == "event-1" + + @pytest.mark.parametrize("first_uid", ["UID;X-P=1:first", "uid:first"]) + def test_earlier_uid_line_is_left_to_the_parser(self, first_uid) -> None: + """A UID line the cheap regex skips still belongs to the first resource.""" + from caldav.datastate import RawDataState + + data = ( + "BEGIN:VCALENDAR\r\nBEGIN:VEVENT\r\n" + first_uid + "\r\nEND:VEVENT\r\n" + "BEGIN:VEVENT\r\nUID:second\r\nEND:VEVENT\r\nEND:VCALENDAR\r\n" + ) + assert RawDataState(data).get_uid() == "first" + def test_escaped_uid_is_unescaped(self) -> None: ical = self._ical("\n ").replace(self.UID[:30], "a\\,b" + self.UID[:30]) event = Event(data=ical) @@ -10222,3 +10278,187 @@ def test_not_when_the_caller_wrote_the_query(self, base) -> None: ) _, requests = self._search(base, expand=False, search_kwargs={"xml": query}) assert requests == ["query"] + + +class TestTzidLinkRewritten: + """Where ``save-load.icalendar.tzid-link`` is ungraceful, the server refuses + a TZID that is a tz database link name (Bedework 5: Europe/Oslo, a link to + Europe/Berlin since tzdata 2022b, is "unknownTimezone", VTIMEZONE or not). + save() then sends the zone the link points to instead.""" + + URL = "http://cal.example.com/dav/user/mycal/" + + @staticmethod + def _vtimezone(tzid): + return ( + "BEGIN:VTIMEZONE\r\n" + f"TZID:{tzid}\r\n" + "BEGIN:STANDARD\r\n" + "DTSTART:19701025T030000\r\n" + "RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU\r\n" + "TZOFFSETFROM:+0200\r\n" + "TZOFFSETTO:+0100\r\n" + "END:STANDARD\r\n" + "END:VTIMEZONE\r\n" + ) + + def _ical(self, *tzids, rid=None): + events = "".join( + "BEGIN:VEVENT\r\n" + "UID:tzlink-1\r\n" + "DTSTAMP:20261101T000000Z\r\n" + f"DTSTART;TZID={tzid}:20261103T100000\r\n" + f"DTEND;TZID={tzid}:20261103T110000\r\n" + + ( + f"RECURRENCE-ID;TZID={tzid}:20261103T100000\r\n" + if i + else f"RRULE:FREQ=DAILY;COUNT=4\r\nEXDATE;TZID={tzid}:20261104T100000,20261105T100000\r\n" + ) + + "SUMMARY:x\r\n" + "END:VEVENT\r\n" + for i, tzid in enumerate(tzids) + ) + zones = "".join(self._vtimezone(t) for t in dict.fromkeys(tzids)) + return ( + f"BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//t//EN\r\n{zones}{events}END:VCALENDAR\r\n" + ) + + def _save(self, data, support="ungraceful", raw=False): + sent = [] + features = {"save-load.icalendar.tzid-link": {"support": support}} + client = DAVClient(url="http://cal.example.com/dav/", features=features) + + def request(url, method="GET", body="", headers=None): + if method.upper() == "PUT": + sent.append(body.decode() if isinstance(body, bytes) else str(body)) + resp = mock.MagicMock() + resp.reason = "whatever" + resp.status_code = 201 if method.upper() == "PUT" else 404 + resp.content = b"" + resp.headers = {} + resp.text = "" + return DAVResponse(resp, client) + + client.request = request + event = Event( + client, url=self.URL + "tzlink-1.ics", data=data, parent=Calendar(client, url=self.URL) + ) + if raw: + event.save(raw_data=data.encode(), no_overwrite=False, no_create=False) + else: + event.save(increase_seqno=False) + [body] = sent + return event, icalendar.Calendar.from_ical(body), body + + @staticmethod + def _tzids(cal): + found = {str(c["TZID"]) for c in cal.walk("VTIMEZONE")} + for component in cal.walk("VEVENT"): + for name in ("DTSTART", "DTEND", "RECURRENCE-ID", "EXDATE"): + if name in component: + found.add(component[name].params["TZID"]) + return found + + def test_a_link_is_sent_as_the_zone_it_points_to(self) -> None: + event, sent, _ = self._save(self._ical("Europe/Oslo", "Europe/Oslo")) + assert self._tzids(sent) == {"Europe/Berlin"} + assert len(sent.walk("VTIMEZONE")) == 1 + assert "Europe/Oslo" in event.data, "the object as the caller holds it is not rewritten" + + def test_a_link_beside_its_target_leaves_one_vtimezone(self) -> None: + _, sent, _ = self._save(self._ical("Europe/Berlin", "Europe/Oslo")) + assert self._tzids(sent) == {"Europe/Berlin"} + assert len(sent.walk("VTIMEZONE")) == 1 + + def test_a_canonical_zone_is_sent_as_it_is(self) -> None: + data = self._ical("Europe/Berlin") + _, _, body = self._save(data) + assert "TZID=Europe/Berlin" in body and "Europe/Oslo" not in body + + def test_nothing_is_rewritten_where_the_server_takes_links(self) -> None: + for support in ("full", "unknown"): + _, sent, _ = self._save(self._ical("Europe/Oslo"), support=support) + assert self._tzids(sent) == {"Europe/Oslo"} + + def test_raw_data_is_sent_unchanged(self) -> None: + data = self._ical("Europe/Oslo") + _, _, body = self._save(data, raw=True) + assert body == data + + @pytest.mark.parametrize("utc", ["UTC", "Zulu", "GMT"]) + def test_a_utc_alias_is_left_alone(self, utc) -> None: + """Rewriting TZID=UTC would put a TZID on a UTC time, which RFC 5545 forbids.""" + from caldav.lib.vcal import canonical_tzids + + data = ( + "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//t//EN\r\nBEGIN:VEVENT\r\n" + f"UID:tzlink-1\r\nDTSTAMP:20261101T000000Z\r\nDTSTART;TZID={utc}:20261103T100000\r\n" + "END:VEVENT\r\nEND:VCALENDAR\r\n" + ) + assert canonical_tzids(data) == data + + def test_a_vtimezone_without_tzid_is_skipped(self) -> None: + from caldav.lib.vcal import canonical_tzids + + data = self._ical("Europe/Oslo") + nameless = self._vtimezone("Europe/Oslo").replace("TZID:Europe/Oslo\r\n", "") + data = data.replace("BEGIN:VEVENT", nameless + "BEGIN:VEVENT", 1) + assert "TZID=Europe/Berlin" in canonical_tzids(data) + + def test_the_links_come_from_the_tz_database(self) -> None: + from caldav.lib.vcal import tzdata_links + + links = tzdata_links() + assert links["Europe/Oslo"] == "Europe/Berlin" + assert links["Asia/Calcutta"] == "Asia/Kolkata" + assert "Europe/Berlin" not in links + + def test_x_lic_location_follows_the_rename(self) -> None: + data = self._ical("Europe/Oslo").replace( + "TZID:Europe/Oslo\r\n", "TZID:Europe/Oslo\r\nX-LIC-LOCATION:Europe/Oslo\r\n" + ) + _, sent, _ = self._save(data) + [tz] = sent.walk("VTIMEZONE") + assert str(tz["X-LIC-LOCATION"]) == "Europe/Berlin" + + def test_due_and_rdate_of_a_task_are_rewritten(self) -> None: + from caldav.lib.vcal import canonical_tzids + + data = ( + "BEGIN:VCALENDAR\r\nVERSION:2.0\r\nPRODID:-//t//EN\r\n" + + self._vtimezone("Europe/Oslo") + + "BEGIN:VTODO\r\nUID:tzlink-2\r\nDTSTAMP:20261101T000000Z\r\n" + "DTSTART;TZID=Europe/Oslo:20261103T100000\r\n" + "DUE;TZID=Europe/Oslo:20261103T110000\r\n" + "RDATE;TZID=Europe/Oslo:20261110T100000\r\n" + "END:VTODO\r\nEND:VCALENDAR\r\n" + ) + out = canonical_tzids(data) + assert "Europe/Oslo" not in out + [todo] = icalendar.Calendar.from_ical(out).walk("VTODO") + assert todo["DUE"].params["TZID"] == "Europe/Berlin" + assert todo["RDATE"].params["TZID"] == "Europe/Berlin" + + def test_a_link_to_a_link_points_on_to_the_zone(self) -> None: + from caldav.lib.vcal import _parse_links + + links = _parse_links( + "Z Europe/Berlin 1\nL Europe/Berlin Europe/Oslo\nL Europe/Oslo Arctic/Longyearbyen\n" + ) + assert links == {"Europe/Oslo": "Europe/Berlin", "Arctic/Longyearbyen": "Europe/Berlin"} + + def test_without_a_tz_database_nothing_is_rewritten(self, monkeypatch) -> None: + from caldav.lib import vcal + + def no_package(name): + raise ModuleNotFoundError(name) + + monkeypatch.setattr(vcal.importlib.resources, "files", no_package) + monkeypatch.setattr(vcal.zoneinfo, "TZPATH", ()) + vcal.tzdata_links.cache_clear() + try: + assert vcal.tzdata_links() == {} + data = self._ical("Europe/Oslo") + assert vcal.canonical_tzids(data) == data + finally: + vcal.tzdata_links.cache_clear() diff --git a/tests/test_vcal.py b/tests/test_vcal.py index 4bd60d48..b4a65be0 100644 --- a/tests/test_vcal.py +++ b/tests/test_vcal.py @@ -389,9 +389,9 @@ def test_vcal_fixups(self): RRULE:FREQ=WEEKLY;UNTIL=20221222T225959Z;BYDAY=WE SUMMARY:Competence Lunch DESCRIPTION: *removed* -ATTENDEE;CN=Tobias Brox;PARTSTAT=TENTATIVE:mailto:tobias@redpill-linpro.com +ATTENDEE;CN=Test User;PARTSTAT=TENTATIVE:mailto:testuser@example.com PRIORITY:9 -ORGANIZER;CN=Someone:mailto:noreply@redpill-linpro.com +ORGANIZER;CN=Someone:mailto:noreply@example.com DTSTART;TZID="Europe/Brussels":20220817T110000 DTEND;TZID="Europe/Brussels":20220817T113000 DURATION:PT30M