Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,11 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0
* A `features="icloud"` profile, from a caldav-server-tester run by a user - the documentation already referred to it, but it did not exist. https://github.com/python-caldav/caldav/issues/730
* Compatibility feature `sync-token.invalid-token`: whether a sync token the server never issued is refused with `DAV:valid-sync-token` as RFC 6578 section 3.2 says. Xandikos and CCS answer some such tokens with a 500, and Stalwart and Ox with a report and no error. https://github.com/jelmer/xandikos/issues/743
* Compatibility feature `propfind.principal-depth-1`: whether a Depth: 1 PROPFIND on the principal URL works. GMX answers it with a 400. https://github.com/python-caldav/caldav/pull/755
* Negated searches: `add_property_filter()` passes the `!=` (alias `<>`), `!contains` and `def` operators of icalendar-searcher 1.1.0 on to the search. They are decided client-side (unless `post_filter=False`): the server query leaves them out, as no server has been probed for `negate-condition` support yet. `build_search_xml_query()` renders `!contains` as a negated `text-match`, and `!=` and `def` as a bare `prop-filter` - a negated `text-match` is a negated *substring* match, so it cannot express `!=`. https://github.com/python-caldav/caldav/issues/568

### Changed

* icalendar-searcher 1.1.0 or later is required (for the negated searches).
* `search()` and `multiget()` (also used to load search results in bulk) ask the server for the ETag and the Schedule-Tag of each object, so a `save()` after a search or multiget now sends `If-Match` (or `If-Schedule-Tag-Match`, RFC 6638 section 3.2), and raises `ETagMismatchError` (or `ScheduleTagMismatchError`) if someone else changed the object in the meantime. Previously such a save overwrote the other change silently.

* Compatibility features `sync-token.limit` and `sync-token.limit.initial-sync`, for DAV:limit and the checkpoint tokens of a truncated sync-collection report (RFC 6578 sections 3.6 and 3.7). Hints for Stalwart, Xandikos, Radicale and Baïkal.
Expand Down
103 changes: 86 additions & 17 deletions caldav/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,21 @@ def _flag_component_type(searcher: "Searcher", flag: str) -> None:
setattr(searcher, flag, True)


## Operators the server is never trusted with: negations and "defined".
## Whether servers honour negate-condition, or a bare prop-filter, has not
## been probed yet (caldav-server-tester), so search() decides these
## client-side. https://github.com/python-caldav/caldav/issues/568
_NEGATED_OR_PRESENCE_OPERATORS = ("!=", "!contains", "def")


def _collation_str(searcher: "Searcher", property: str) -> str:
"""The CalDAV collation name for a property filter."""
if hasattr(searcher, "_property_collation") and property in searcher._property_collation:
case_sensitive = searcher._property_case_sensitive.get(property, True)
return _collation_to_caldav(searcher._property_collation[property], case_sensitive)
return "i;octet"


def _build_search_xml_query(
searcher: "Searcher",
server_expand: bool = False,
Expand Down Expand Up @@ -208,10 +223,32 @@ def _build_search_xml_query(
comp_filter = None

for property in searcher._property_operator:
if searcher._property_operator[property] == "undef":
match = cdav.NotDefined()
prop_name = "CATEGORIES" if property.lower() == "category" else property.upper()
filters.append(cdav.PropFilter(prop_name) + match)
operator = searcher._property_operator[property]
prop_name = (
"CATEGORIES" if property.lower() in ("category", "categories") else property.upper()
)
if operator == "undef":
filters.append(cdav.PropFilter(prop_name) + cdav.NotDefined())
elif operator == "!contains" and prop_name != "CATEGORIES":
## RFC 4791 9.7.5: a negated text-match is a negated substring match
value = searcher._property_filters[property]
if hasattr(value, "to_ical"):
value = value.to_ical()
filters.append(
cdav.PropFilter(prop_name)
+ cdav.TextMatch(value, collation=_collation_str(searcher, property), negate=True)
)
elif operator in _NEGATED_OR_PRESENCE_OPERATORS:
## A bare prop-filter matches any defined property (RFC 4791
## 9.7.2). That is all CalDAV can say about != (a negated
## text-match means "does not contain", not "not equal") and
## about def. Nor does a negated text-match fit !contains on
## categories: the plural form means "not all of these", which
## is not the AND of one negation per category, and the
## singular form matches within one category, which a match
## against the whole comma-joined property value is not. The
## client-side filter has to decide the rest.
filters.append(cdav.PropFilter(prop_name))
else:
value = searcher._property_filters[property]
property_ = property.upper()
Expand All @@ -226,17 +263,7 @@ def _build_search_xml_query(
if hasattr(value, "to_ical"):
value = value.to_ical()

collation_str = "i;octet"
if (
hasattr(searcher, "_property_collation")
and property in searcher._property_collation
):
case_sensitive = searcher._property_case_sensitive.get(property, True)
collation_str = _collation_to_caldav(
searcher._property_collation[property], case_sensitive
)

match = cdav.TextMatch(value, collation=collation_str)
match = cdav.TextMatch(value, collation=_collation_str(searcher, property))
filters.append(cdav.PropFilter(property_) + match)

if comp_filter and filters:
Expand Down Expand Up @@ -435,7 +462,8 @@ def add_property_filter(
Special virtual property "category" (singular) is also supported
for substring matching within category names
:param value: Filter value, should adhere to the type defined in the RFC
:param operator: Comparison operator ("contains", "==", "undef"). If not
:param operator: Comparison operator ("contains", "==", "!contains",
"!=", "def", "undef"; see below). If not
specified, the server decides the matching behavior (usually
substring search per RFC). If explicitly set to "contains"
and the server doesn't support substring search, client-side
Expand All @@ -454,18 +482,31 @@ def add_property_filter(
* **contains** - substring match (e.g., "rain" matches "Training session"
and "Singing in the rain")
* **==** - exact match required, enforced client-side
* **!contains** - no substring match; the property must be defined
* **!=** (or **<>**) - not an exact match; the property must be defined
* **def** - matches if property is defined (value parameter ignored)
* **undef** - matches if property is not defined (value parameter ignored)

The negations and ``def`` need icalendar-searcher 1.1.0 or later.
search() decides them client-side and leaves them out of the
server query, unless ``post_filter=False``; then, as with
build_search_xml_query() on its own, they are sent as the
closest CalDAV filter that never excludes a match, and != and
def match any object where the property is defined.

**Special handling for categories:**

- **"categories"** (plural): Exact category name matching
- "contains": subset check (all filter categories must be in component)
- "==": exact set equality (same categories, order doesn't matter)
- "!contains": not all of the filter categories are in the component
- "!=": set inequality
- Commas in filter values split into multiple categories

- **"category"** (singular): Substring matching within category names
- "contains": substring match (e.g., "out" matches "outdoor")
- "==": exact match to at least one category name
- "!contains" / "!=": no category name contains / equals the value
- Commas in filter values treated as literal characters

Examples:
Expand Down Expand Up @@ -629,7 +670,10 @@ def _search_impl(
or self.expand
or "categories" in self._property_filters
or "category" in self._property_filters
or any(op == "==" for op in self._property_operator.values())
or any(
op == "==" or op in _NEGATED_OR_PRESENCE_OPERATORS
for op in self._property_operator.values()
)
or not calendar.client.features.is_supported("search.text.case-sensitive")
or not calendar.client.features.is_supported("search.time-range.accurate")
)
Expand Down Expand Up @@ -699,6 +743,31 @@ def _search_impl(
)
return

## Negated and "defined" filters are left out of the server query and
## decided client-side; see _NEGATED_OR_PRESENCE_OPERATORS.
if cw and post_filter is not False:
untrusted = [
prop
for prop, op in self._property_operator.items()
if op in _NEGATED_OR_PRESENCE_OPERATORS
]
if untrusted:
clone = self._clone_without_filters(untrusted)
objects = yield (
SearchAction.RECURSIVE_SEARCH,
(clone, calendar, server_expand, split_expanded, props, xml, None, None),
)
yield (
SearchAction.RETURN,
self.filter(
objects,
post_filter=True,
split_expanded=split_expanded,
server_expand=server_expand,
),
)
return

## special compatibility-case for servers that do not support is-not-defined
## for specific properties (e.g. search.is-not-defined.category or .dtend)
if cw and post_filter is not False:
Expand Down
4 changes: 4 additions & 0 deletions docs/design/TODO.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Known Issues and TODO Items

## Server-side negated searches

`search()` decides the `!=` / `<>`, `!contains` and `def` operators client-side and leaves them out of the server query (https://github.com/python-caldav/caldav/issues/568). A caldav-server-tester probe for `negate-condition` support, and the compatibility_hints flag it would feed, would let `search()` send `!contains` as a negated `text-match` to the servers that honour it.

## Calendar cleanup in integration tests

For servers where calendars cannot be created, the test will take the first available calendar it finds and wipe it. That is not acceptable without the user explicitly configuring that this is OK.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ dependencies = [
"recurring-ical-events>=2.0.0",
"typing_extensions;python_version<'3.11'",
"icalendar>6.0.0",
"icalendar-searcher>=1.0.5,<2",
"icalendar-searcher>=1.1.0,<2",
"dnspython",
"python-dateutil",
"PyYAML",
Expand Down
Loading
Loading