[sertugkaya]

Upstash Redis Search now has HYBRID fields and the $ann / $rrf retrievers. This PR lets the Hermes memory provider use them for recall, behind a flag. Nothing changes without it.

Why

Keyword recall ($smart) only finds a memory that shares words with the message. On 12 paraphrased questions against 12 facts (throwaway database, synthetic data), the right memory was among the 3 recalled for 3 of 12 with keyword search and 11 of 12 with hybrid search on the first attempt (12 of 12 when a server-side failure was retried; see below). Same results with upstash-redis 1.8.x and 1.9.0.

What changes (with UPSTASH_HERMES_SEMANTIC=1 / AgentMemory(semantic=True))

• A new shared index is created with text HYBRID. Only the memory text changes: chat transcripts stay TEXT, because a HYBRID field gets one embedding for its whole value.
• Recall runs $rrf over {$smart} and {$ann}, both with the full user/kind filter. If a hybrid query errors, recall falls back to keyword for that call. Records carry retrieval: "hybrid" | "keyword". min_score applies only to keyword results, because RRF scores are not on the same scale.
• Automatic recall injects 3 memories instead of 5. $ann has no threshold, and its similarity scores (about 0.55 to 0.77 for related and unrelated text alike) did not support a cutoff.
• An existing index is never dropped at runtime. With the flag on and a TEXT index, recall stays keyword and logs a warning that names the migration command. A HYBRID index also works with the flag off.
• New python -m upstash_hermes_memory.migrate --to hybrid|keyword, run with clients stopped. It reads credentials from the environment only (catalog rule 11).
  • It refuses an index this package didn't create, and refuses aliases.
  • It waits for pending indexing, then saves a baseline at <ns>:migrate:baseline: counts plus a sample of memory keys.
  • It drops the index, reconciling with DESCRIBE afterwards, then recreates it, retrying and reconciling ambiguous CREATE, DESCRIBE and WAITINDEXING steps.
  • It checks the new index against the baseline, by counts and sampled keys.
  • An interrupted run resumes when the same command runs again, or rolls back with --to <previous> against the original baseline. The check pages within LIMIT 1,000 and OFFSET 10,000.
  • --to keyword is the rollback, needed before reinstalling 0.2.x, whose EXISTSOK create fails on a HYBRID index.
• Recall falls back on UpstashError and httpx.HTTPError only. Programming errors propagate.
• SearchIndex.ensure() now requires WAITINDEXING == 1 and checks the resolved index name.

Also fixed

• SearchIndex.query checked for a missing index with SEARCH.COUNT on the query itself. That is invalid for retriever queries, because Search rejects $ann in COUNT. It now counts {}.
• Comments claimed the shared agentkit index matches a TS SharedIndexSchema. It doesn't exist: the TS SDK creates agentkit_memory / agentkit_chat. The README already said so; the comments now agree.

Found while testing (server side, for the Search team)

• $ann intermittently fails with ERR dot metric only accepts unit-length vectors, got length 0.9994862. The same query failed 3 of 4 times, with or without a filter, and $rrf containing it fails too. That hit 2 to 3 of every 12 to 15 hybrid queries in the eval. The fallback covers it, but those calls lose semantic recall.
• In a small store, $ann returns every document, so equal-weight RRF often ranks a keyword-only match first. In the eval, the right memory was in the top 3 every time but first only 3 of 12 times. Retriever weights could help; I left them at the default pending real data.

Follow-up proposed

• A recall time budget. upstash-redis builds httpx.Client(timeout=None) for every command, which predates this PR. A stalled request can outlive Hermes's 8 s prefetch wait. Bounding it needs an upstash-redis timeout option, or reaching into the client's private internals, so it's a separate change.
• Retriever weights. Tune them on real data.

Not in this PR

• The TS SDK (still TEXT).
• The Hermes catalog entry, whose description says "no vector database or embeddings". Suggested disclosure text: "With UPSTASH_HERMES_SEMANTIC=1, memory text and each recall query are embedded by Upstash Redis Search's managed embeddings in the configured database; no separate vector database is used. Off by default." The embedding model, its processing location and its cost are not in the docs, so the disclosure should not claim them.

Tests

• 49 new unit tests, using a filter-aware fake that also covers the client-parsed DESCRIBE. They cover command shapes, the accepted-schema and alias checks, WAITINDEXING == 0, fallback on server and transport errors, a failed fallback, the COUNT probe and provider wiring. On the migration side: pending updates, a failed drop, a failed create then resume, a count change, sample loss, rollback of an interrupted migration, stores past one 1,000-result page, and wait_indexing recovery.
• 6 new live tests in test_semantic_redis.py: new index, meaning recall, user and chat isolation, empty store, and a migration round trip ending in a 0.2.2-style create.
• The full suite ran against a throwaway database: 130 passed with upstash-redis 1.8.0 and with 1.9.0. ruff, prettier and uv lock --locked are clean.
• Live tests that rebuild indexes need a database of their own, as the README now says.