Repository navigation
Conversation
The discover-reply handler at server.py:4058 called salt.crypt.PublicKeyString(payload["cluster_pub"]) without guarding against None, so a joiner master that answered another joiner's discover before its own join-reply from the founder had landed would crash the recipient's publish daemon with AttributeError: 'NoneType' object has no attribute 'encode'. Fix both sides of the handshake: * Receiver: early-return when the discover-reply payload lacks cluster_pub. A debug log carries the peer_id so the retry can be observed. Previously the recipient's publish daemon died outright via an unhandled AttributeError, after which the joiner never started its Raft learner and the cluster never stabilised. * Sender: skip the discover-reply when cluster_public_key() returns None (the joiner has not installed cluster.pub yet). Prevents the peer from having to reject or defer; the next discover round picks up the key once join-reply has landed. The bug is pre-existing but was latent on resources-2 because the publish-daemon event-loop ordering happened to always process join-reply before any peer-discover trade. The cache-replication work on this branch shifts that ordering (new publish branch at server.py:3547 and the apply_master_config default flip for cluster_cache_replication) and surfaces the crash deterministically. See agents/reports/cache-replicate-alpha-diagnosis.md.
discover_peers() in salt.master.Master.start is fire-and-forget: it sends one cluster/peer/discover event per peer over the local IPC bus. The publish daemon forwards that event to each peer's pool-puller. If the peer's pool-puller TCP listener is not yet bound on cluster_pool_port when the event is forwarded (a tight startup race narrowed by the extra event-loop work the cache-replication machinery puts on the publish daemon), the push raises StreamClosedError and the discover is lost silently. No discover-reply, no cluster.pub install, no Raft election, no cluster_aes rotation, no session keys -- the joiner waits forever on _discover_event and the whole bootstrap stalls. Add a periodic retry owned by MasterPubServerChannel. When a non- founder enters _publish_daemon, _schedule_discover_retry installs a tornado PeriodicCallback that re-sends discover to each peer whose cluster/peer/discover-reply has not yet landed. The retry stops as soon as every configured peer has replied, or after cluster_join_discover_retry_timeout seconds have elapsed; the interval is tunable via cluster_join_discover_retry_interval (default 2 s, timeout default 60 s). The cancel signal is a dedicated self._discover_replies set populated in the discover-reply branch of handle_pool_publish after the peer signature and cluster-pub fingerprint are verified. Earlier drafts cancelled on peer_keys growth, but peer_keys is populated by the AES- key broadcast (send_aes_key_event) independently of the discover protocol: in a 3-master bring-up every joiner sees the founder's AES key arrive well before any discover-reply, so a peer_keys-keyed cancel fires on the very first tick and the discover stays lost. The cache-replication work on wip/cache-cluster-replicate surfaced this race by shifting the publish-daemon event-loop ordering during startup, but the one-shot discover is a pre-existing bug in the cluster bootstrap path: any added startup latency (slow disk, loaded CI runner, extra pre-fork work) can push the handshake over the cliff on the base branch too. Covered by tests/pytests/unit/channel/test_cluster_discover_retry.py, including a dedicated regression test that AES-key arrival alone does not cancel the retry.
Every ``cache.store`` on a clustered master opens a fresh ``MasterEvent`` (connect + fire + close) before emitting the ``cluster/runner/delegate_cache_write`` event. Measured against 200K-resource register loads on a 3-master isolated-FS cluster, the per-write connect cost adds roughly 70 seconds per master to the replication tail (hot banks are ``grains``, ``minions``, ``resource_registry`` -- every one hits the facade). Cache the handle in a module-local dict keyed by ``(sock_dir, opts["id"], os.getpid())``. First call in each process creates the handle lazily; subsequent calls reuse it. Fork-safety comes from pid-keying: a worker that inherits a parent's dict entry doesn't reuse the parent's stale fds, it constructs its own on first fire. Teardown is best-effort via ``atexit`` -- replication events are not critical on shutdown. On a fire failure the cached handle is invalidated so the next call re-opens the socket (e.g. after a master restart yanks the underlying unix socket).
Adds the sender/receiver building blocks for join-time replicate-mode cache-bank catch-up so a newly joining master receives every replicate-mode bank's current contents before it starts applying live delegate-cache-write deltas: * BANK_SYNC_CHANNEL_PREFIX (bank_sync:<bank>) wire channel distinct from the operator-collect bank: prefix so the receiver can apply with _already_replicated=True. * BANK_SYNC_SKIP_PREFIXES guards ring-routed / consensus-owned banks from the catch-up list. * enumerate_replicate_banks walks the cachedir and returns the top-level banks whose per-bank routing resolves to replicate. * iter_cache_bank_chunks chunks a bank by cumulative byte budget. * install_bank_sync_chunk applies a chunk with the already- replicated flag set so the receiver doesn't re-broadcast.
The salt.utils.resource_registry.ResourceRegistry.register_minion write path bypasses salt.cache.Cache (hash-indexed mmap with the SRN key shape and uint32 roster) so the facade-level fan-out added in earlier commits does not cover it. On a 3-master cluster with haproxy TCP round-robin, each master was writing its own slice of the resource_index.by_id.mmap, producing three different mmap files. Close the gap with the same architectural shape as cache replication but a dedicated event tag so the chunk metadata path can diverge: * Fire cluster/runner/delegate_resource_register_write from AESFuncs.__register_resources_sync right after the local update_resource_index call lands. * Receiver side (MasterPubServerChannel) decrypts the cluster_aes-encrypted cluster/peer/delegate-resource-register-write event and applies via update_resource_index locally. Loop-guard on originating_master == self.opts['id'] prevents bus echo from re-applying. * Gated on the same preconditions as cache fan-out: cluster_id set, cluster_isolated_filesystem True, cluster_cache_replication not forced False. The resource_grains cache writes at AESFuncs.__register_resources_sync line 2942 already go through cache.store and are covered by the existing facade-level fan-out -- no additional hook needed there.
Switch the minion resource subsystem off pillar and onto a per-type
opts-key model. Resource types are declared via the ``resource_types``
minion opt; per-type modules read their own opts (e.g. the dummy module
reads ``dummy_resource_count``) rather than ``opts["pillar"]["resources"][type]``.
Key invariants enforced by this change:
* The minion never materializes ``opts["resources"]``. Every call site
(gen_modules loader/init loop, _post_master_init, pillar_refresh,
resource_refresh event, target resolution, grain collection) now
iterates ``salt.utils.resources.enabled_resource_types(opts)`` + the
per-type ``discover(opts)`` generator.
* ``discover(opts)`` is now a streaming contract -- implementations
MAY return a generator. The dummy module yields
``f"dummy-{i:07d}"`` for i in ``range(1, dummy_resource_count + 1)``
so a 200K-resource minion never holds the full ID list in memory.
* ``_register_resources_with_master`` is rewritten as a chunked loop
that drains the per-type generator into fixed-size chunks
(``resource_register_chunk_size``, default 10000), ships each chunk
with a ``chunk_rtype`` / ``chunk_index`` / ``chunk_final`` header,
renders grains per-chunk, and drops the chunk reference before
streaming the next one. Per-type peak is bounded by
``chunk_size * per-SRN-cost`` rather than ``total * per-SRN-cost``.
* ``salt.utils.resources.pillar_resources_tree`` and
``resource_pillar_key`` are deleted -- with pillar out of the picture
the remap is meaningless. A new helper
``enabled_resource_types(opts)`` returns the opt-driven type list.
``bare_resource_id_in_minion_data_cache`` keeps the grains-cache
branch (unchanged behavior for offline-minion targeting) and loses
its pillar-cache branch.
Companion to the no-pillar-resources design
(``agents/reports/no-pillar-design.md``). ``salt/resources/ssh/``
still reads pillar and is handled in a follow-up.
Closes Phase 3 reviewer concern #3: ResourceRegistry._pending_registers has no GC/TTL, so a minion that ships chunks 0..k and crashes before chunk_final=True leaves a stuck entry forever. For 200K-register workloads with crash-looping minions, these windows accumulate unboundedly. Fix shape: * Each per-scope window state (_pending_registers[minion_id][scope]) carries a created_ts + last_activity_ts. Both are stamped from the same time.monotonic() reading that the sweep uses, so a just-created entry cannot be evicted on its own creation call. * last_activity_ts refreshes on every chunk arrival for that scope, so active-but-slow registers never evict their own window. A stalled minion is caught; a slow one is not. * On every chunk arrival, before touching the incoming window state, the write path runs a sweep over _pending_registers and drops any scope whose last_activity_ts is older than the TTL. Each eviction logs at WARNING level naming (minion_id, scope, age_seconds, srn_count) so operators can identify the crash-loop source. * No background task required: every register_minion_chunk call runs the sweep under the existing _pending_registers_lock. Sweep cost is O(live pending scopes) which is bounded by N_minions * N_types. Config: * New master opt cluster_resource_register_pending_ttl (float seconds, default 300). Added to VALID_OPTS + DEFAULT_MASTER_OPTS so schema docs and the config validator both know about it. The default comfortably exceeds the 200K-register window observed on the 3-node cluster. Tests: * test_register_minion_chunk_pending_ttl_evicts_abandoned_window: seeds a crash-loop minion's pending window, rewinds its last_activity_ts past the TTL, fires a chunk for a second minion, asserts the abandoned window is evicted + the WARNING is emitted. * test_register_minion_chunk_pending_ttl_does_not_evict_active_window: negative counterpart -- a window with activity inside the TTL must not be evicted even when the TTL is small. Timestamp manipulation (rewinding last_activity_ts directly) keeps the tests deterministic with no sleep + no mock-time machinery.
Phase-3 verifier finding #2: ``_save_state`` + ``_load_state`` each emit a ``log.warning`` with ``pprint.pformat(state_dict)`` per resource. At 200K resources that's 200K WARNING lines per full enumeration, which: - Drives docker container log circular-buffer rotation during scale tests. - Contributes to 62-68K "dummy-0" strings in the pid-10 anon-memory scan (Python's logging / format interning). - Makes real WARNING-level issues hard to spot in the firehose. Demote both call sites to DEBUG. Fixture noise, not production bug, but makes the scan-and-verify loop measurably quieter and shrinks the memory-scan false-positive footprint. The ``log.warning`` at line 52 (``dummy_resource_count must be an int``) stays at WARNING -- that's a one-shot config-error log and is correctly loud.
A plain import salt.cache._cluster_replicator inside handle_pool_publish silently rebound salt as a function-local name for the entire body. Every earlier reference in the same function (salt.utils.event.SaltEvent.unpack, salt.cluster.consensus.rpc.is_raft_tag, etc.) then raised UnboundLocalError: local variable 'salt' referenced before assignment on each incoming cluster/peer event, which dropped every join-reply / pre-vote-reply. Masters 2+3 never discovered master-1 and the Raft log never committed any membership entry -- blocking cluster formation from a fresh docker compose up -d. Switch to importlib.import_module so the submodule is loaded into sys.modules without introducing a local binding for salt.
Previously the master-side register handler called cache.store() in a per-SR loop for the RESOURCE_GRAINS_BANK, producing one cluster/peer/delegate-cache-write event per SR. For a 10K-SR chunk that is 10,000 individual events fanning out to all peer masters via the replicate routing default -- a 10,000x event amplifier vs the registry path (which uses one event per chunk). Switch to cache.store_many(bank, items) so one chunk fires a single batched event (further sub-batched inside store_many by cluster_cache_replicate_batch_size, default 1024). Semantics unchanged -- same SR+grain pairs land in the same bank; the delegate-cache-write dispatch batches transparently. Affects both async (__register_resources_sync) and MWorker-sync (_sync_register_resources) paths.
… ring
Replaces the TODO-fallback in _run_delegate_cache_write for the "ring"
write_kind with actual 1-of-N targeted delivery. Resolves the owning
master for each (bank, key) via the resource_index HashRing and pushes
the delegate-write event only to that peer. Self-owned writes are
dropped (originator already applied the local write); unresolved
owners fall back to broadcast so no update is lost.
Batched payloads partition by owner: each remote owner receives a
batch containing only its own [key, data] pairs. publish_cache_write
grows a target_master kwarg so a single peer can be addressed without
fanning out to the whole pool, mirroring publish_resource_register_write.
Enables cluster_cache_routing: {resource_grains: ring} to actually
cut grain fan-out from N-of-N to 1-of-N during register at scale.
Normalise gdict["resource_id"] to the id portion of the SRN before cache.store_many in both __register_resources_sync and _sync_register_resources. The bank key <type>:<id> is authoritative; any resource module whose grains() emitted a divergent or missing resource_id is now corrected on write. Makes the grain-target "resource_id:<glob>" fast path structurally safe: a bank-key glob match is now proof of the resource_id grain value, independent of the resource module's grains() implementation.
_augment_grain_match_with_resource_grains used to pickle-load every grain dict in the bank and run subdict_match -- O(bank-size) pickle IO per query regardless of match count. At 500K-800K grain files per master, that is ~13-46 s per grain-target query. For predicates of the form "resource_id:<pattern>", the bank key <type>:<id> carries the match answer. The new _resource_id_candidates helper narrows the candidate set by fnmatch / regex against the id portion of each bank key, and the main loop skips the per-match pickle fetch entirely -- the master-side enforcement of gdict["resource_id"] == <id> (see salt/master.py register handlers) makes the shortcut provably correct. Observed on a 3-master cluster at 1M SRs: grain-target queries drop from ~46 s cold to ~0.5 s regardless of match count (100 or 300K). Non-resource_id predicates (nested keys, other grain fields) still go through the full fetch loop.
Pre-commit check-docs flagged salt.cache._cluster_replicator as missing from doc/ref/cache/all/index.rst. Add it to the autosummary. Prepare Release man-page build fails under sphinx -W because doc/topics/releases/3009.0.md (generated during release prep) is orphan. Add the release stub, the matching template, and 3009.* to the releases toctree glob so the man build is clean.
Cache.store_many previously looped self.store() per key regardless
of driver; the batched event path from _maybe_replicate_write_many
was in place but the local write still took the per-key lock on
mmap_cache. At scale (5 minions x 1024-key batches) this serialised
every MWorker on the per-bank .mmap_cache.idx.lock and reduced
mmap-backed register throughput to a trickle.
Two changes:
- Cache.store_many now dispatches to a driver-native "store_many"
function if the loaded driver exposes one, falling back to the
per-item loop otherwise (preserves localfs / redis / consul
compat).
- salt/cache/mmap_cache.py gains a store_many that routes the
batch through MmapCache.put_many so the whole set takes a
single flock + batched fsync/msync set.
Ports the driver-level pieces from saltstack#70752 (closed) so the mmap
bank can keep up with the cross-master batched replicate path
this branch already ships.
``_augment_grain_match_with_resource_grains`` previously resolved every ``resource_id:<pattern>`` target by listing the entire ``resource_grains`` bank and iterating -- O(bank-size) per call (~400 ms on a 400K-key bank). Literal predicates (``resource_id:a-dummy-0000100`` with no wildcards) are now routed to an O(T) hash probe across the configured resource types via ``self.cache.contains(bank, "<rtype>:<rid>")`` where T is ``opts['resource_types']`` (small, typically 1-20). The master already enforces ``bank-key == "<rtype>:<rid>"`` at register time so a probe hit is the authoritative answer -- no bank list / per-key fetch. Glob predicates (``resource_id:a-*``), nested predicates (``resource_id:a:b``), regex form, non-``resource_id`` keys, and the empty-pattern form all fall through to the existing behaviour. Scope: literal-exact probe only. Sorted-key / reverse-value indices and consistency metadata are out of scope for this change.
Adds an in-process sorted-key index of the ``resource_grains`` cache
bank so ``resource_id:<prefix>*`` style grain-target predicates can
binary-search the prefix range instead of linearly scanning + fnmatching
every entry. On a 400K-key bank a prefix query that matches ~100 keys
drops from ~500 ms to target <10 ms.
New module ``salt.utils.grain_target_cache`` holds
``{(cachedir, bank): (generation, sorted_keys)}`` guarded by a
module-level ``RLock``. Invalidation is driven by a cheap O(1)
per-bank generation signal -- the mmap_cache ``occupied_count`` header
field when available, otherwise the bank directory's ``st_mtime_ns``
(advances on any ``.p`` file add/remove/rename in the localfs driver).
Banks without a derivable signal rebuild on every query so correctness
is preserved.
Dispatch in ``_augment_grain_match_with_resource_grains`` is tiered:
1. Mission 1 literal probe for ``resource_id:<exact-id>`` (unchanged).
2. (new) Prefix fast path for ``resource_id:<glob>`` with a non-empty
fixed prefix. Walks the sorted-key range per enabled rtype; for each
candidate key confirms the full fnmatch pattern (handles the tail
wildcards). Falls through if ``enabled_resource_types`` is empty so
the linear scan still runs.
3. Existing linear ``cache.list()`` + fnmatch fallback.
Edge cases:
* ``resource_id:*foo*`` or ``resource_id:[abc]-*`` -- prefix is empty,
dispatch falls through to the linear scan. Char-class enumeration
was explicitly skipped (noted in the design spec); the existing
fallback path handles it.
* PCRE (``regex_match=True``) -- prefix-of-regex analysis isn't worth
the complexity; always falls through.
* Bank keys are ``<rtype>:<rid>`` composites; the fast path intersects
per rtype so a single binary-search bound only covers one type's
stride, giving O(K * log N + matches).
Adds an in-process reverse-value index keyed by (cachedir, bank, frozenset(declared_keys)) that maps (grain_key, value) -> [sr_key, ...] for the resource_grains bank. The new opts list ``index_grain_keys`` names which grain keys to index; a flat <grain_key>:<value> predicate on any declared key is dispatched to the reverse-index lookup before falling into the slow-path per-key cache.fetch + subdict_match loop. Literal values are an O(matches) list read; glob and regex patterns walk the indexed value set applying fnmatch / re.search and union the matching sr_key lists. Lists / tuples are indexed per element so predicates like 'tags:prod' match grains such as tags = ['prod', 'us-east']. Nested dict values are skipped. The index rebuilds lazily on the first lookup after a bank-generation bump (mmap occupied-count header or bank-dir mtime, same signal as Mission 2) and swallows build errors into an empty map so the slow path still serves queries. Changing index_grain_keys forces a rebuild (the declared-key frozenset is part of the cache key). Empty / missing index_grain_keys keeps the legacy behaviour untouched.
… build Add a persistent mmap-backed scalar-leaf index over the resource_grains bank so grain-target predicates of arbitrary shape (nested dns:nameservers:1.1.1.1, operator-undeclared keys, PCRE / glob values) can be answered without walking the bank or holding a full reverse-map in process RAM. The index is a MmapCache keyed by b"<path-joined-by-\\x1e>\\x1f<value>" holding msgpack-packed lists of sr_keys. High-cardinality buckets are replaced with a msgpack-dict cap sentinel at grain_trie_max_list_size (default 100000), which the dispatcher detects and treats as "fall back to slow path" so operator-visible results stay correct. Rebuilds stream the primary bank one SR at a time, accumulate into a bounded in-memory dict (default 1024 pending sr_keys), and flush via MmapCache.put_many with merge-on-flush semantics so periodic flushes inside a single rebuild do not overwrite earlier batches for the same (path, value) key. Build output is atomic: files are written with a .new suffix and os.replace()d into live names (heap, roster, index, meta) so queries racing the rebuild see either the pre- or post-rebuild state, never a torn mix. Opt-in via enable_grain_trie_index; grain_trie_max_list_size and grain_trie_build_batch_size tune the cardinality cap and flush cadence.
…xed binary
Mission 4 live measurement showed wide-match query latency on a 600K-element
sr_key list was ~78 s end-to-end, dominated by msgpack.unpackb; Mission 3's
in-process Python list was ~500 ms for similar counts. Replace the
msgpack.packb([sr_key, ...]) leaf encoding with a compact length-prefixed
binary format:
byte 0 : 0xC1 (msgpack-reserved "never used" magic)
byte 1..4 : uint32 LE count of sr_keys
byte 5.. : repeated:
uint16 LE key length
<utf-8 key bytes>
Decode is a tight struct.unpack_from loop with no per-element Python object
overhead. _unpack_entry dispatches on the first byte: 0xC1 -> binary list;
otherwise -> msgpack (cap sentinel, and legacy msgpack-list entries from any
pre-upgrade build still decode). The cap sentinel keeps its msgpack dict
encoding; its leading bytes (0x80-0x8F or 0xDE/0xDF) never collide with 0xC1.
Also rejects sr_keys whose UTF-8 encoding exceeds the uint16 length field
with a friendly ValueError (programming error, not a data concern -- SR keys
are tens of bytes in practice).
Public API unchanged. Existing Mission 1-4 tests pass unchanged.
Mission 5 switches trie leaf-value sr_key lists from msgpack to a length-prefixed binary format (magic byte 0xC1 + uint32 LE count + repeated uint16 LE key length + UTF-8 key bytes). Cap sentinel stays msgpack-encoded so the decode dispatcher branches on raw[0:1]. Adds seven tests: * round-trip correctness across empty / 1 / 10 / 1000 / unicode cases * magic-byte distinguishes from the cap sentinel (0xC1 is a reserved msgpack lead byte, chosen for exactly this purpose) * short-buffer rejection (struct.error / ValueError / IndexError) * 10K wide-list decode must be >=2x faster than msgpack (slow_test; the whole point of the Mission 5 switch) * 10-element narrow-list decode must not regress >2x (overhead floor) * end-to-end rebuild+lookup smoke on 10K SRs sharing a bucket * mixed-magic-byte dispatch: list bucket + cap sentinel coexist and each routes through the correct decoder Uses an import shim (_pack_list / _unpack_list) that probes a handful of plausible helper names so the tests remain resilient to naming drift and skip cleanly when the helpers are not yet landed.
…th-prefixed binary" This reverts commit 201fb0a.
Mission 6 of the trie-index work. Previously a bucket whose total
sr_key count exceeded ``grain_trie_max_list_size`` was replaced with
a cap sentinel (``{"_capped": True, "count": N}``), discarding the
sr_keys and forcing queries to fall through to the legacy O(N)
``cache.fetch`` loop. A live-cluster ``dummy_grain_1:one`` match of
600K SRs took ~32 s that way.
Replace the "give up at cap" layout with chunked storage:
* The natural trie key stores a chunk marker:
``{"_chunks": N, "_total": T}`` (msgpack map; still distinguishable
from a regular msgpack-array leaf).
* Chunk ``i`` lives under a derived key
``<natural_key> + b"\x1d" + f"{i:08X}".encode("ascii")``. The
8-char uppercase-hex chunk index avoids NUL bytes -- MmapCache
truncates keys at the first NUL, which would silently collapse
chunk-0 (little-endian uint32 0 == four NULs) and all chunks after
it onto the same slot. The ``\x1d`` separator is distinct from the
Mission-4 path separators ``\x1e`` / ``\x1f``.
* Each chunk's value is a plain msgpack list, same shape as a small
non-chunked bucket.
Flush path:
* No prior entry / legacy cap sentinel: write chunks directly if the
new bucket exceeds ``max_list_size``, else a plain list.
* Prior list that overflows after merge: promote to chunks (rewrite
everything -- rare, only on the growth transition).
* Prior chunks: touch only the LAST chunk until full, then roll over
and start a new chunk. Earlier chunks are immutable within a
rebuild, giving O(M) cumulative merge cost for a bucket growing to
size M instead of the old O(M**2 / batch_size).
Query path:
* ``lookup_literal`` fans out to every chunk entry on a chunk-marker
hit, concatenates the sr_key lists, and returns them to
``dispatch_predicate`` as if it were a single wide list.
* ``values_for_path`` skips chunk sub-keys (they carry no new value
information; the natural key's marker still sits in ``list_keys``).
* An orphaned chunk marker or a missing chunk logs a WARNING and
returns ``None`` so the caller falls back to the slow path until
the next rebuild.
Back-compat: legacy ``_capped`` sentinels on disk from older builds
are recognised by the decoder and treated as "data was lost, trigger
slow path once". The next successful rebuild writes the bucket in
chunked form.
Benchmark (``FakeCache`` with 600_000 SRs matching one grain value,
cap 100_000):
* Build: ~3.4 s (unchanged within noise).
* Warm lookup: 29.7 ms, returns the full 600_000 sr_key list.
Down from 32 s via the legacy slow path -- a ~1000x speedup on the
live-cluster workload.
Memory discipline preserved: the only extra allocation during a flush
is the chunk-key bytes (~17 B each), bounded by the number of chunks
touched by that flush, not by total fleet size.
…tory The branch's earlier commits (64657af, 2e01cc7, c380f5c) replaced ``opts["pillar"]["resources"]`` with an opts-driven inventory API (``resource_types`` + each type's ``discover(opts)``), but this test file still seeded inventory via the deleted pillar path. Result: ``_resolve_resource_targets`` returned ``[]`` and 7 of 8 tests failed on CI. * Fixture now sets ``resource_types = ["dummy"]`` + ``dummy_resource_count = 3`` so ``salt.resources.dummy.discover()`` streams the three synthetic IDs the tests exercise. * ID assertions updated from ``dummy-01..03`` to ``dummy-0000001..03`` to match the module's fixed ``dummy-NNNNNNN`` (7-digit zero-padded) output -- the pre-refactor assertions hard-coded the shorter form that pillar seeding supplied. * ``salt/cli/caller.py`` comment at the ``_resolve_resource_targets`` call site refreshed to describe the opts-driven inventory API instead of the removed pillar-reading path.
Mission 1's commit ``master: enforce resource_id == bank-key id at register time`` normalizes ``resource_id`` into every grain dict stored by ``__register_resources_sync`` -- the id portion of the bank key (``<type>:<id>``) is authoritative so grain targeting's ``resource_id:<glob>`` fast path can skip a per-match fetch. Four ``test_master.py`` assertions compared the stored grain dict to the raw ``resource_grains`` payload the test sent, which no longer matches once the register path adds the normalized ``resource_id`` field. Update each asserted dict to include it.
Make the resource-register entry point defensive about ``self.tok``: if auth has not populated the token yet, bail (debug-log) and let the next trigger (pillar_refresh / tune_in) retry. The attribute is assigned on a successful ``connect_master`` handshake; a ``pillar_refresh`` that fires before the first handshake, a proxy minion that has not yet attached its pub channel, or a bare ``Minion`` built for a unit test can all reach ``_send_register_chunk`` without ``tok`` set, which previously crashed with ``AttributeError: 'Minion' object has no attribute 'tok'``. This fixes the 7 metaproxy functional tests + 1 unit test (``test_minion_keeps_schedule_on_master_reconnect``) that regressed when ``_register_resources_with_master`` was wired into ``pillar_refresh``. Also adapt the ``_threaded_minion_61830`` fixture in ``tests/pytests/unit/test_minion.py`` to the opts-driven resource inventory API (see commit ``resources: remove pillar as resource-inventory source on the minion``). The pre-refactor fixture set resources via pillar; after the refactor ``enabled_resource_types()`` reads ``opts["resource_types"]`` instead, so the dummy type was never discovered and the sibling-job trigger never fired -- ``AssertionError: sibling job never ran in the window``. Derive ``resource_types`` from the pillar dict and stub ``_iter_resource_ids`` to yield each type's declared IDs, same shim pattern used in ``test_minion_resources.py``. Fixes all 6 ``test_thread_return_*_resource_job_owns_job_generation_61830`` parametrizations.
The ``resources/`` integration suite still seeded dummy-resource inventory via ``opts["pillar"]["resources"]``, which earlier branch commits (64657af, 2e01cc7, c380f5c) removed in favour of the ``resource_types`` + per-type opts API consumed by ``salt.utils.resources.enabled_resource_types``. Mirror the pattern landed in 07c6b0b for the unit suite. * Replace the pillar-seeded ``DUMMY_RESOURCES`` fixture with opts overrides on the minion fixtures: ``resource_types: ["dummy"]`` + ``dummy_resource_count: 3`` for the primary minion, and the same pair plus ``dummy_resource_id_prefix: "m2-"`` on the second minion so the two IDs sets stay disjoint. * Update the shared ``DUMMY_RESOURCES`` constants to the dummy module's fixed ``dummy-NNNNNNN`` (7-digit zero-padded) output -- ``dummy-0000001..3`` for minion 1, ``m2-dummy-0000001..2`` for minion 2 -- and refresh every hard-coded ID assertion across ``test_dummy_resource.py``, ``test_multi_minion_grain_targeting.py``, and ``test_resource_loader_strict.py``. The grain-PCRE test's ``^dummy-0[12]$`` pattern becomes ``^dummy-000000[12]$``. * Add ``resource_types: ["dummy", "dynamic_test"]`` to the salt_master fixture so the master-side literal ``resource_id:<id>`` fast-path in ``salt.utils.minions._augment_grain_match_with_resource_grains`` can probe the bank keys per enabled rtype -- without it, the master never resolves single-id grain targets and the CLI exits with ``ERROR: No return received``. * Rewrite ``test_pillar_addition_at_runtime_registers_new_resource`` (pillar-only mechanism now gone) as ``test_all_configured_resources_registered_with_master`` -- same intent (every configured dummy ID reaches the master's ``resource_grains`` bank) without the dead pillar-mutation path. * Rewrite ``test_minion_has_resources_configured`` to probe ``config.get resource_types`` + ``config.get dummy_resource_count`` instead of the removed ``config.get resources`` path. * ``test_cli_offline_expands_resource_targets.py``: declare inventory through opts (prefix ``off-cli-`` + count 3) and write the ``salt_resources`` subtree directly into the master's grains cache -- the in-tree ``salt/grains/resources.py`` still reads ``__opts__.get("resources")`` and under opts-driven inventory that is empty, so the master's primed grains cache does not carry the bank naturally. Fixing that grain module is a separate change outside this suite's scope. * The dynamic_test fixture keeps its pillar-reading contract (the whole point of the synthetic type is "discover() is authoritative"; it legitimately reads a non-``resources`` pillar key) but now also sets ``resource_types: ["dynamic_test"]`` so the opts-driven loader enables the type.
twangboy
previously approved these changes
Nov 11, 2026
Teaches for_bank to read the previously-documented-but-unwired grain_trie_mmap_size knob and forward it to :class:. Each bank's sparse trie.idx file is mmap_size * slot_size bytes; the production default of 2_000_000 slots x 160-byte slots yields a 320 MiB sparse file per bank. On CI runners with a tmpfs-backed /tmp, a fleet of ~40 test_mmap_trie_index tests plus the integration-test setup copying hundreds of MiB of fixtures exhausts the tmpfs before the trie suite finishes, after which MmapCache key-store writes silently drop and mm.get(key) returns None for every lookup. That surfaced as test_chunk_key_derivation_uses_1d_separator, test_small_bucket_still_regular_list, and the other wide-bucket assertions reporting the trie as empty on CI while all tests passed locally. The test adapter now passes grain_trie_mmap_size=200_000 (32 MiB per bank) through the opts dict, and test_rebuild_swaps_every_rolled_heap_segment which bypasses the adapter for its grain_trie_mmap_max_segment_bytes override now also passes grain_trie_mmap_size=50_000. Full file footprint drops from ~12 GiB to ~1.3 GiB per run.
…emantics PR saltstack#70755 added commit_index + last_applied persistence to the Raft log (commits 3e87ddd, backport 7af42dd); the test's docstring and Phase 2 assertions still described the pre-fix behaviour where commit_index came back as -1 after restart. Update preconditions to match persisted state, and rewire the on_ready callback since RaftService.__init__ now fires reconcile_membership during construction (which left fired2 empty when the callback was wired post-construction).
test_minion_manager_destroy_after_closed_io_loop_does_not_raise closed its ``manager.io_loop`` without first draining the manager's event publisher and cancelling background tasks. The PubServer's per- subscriber ``_stream_read`` tasks are not part of ``manager._background_tasks``, so they survived loop teardown and later asyncio emitted ``Task was destroyed but it is pending!`` from a subsequent test's caplog, breaking saltstack#70748's regression check purely as a test-cleanup artifact. On CPython 3.14 the GC timing changed by the resource-discovery streaming refactor makes the ghost task reach the later test's capture window deterministically. Close the event publisher first so each subscriber's ``_stream_read`` exits via ``StreamClosedError``, then cancel + gather the manager's tracked background tasks, then close the loop. Mirrors the pattern already in ``spawn_minions_keeps_task_reference`` and ``bind_keeps_publisher_task_reference``.
This branch is being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cluster_cache_routing(jobs→ ring,cluster/consensus/cluster/state_sync→ none). Resolver only turns replication on when the cluster uses an isolated per-master filesystem.store_many) with mmap driver-native dispatch; MmapCache gains aput_manyfast path.salt/utils/mmap_trie_index.py) with chunked wide buckets; routed fromCkMinions._check_cache_minionsfor thegrainsbank and from_augment_grain_match_with_resource_grainsforresource_grains.resource_idenforcement, dedicated cross-master replication event, pillar removed as the minion-side inventory source.MasterEventhandle reused across stores, per-minion peer-apply executor, per-event per-peer publish log demoted to DEBUG.Test plan
wip/cache-cluster-replicate).jobsring-routes andgrains/resource_grainsfan out N-of-N under the new defaults.test_cache_replicationsuite undertests/pytests/integration/cluster/passes on Ubuntu 24.04 zmq3 (the one previously-flaky job).