Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 38 additions & 8 deletions lifecycle/graph_miners.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@

Miners read existing data (epi_tags, node contents, L0 user-message rows) and
write edges into epi_edges tagged `heuristic:<name>` (rollback: DELETE WHERE
tags LIKE '%heuristic:%'). All inserts are INSERT OR IGNORE against the
epi_edges PK — re-runs are no-ops. No LLM calls anywhere.
tags LIKE '%heuristic:%'). Edges go through `_insert_edge`, an upsert on the
epi_edges PK that keeps the strongest evidence: a re-run at an equal or weaker
weight is a no-op, a strictly stronger one rewrites the weight and moves
`created_at` (see that function for what the age means). No LLM calls anywhere.
"""

from __future__ import annotations
Expand Down Expand Up @@ -176,19 +178,47 @@ async def _inhibit_dirty(conn: Any, dirty: set[int]) -> int:


async def _insert_edge(conn: Any, a: int, b: int, relation: str, weight: float, heuristic: str, dirty: set[int] | None = None) -> int:
"""UPSERT into epi_edges; returns rows actually written (re-run → 0).
"""UPSERT into epi_edges; returns rows actually written (equal repeat → 0).

Audit 05.09 (P1): INSERT OR IGNORE froze the weight — a miner re-run
never strengthened a link. The upsert now takes max(weight) (strongest
evidence wins) and refreshes created_at. After a heuristic edge is
inserted, lateral inhibition applies (G5, SYNAPSE): a weak edge is
suppressed by the cluster of stronger neighbors around the node.

19.09 (issue G): inhibition runs ONLY for newly created rows. A repeat
upsert used to re-run the quadratic lateral_inhibition over the whole
hub on every nightly (640k edges → 9h of 100% main-thread CPU that
never converges: upsert re-raises what inhibition lowered). Return
contract is unchanged (rows written, 0/1).
`created_at` is the edge's BIRTH or its last WEIGHT-INCREASING
re-derivation, whichever is later — an equal or weaker re-mine leaves it
alone (measured 2026-11-03, F42; see
`.compose/reports/2026-11-03-f42-dryrun.md`). There is no separate birth
column and `valid_from` is NULL on every row of all three house bases, so an
edge's true age cannot be recovered. Consequences, pinned by the test
`test_remine_after_inhibition_refreshes_the_edge_age`:

- a repeat at an EQUAL-or-lower weight is a true no-op (0 rows written,
weight and age untouched) — the 19.09 gate below. This matters: the
miners re-derive the same pairs every night, and if an equal repeat moved
the age, nothing would ever settle;
- a repeat at a HIGHER weight writes the row and moves the age. That is
why an edge that inhibition drove to zero comes back with a fresh age
when its miner re-derives it at full strength, and it is the only way
this upsert keeps an edge out of the NREM stale band
(`graph_enrich._dream_nrem`, `created_at < now - 30 d`).

F42 read the second bullet as prune starvation: if a re-mined dead edge
never reaches 30 days, it never dies. Measured on the live bases, that does
not happen. On 2026-11-03 at 06:59 CET the dead heuristic edges sat almost
entirely inside the window — Hermes 56 650 in the 25-30 d band against 189
just past it (30-31 d) and none beyond; CowAgent 1 765 against 2 and none.
The boundary is therefore crossed in batches and cleared on the NEXT NREM
pass, so pruning lags by up to one pass rather than starving. Sampling the
same Hermes base eight minutes apart is enough to see the lag move (56 839 /
0 → 56 650 / 189), so a single "0 past 30 d" reading is a moment, not a
property.

One exception is measured and unresolved: Mimocode holds 120 dead
`heuristic:tokens` edges at 44.7 d, all stamped within ~0.9 s on 2026-09-19.
Skipping NREM across a boundary would explain it, but that is a hypothesis
about that base's schedule, not something these rows prove.

19.09 (issue G, part 3): the upsert is gated — a weaker-or-equal repeat
is a true no-op (no row touched, created_at stable, returns 0). Without
Expand Down
75 changes: 75 additions & 0 deletions tests/test_lifecycle/test_graph_miners.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,81 @@ async def _edge(a: int, b: int, relation: str = "mentions", weight: float = 0.8)
await conn.commit()


# --- edge age semantics: created_at advances only on a weight-increasing re-mine ---


@pytest.mark.asyncio
async def test_remine_after_inhibition_refreshes_the_edge_age(db) -> None:
"""Pin the measured contract of `created_at` on heuristic edges.

F42 asked whether a re-mine resets an edge's age so the 30-day NREM prune
never fires. The upsert does refresh the age — but dead edges do NOT
accumulate past the boundary: measured 2026-11-03 (see
`.compose/reports/2026-11-03-f42-dryrun.md`), ~57k dead heuristic edges sit
in Hermes' 25-30 d band and only ~190 sit just past 30 d, waiting for the
next NREM pass. Pruning lags by up to one pass; it does not starve. (One
measured exception, not explained here: Mimocode holds 120 dead edges at
44.7 d — see the report.)

The contract pinned here:

- `created_at` is the edge's birth or its last WEIGHT-INCREASING
re-derivation, whichever is later. It is not "last time seen": an equal
re-mine is common (the miner re-derives the same pair every night) and
must not move it, or no edge would ever settle;
- equal and weaker repeats are true no-ops, so a fading edge still ages;
- a stronger repeat writes the row and refreshes the age, which is why an
edge that inhibition drove to zero comes back with a fresh age when its
miner re-derives it at full strength.

Changing the last of these is a behaviour decision, not a bugfix: a
re-derived edge would then start decaying 30 days after its first sighting
instead of being refreshed, and would die once enough NREM passes had
subtracted below the floor.
"""
from lifecycle.graph_miners import _insert_edge

conn = await db.get(DB_NAME)
old = T - 40 * 86400
a = await _node("alpha beta gamma", T, ["x"])
b = await _node("alpha beta gamma", T, ["x"])
await conn.execute(
"INSERT INTO epi_edges (source_id, target_id, relation, weight, created_at, tags)"
" VALUES (?, ?, 'topic_overlap', 0.8, ?, '[\"heuristic:tokens\"]')",
(a, b, old),
)
await conn.commit()

async def state() -> tuple[int, float]:
row = await (await conn.execute("SELECT weight, created_at FROM epi_edges WHERE source_id=? AND target_id=?", (a, b))).fetchone()
return int(row["weight"] * 100), float(row["created_at"])

# lateral inhibition drives the weight down; the edge is now dead but old
await conn.execute("UPDATE epi_edges SET weight=0 WHERE source_id=? AND target_id=?", (a, b))
await conn.commit()

# a stronger re-mine is a real write and refreshes the age
assert await _insert_edge(conn, a, b, "topic_overlap", 0.8, "tokens") == 1
weight, settled = await state()
assert weight == 80
assert settled > old + 86400, "created_at tracks the last weight-increasing evidence, not the birth"

# an equal repeat is a genuine no-op: a settled edge stops churning
assert await _insert_edge(conn, a, b, "topic_overlap", 0.8, "tokens") == 0
assert await state() == (80, settled), "an equal repeat must not touch weight or age"

# a weaker repeat is also a no-op: the stronger stored evidence wins, and a
# fading edge must still be allowed to age out
assert await _insert_edge(conn, a, b, "topic_overlap", 0.5, "tokens") == 0
assert await state() == (80, settled), "a weaker repeat must not touch weight or age"

# a strictly stronger repeat refreshes the age again
assert await _insert_edge(conn, a, b, "topic_overlap", 0.81, "tokens") == 1
weight2, refreshed = await state()
assert weight2 == 81
assert refreshed > settled, "a weight-increasing re-mine moves created_at forward"


# --- (a) miner_tags: общий тег → tagged, weight = min(0.3+0.1*shared, 0.6) ---


Expand Down
Loading