Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 72 additions & 37 deletions backend/norm/docs/events.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

How `norm` turns OTel **log records** (events) into knowledge-graph nodes and edges. It covers
the streaming path, `StreamNormalizer.process(record, kind="event")`, used by the
[rt-norm-worker](../../workers/rt-norm-worker/README.md). Spans take a different path
(`dispatch_span`, see [normalization.md](normalization.md)).
[rt-norm-worker](../../workers/rt-norm-worker/README.md). Completed spans use
[`dispatch_span`](../src/norm/ioa_observe/build.py); both paths share persisted reconciliation.

Code: [`events.py`](../src/norm/ioa_observe/events.py) routes on `LogAttributes["event.name"]`;
the handlers are in [`handlers/`](../src/norm/ioa_observe/handlers/).
Expand All @@ -12,26 +12,32 @@ the handlers are in [`handlers/`](../src/norm/ioa_observe/handlers/).

1. A dry run on an empty registry finds which node ids (and agent names) the event touches.
2. The reader loads them from the graph store, together with the event's session subgraph.
3. The handler runs against the loaded graph.
4. Only the difference is returned as a `StreamDelta` and written back.
3. The handler merges provisional event data or authoritative completed-span data.
4. Ownership, pending messages/tools/handoffs and derived trajectories are reconciled.
5. Only the difference is returned as a `StreamDelta` and written back.

Nothing is kept between events: the stored graph is the state. Consequences:
The normalizer itself keeps no state between events: it uses the stored graph and
adapter-managed trace context. Consequences:

- Handlers are **idempotent**. Replaying an event changes nothing.
- Handlers are **order-tolerant**. Most of them create whatever is missing above the node they
need (MAS, Session, MASCall, Agent, AgentCall), so a `completed` event arriving alone builds
what its `started` would have.
- A node that already exists is **reused, not overwritten**. `startTime` never changes;
`endTime` only moves forward. Initial and final States are set once, by whichever event
arrives first.
- Existing identities are reused. Late starts can correct `startTime` and input; completed
spans replace provisional timing, content and metrics without creating a second call.
Container start/end times encompass their children, and final states follow the child
ending last, not the child starting last.
- Compact `rtNormalization` metadata retains pending relationships, canonical trajectory
identities and processed event ids. Existing persisted worker metadata and the
`handoffSourceSpanIds` property remain readable; no raw telemetry replay buffer is kept.
- An event with no `session.id` (or no agent / tool name where one is needed) is ignored.

## Identity

| Node | Id |
|---|---|
| `Session` | `session.id` without its `<application_id>_` prefix |
| `MAS` | `application_id` (falls back to `ServiceName`) |
| `MAS` | originating application in a propagated `<application>_<session>`; otherwise `application_id`, then `ServiceName` |
| `MASCall` | `mascall:<session id>` |
| `Agent` | the agent's **name** (`agent.name`, `source.agent`, `target.agent`, `tool.name` of an a2a tool, the card's `name`). The same name in another MAS is another Agent, with id `<mas>:<name>`. The span id is the AgentCall's, not the Agent's. |
| `AgentCall` | hash of the Agent's id and the span id (`topology.node.*` `span.id`, or `ioa_observe.agent.span_id` of the other events) |
Expand All @@ -42,7 +48,35 @@ Nothing is kept between events: the stored graph is the state. Consequences:
| `State`, `Transition` | hashes of the owner's span and the content |

Events that carry only an `ioa_observe.agent.span_id` (`tool.*`, `mcp.tools.discovered`) find the
Agent through the AgentCall of that span.
Agent through the AgentCall of that span. Ownership is backfilled when that call arrives.
Application names may contain underscores; participating MCP services do not create a
second MAS merely because their application or service name differs.

Span ownership also follows parent chains through MCP/A2A/framework transport spans.
An explicit agent-owner span id takes precedence over that ancestor's ordinary parent.
Intermediate spans need not have a session id: private trace context retains their
`TraceId`, `SpanId`, parent, owner and optional session id, without storing raw payloads
or manufacturing ontology executions. Known session boundaries and trace ids are
respected; missing links and cyclic parent chains remain unresolved rather than guessed.

Streaming adapters must implement `GraphReader.load_span_contexts(trace_id)` and retain
`StreamDelta.span_contexts` separately from ontology deltas. The realtime worker retains
this context in memory on its store instance, not in Neo4j. It is lost on restart and is
not shared across workers; replay the relevant spans/events into the same instance to
rebuild it. A sessionless transport arriving
last rechecks sessions referenced by that trace. Without `TraceId`, ancestry is scoped
to the explicit session; a span with neither identifier cannot be correlated safely.
Batch normalization uses the same ancestry resolver.

Agent and Tool identities may share a name/id while remaining different ontology types.
Streaming edge endpoints are resolved using their relationship type (and the concrete
execution relationship for generic `executes`), not whichever same-id node was loaded last.
The existing verifier can still report domain mismatches when it collapses such node ids;
that diagnostic limitation is separate from the typed relationships persisted to Neo4j.

Execution graph metadata is retained on the MAS. A graph node becomes a declared Agent
only when it matches an observed agent's identity, name or runtime alias. Routers and
tool-serving graph nodes are not assumed to be agents.

## States

Expand Down Expand Up @@ -92,16 +126,12 @@ Same nodes as `node.started`.

### `topology.edge.updated`

Reads `source.agent.span.id`, `target.agent`. Creates nothing.
Reads `source.agent.span.id`, `target.agent`.

- Adds `executesAgent` from the **AgentCall** of the source span to the **Agent** named
`target.agent`.
- Ignored when either does not exist: a target that is not an agent (an MCP server such as
`debug`) matches nothing.

`executesAgent` otherwise means "this call is an instance of that agent"; here it also records
the handoff. A call's own agent is therefore the one it was created for, not any `executesAgent`
target (`_own_agent_id`).
- Retains a pending handoff hint until both agent calls exist.
- Connects their execution trajectories through the shared handoff heuristic.
- Never adds a second `executesAgent` target to the sender or invents an Agent for a
tool-serving target. Explicit MCP/tool topology edges are ignored as agent handoffs.

### `llm.started`

Expand All @@ -119,6 +149,15 @@ Same nodes as `llm.started`.
- Moves the LLMCall's `endTime` forward and sets `duration`.
- Creates the final State from `llm.output` and the Transition, once.

### `llm.tools.presented`

Reads `capability.content` (JSON tools, including OpenAI-style function descriptors) and
`ioa_observe.agent.span_id`, falling back to the record's span.

- Creates or enriches Tool descriptions.
- Persists the offered tool ids and owner span, linking them with `usesTool` when the
owning AgentCall arrives.

### `tool.started`

Reads `tool.name`, `tool.input`, `tool.type`, `tool.call.id`, `ioa_observe.agent.span_id`.
Expand Down Expand Up @@ -147,8 +186,7 @@ Same nodes as `tool.started`.
Reads `session.id`, `capability.content` (JSON): `name`, `description`.

- Ignored without a `session.id`.
- Creates `MAS` (the first part of `session.id`, `<masname>_<sessionid>`) and `Session` (the
rest), linked by `executesSession`.
- Creates the originating `MAS` and `Session`, linked by `executesSession`.
- Adds the `Agent` named by the card, keyed on its **name** (the card has no span id of the
agent it describes), with `belongsToMAS`. An Agent of that name already in the MAS is reused
and only gets its description. The ontology has no
Expand All @@ -170,21 +208,20 @@ Reads `source.agent`, `source.agent.span.id`, `target.agent`, `message.id`, `mes
- Keeps `message.input` as a State of the sender's AgentCall (`hasState` only, so its initial
and final States are untouched).

### `a2a.message.received`

Same nodes as `a2a.message.sent`.
### `a2a.message.received` / `a2a.message.completed`

- Keeps `message.input` as a State of the **recipient's** latest AgentCall, only if that call
already exists. The event does not carry the recipient's span id, and its call is normally
created a few milliseconds later by its own `topology.node.started`, so usually nothing of
the message is kept.
- Retains incoming `message.input` and completion `message.output` as separate message
States, even before the corresponding AgentCall exists.
- Received messages are reconciled with the recipient's call by name and event time;
completion replies attach to the source agent span.
- Does not fabricate a recipient AgentCall or replace an agent's own final State.

### `mcp.message.sent` / `mcp.message.received`

Reads `message.input` (JSON), `message.id`, `source.agent`, `source.agent.span.id`.

- Ignored when `message.input` is empty or has no `name` (a `tools/list` request).
- Ensures the sender's chain: MAS (from `application_id`, else the first part of `session.id`),
- Ensures the sender's chain: originating MAS,
Session, MASCall, Agent, AgentCall.
- Creates `Tool` (`message.input.name`) and `ToolCall` keyed on `message.id`, with
`executesTool`, `hasToolCall` and `usesTool`.
Expand All @@ -206,13 +243,13 @@ Same nodes as `mcp.message.sent`.
Reads `ioa_observe.agent.span_id`, `session.id`, `capability.content` (JSON with a `tools` list).

- Ignored without `ioa_observe.agent.span_id` or a session.
- Creates `MAS` (named by the first part of `session.id`, `<masname>_<sessionid>`) and `Session`.
- Creates the originating `MAS` and `Session`.
- Creates a `Tool` for every entry in `tools` with its `description`, and fills the description
of a Tool that already exists without one.
- The event names the agent only by span id, so the agent is the one of the AgentCall of that
span, and every tool is linked to it with `usesTool`. If that call does not exist yet (the
event arrives before the agent's `topology.node.started`), the tools are created **without**
the link: nothing is remembered to link them later.
event arrives before the agent's `topology.node.started` or completed span), owner hints
and tool ids are persisted so the link is backfilled.

## Not handled

Expand All @@ -222,15 +259,13 @@ These are ignored; `dispatch_event` returns `False` for names it does not know.
|---|---|
| `slim.message.sent`, `slim.message.received` | `TODO` |
| `mcp.tools.published` | `TODO` |
| `a2a.message.completed` | no case |
| `llm.tools.presented` | no case |

## Known gaps

- The handoff between agents is only the `executesAgent` edge from `topology.edge.updated`. The
ontology has no Agent-to-Agent relationship.
- `tool.*` events without `tool.call.id` and without `ioa_observe.agent.span_id` create a
ToolCall that nothing links to an agent.
- When an agent runs twice it has one `Agent` and two `AgentCall` nodes. The same agent name in
another MAS is a different Agent.
- `mcp.tools.discovered` arriving before the agent's `topology.node.started` does not link its tools.
- Previously misclassified graph data is not automatically migrated or deleted. Compare
end-to-end runs on a fresh graph. Ontology/SHACL findings are separate diagnostics;
normalization does not weaken the verifier to hide them.
4 changes: 3 additions & 1 deletion backend/norm/src/norm/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
# SPDX-License-Identifier: Apache-2.0
from .compare import KGCompareResult, compare_kg, read_kg_json, write_kg_json
from .ioa_observe import build_kg
from .ioa_observe.stream import GraphReader, InMemoryGraph, Kind, StreamDelta, StreamNormalizer
from .ioa_observe.otel_io import infer_run_id, load_otel_export
from .ioa_observe.registry import SpanContext
from .ioa_observe.stream import GraphReader, InMemoryGraph, Kind, StreamDelta, StreamNormalizer
from .normalizer import Normalizer, OnNormalizedCallback, configure, dump_jsonld, normalize
from .verifier import (
KGCheckSkipped,
Expand All @@ -26,6 +27,7 @@
"InMemoryGraph",
"Kind",
"StreamDelta",
"SpanContext",
"StreamNormalizer",
"build_kg",
"check_edge_domain_range",
Expand Down
Loading