diff --git a/backend/norm/docs/events.md b/backend/norm/docs/events.md index c3cef5a..319aa0a 100644 --- a/backend/norm/docs/events.md +++ b/backend/norm/docs/events.md @@ -2,8 +2,8 @@ How `norm` turns OTel **log records** (events) into knowledge-graph nodes and edges. It covers the streaming path, `StreamNormalizer.process(record, kind="event")`, used by the -[rt-norm-worker](../../workers/rt-norm-worker/README.md). Spans take a different path -(`dispatch_span`, see [normalization.md](normalization.md)). +[rt-norm-worker](../../workers/rt-norm-worker/README.md). Completed spans use +[`dispatch_span`](../src/norm/ioa_observe/build.py); both paths share persisted reconciliation. Code: [`events.py`](../src/norm/ioa_observe/events.py) routes on `LogAttributes["event.name"]`; the handlers are in [`handlers/`](../src/norm/ioa_observe/handlers/). @@ -12,18 +12,24 @@ the handlers are in [`handlers/`](../src/norm/ioa_observe/handlers/). 1. A dry run on an empty registry finds which node ids (and agent names) the event touches. 2. The reader loads them from the graph store, together with the event's session subgraph. -3. The handler runs against the loaded graph. -4. Only the difference is returned as a `StreamDelta` and written back. +3. The handler merges provisional event data or authoritative completed-span data. +4. Ownership, pending messages/tools/handoffs and derived trajectories are reconciled. +5. Only the difference is returned as a `StreamDelta` and written back. -Nothing is kept between events: the stored graph is the state. Consequences: +The normalizer itself keeps no state between events: it uses the stored graph and +adapter-managed trace context. Consequences: - Handlers are **idempotent**. Replaying an event changes nothing. - Handlers are **order-tolerant**. Most of them create whatever is missing above the node they need (MAS, Session, MASCall, Agent, AgentCall), so a `completed` event arriving alone builds what its `started` would have. -- A node that already exists is **reused, not overwritten**. `startTime` never changes; - `endTime` only moves forward. Initial and final States are set once, by whichever event - arrives first. +- Existing identities are reused. Late starts can correct `startTime` and input; completed + spans replace provisional timing, content and metrics without creating a second call. + Container start/end times encompass their children, and final states follow the child + ending last, not the child starting last. +- Compact `rtNormalization` metadata retains pending relationships, canonical trajectory + identities and processed event ids. Existing persisted worker metadata and the + `handoffSourceSpanIds` property remain readable; no raw telemetry replay buffer is kept. - An event with no `session.id` (or no agent / tool name where one is needed) is ignored. ## Identity @@ -31,7 +37,7 @@ Nothing is kept between events: the stored graph is the state. Consequences: | Node | Id | |---|---| | `Session` | `session.id` without its `_` prefix | -| `MAS` | `application_id` (falls back to `ServiceName`) | +| `MAS` | originating application in a propagated `_`; otherwise `application_id`, then `ServiceName` | | `MASCall` | `mascall:` | | `Agent` | the agent's **name** (`agent.name`, `source.agent`, `target.agent`, `tool.name` of an a2a tool, the card's `name`). The same name in another MAS is another Agent, with id `:`. The span id is the AgentCall's, not the Agent's. | | `AgentCall` | hash of the Agent's id and the span id (`topology.node.*` `span.id`, or `ioa_observe.agent.span_id` of the other events) | @@ -42,7 +48,35 @@ Nothing is kept between events: the stored graph is the state. Consequences: | `State`, `Transition` | hashes of the owner's span and the content | Events that carry only an `ioa_observe.agent.span_id` (`tool.*`, `mcp.tools.discovered`) find the -Agent through the AgentCall of that span. +Agent through the AgentCall of that span. Ownership is backfilled when that call arrives. +Application names may contain underscores; participating MCP services do not create a +second MAS merely because their application or service name differs. + +Span ownership also follows parent chains through MCP/A2A/framework transport spans. +An explicit agent-owner span id takes precedence over that ancestor's ordinary parent. +Intermediate spans need not have a session id: private trace context retains their +`TraceId`, `SpanId`, parent, owner and optional session id, without storing raw payloads +or manufacturing ontology executions. Known session boundaries and trace ids are +respected; missing links and cyclic parent chains remain unresolved rather than guessed. + +Streaming adapters must implement `GraphReader.load_span_contexts(trace_id)` and retain +`StreamDelta.span_contexts` separately from ontology deltas. The realtime worker retains +this context in memory on its store instance, not in Neo4j. It is lost on restart and is +not shared across workers; replay the relevant spans/events into the same instance to +rebuild it. A sessionless transport arriving +last rechecks sessions referenced by that trace. Without `TraceId`, ancestry is scoped +to the explicit session; a span with neither identifier cannot be correlated safely. +Batch normalization uses the same ancestry resolver. + +Agent and Tool identities may share a name/id while remaining different ontology types. +Streaming edge endpoints are resolved using their relationship type (and the concrete +execution relationship for generic `executes`), not whichever same-id node was loaded last. +The existing verifier can still report domain mismatches when it collapses such node ids; +that diagnostic limitation is separate from the typed relationships persisted to Neo4j. + +Execution graph metadata is retained on the MAS. A graph node becomes a declared Agent +only when it matches an observed agent's identity, name or runtime alias. Routers and +tool-serving graph nodes are not assumed to be agents. ## States @@ -92,16 +126,12 @@ Same nodes as `node.started`. ### `topology.edge.updated` -Reads `source.agent.span.id`, `target.agent`. Creates nothing. +Reads `source.agent.span.id`, `target.agent`. -- Adds `executesAgent` from the **AgentCall** of the source span to the **Agent** named - `target.agent`. -- Ignored when either does not exist: a target that is not an agent (an MCP server such as - `debug`) matches nothing. - -`executesAgent` otherwise means "this call is an instance of that agent"; here it also records -the handoff. A call's own agent is therefore the one it was created for, not any `executesAgent` -target (`_own_agent_id`). +- Retains a pending handoff hint until both agent calls exist. +- Connects their execution trajectories through the shared handoff heuristic. +- Never adds a second `executesAgent` target to the sender or invents an Agent for a + tool-serving target. Explicit MCP/tool topology edges are ignored as agent handoffs. ### `llm.started` @@ -119,6 +149,15 @@ Same nodes as `llm.started`. - Moves the LLMCall's `endTime` forward and sets `duration`. - Creates the final State from `llm.output` and the Transition, once. +### `llm.tools.presented` + +Reads `capability.content` (JSON tools, including OpenAI-style function descriptors) and +`ioa_observe.agent.span_id`, falling back to the record's span. + +- Creates or enriches Tool descriptions. +- Persists the offered tool ids and owner span, linking them with `usesTool` when the + owning AgentCall arrives. + ### `tool.started` Reads `tool.name`, `tool.input`, `tool.type`, `tool.call.id`, `ioa_observe.agent.span_id`. @@ -147,8 +186,7 @@ Same nodes as `tool.started`. Reads `session.id`, `capability.content` (JSON): `name`, `description`. - Ignored without a `session.id`. -- Creates `MAS` (the first part of `session.id`, `_`) and `Session` (the - rest), linked by `executesSession`. +- Creates the originating `MAS` and `Session`, linked by `executesSession`. - Adds the `Agent` named by the card, keyed on its **name** (the card has no span id of the agent it describes), with `belongsToMAS`. An Agent of that name already in the MAS is reused and only gets its description. The ontology has no @@ -170,21 +208,20 @@ Reads `source.agent`, `source.agent.span.id`, `target.agent`, `message.id`, `mes - Keeps `message.input` as a State of the sender's AgentCall (`hasState` only, so its initial and final States are untouched). -### `a2a.message.received` - -Same nodes as `a2a.message.sent`. +### `a2a.message.received` / `a2a.message.completed` -- Keeps `message.input` as a State of the **recipient's** latest AgentCall, only if that call - already exists. The event does not carry the recipient's span id, and its call is normally - created a few milliseconds later by its own `topology.node.started`, so usually nothing of - the message is kept. +- Retains incoming `message.input` and completion `message.output` as separate message + States, even before the corresponding AgentCall exists. +- Received messages are reconciled with the recipient's call by name and event time; + completion replies attach to the source agent span. +- Does not fabricate a recipient AgentCall or replace an agent's own final State. ### `mcp.message.sent` / `mcp.message.received` Reads `message.input` (JSON), `message.id`, `source.agent`, `source.agent.span.id`. - Ignored when `message.input` is empty or has no `name` (a `tools/list` request). -- Ensures the sender's chain: MAS (from `application_id`, else the first part of `session.id`), +- Ensures the sender's chain: originating MAS, Session, MASCall, Agent, AgentCall. - Creates `Tool` (`message.input.name`) and `ToolCall` keyed on `message.id`, with `executesTool`, `hasToolCall` and `usesTool`. @@ -206,13 +243,13 @@ Same nodes as `mcp.message.sent`. Reads `ioa_observe.agent.span_id`, `session.id`, `capability.content` (JSON with a `tools` list). - Ignored without `ioa_observe.agent.span_id` or a session. -- Creates `MAS` (named by the first part of `session.id`, `_`) and `Session`. +- Creates the originating `MAS` and `Session`. - Creates a `Tool` for every entry in `tools` with its `description`, and fills the description of a Tool that already exists without one. - The event names the agent only by span id, so the agent is the one of the AgentCall of that span, and every tool is linked to it with `usesTool`. If that call does not exist yet (the - event arrives before the agent's `topology.node.started`), the tools are created **without** - the link: nothing is remembered to link them later. + event arrives before the agent's `topology.node.started` or completed span), owner hints + and tool ids are persisted so the link is backfilled. ## Not handled @@ -222,15 +259,13 @@ These are ignored; `dispatch_event` returns `False` for names it does not know. |---|---| | `slim.message.sent`, `slim.message.received` | `TODO` | | `mcp.tools.published` | `TODO` | -| `a2a.message.completed` | no case | -| `llm.tools.presented` | no case | ## Known gaps -- The handoff between agents is only the `executesAgent` edge from `topology.edge.updated`. The - ontology has no Agent-to-Agent relationship. - `tool.*` events without `tool.call.id` and without `ioa_observe.agent.span_id` create a ToolCall that nothing links to an agent. - When an agent runs twice it has one `Agent` and two `AgentCall` nodes. The same agent name in another MAS is a different Agent. -- `mcp.tools.discovered` arriving before the agent's `topology.node.started` does not link its tools. +- Previously misclassified graph data is not automatically migrated or deleted. Compare + end-to-end runs on a fresh graph. Ontology/SHACL findings are separate diagnostics; + normalization does not weaken the verifier to hide them. diff --git a/backend/norm/src/norm/__init__.py b/backend/norm/src/norm/__init__.py index 3192edf..c31b530 100644 --- a/backend/norm/src/norm/__init__.py +++ b/backend/norm/src/norm/__init__.py @@ -2,8 +2,9 @@ # SPDX-License-Identifier: Apache-2.0 from .compare import KGCompareResult, compare_kg, read_kg_json, write_kg_json from .ioa_observe import build_kg -from .ioa_observe.stream import GraphReader, InMemoryGraph, Kind, StreamDelta, StreamNormalizer from .ioa_observe.otel_io import infer_run_id, load_otel_export +from .ioa_observe.registry import SpanContext +from .ioa_observe.stream import GraphReader, InMemoryGraph, Kind, StreamDelta, StreamNormalizer from .normalizer import Normalizer, OnNormalizedCallback, configure, dump_jsonld, normalize from .verifier import ( KGCheckSkipped, @@ -26,6 +27,7 @@ "InMemoryGraph", "Kind", "StreamDelta", + "SpanContext", "StreamNormalizer", "build_kg", "check_edge_domain_range", diff --git a/backend/norm/src/norm/ioa_observe/build.py b/backend/norm/src/norm/ioa_observe/build.py index 3038ff7..bf3f23c 100644 --- a/backend/norm/src/norm/ioa_observe/build.py +++ b/backend/norm/src/norm/ioa_observe/build.py @@ -12,9 +12,34 @@ from __future__ import annotations +from copy import deepcopy from typing import Any -from .fields import get_span_attrs +from scope_ontology.models.edges import ( + executes, + executesAgent, + executesLLM, + executesMAS, + executesProcessing, + executesSession, + executesTool, +) +from scope_ontology.models.nodes import ( + Agent, + AgentCall, + LLMCall, + Session, + Tool, + ToolCall, +) + +from .fields import ( + _to_float, + get_span_attrs, + get_span_session_id, + get_span_start_time, + normalize_record, +) from .handlers import ( handle_agent, handle_chat, @@ -23,13 +48,30 @@ handle_session_start, handle_tool, ) +from .handlers.agent import _agent_id_for from .heuristics import ( assign_container_boundary_states, chain_agent_handoffs_fallback, chain_capability_calls, link_agent_handoffs, ) -from .registry import Registry +from .heuristics.handoff import _source_span_ids +from .reconciliation import ( + handoff_spans, + reconcile_relationships, + remember_span_ancestry, + strip_derived, +) +from .registry import HANDOFF_SOURCES_PROP, Registry +from .trajectory import ( + _clear_boundaries, + _remember_trajectory, + prune_trajectories, + restore_trajectories, +) + +_CALLS = (AgentCall, LLMCall, ToolCall) + _HANDLERS: dict[str, Any] = { "session.start": handle_session_start, @@ -37,7 +79,7 @@ } -def dispatch_span(span: dict[str, Any], registry: Registry) -> None: +def _dispatch_span(span: dict[str, Any], registry: Registry) -> None: """Run the one per-span handler matching ``span``'s ``SpanName`` (if any).""" span_name = str(span.get("SpanName") or "") span_attrs = get_span_attrs(span) @@ -57,14 +99,39 @@ def dispatch_span(span: dict[str, Any], registry: Registry) -> None: def apply_heuristics(spans: list[dict[str, Any]], registry: Registry) -> None: """Cross-span clean-up of the trajectory, once ``spans`` are all handled.""" + reconcile_relationships(registry) + strip_derived(registry) + restore_trajectories(registry, registry.metadata_for) + registry.nodes = dict(sorted(registry.nodes.items())) + registry.edges = dict( + sorted(registry.edges.items(), key=lambda item: (item[0][0].__name__, *item[0][1:])) + ) # Agent handoffs - explicit_handoff_targets = link_agent_handoffs(spans, registry) + explicit_handoff_targets = link_agent_handoffs(handoff_spans(registry), registry) # Fallback: timestamp-order sibling AgentCalls the above left unlinked chain_agent_handoffs_fallback(registry, explicit_handoff_targets) # Capability calls within agent call chain_capability_calls(registry) # Session and MAS call i/o assign_container_boundary_states(registry) + prune_trajectories(registry, registry.metadata_for) + typed_execution = ( + executesAgent, + executesLLM, + executesTool, + executesProcessing, + executesMAS, + executesSession, + ) + registry.remove_edges_where( + lambda edge, source, target: ( + edge is executes + and not any((typed, source, target) in registry.edges for typed in typed_execution) + ) + ) + for edge, source, target in list(registry.edges): + if edge in typed_execution: + registry.add_edge(executes(source_id=source, target_id=target)) def serialize_registry( @@ -86,6 +153,8 @@ def serialize_registry( value = getattr(item, attr, None) if value is not None: payload[attr] = value + extra = registry.extras.get((type(item).__name__, item.id), {}) + payload.update({key: value for key, value in extra.items() if value}) nodes.append(payload) return nodes, edges @@ -99,3 +168,130 @@ def build_kg( dispatch_span(span, registry) apply_heuristics(spans, registry) return serialize_registry(registry) + + +def dispatch_span(record: dict[str, Any], registry: Registry) -> None: + record, attrs = normalize_record(record, "span") + remember_span_ancestry(record, attrs, registry) + meta = registry.metadata_for + name = str(record.get("SpanName") or "") + if name in ("session.start", "session.end"): + session = registry.get(Session, get_span_session_id(record, attrs)) + if session is None: + initial_attrs = attrs + if name == "session.end": + initial_attrs = { + **attrs, + "ioa_start_time": _to_float(attrs.get("session.ended_at")) + or get_span_start_time(record, attrs), + } + handle_session_start(record, initial_attrs, registry) + session = registry.get(Session, get_span_session_id(record, attrs)) + if session is not None: + _dispatch_span(record, registry) + if name == "session.start": + meta(session)["sessionSpanId"] = str(record.get("SpanId") or "") + start = min(session.startTime, get_span_start_time(record, attrs)) + registry.upsert( + session.model_copy( + update={ + "startTime": start, + "duration": (session.endTime - start) * 1000, + } + ) + ) + return + if name.endswith(".graph"): + handle_graph(record, attrs, registry) + return + if name.endswith(".agent"): + display = str( + attrs.get("ioa_observe.entity.name") + or attrs.get("agent_id") + or record.get("AgentId") + or "" + ) + application = str(attrs.get("application_id") or "") + runtime_id = str(record.get("AgentId") or attrs.get("agent_id") or "") + attrs = { + **attrs, + "agent_id": _agent_id_for(registry, display, (application,)), + } + else: + runtime_id = "" + scratch = Registry() + _dispatch_span( + { + **record, + "SpanAttributes": attrs, + **({"AgentId": attrs["agent_id"]} if name.endswith(".agent") else {}), + }, + scratch, + ) + for cls in _CALLS: + for call in scratch.all_of(cls): + existing = registry.find( + cls, + lambda n: ( + n.sessionId == call.sessionId + and (n.spanId == call.spanId or call.spanId in meta(n).get("spanIds", [])) + ), + ) + tool_call_id = str(attrs.get("gen_ai.tool.call.id") or "") if cls is ToolCall else "" + if existing is None and tool_call_id: + existing = registry.get(ToolCall, tool_call_id) + canonical_id = tool_call_id or (existing.id if existing is not None else call.id) + if existing is not None and existing.id != canonical_id: + old_metadata = deepcopy(meta(existing)) + registry.merge_node_into(cls, existing.id, canonical_id) + else: + old_metadata = {} + if call.id != canonical_id: + scratch.merge_node_into(cls, call.id, canonical_id) + call = call.model_copy(update={"id": canonical_id}) + scratch.add(call) + meta(call).update(old_metadata) + if record.get("TraceId"): + meta(call)["traceId"] = str(record["TraceId"]) + if meta(call).get("spanComplete") and registry.get(cls, call.id) is not None: + return + # The completed span replaces provisional event boundaries, not message States. + _clear_boundaries(registry, call.id) + registry.upsert(call) + _remember_trajectory(scratch, call, meta) + for key in ( + "spanIds", + "spanPriority", + "input", + "inputPriority", + "output", + "outputPriority", + "complete", + ): + meta(call).pop(key, None) + meta(call)["spanComplete"] = True + owner = str(attrs.get("ioa_observe.agent.span_id") or call.parentSpanId or "") + if owner: + meta(call)["ownerSpanId"] = owner + owner_agent = str(record.get("AgentId") or attrs.get("agent_id") or "") + if owner_agent and cls in (LLMCall, ToolCall): + meta(call)["ownerAgentId"] = owner_agent + if cls is AgentCall: + sources = _source_span_ids(attrs) + if sources: + registry.extras.setdefault(("AgentCall", call.id), {})[HANDOFF_SOURCES_PROP] = ( + sources + ) + for node in scratch.nodes.values(): + existing_node = registry.add(node) + if ( + isinstance(node, (Agent, Tool)) + and isinstance(existing_node, (Agent, Tool)) + and node.description + and not existing_node.description + ): + registry.upsert(existing_node.model_copy(update={"description": node.description})) + if isinstance(node, Agent) and runtime_id: + meta(node)["aliases"] = sorted(set(meta(node).get("aliases", []) + [runtime_id])) + for edge in scratch.edges.values(): + registry.add_edge(edge) diff --git a/backend/norm/src/norm/ioa_observe/events.py b/backend/norm/src/norm/ioa_observe/events.py index 08c7c1e..e280e56 100644 --- a/backend/norm/src/norm/ioa_observe/events.py +++ b/backend/norm/src/norm/ioa_observe/events.py @@ -4,14 +4,22 @@ from __future__ import annotations +import logging +from copy import deepcopy from typing import Any +from scope_ontology.models.edges import executesTool +from scope_ontology.models.nodes import AgentCall, LLMCall, ToolCall + +from .fields import get_event_start_time, normalize_record from .handlers import ( handle_a2a_agent_card, + handle_a2a_message_completed, handle_a2a_message_received, handle_a2a_message_sent, handle_llm_completed, handle_llm_started, + handle_llm_tools_presented, handle_mcp_message_completed, handle_mcp_message_received, handle_mcp_message_sent, @@ -24,10 +32,30 @@ handle_topology_session_completed, handle_topology_session_started, ) +from .reconciliation import event_context, merge_span_context from .registry import Registry +from .trajectory import ( + _clear_boundaries, + _remember_trajectory, + add_final_state_transition, + add_initial_state, +) + +logger = logging.getLogger(__name__) +_CALLS = (AgentCall, LLMCall, ToolCall) def dispatch_event(event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry) -> bool: + """Apply a lifecycle record without overwriting an authoritative completed span.""" + event, event_attrs = normalize_record({**event, "LogAttributes": event_attrs}, "event") + context = event_context(event, event_attrs) + if context is not None: + key = (context.trace_id, context.span_id) + registry.span_contexts[key] = merge_span_context(context, registry.span_contexts.get(key)) + return apply_event(event, event_attrs, registry) + + +def _dispatch_event(event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry) -> bool: """Run the case for the event's ``event.name``. False if the name is unknown.""" match str(event_attrs.get("event.name") or ""): case "topology.session.started": @@ -39,13 +67,13 @@ def dispatch_event(event: dict[str, Any], event_attrs: dict[str, Any], registry: case "topology.node.completed": handle_topology_node_completed(event, event_attrs, registry) case "topology.edge.updated": - # TODO: handle topology.edge.updated - #handle_topology_edge_updated(event, event_attrs, registry) - pass + handle_topology_edge_updated(event, event_attrs, registry) case "llm.started": handle_llm_started(event, event_attrs, registry) case "llm.completed": handle_llm_completed(event, event_attrs, registry) + case "llm.tools.presented": + handle_llm_tools_presented(event, event_attrs, registry) case "tool.started": handle_tool_started(event, event_attrs, registry) case "tool.completed": @@ -54,6 +82,8 @@ def dispatch_event(event: dict[str, Any], event_attrs: dict[str, Any], registry: handle_a2a_message_sent(event, event_attrs, registry) case "a2a.message.received": handle_a2a_message_received(event, event_attrs, registry) + case "a2a.message.completed": + handle_a2a_message_completed(event, event_attrs, registry) case "a2a.agent_card": handle_a2a_agent_card(event, event_attrs, registry) case "slim.message.sent": @@ -76,3 +106,104 @@ def dispatch_event(event: dict[str, Any], event_attrs: dict[str, Any], registry: case _: return False return True + + +def apply_event(record: dict[str, Any], attrs: dict[str, Any], registry: Registry) -> bool: + + meta = registry.metadata_for + name = str(attrs.get("event.name") or "") + tool_call_id = str(attrs.get("tool.call.id") or attrs.get("message.id") or "") + event_span_id = str(record.get("SpanId") or "") + existing_tool = registry.find(ToolCall, lambda n: n.spanId == event_span_id) + if ( + tool_call_id + and existing_tool is not None + and existing_tool.id != tool_call_id + and registry.get(ToolCall, tool_call_id) is None + ): + old_metadata = deepcopy(meta(existing_tool)) + registry.merge_node_into(ToolCall, existing_tool.id, tool_call_id) + renamed = existing_tool.model_copy(update={"id": tool_call_id}) + registry.add(renamed) + meta(renamed).update(old_metadata) + authoritative = [ + n.model_copy(deep=True) + for cls in _CALLS + for n in registry.all_of(cls) + if meta(n).get("spanComplete") + ] + if not _dispatch_event(record, attrs, registry): + logger.debug("Unsupported runtime event %s", name) + return False + for node in authoritative: + registry.upsert(node) + event_time = get_event_start_time(record, attrs) + span_id = str(attrs.get("llm.call.id") or record.get("SpanId") or "") + prefix = name.rsplit(".", 1)[0] + cls = { + "topology.node": AgentCall, + "llm": LLMCall, + "tool": ToolCall, + "mcp.message": ToolCall, + }.get(prefix) + if cls is not None: + call_id = str(attrs.get("tool.call.id") or attrs.get("message.id") or "") + call = registry.get(cls, call_id) if call_id else None + call = call or registry.find(cls, lambda n: n.spanId == span_id) + if call is not None and not meta(call).get("spanComplete"): + start = min(call.startTime, event_time) + end = max(call.endTime, event_time) + call_meta = meta(call) + call_meta["spanIds"] = sorted(set(call_meta.get("spanIds", []) + [span_id])) + priority = 2 if prefix != "mcp.message" else (0 if name.endswith(".received") else 1) + previous = call_meta.get("spanPriority", -1) + chosen_span = call.spanId + if priority > previous or (priority == previous and span_id < chosen_span): + chosen_span = span_id + call_meta["spanPriority"] = priority + call = call.model_copy( + update={ + "startTime": start, + "endTime": end, + "duration": (end - start) * 1000, + "spanId": chosen_span, + } + ) + if isinstance(call, ToolCall): + tool = registry.edge_target(executesTool, call.id) or "" + call = call.model_copy(update={"name": f"Tool call ({tool}) {chosen_span}"}) + registry.upsert(call) + field_prefix = { + "topology.node": "agent", + "mcp.message": "message", + }.get(prefix, prefix) + priority = 1 if prefix == "mcp.message" else 2 + for field in ("input", "output"): + key = f"{field_prefix}.{field}" + if key in attrs and priority >= call_meta.get(f"{field}Priority", 0): + call_meta[field] = str(attrs[key] or "") + call_meta[f"{field}Priority"] = priority + if name.endswith(".completed"): + call_meta["complete"] = True + _clear_boundaries(registry, call.id) + input_text = call_meta.get("input", "") + initial = add_initial_state( + registry, + call.id, + call.sessionId, + call.spanId, + input_text, + ) + if call_meta.get("complete"): + add_final_state_transition( + registry, + call.id, + call.sessionId, + call.spanId, + initial.id, + input_text, + call_meta.get("output", ""), + ) + _remember_trajectory(registry, call, meta) + + return True diff --git a/backend/norm/src/norm/ioa_observe/fields.py b/backend/norm/src/norm/ioa_observe/fields.py index 12b32c8..676a72f 100644 --- a/backend/norm/src/norm/ioa_observe/fields.py +++ b/backend/norm/src/norm/ioa_observe/fields.py @@ -5,18 +5,21 @@ from __future__ import annotations import hashlib +import json +import logging from datetime import datetime, timezone from typing import Any # Span field derivation functions + def get_span_attrs(span: dict[str, Any]) -> dict[str, Any]: """Return a span's SpanAttributes map, or {} if absent.""" return span.get("SpanAttributes") or {} def get_span_application_id(span: dict[str, Any], span_attrs: dict[str, Any]) -> str: - return str(span_attrs.get("application_id") or span.get("ServiceName") or "") + return _application_id(span, span_attrs) def get_span_session_id(span: dict[str, Any], span_attrs: dict[str, Any]) -> str: @@ -109,14 +112,17 @@ def _hash_id(prefix: str, *parts: str) -> str: digest = hashlib.sha1("\x1f".join(str(p) for p in parts).encode("utf-8")).hexdigest()[:12] return f"{prefix}:{digest}" + # Event field derivation functions + def get_event_attrs(event: dict[str, Any]) -> dict[str, Any]: """Return an event's (OTel log record's) LogAttributes map, or {} if absent.""" return event.get("LogAttributes") or {} + def get_event_application_id(event: dict[str, Any], event_attrs: dict[str, Any]) -> str: - return str(event_attrs.get("application_id") or event.get("ServiceName") or "") + return _application_id(event, event_attrs) def get_event_session_id(event: dict[str, Any], event_attrs: dict[str, Any]) -> str: @@ -146,3 +152,41 @@ def get_event_start_time(event: dict[str, Any], event_attrs: dict[str, Any]) -> pass timestamp = str(event.get("Timestamp") or "") return _parse_timestamp_ns(timestamp) / 1e9 if timestamp else 0.0 + + +logger = logging.getLogger(__name__) + + +def _application_id(record: dict[str, Any], attrs: dict[str, Any]) -> str: + application = str(attrs.get("application_id") or record.get("ServiceName") or "") + session = str(record.get("SessionId") or attrs.get("session.id") or "") + # Observe propagates the originating application's session into participating services. + if "_" in session and not session.startswith(f"{application}_"): + return session.rpartition("_")[0] + return application + + +def normalize_record(record: dict[str, Any], kind: str) -> tuple[dict[str, Any], dict[str, Any]]: + key = "SpanAttributes" if kind == "span" else "LogAttributes" + attrs = record.get(key) or {} + if isinstance(attrs, str): + attrs = json.loads(attrs) + if not isinstance(attrs, dict): + raise ValueError(f"{key} must be an object") + attrs = dict(attrs) + application = _application_id(record, attrs) + if application: + attrs["application_id"] = application + return {**record, key: attrs}, attrs + + +def parse_json_object(raw: Any, label: str) -> dict[str, Any]: + try: + value = json.loads(raw) if isinstance(raw, str) else raw + except ValueError as exc: + logger.warning("Invalid %s: %s", label, exc) + return {} + if not isinstance(value, dict): + logger.warning("%s must be an object", label) + return {} + return value diff --git a/backend/norm/src/norm/ioa_observe/handlers/__init__.py b/backend/norm/src/norm/ioa_observe/handlers/__init__.py index e7185c9..ce44632 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/__init__.py +++ b/backend/norm/src/norm/ioa_observe/handlers/__init__.py @@ -2,17 +2,22 @@ # SPDX-License-Identifier: Apache-2.0 """One handler module per ``SpanName`` shape -- see build.py for dispatch.""" -from .a2a import handle_a2a_agent_card, handle_a2a_message_received, handle_a2a_message_sent +from .a2a import ( + handle_a2a_agent_card, + handle_a2a_message_completed, + handle_a2a_message_received, + handle_a2a_message_sent, +) from .agent import handle_agent from .chat import handle_chat from .graph import handle_graph +from .llm import handle_llm_completed, handle_llm_started, handle_llm_tools_presented from .mcp import ( handle_mcp_message_completed, handle_mcp_message_received, handle_mcp_message_sent, handle_mcp_tools_discovered, ) -from .llm import handle_llm_completed, handle_llm_started from .session import handle_session_end, handle_session_start from .tool import handle_tool from .tool_events import handle_tool_completed, handle_tool_started @@ -28,11 +33,13 @@ "handle_a2a_agent_card", "handle_a2a_message_received", "handle_a2a_message_sent", + "handle_a2a_message_completed", "handle_agent", "handle_chat", "handle_graph", "handle_llm_completed", "handle_llm_started", + "handle_llm_tools_presented", "handle_mcp_message_completed", "handle_mcp_message_received", "handle_mcp_message_sent", diff --git a/backend/norm/src/norm/ioa_observe/handlers/a2a.py b/backend/norm/src/norm/ioa_observe/handlers/a2a.py index 2a2519c..ca6a65a 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/a2a.py +++ b/backend/norm/src/norm/ioa_observe/handlers/a2a.py @@ -10,9 +10,9 @@ from scope_ontology.models.edges import belongsToMAS, hasState from scope_ontology.models.nodes import Agent, AgentCall, State -from ..fields import _hash_id +from ..fields import _hash_id, get_event_application_id, get_event_session_id, get_event_start_time from ..registry import Registry -from .agent import _agent_id_for, _ensure_agent, _latest_call_of +from .agent import _agent_id_for, _ensure_agent from .topology import _ensure_agent_call, _ensure_session_and_mas @@ -61,8 +61,13 @@ def handle_a2a_message_sent( state = registry.add( State( id=_hash_id( - "state", "a2a.message.sent", session_id, source_span_id, - target_name, message_id, content, + "state", + "a2a.message.sent", + session_id, + source_span_id, + target_name, + message_id, + content, ), sessionId=session_id, content=content, @@ -74,35 +79,11 @@ def handle_a2a_message_sent( def handle_a2a_message_received( event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry ) -> None: - """Ensure the same nodes as ``a2a.message.sent``, and retain the incoming message as a - State of the recipient's AgentCall without changing its initial or final State. - - The event names the recipient only by ``target.agent`` -- its span is not in it, and - its AgentCall is normally created a few milliseconds later by its own - ``topology.node.started`` -- so the State is added only if that call is already there; - otherwise nothing of the message is kept.""" - parties = _ensure_message_parties(event, event_attrs, registry) - if parties is None: - return - _, session_id, source_span_id, target_name, message_id, content = parties - receiver_call = _latest_call_of(registry, target_name, session_id) - if receiver_call is None or not content: - return - state = registry.add( - State( - id=_hash_id( - "state", "a2a.message.received", session_id, source_span_id, - target_name, message_id, content, - ), - sessionId=session_id, - content=content, - ) - ) - registry.add_edge(hasState(source_id=receiver_call.id, target_id=state.id)) + _handle_a2a_message(event, event_attrs, registry) def _agent_card(event_attrs: dict[str, Any]) -> dict[str, Any]: - """The agent card carried in ``capability.content`` (a JSON object), or {} if absent or malformed.""" + """The agent card in ``capability.content``, or {} if absent or malformed.""" content = event_attrs.get("capability.content") if isinstance(content, str): try: @@ -126,10 +107,13 @@ def handle_a2a_agent_card( one of that name already there is reused and only gets its description.""" card = _agent_card(event_attrs) name = str(card.get("name") or "") - mas_name = str(event_attrs.get("session.id") or "").partition("_")[0] + mas_name = get_event_application_id(event, event_attrs) if not name or not mas_name: return - if _ensure_session_and_mas(event, {**event_attrs, "application_id": mas_name}, registry) is None: + if ( + _ensure_session_and_mas(event, {**event_attrs, "application_id": mas_name}, registry) + is None + ): return description = str(card.get("description") or "") agent = _ensure_agent(registry, _agent_id_for(registry, name, (mas_name,)), name, (mas_name,)) @@ -164,3 +148,46 @@ def handle_a2a_tool_started( # target. The ontology has no Agent -> Agent relationship; a handoff is otherwise a # bridge ProcessingCall between the two AgentCalls (heuristics/handoff.py), and the # target's AgentCall does not exist yet when this event arrives. + + +def _handle_a2a_message( + event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry +) -> None: + record, attrs = event, event_attrs + meta = registry.metadata_for + name = str(attrs.get("event.name") or "") + source = str(attrs.get("source.agent.span.id") or "") + call = registry.find(AgentCall, lambda n: n.spanId == source) + content = str( + attrs.get("message.output" if name.endswith("completed") else "message.input") or "" + ) + session_id = get_event_session_id(record, attrs) + if session_id and content: + _ensure_session_and_mas(record, attrs, registry) + state = registry.add( + State( + id=_hash_id( + "state", + name, + session_id, + str(attrs.get("message.id") or ""), + content, + ), + sessionId=session_id, + content=content, + ) + ) + if name.endswith("received"): + meta(state)["receiverName"] = str(attrs.get("target.agent") or "") + meta(state)["eventTime"] = get_event_start_time(record, attrs) + else: + meta(state)["ownerSpanId"] = source + if call is not None and name.endswith("completed"): + registry.add_edge(hasState(source_id=call.id, target_id=state.id)) + return + + +def handle_a2a_message_completed( + event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry +) -> None: + _handle_a2a_message(event, event_attrs, registry) diff --git a/backend/norm/src/norm/ioa_observe/handlers/graph.py b/backend/norm/src/norm/ioa_observe/handlers/graph.py index cbba519..65b8c29 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/graph.py +++ b/backend/norm/src/norm/ioa_observe/handlers/graph.py @@ -1,6 +1,6 @@ # Copyright (c) 2026 Cisco Systems, Inc. and its affiliates # SPDX-License-Identifier: Apache-2.0 -"""Handler for the "*.graph" span name: declared MAS/Agent structural nodes. +"""Retain graph metadata and declare only matching observed agents. This is the one authoritative source of "declared" structural data, so it always overwrites (``registry.upsert``) rather than fills-in-if-absent -- @@ -11,19 +11,15 @@ from __future__ import annotations -import json -import logging from typing import Any from scope_ontology.models.base import KGBase from scope_ontology.models.edges import belongsToMAS from scope_ontology.models.nodes import MAS, Agent, StructuralElement -from ..fields import get_span_application_id +from ..fields import get_span_application_id, parse_json_object from ..registry import Registry -log = logging.getLogger(__name__) - def _declare( registry: Registry, @@ -69,21 +65,26 @@ def handle_graph( graph_raw = span_attrs.get("gen_ai.ioa.graph") if not graph_raw: return - try: - graph_data: dict = json.loads(graph_raw) - except (TypeError, ValueError, json.JSONDecodeError): - log.warning( - "Could not parse gen_ai.ioa.graph for span %s -- skipping structural " - "graph extraction (span name %s)", - span.get("SpanId"), - span.get("SpanName"), - ) - return + graph_data = parse_json_object(graph_raw, "gen_ai.ioa.graph") + graph_nodes = graph_data.get("nodes", {}) + if not isinstance(graph_nodes, dict): + raise ValueError("gen_ai.ioa.graph.nodes must be an object") + mas = registry.get(MAS, application_id) + if mas is not None and graph_data: + registry.metadata_for(mas)["graphNodes"] = graph_nodes + - graph_nodes: dict[str, dict] = graph_data.get("nodes") or {} - for node_id, node in graph_nodes.items(): - if node_id in ("__start__", "__end__"): - continue - agent_name = str(node.get("name") or node_id) - agent: Agent = _declare(registry, Agent, node_id, name=agent_name) - registry.add_edge(belongsToMAS(source_id=agent.id, target_id=application_id)) +def reconcile_graph_agents(registry: Registry) -> None: + """Graph nodes describe execution, not roles; require runtime agent evidence.""" + for mas in registry.all_of(MAS): + graph_nodes = registry.metadata_for(mas).get("graphNodes", {}) + for agent in registry.all_of(Agent): + if mas.id not in registry.edges_from(belongsToMAS, agent.id): + continue + aliases = [agent.id, agent.name, *registry.metadata_for(agent).get("aliases", [])] + if any( + key in aliases or str(value.get("name") or key) in aliases + for key, value in graph_nodes.items() + if isinstance(value, dict) + ): + registry.upsert(agent.model_copy(update={"declared": True})) diff --git a/backend/norm/src/norm/ioa_observe/handlers/llm.py b/backend/norm/src/norm/ioa_observe/handlers/llm.py index a2ce4b8..3f29fbe 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/llm.py +++ b/backend/norm/src/norm/ioa_observe/handlers/llm.py @@ -13,9 +13,9 @@ hasLLMCall, usesLLM, ) -from scope_ontology.models.nodes import LLM, LLMCall +from scope_ontology.models.nodes import LLM, LLMCall, Session, Tool -from ..fields import _hash_id +from ..fields import _hash_id, get_event_session_id, parse_json_object from ..registry import Registry from ..trajectory import add_final_state_transition, add_initial_state from .agent import _own_agent_id @@ -39,7 +39,9 @@ def _ensure_llm_call( return None agent_call, session_id, _, event_time = ensured agent_id = _own_agent_id(registry, agent_call) or str(event_attrs.get("agent.name") or "") - span_id = str(event_attrs.get("llm.call.id") or event.get("SpanId") or event_attrs.get("span.id") or "") + span_id = str( + event_attrs.get("llm.call.id") or event.get("SpanId") or event_attrs.get("span.id") or "" + ) # llm.name is "/" (as in "azure/gpt-4o"); handle_chat gets them separately. provider, _, model = llm_name.partition("/") @@ -113,3 +115,40 @@ def handle_llm_completed( input_text, str(event_attrs.get("llm.output") or ""), ) + + +def handle_llm_tools_presented( + event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry +) -> None: + record, attrs = event, event_attrs + meta = registry.metadata_for + content = parse_json_object(attrs.get("capability.content") or {}, "capability.content") + owner_span = str(attrs.get("ioa_observe.agent.span_id") or record.get("SpanId") or "") + for item in content.get("tools", []): + if not isinstance(item, dict): + continue + function = item.get("function") or item + if not isinstance(function, dict): + raise ValueError("Presented tool function must be an object") + tool_name = str(function.get("name") or "") + if tool_name: + tool = registry.add( + Tool( + id=tool_name, + name=tool_name, + description=str(function.get("description") or ""), + ) + ) + description = str(function.get("description") or "") + if description and not tool.description: + registry.upsert(tool.model_copy(update={"description": description})) + session = registry.get(Session, get_event_session_id(record, attrs)) + if session is not None: + meta(session)["presentedToolIds"] = sorted( + set(meta(session).get("presentedToolIds", []) + [tool_name]) + ) + if owner_span: + meta(tool)["ownerSpanIds"] = sorted( + set(meta(tool).get("ownerSpanIds", []) + [owner_span]) + ) + return diff --git a/backend/norm/src/norm/ioa_observe/handlers/mcp.py b/backend/norm/src/norm/ioa_observe/handlers/mcp.py index 861f3c0..0d48885 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/mcp.py +++ b/backend/norm/src/norm/ioa_observe/handlers/mcp.py @@ -10,6 +10,7 @@ from scope_ontology.models.edges import hasFinalState, hasInitialState, usesTool from scope_ontology.models.nodes import AgentCall, Tool, ToolCall +from ..fields import get_event_application_id from ..registry import Registry from ..trajectory import add_final_state_transition, add_initial_state from .agent import _own_agent_id @@ -36,12 +37,12 @@ def handle_mcp_tools_discovered( """Ensure the MAS and Session, then create every tool in ``capability.content`` and link each to the agent that discovered them (``usesTool``). - Skipped without an ``ioa_observe.agent.span_id`` or a session. The MAS is named by the - first part of ``session.id`` (``_``). The event names the agent only by + Skipped without an ``ioa_observe.agent.span_id`` or a session. The MAS is the originating + application. The event names the agent only by its span id, but an Agent is keyed on its name, so the agent is the one of the AgentCall of - that span. If that call does not exist yet, the tools are created and not linked.""" + that span. If that call does not exist yet, ownership is retained for reconciliation.""" agent_span_id = str(event_attrs.get("ioa_observe.agent.span_id") or "") - mas_name = str(event_attrs.get("session.id") or "").partition("_")[0] + mas_name = get_event_application_id(event, event_attrs) if not agent_span_id or not mas_name: return session = _ensure_session_and_mas(event, {**event_attrs, "application_id": mas_name}, registry) @@ -60,6 +61,12 @@ def handle_mcp_tools_discovered( tool, _ = registry.add_if_not_exists(Tool(id=name, name=name, description=description)) if description and not tool.description: # created earlier by a tool event, without one tool.description = description # type: ignore[attr-defined] + registry.metadata_for(tool)["ownerSpanIds"] = sorted( + set(registry.metadata_for(tool).get("ownerSpanIds", []) + [agent_span_id]) + ) + registry.metadata_for(session)["presentedToolIds"] = sorted( + set(registry.metadata_for(session).get("presentedToolIds", []) + [tool.id]) + ) if agent_id: registry.add_edge(usesTool(source_id=agent_id, target_id=tool.id)) @@ -93,11 +100,13 @@ def _ensure_mcp_tool_call( tool_name = _message_tool_name(message_input) source_name = str(event_attrs.get("source.agent") or "") source_span_id = str( - event_attrs.get("source.agent.span.id") or event_attrs.get("ioa_observe.agent.span_id") or "" + event_attrs.get("source.agent.span.id") + or event_attrs.get("ioa_observe.agent.span_id") + or "" ) if not tool_name or not source_name or not source_span_id: return None - mas_name = str(event_attrs.get("session.id") or "").partition("_")[0] + mas_name = get_event_application_id(event, event_attrs) agent_attrs = { **event_attrs, "application_id": event_attrs.get("application_id") or mas_name, @@ -111,7 +120,7 @@ def _ensure_mcp_tool_call( **agent_attrs, "tool.name": tool_name, "ioa_observe.agent.span_id": source_span_id, - # the message id is the id of the tool call: tool.started/completed carry it as tool.call.id + # MCP and tool lifecycle events share the same natural call id. "tool.call.id": event_attrs.get("message.id"), }, registry, diff --git a/backend/norm/src/norm/ioa_observe/handlers/session.py b/backend/norm/src/norm/ioa_observe/handlers/session.py index 7d17ab1..75433a2 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/session.py +++ b/backend/norm/src/norm/ioa_observe/handlers/session.py @@ -68,8 +68,8 @@ def handle_session_end( span, span_attrs, ) - session.endTime = ended_at # type: ignore[attr-defined] - session.duration = max(0.0, (ended_at - session.startTime) * 1000.0) # type: ignore[attr-defined] + session.endTime = max(session.endTime, ended_at) # type: ignore[attr-defined] + session.duration = max(0.0, (session.endTime - session.startTime) * 1000.0) # type: ignore[attr-defined] end_success = get_span_success(span, span_attrs) if end_success is not None: session.success = end_success # type: ignore[attr-defined] diff --git a/backend/norm/src/norm/ioa_observe/handlers/topology.py b/backend/norm/src/norm/ioa_observe/handlers/topology.py index 79f466f..e454b44 100644 --- a/backend/norm/src/norm/ioa_observe/handlers/topology.py +++ b/backend/norm/src/norm/ioa_observe/handlers/topology.py @@ -17,7 +17,7 @@ hasToolCall, usesTool, ) -from scope_ontology.models.nodes import MAS, Agent, AgentCall, Session, State, ToolCall +from scope_ontology.models.nodes import MAS, AgentCall, Session, State, ToolCall from ..fields import ( _hash_id, @@ -27,30 +27,29 @@ ) from ..registry import Registry from ..trajectory import add_final_state_transition, add_initial_state -from .agent import _agent_id_for, _ensure_agent, _ensure_mas_call, _own_agent_id +from .agent import _agent_id_for, _ensure_agent, _ensure_mas_call def handle_topology_edge_updated( event: dict[str, Any], event_attrs: dict[str, Any], registry: Registry ) -> None: - """The handoff from one agent to another: an ``executesAgent`` edge from the source's - AgentCall to the target Agent. - - The source is the AgentCall of ``source.agent.span.id``; the target is named only - (``target.agent``), so it is the Agent of that name. Nothing is created: the agents' - own events do that, and an edge to something that is not an agent (an MCP server, say) - matches nothing.""" - source_span_id = str(event_attrs.get("source.agent.span.id") or "") - target_name = str(event_attrs.get("target.agent") or "") - session_id = get_event_session_id(event, event_attrs) - if not source_span_id or not target_name or not session_id: + record, attrs = event, event_attrs + meta = registry.metadata_for + if attrs.get("network.protocol.name") == "mcp" or attrs.get("topology.edge.kind") in ( + "tool_call", + "mcp_message", + ): return - source = registry.find( - AgentCall, lambda call: call.spanId == source_span_id and call.sessionId == session_id - ) - target = registry.find(Agent, lambda agent: agent.name == target_name) - if source is not None and target is not None and target.id != _own_agent_id(registry, source): - registry.add_edge(executesAgent(source_id=source.id, target_id=target.id)) + source = str(attrs.get("source.agent.span.id") or "") + target = str(attrs.get("target.agent") or "") + session = _ensure_session_and_mas(record, attrs, registry) + if session is not None and source and target: + pending = meta(session).setdefault("handoffs", []) + hint = {"source": source, "target": target} + if hint not in pending: + pending.append(hint) + pending.sort(key=lambda item: (item["source"], item["target"])) + return def _ensure_session_and_mas( @@ -98,6 +97,14 @@ def handle_topology_session_started( """Ensure the Session and its MAS, and give the Session its initial State.""" session = _ensure_session_and_mas(event, event_attrs, registry) if session is not None: + start = min(session.startTime, get_event_start_time(event, event_attrs)) + session = session.model_copy( + update={ + "startTime": start, + "duration": (session.endTime - start) * 1000, + } + ) + registry.upsert(session) _ensure_session_initial_state(registry, session) @@ -118,7 +125,9 @@ def handle_topology_session_completed( # Session.endTime only ever moves forward, so it is the latest completion seen so far # (it starts at startTime, hence the check for no final State yet). - is_latest = registry.edge_target(hasFinalState, session.id) is None or end_time > session.endTime + is_latest = ( + registry.edge_target(hasFinalState, session.id) is None or end_time > session.endTime + ) if end_time > session.endTime: session.endTime = end_time # type: ignore[attr-defined] session.duration = (end_time - session.startTime) * 1000.0 # type: ignore[attr-defined] @@ -163,8 +172,7 @@ def _ensure_agent_call( # The Agent is keyed on the agent's name, so the Agent that an a2a.agent_card or a handoff # created under that name is the one reused. (The same name in another MAS is another # Agent: see _agent_id_for.) The AgentCall is what carries the span id. - mas_prefix = str(event_attrs.get("session.id") or "").partition("_")[0] - mas_ids = (application_id, mas_prefix) + mas_ids = (application_id,) agent = _ensure_agent(registry, _agent_id_for(registry, agent_id, mas_ids), agent_id, mas_ids) if application_id: registry.add_edge(belongsToMAS(source_id=agent.id, target_id=application_id)) diff --git a/backend/norm/src/norm/ioa_observe/heuristics/container_boundary_states.py b/backend/norm/src/norm/ioa_observe/heuristics/container_boundary_states.py index cecb39e..da7d365 100644 --- a/backend/norm/src/norm/ioa_observe/heuristics/container_boundary_states.py +++ b/backend/norm/src/norm/ioa_observe/heuristics/container_boundary_states.py @@ -20,38 +20,54 @@ from scope_ontology.models.nodes import AgentCall, MASCall, Session from ..registry import Registry -from ..trajectory import adopt_boundary_states +from ..trajectory import _clear_boundaries, adopt_boundary_states def assign_container_boundary_states(registry: Registry) -> None: - for mas_call in registry.all_of(MASCall): - children = [ - registry.get(AgentCall, target_id) - for target_id in registry.edges_from(hasAgentCall, mas_call.id) - ] - ordered = sorted((c for c in children if c is not None), key=lambda c: c.startTime) - if not ordered: - continue - initial_state_id = registry.edge_target(hasInitialState, ordered[0].id) - final_state_id = registry.edge_target(hasFinalState, ordered[-1].id) - if initial_state_id is None or final_state_id is None: - continue - adopt_boundary_states( - registry, mas_call.id, mas_call.sessionId, initial_state_id, final_state_id - ) - - for session in registry.all_of(Session): - children = [ - registry.get(MASCall, target_id) - for target_id in registry.edges_from(hasMASCall, session.id) - ] - ordered = sorted((c for c in children if c is not None), key=lambda c: c.startTime) - if not ordered: - continue - initial_state_id = registry.edge_target(hasInitialState, ordered[0].id) - final_state_id = registry.edge_target(hasFinalState, ordered[-1].id) - if initial_state_id is None or final_state_id is None: - continue - adopt_boundary_states( - registry, session.id, session.sessionId, initial_state_id, final_state_id - ) + for cls, child_cls, edge in ( + (MASCall, AgentCall, hasAgentCall), + (Session, MASCall, hasMASCall), + ): + for container in registry.all_of(cls): + children = [ + registry.get(child_cls, node_id) + for node_id in registry.edges_from(edge, container.id) + ] + children = [child for child in children if child is not None] + if not children: + continue + first = min(children, key=lambda node: (node.startTime, node.id)) + last = max(children, key=lambda node: (node.endTime, node.id)) + initial = registry.edge_target(hasInitialState, first.id) + final = registry.edge_target(hasFinalState, last.id) + if initial and final: + _clear_boundaries(registry, container.id) + adopt_boundary_states(registry, container.id, container.sessionId, initial, final) + + +def update_container_times(registry: Registry) -> None: + meta = registry.metadata_for + for cls, child_cls, edge in ( + (MASCall, AgentCall, hasAgentCall), + (Session, MASCall, hasMASCall), + ): + for container in registry.all_of(cls): + children = [registry.get(child_cls, i) for i in registry.edges_from(edge, container.id)] + children = [c for c in children if c is not None] + if children: + start = min(container.startTime, *(c.startTime for c in children)) + end = max(container.endTime, *(c.endTime for c in children)) + registry.upsert( + container.model_copy( + update={ + "startTime": start, + "endTime": end, + "duration": (end - start) * 1000, + **( + {"spanId": meta(container).get("sessionSpanId", "")} + if cls is Session + else {} + ), + } + ) + ) diff --git a/backend/norm/src/norm/ioa_observe/reconciliation.py b/backend/norm/src/norm/ioa_observe/reconciliation.py new file mode 100644 index 0000000..90a647f --- /dev/null +++ b/backend/norm/src/norm/ioa_observe/reconciliation.py @@ -0,0 +1,299 @@ +# Copyright (c) 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Persisted lifecycle and cross-record relationship reconciliation.""" + +from __future__ import annotations + +import json +import logging +from dataclasses import replace +from typing import Any + +from scope_ontology.models.edges import ( + belongsToMASCall, + correspondsToTransition, + executesAgent, + executesLLM, + executesSession, + executesTool, + hasAgentCall, + hasFinalState, + hasInitialState, + hasLLMCall, + hasMASCall, + hasState, + hasToolCall, + usesLLM, + usesTool, +) +from scope_ontology.models.nodes import ( + Agent, + AgentCall, + LLMCall, + MASCall, + ProcessingCall, + Session, + State, + Tool, + ToolCall, + Transition, +) + +from .fields import _hash_id, get_span_session_id +from .handlers.agent import _agent_id_for, _own_agent_id +from .handlers.graph import reconcile_graph_agents +from .handlers.mcp import _message_tool_name +from .heuristics.container_boundary_states import update_container_times +from .registry import HANDOFF_SOURCES_PROP, Registry, SpanContext + +logger = logging.getLogger(__name__) +_CALLS = (AgentCall, LLMCall, ToolCall) + + +def span_context(record: dict[str, Any], attrs: dict[str, Any]) -> SpanContext | None: + session_id = get_span_session_id(record, attrs) + span_id = str(record.get("SpanId") or "") + trace_id = str(record.get("TraceId") or (f"session:{session_id}" if session_id else "")) + if not trace_id or not span_id: + return None + return SpanContext( + trace_id=trace_id, + span_id=span_id, + parent_span_id=str(record.get("ParentSpanId") or ""), + owner_span_id=str(attrs.get("ioa_observe.agent.span_id") or ""), + session_id=session_id, + ) + + +def event_context(record: dict[str, Any], attrs: dict[str, Any]) -> SpanContext | None: + if attrs.get("event.name") not in ( + "mcp.message.sent", + "mcp.message.received", + "mcp.message.completed", + ): + return None + operation = str(attrs.get("operation.name") or "") + if operation and operation != "tools/call": + return None + if not operation and not _message_tool_name(attrs.get("message.input")): + return None + source_trace = str(attrs.get("source.agent.trace.id") or "") + source_owner = ( + attrs.get("source.agent.span.id") + if not source_trace or source_trace == record.get("TraceId") + else "" + ) + owner = attrs.get("ioa_observe.agent.span_id") or source_owner + if not owner: + return None + return span_context( + record, + { + **attrs, + "ioa_observe.agent.span_id": owner, + }, + ) + + +def merge_span_context(context: SpanContext, previous: SpanContext | None) -> SpanContext: + if previous is None: + return context + return replace( + context, + parent_span_id=context.parent_span_id or previous.parent_span_id, + owner_span_id=context.owner_span_id or previous.owner_span_id, + session_id=context.session_id or previous.session_id, + ) + + +def remember_span_ancestry( + record: dict[str, Any], attrs: dict[str, Any], registry: Registry +) -> None: + context = span_context(record, attrs) + if context is not None: + key = (context.trace_id, context.span_id) + registry.span_contexts[key] = merge_span_context(context, registry.span_contexts.get(key)) + + +def _owning_agent_call( + registry: Registry, child: LLMCall | ToolCall, agents: list[AgentCall] +) -> AgentCall | None: + meta = registry.metadata_for + owner = str(meta(child).get("ownerSpanId") or child.parentSpanId or "") + trace_id = str(meta(child).get("traceId") or "") + context_trace = trace_id or f"session:{child.sessionId}" + candidates = { + agent.spanId: agent + for agent in agents + if agent.sessionId == child.sessionId + and not (trace_id and meta(agent).get("traceId") and meta(agent)["traceId"] != trace_id) + } + visited: set[str] = set() + while owner and owner not in visited: + visited.add(owner) + ancestor = registry.span_contexts.get((context_trace, owner)) + if ancestor is not None and ancestor.session_id not in ("", child.sessionId): + return None + if owner in candidates: + return candidates[owner] + owner = (ancestor.owner_span_id or ancestor.parent_span_id) if ancestor is not None else "" + return None + + +def handoff_sources(registry: Registry) -> dict[str, list[str]]: + return { + call.id: list(registry.extras.get(("AgentCall", call.id), {}).get(HANDOFF_SOURCES_PROP, [])) + for call in registry.all_of(AgentCall) + } + + +def reconcile_relationships(registry: Registry) -> None: + meta = registry.metadata_for + signals = handoff_sources(registry) + agents = registry.all_of(AgentCall) + for cls, contains, execution_edge, uses in ( + (LLMCall, hasLLMCall, executesLLM, usesLLM), + (ToolCall, hasToolCall, executesTool, usesTool), + ): + for child in registry.all_of(cls): + parent = _owning_agent_call(registry, child, agents) + if parent is None: + application = registry.edge_target(executesSession, child.sessionId) or "" + owner_agent = str(meta(child).get("ownerAgentId") or "") + agent_id = ( + _agent_id_for(registry, owner_agent, (application,)) if owner_agent else "" + ) + target = registry.edge_target(execution_edge, child.id) + if target and registry.get(Agent, agent_id) is not None: + registry.add_edge(uses(source_id=agent_id, target_id=target)) + continue + registry.add_edge(contains(source_id=parent.id, target_id=child.id)) + agent = _own_agent_id(registry, parent) + target = registry.edge_target(execution_edge, child.id) + if agent and target: + registry.add_edge(uses(source_id=agent, target_id=target)) + registry.remove_edges_where( + lambda edge, source, _: edge in (usesTool, usesLLM) and registry.get(Agent, source) is None + ) + for tool in registry.all_of(Tool): + for owner in meta(tool).get("ownerSpanIds", []): + for call in agents: + agent = _own_agent_id(registry, call) + if call.spanId == owner and agent: + registry.add_edge(usesTool(source_id=agent, target_id=tool.id)) + reconcile_graph_agents(registry) + for state in registry.all_of(State): + receiver = meta(state).get("receiverName") + owner = meta(state).get("ownerSpanId") + candidates = [] + for call in agents: + agent = registry.get(Agent, _own_agent_id(registry, call) or "") + if call.sessionId == state.sessionId and ( + (owner and call.spanId == owner) + or (receiver and agent is not None and agent.name == receiver) + ): + candidates.append(call) + if candidates: + event_time = meta(state).get("eventTime", 0) + call = min( + candidates, + key=lambda a: ( + a.startTime < event_time, + abs(a.startTime - event_time), + a.id, + ), + ) + registry.remove_edges_where( + lambda edge, _, target: edge is hasState and target == state.id + ) + registry.add_edge(hasState(source_id=call.id, target_id=state.id)) + for session in registry.all_of(Session): + for hint in meta(session).get("handoffs", []): + source = next( + (a for a in agents if a.spanId == hint["source"] and a.sessionId == session.id), + None, + ) + if source is None: + continue + targets = [ + a + for a in agents + if a.sessionId == session.id + and a.id != source.id + and (agent := registry.get(Agent, _own_agent_id(registry, a) or "")) is not None + and agent.name == hint["target"] + ] + if targets: + target = min( + targets, + key=lambda a: ( + a.startTime < source.startTime, + abs(a.startTime - source.startTime), + a.id, + ), + ) + signals[target.id] = sorted(set(signals.get(target.id, []) + [source.spanId])) + for edge, source, target in list(registry.edges): + if edge is hasAgentCall: + registry.add_edge(belongsToMASCall(source_id=target, target_id=source)) + + for call_id, sources in signals.items(): + if sources: + registry.extras.setdefault(("AgentCall", call_id), {})[HANDOFF_SOURCES_PROP] = sources + update_container_times(registry) + + +def strip_derived(registry: Registry) -> None: + """Drop everything ``apply_heuristics`` derives, so it can be rebuilt.""" + # Session events own native states until complete child boundaries can replace them. + containers = { + n.id + for cls, edge in ((Session, hasMASCall), (MASCall, hasAgentCall)) + for n in registry.all_of(cls) + if registry.edges_from(edge, n.id) + and not ( + cls is Session + and registry.edge_target(hasInitialState, n.id) + == _hash_id("state", n.id, "initial", "") + ) + } + bridges = {n.id for n in registry.all_of(ProcessingCall)} + owners = containers | bridges + transitions = { + target + for etype, source, target in registry.edges + if etype is correspondsToTransition and source in owners + } + for bridge_id in bridges: + registry.remove_node(ProcessingCall, bridge_id) + for transition_id in transitions: + registry.remove_node(Transition, transition_id) + gone = bridges | transitions + boundary_edges = (hasInitialState, hasFinalState, hasState) + registry.remove_edges_where( + lambda etype, source, target: ( + source in gone or target in gone or (etype in boundary_edges and source in containers) + ) + ) + + +def handoff_spans(registry: Registry) -> list[dict[str, Any]]: + """Minimal ``*.agent`` spans carrying only the handoff signal, rebuilt from stored state.""" + signals = handoff_sources(registry) + spans: list[dict[str, Any]] = [] + for call in registry.all_of(AgentCall): + sources = signals.get(call.id) + agent_id = registry.edge_target(executesAgent, call.id) + if not sources or agent_id is None: + continue + spans.append( + { + "SpanName": "handoff.agent", + "SpanId": call.spanId, + "SpanAttributes": { + "agent_id": agent_id, + "ioa_observe.handoff.source.span_ids": json.dumps(sources), + }, + } + ) + return spans diff --git a/backend/norm/src/norm/ioa_observe/registry.py b/backend/norm/src/norm/ioa_observe/registry.py index c2e2f52..ef1a0b0 100644 --- a/backend/norm/src/norm/ioa_observe/registry.py +++ b/backend/norm/src/norm/ioa_observe/registry.py @@ -11,17 +11,39 @@ from __future__ import annotations from collections.abc import Callable -from typing import TypeVar, cast +from dataclasses import dataclass +from typing import Any, TypeVar, cast from scope_ontology.models.base import KGBase T = TypeVar("T", bound=KGBase) +MetadataGetter = Callable[[KGBase], dict[str, Any]] +NORMALIZATION_PROP = "rtNormalization" +HANDOFF_SOURCES_PROP = "handoffSourceSpanIds" + + +@dataclass(frozen=True) +class SpanContext: + """Private ancestry evidence, not an ontology node or raw telemetry buffer.""" + + trace_id: str + span_id: str + parent_span_id: str + owner_span_id: str + session_id: str class Registry: def __init__(self) -> None: self.nodes: dict[tuple[str, str], KGBase] = {} self.edges: dict[tuple[type, str, str], KGBase] = {} + self.extras: dict[tuple[str, str], dict[str, Any]] = {} + self.span_contexts: dict[tuple[str, str], SpanContext] = {} + + def metadata_for(self, node: KGBase) -> dict[str, Any]: + """Persist reconciliation evidence without changing ontology models.""" + extra = self.extras.setdefault((type(node).__name__, node.id), {}) + return extra.setdefault(NORMALIZATION_PROP, {}) def get(self, cls: type[T], node_id: str) -> T | None: return cast("T | None", self.nodes.get((cls.__name__, node_id))) @@ -55,6 +77,7 @@ def add_edge(self, edge: KGBase) -> None: def remove_node(self, cls: type[KGBase], node_id: str) -> None: self.nodes.pop((cls.__name__, node_id), None) + self.extras.pop((cls.__name__, node_id), None) def remove_edges_where(self, predicate: Callable[[type, str, str], bool]) -> None: """Drop every edge for which predicate(edge_class, source_id, target_id) holds.""" @@ -104,6 +127,10 @@ def merge_node_into(self, cls: type[KGBase], drop_id: str, keep_id: str) -> None moved = edge.model_copy(update={"source_id": new_source, "target_id": new_target}) self.edges.setdefault((etype, new_source, new_target), moved) self.nodes.pop((cls.__name__, drop_id), None) + extra = self.extras.pop((cls.__name__, drop_id), {}) + kept = self.extras.setdefault((cls.__name__, keep_id), {}) + for key, value in extra.items(): + kept.setdefault(key, value) def all_of(self, cls: type[T]) -> list[T]: """Return every node of this class.""" diff --git a/backend/norm/src/norm/ioa_observe/stream.py b/backend/norm/src/norm/ioa_observe/stream.py index 2f8b612..0a41135 100644 --- a/backend/norm/src/norm/ioa_observe/stream.py +++ b/backend/norm/src/norm/ioa_observe/stream.py @@ -1,10 +1,10 @@ # Copyright (c) 2026 Cisco Systems, Inc. and its affiliates # SPDX-License-Identifier: Apache-2.0 -"""Stateless, span-at-a-time normalization against a graph database. +"""Stateless, record-at-a-time normalization against a graph database. ``build_kg`` needs the whole span array. ``StreamNormalizer.process`` takes a -single span and keeps nothing between calls: the graph already stored in the -database *is* the state. For each span it +single span and keeps nothing between calls: the stored graph and adapter-managed +trace context provide its state. For each span it 1. loads what it needs from the database through a ``GraphReader`` (the span's session subgraph plus the structural nodes the span touches), @@ -14,48 +14,52 @@ 4. returns only the difference against what was loaded, as a ``StreamDelta``, for the caller to write back. +Transport ancestry is loaded separately, including spans without session ids. +It is retained through ``StreamDelta.span_contexts``, never as ontology executions. +Its lifetime and durability depend on the adapter. + The heuristics' derived items (bridge ProcessingCalls, container-boundary Transitions and States) are dropped from the loaded graph and rebuilt every time, so a late-arriving span replaces stale ones instead of piling up. -The one thing a node cannot rebuild is a span's handoff signal -(``ioa_observe.handoff.source.span_ids``): ontology models forbid extra -fields, so it is carried on the AgentCall payload as the non-ontology -``handoffSourceSpanIds`` property. +Reconciliation evidence is persisted as ``rtNormalization`` extras because +ontology models forbid extra fields. Legacy ``handoffSourceSpanIds`` on +AgentCall payloads remain supported. """ from __future__ import annotations import json from collections.abc import Sequence +from copy import deepcopy from dataclasses import dataclass, field from typing import Any, Literal, Protocol from pydantic import ValidationError from scope_ontology.models import edges as edge_models from scope_ontology.models import nodes as node_models -from scope_ontology.models.base import KGBase -from scope_ontology.models.edges import ( - correspondsToTransition, - executesAgent, - hasFinalState, - hasInitialState, - hasState, +from scope_ontology.models.nodes import ( + Agent, + Session, ) -from scope_ontology.models.nodes import Agent, AgentCall, MASCall, ProcessingCall, Session, Transition +from ..verifier import PROPERTY_DOMAIN_RANGE from .build import apply_heuristics, dispatch_span, serialize_registry from .events import dispatch_event -from .fields import get_event_attrs, get_event_session_id, get_span_attrs, get_span_session_id -from .heuristics.handoff import _source_span_ids -from .registry import Registry -from ..verifier import PROPERTY_DOMAIN_RANGE +from .fields import ( + _hash_id, + get_event_session_id, + get_span_session_id, + normalize_record, + parse_json_object, +) +from .handlers.topology import _ensure_session_and_mas +from .reconciliation import event_context, merge_span_context, span_context +from .registry import HANDOFF_SOURCES_PROP, NORMALIZATION_PROP, Registry, SpanContext Kind = Literal["span", "event"] KINDS = ("span", "event") -HANDOFF_SOURCES_PROP = "handoffSourceSpanIds" - @dataclass class StreamDelta: @@ -64,16 +68,22 @@ class StreamDelta: ``nodes``/``edges`` are new or modified items; ``removed_*`` are items the database holds that the recomputed graph no longer contains. Edges carry ``from_type``/``to_type`` so they can be written without their endpoint - nodes being part of the same delta. + nodes being part of the same delta. ``span_contexts`` is private ancestry + evidence that adapters must retain separately from ontology nodes. """ nodes: list[dict[str, Any]] = field(default_factory=list) edges: list[dict[str, Any]] = field(default_factory=list) removed_edges: list[dict[str, Any]] = field(default_factory=list) removed_nodes: list[dict[str, Any]] = field(default_factory=list) + span_contexts: list[SpanContext] = field(default_factory=list) class GraphReader(Protocol): + def load_span_contexts(self, trace_id: str) -> list[SpanContext]: + """Load private ancestry evidence, including sessionless transport spans.""" + ... + def load( self, session_id: str, node_ids: Sequence[str] ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: @@ -97,11 +107,8 @@ def _edge_key(edge: dict[str, Any]) -> tuple[str, str, str, str]: return ("edge", str(edge["edge_type"]), str(edge["from_id"]), str(edge["to_id"])) -def _rehydrate( - nodes: list[dict[str, Any]], edges: list[dict[str, Any]] -) -> tuple[Registry, dict[str, list[str]]]: +def _rehydrate(nodes: list[dict[str, Any]], edges: list[dict[str, Any]]) -> Registry: registry = Registry() - signals: dict[str, list[str]] = {} for raw in nodes: cls = getattr(node_models, str(raw["node_type"]), None) if cls is None: @@ -110,64 +117,25 @@ def _rehydrate( try: node = cls.model_validate(data) except ValidationError: - continue # partial node (only an id) written ahead of its own span; that span will fill it in + # A partial node written ahead of its span will be filled in by that span. + continue registry.upsert(node) - sources = raw.get(HANDOFF_SOURCES_PROP) - if sources: - signals[node.id] = [str(item) for item in sources] + extra = { + key: deepcopy(raw[key]) + for key in (NORMALIZATION_PROP, HANDOFF_SOURCES_PROP) + if raw.get(key) + } + if extra: + registry.extras[(type(node).__name__, node.id)] = extra for raw in edges: cls = getattr(edge_models, str(raw["edge_type"]), None) if cls is None: continue registry.add_edge(cls(source_id=str(raw["from_id"]), target_id=str(raw["to_id"]))) - return registry, signals - - -def _strip_derived(registry: Registry) -> None: - """Drop everything ``apply_heuristics`` derives, so it can be rebuilt.""" - containers = {n.id for n in registry.all_of(Session)} | {n.id for n in registry.all_of(MASCall)} - bridges = {n.id for n in registry.all_of(ProcessingCall)} - owners = containers | bridges - transitions = { - target - for etype, source, target in registry.edges - if etype is correspondsToTransition and source in owners - } - for bridge_id in bridges: - registry.remove_node(ProcessingCall, bridge_id) - for transition_id in transitions: - registry.remove_node(Transition, transition_id) - gone = bridges | transitions - boundary_edges = (hasInitialState, hasFinalState, hasState) - registry.remove_edges_where( - lambda etype, source, target: ( - source in gone or target in gone or (etype in boundary_edges and source in containers) - ) - ) + return registry -def _handoff_spans(registry: Registry, signals: dict[str, list[str]]) -> list[dict[str, Any]]: - """Minimal ``*.agent`` spans carrying only the handoff signal, rebuilt from stored state.""" - spans: list[dict[str, Any]] = [] - for call in registry.all_of(AgentCall): - sources = signals.get(call.id) - agent_id = registry.edge_target(executesAgent, call.id) - if not sources or agent_id is None: - continue - spans.append( - { - "SpanName": "handoff.agent", - "SpanId": call.spanId, - "SpanAttributes": { - "agent_id": agent_id, - "ioa_observe.handoff.source.span_ids": json.dumps(sources), - }, - } - ) - return spans - - -def _endpoint_type(edge_type: str, side: int, node_id: str, node_types: dict[str, str]) -> str: +def _endpoint_type(edge_type: str, side: int, node_id: str, node_types: dict[str, set[str]]) -> str: """Type of an edge endpoint: its node's own type, else the one class the ontology allows there (``side`` 0 = domain, 1 = range). "" if ambiguous. @@ -175,28 +143,48 @@ def _endpoint_type(edge_type: str, side: int, node_id: str, node_types: dict[str emitted by a chat span before the agent span that creates the Agent -- in which case the database holds a partial (id-only) node until then. """ - if node_id in node_types: - return node_types[node_id] candidates = PROPERTY_DOMAIN_RANGE.get(edge_type, ([], []))[side] + actual = node_types.get(node_id, set()) + compatible = { + name + for name in actual + if any(base.__name__ in candidates for base in getattr(node_models, name).__mro__) + } + if len(compatible) == 1: + return next(iter(compatible)) + if len(actual) == 1: + return next(iter(actual)) + if len(actual) > 1: + raise ValueError(f"Ambiguous {edge_type} endpoint {node_id!r}: {sorted(actual)}") return str(candidates[0]) if len(candidates) == 1 else "" -def _snapshot(registry: Registry, signals: dict[str, list[str]]) -> dict[tuple[str, ...], dict[str, Any]]: +def _snapshot(registry: Registry) -> dict[tuple[str, ...], dict[str, Any]]: nodes, edges = serialize_registry(registry) - node_types = {str(n["id"]): str(n["node_type"]) for n in nodes} + node_types: dict[str, set[str]] = {} + for node in nodes: + node_types.setdefault(str(node["id"]), set()).add(str(node["node_type"])) + execution_types = { + (source, target): cls.__name__ + for cls, source, target in registry.edges + if cls.__name__.startswith("executes") and cls.__name__ != "executes" + } items: dict[tuple[str, ...], dict[str, Any]] = {} for node in nodes: - if node["node_type"] == "AgentCall" and signals.get(str(node["id"])): - node[HANDOFF_SOURCES_PROP] = signals[str(node["id"])] items[_node_key(node)] = node for edge in edges: - edge["from_type"] = _endpoint_type(edge["edge_type"], 0, str(edge["from_id"]), node_types) - edge["to_type"] = _endpoint_type(edge["edge_type"], 1, str(edge["to_id"]), node_types) + edge_type = edge["edge_type"] + if edge_type == "executes": + edge_type = execution_types.get((edge["from_id"], edge["to_id"]), edge_type) + edge["from_type"] = _endpoint_type(edge_type, 0, str(edge["from_id"]), node_types) + edge["to_type"] = _endpoint_type(edge_type, 1, str(edge["to_id"]), node_types) items[_edge_key(edge)] = edge return items -def _diff(before: dict[tuple[str, ...], dict[str, Any]], after: dict[tuple[str, ...], dict[str, Any]]) -> StreamDelta: +def _diff( + before: dict[tuple[str, ...], dict[str, Any]], after: dict[tuple[str, ...], dict[str, Any]] +) -> StreamDelta: delta = StreamDelta() for key, item in after.items(): if before.get(key) != item: @@ -212,109 +200,168 @@ def __init__(self, reader: GraphReader) -> None: self._reader = reader def process(self, record: dict[str, Any], kind: Kind = "span") -> StreamDelta: - """Normalize one item against the stored graph and return what changed. - - ``kind`` says what ``record`` is: ``"span"`` (an OTel span, the default) or - ``"event"`` (an OTel log record). Events are routed by ``event.name`` (see events.py); their - handlers are stubs for now, so they change nothing. - """ - if kind == "event": - return self._process_event(record) - elif kind == "span": - return self._process_trace(record) - else: - raise ValueError(f"Unknown kind {kind!r}, expected one of {KINDS}") - - def _process_event(self, event: dict[str, Any]) -> StreamDelta: - event_attrs = get_event_attrs(event) - session_id = get_event_session_id(event, event_attrs) - - # Dry run on an empty registry to learn which nodes this event touches. - scratch = Registry() - if not dispatch_event(event, event_attrs, scratch): - return StreamDelta() - touched = {node.id for node in scratch.nodes.values()} - for _etype, source_id, target_id in scratch.edges: - touched.update((source_id, target_id)) - # Agents are keyed on a span id but matched by name (an a2a.agent_card has no span id), - # so the reader is also handed their names: it loads the Agents called that. - touched.update(node.name for node in scratch.all_of(Agent)) - - # The dry run can't see what depends on stored state (e.g. an edge to an agent found - # through an existing AgentCall), so if the real run touches more, load those too - # and run again -- otherwise their stored copies would be missed and re-sent. - for _ in range(2): - stored_nodes, stored_edges = self._reader.load(session_id, sorted(touched)) - registry, signals = _rehydrate(stored_nodes, stored_edges) - before = _snapshot(registry, signals) - dispatch_event(event, event_attrs, registry) - after = _snapshot(registry, signals) - produced: set[str] = set() - for key, item in after.items(): - if before.get(key) != item: - produced.update( - (str(item["from_id"]), str(item["to_id"])) - if key[0] == "edge" - else (str(item["id"]),) - ) - if produced <= touched: - break - touched |= produced - return _diff(before, after) - - def _process_trace(self, span: dict[str, Any]) -> StreamDelta: - span_attrs = get_span_attrs(span) - session_id = get_span_session_id(span, span_attrs) - - # Dry run on an empty registry to learn which nodes this span touches. + if kind not in ("span", "event"): + raise ValueError(f"Unknown kind {kind!r}") + record, attrs = normalize_record(record, kind) + session_id = ( + get_span_session_id(record, attrs) + if kind == "span" + else get_event_session_id(record, attrs) + ) + context = span_context(record, attrs) if kind == "span" else event_context(record, attrs) + trace_id = str(record.get("TraceId") or (f"session:{session_id}" if session_id else "")) + contexts = ( + { + (item.trace_id, item.span_id): item + for item in self._reader.load_span_contexts(trace_id) + } + if trace_id + else {} + ) + delta = StreamDelta() + if context is not None: + key = (context.trace_id, context.span_id) + context = merge_span_context(context, contexts.get(key)) + if contexts.get(key) != context: + delta.span_contexts.append(context) + contexts[key] = context + sessions = ( + {session_id} + if session_id + else {item.session_id for item in contexts.values() if item.session_id} + ) + # A sessionless transport arriving last may complete ancestry in several sessions. + for current_session in sorted(sessions or {""}): + change = self._process(record, attrs, kind, current_session, contexts) + delta.nodes.extend(change.nodes) + delta.edges.extend(change.edges) + delta.removed_nodes.extend(change.removed_nodes) + delta.removed_edges.extend(change.removed_edges) + return delta + + def _process( + self, + record: dict[str, Any], + attrs: dict[str, Any], + kind: Kind, + session_id: str, + contexts: dict[tuple[str, str], SpanContext], + ) -> StreamDelta: + application = str(attrs.get("application_id") or "") + touched = {session_id, application, str(record.get("SpanId") or "")} + touched.update( + str(attrs.get(k) or "") + for k in ( + "agent_id", + "agent.name", + "source.agent", + "target.agent", + "ioa_observe.entity.name", + ) + ) + graph_data = ( + parse_json_object(attrs["gen_ai.ioa.graph"], "gen_ai.ioa.graph") + if attrs.get("gen_ai.ioa.graph") + else {} + ) + if not isinstance(graph_data.get("nodes", {}), dict): + raise ValueError("gen_ai.ioa.graph.nodes must be an object") + for node_id, node in graph_data.get("nodes", {}).items(): + touched.add(str(node_id)) + if isinstance(node, dict): + touched.add(str(node.get("name") or node_id)) + # Structural ids (tools, models, cards) need loading even before a session exists. scratch = Registry() - dispatch_span(span, scratch) - touched = {node.id for node in scratch.nodes.values()} - for _etype, source_id, target_id in scratch.edges: - touched.update((source_id, target_id)) - - stored_nodes, stored_edges = self._reader.load(session_id, sorted(touched)) - registry, signals = _rehydrate(stored_nodes, stored_edges) - - # Already processed (re-delivery / re-run): handlers are not order-independent, - # so replaying a span into a complete graph would change it. Make it a no-op. - span_id = str(span.get("SpanId") or "") - if span_id and any(getattr(n, "spanId", None) == span_id for n in registry.nodes.values()): - return StreamDelta() - - before = _snapshot(registry, signals) - - dispatch_span(span, registry) - if str(span.get("SpanName") or "").endswith(".agent"): - sources = _source_span_ids(span_attrs) - call = registry.find(AgentCall, lambda a: a.spanId == span_id) - if sources and call is not None: - signals[call.id] = sources - - _strip_derived(registry) - apply_heuristics(_handoff_spans(registry, signals), registry) - after = _snapshot(registry, signals) - - return _diff(before, after) + recognized = False + if kind == "span": + dispatch_span( + { + **record, + "SpanAttributes": {k: v for k, v in attrs.items() if k != "gen_ai.ioa.graph"}, + }, + scratch, + ) + else: + recognized = dispatch_event(record, attrs, scratch) + if not recognized or (not scratch.nodes and event_context(record, attrs) is None): + return StreamDelta() + touched.update(n.id for n in scratch.nodes.values()) + touched.update(n.name for n in scratch.all_of(Agent)) + for _, source, target in scratch.edges: + touched.update((source, target)) + stored_nodes, stored_edges = self._reader.load(session_id, sorted(touched - {""})) + pending_tools = { + tool_id + for node in stored_nodes + if node["node_type"] == "Session" and node["id"] == session_id + for tool_id in (node.get(NORMALIZATION_PROP) or {}).get("presentedToolIds", []) + } + structural_ids = {str(node["id"]) for node in stored_nodes if not node.get("sessionId")} + related_ids = pending_tools | structural_ids + if not related_ids <= touched: + touched.update(related_ids) + stored_nodes, stored_edges = self._reader.load(session_id, sorted(touched - {""})) + stored_nodes = [ + n for n in stored_nodes if not n.get("sessionId") or n["sessionId"] == session_id + ] + ids = {n["id"] for n in stored_nodes} + stored_edges = [e for e in stored_edges if e["from_id"] in ids and e["to_id"] in ids] + registry = _rehydrate(stored_nodes, stored_edges) + registry.span_contexts.update(contexts) + meta = registry.metadata_for + before = deepcopy(_snapshot(registry)) + event_id = str( + attrs.get("event.id") or _hash_id("event", json.dumps(record, sort_keys=True)) + ) + session = registry.get(Session, session_id) + if ( + kind == "event" + and session is not None + and event_id in meta(session).get("eventIds", []) + ): + apply_heuristics([], registry) + return _diff(before, _snapshot(registry)) + if kind == "span": + dispatch_span(record, registry) + else: + if recognized or attrs.get("event.name") in ( + "llm.tools.presented", + "a2a.message.completed", + ): + _ensure_session_and_mas(record, attrs, registry) + dispatch_event(record, attrs, registry) + session = registry.get(Session, session_id) + if session is not None: + meta(session)["eventIds"] = sorted( + set(meta(session).get("eventIds", []) + [event_id]) + ) + apply_heuristics([], registry) + return _diff(before, _snapshot(registry)) class InMemoryGraph: """A ``GraphReader`` plus writer over plain dicts; for tests and dry runs.""" - # Set on creation and never changed afterwards, same as the Neo4j store (endTime may advance). - IMMUTABLE_PROPS = frozenset({"startTime"}) - def __init__(self) -> None: self.nodes: dict[tuple[str, str], dict[str, Any]] = {} self.edges: dict[tuple[str, str, str], dict[str, Any]] = {} + self.span_contexts: dict[tuple[str, str], SpanContext] = {} + + def load_span_contexts(self, trace_id: str) -> list[SpanContext]: + return [item for (trace, _), item in self.span_contexts.items() if trace == trace_id] def load( self, session_id: str, node_ids: Sequence[str] ) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: ids = set(node_ids) - by_id = {node["id"]: node for node in self.nodes.values()} - in_session = {i for i, n in by_id.items() if session_id and n.get("sessionId") == session_id} - named = {i for i, n in by_id.items() if n["node_type"] == "Agent" and n.get("name") in ids} + in_session = { + n["id"] for n in self.nodes.values() if session_id and n.get("sessionId") == session_id + } + named = { + n["id"] + for n in self.nodes.values() + if n["node_type"] == "Agent" and n.get("name") in ids + } edges = [ e for e in self.edges.values() @@ -327,10 +374,12 @@ def load( wanted = ( in_session | named | ids | {e["from_id"] for e in edges} | {e["to_id"] for e in edges} ) - nodes = [dict(by_id[i]) for i in wanted if i in by_id] + nodes = [dict(node) for node in self.nodes.values() if node["id"] in wanted] return nodes, [dict(e) for e in edges] def apply(self, delta: StreamDelta) -> None: + for context in delta.span_contexts: + self.span_contexts[(context.trace_id, context.span_id)] = context for edge in delta.removed_edges: self.edges.pop((edge["edge_type"], edge["from_id"], edge["to_id"]), None) for node in delta.removed_nodes: @@ -344,6 +393,6 @@ def apply(self, delta: StreamDelta) -> None: if existing is None: self.nodes[key] = dict(node) else: - existing.update({k: v for k, v in node.items() if k not in self.IMMUTABLE_PROPS}) + existing.update(node) for edge in delta.edges: self.edges[(edge["edge_type"], edge["from_id"], edge["to_id"])] = dict(edge) diff --git a/backend/norm/src/norm/ioa_observe/trajectory.py b/backend/norm/src/norm/ioa_observe/trajectory.py index 69b81fa..10a8f6d 100644 --- a/backend/norm/src/norm/ioa_observe/trajectory.py +++ b/backend/norm/src/norm/ioa_observe/trajectory.py @@ -6,6 +6,7 @@ from typing import Literal +from scope_ontology.models.base import KGBase from scope_ontology.models.edges import ( correspondsToTransition, executesProcessing, @@ -17,10 +18,19 @@ leadsTo, representsExecution, ) -from scope_ontology.models.nodes import Processing, ProcessingCall, State, Transition +from scope_ontology.models.nodes import ( + AgentCall, + LLMCall, + Processing, + ProcessingCall, + Session, + State, + ToolCall, + Transition, +) from .fields import _hash_id -from .registry import Registry +from .registry import MetadataGetter, Registry def add_initial_state( @@ -222,3 +232,94 @@ def adopt_boundary_states( registry.add_edge(leadsTo(source_id=transition.id, target_id=final_state_id)) registry.add_edge(correspondsToTransition(source_id=owner_id, target_id=transition.id)) registry.add_edge(representsExecution(source_id=transition.id, target_id=owner_id)) + + +def _clear_boundaries(registry: Registry, owner_id: str) -> None: + boundaries = { + target + for edge, source, target in registry.edges + if source == owner_id and edge in (hasInitialState, hasFinalState) + } + transitions = set(registry.edges_from(correspondsToTransition, owner_id)) + registry.remove_edges_where( + lambda edge, source, target: ( + source == owner_id + and ( + edge in (hasInitialState, hasFinalState, correspondsToTransition) + or (edge is hasState and target in boundaries) + ) + ) + ) + for transition_id in transitions: + if not registry.edge_source(correspondsToTransition, transition_id): + registry.remove_node(Transition, transition_id) + registry.remove_edges_where(lambda _, source, target: transition_id in (source, target)) + + +def _remember_trajectory(registry: Registry, call: KGBase, meta: MetadataGetter) -> None: + meta(call)["trajectory"] = { + "initial": registry.edge_target(hasInitialState, call.id), + "final": registry.edge_target(hasFinalState, call.id), + "transition": registry.edge_target(correspondsToTransition, call.id), + } + + +def restore_trajectories(registry: Registry, meta: MetadataGetter) -> None: + # Shared heuristics merge equal-content States. Recreate each call's canonical + # boundaries first so a transient partial graph cannot determine their final ids. + for cls in _CALLS: + for call in registry.all_of(cls): + trajectory = meta(call).get("trajectory") + if not trajectory: + continue + initial = registry.get(State, registry.edge_target(hasInitialState, call.id) or "") + final = registry.get(State, registry.edge_target(hasFinalState, call.id) or "") + if initial is None: + continue + _clear_boundaries(registry, call.id) + initial = registry.add(initial.model_copy(update={"id": trajectory["initial"]})) + registry.add_edge(hasInitialState(source_id=call.id, target_id=initial.id)) + registry.add_edge(hasState(source_id=call.id, target_id=initial.id)) + if final is not None and trajectory["final"] and trajectory["transition"]: + final = registry.add(final.model_copy(update={"id": trajectory["final"]})) + registry.add_edge(hasFinalState(source_id=call.id, target_id=final.id)) + registry.add_edge(hasState(source_id=call.id, target_id=final.id)) + transition = registry.add( + Transition( + id=trajectory["transition"], + sessionId=call.sessionId, + ) + ) + for edge, source, target in ( + (inputTo, initial.id, transition.id), + (leadsTo, transition.id, final.id), + (correspondsToTransition, call.id, transition.id), + (representsExecution, transition.id, call.id), + ): + registry.add_edge(edge(source_id=source, target_id=target)) + + +def prune_trajectories(registry: Registry, meta: MetadataGetter) -> None: + # A replaced event trajectory must not leave disconnected States or Transitions behind. + transitions = {target for edge, _, target in registry.edges if edge is correspondsToTransition} + for node in registry.all_of(Transition): + if node.id not in transitions: + registry.remove_node(Transition, node.id) + registry.remove_edges_where(lambda _, source, target: node.id in (source, target)) + referenced = {i for _, source, target in registry.edges for i in (source, target)} + for cls in (State, Processing): + for node in registry.all_of(cls): + pending = isinstance(node, State) and ( + meta(node).get("receiverName") or meta(node).get("ownerSpanId") + ) + if ( + node.id not in referenced + and pending + and registry.get(Session, node.sessionId) is not None + ): + registry.add_edge(hasState(source_id=node.sessionId, target_id=node.id)) + if node.id not in referenced and not pending: + registry.remove_node(cls, node.id) + + +_CALLS = (AgentCall, LLMCall, ToolCall) diff --git a/backend/norm/tests/test_normalizer.py b/backend/norm/tests/test_normalizer.py index e029053..47037d0 100644 --- a/backend/norm/tests/test_normalizer.py +++ b/backend/norm/tests/test_normalizer.py @@ -97,8 +97,13 @@ def test_stream_normalizer_converges_to_batch_result() -> None: graph.apply(StreamNormalizer(graph).process(span)) batch_nodes, batch_edges = normalize(spans) - assert {i for _, i in graph.nodes} == {n["id"] for n in batch_nodes} - assert set(graph.edges) == {(e["edge_type"], e["from_id"], e["to_id"]) for e in batch_edges} + assert graph.nodes == {(n["node_type"], n["id"]): n for n in batch_nodes} + assert { + key: {field: value for field, value in edge.items() if field not in ("from_type", "to_type")} + for key, edge in graph.edges.items() + } == { + (edge["edge_type"], edge["from_id"], edge["to_id"]): edge for edge in batch_edges + } def test_stream_normalizer_ignores_already_processed_span() -> None: @@ -160,7 +165,7 @@ def test_topology_session_started_creates_session_and_mas() -> None: assert session["startTime"] == 1792075846.93875 assert session["spanId"] == "211fdd62f03f330c" assert sorted(e["edge_type"] for e in delta.edges) == [ - "executesSession", "hasInitialState", "hasState" + "executes", "executesSession", "hasInitialState", "hasState" ] assert not any(e["edge_type"] == "hasFinalState" for e in delta.edges) @@ -205,7 +210,10 @@ def test_topology_node_events_build_agent_call_states_and_transition() -> None: graph.apply(normalizer.process(event, kind="event")) types = sorted(t for t, _ in graph.nodes) - assert types == ["Agent", "AgentCall", "MAS", "MASCall", "Session", "State", "State", "Transition"] + assert types == [ + "Agent", "AgentCall", "MAS", "MASCall", "Session", "State", "State", + "Transition", "Transition", "Transition", + ] contents = sorted(n["content"] for (t, _), n in graph.nodes.items() if t == "State") assert contents == ["bye", "hello"] edge_types = {e["edge_type"] for e in graph.edges.values()} @@ -255,7 +263,8 @@ def test_topology_session_completed_creates_missing_nodes_and_final_state() -> N assert {"executesSession", "hasInitialState", "hasFinalState", "inputTo", "leadsTo"} <= edge_types assert normalizer.process(completed, kind="event") == StreamDelta() - # started afterwards adds nothing; started-then-completed ends the session later + # Late starts correct timing without replacing the completion trajectory. + graph.apply(normalizer.process(started, kind="event")) assert normalizer.process(started, kind="event") == StreamDelta() graph2 = InMemoryGraph() n2 = StreamNormalizer(graph2) @@ -263,6 +272,8 @@ def test_topology_session_completed_creates_missing_nodes_and_final_state() -> N graph2.apply(n2.process(completed, kind="event")) session = graph2.nodes[("Session", "213dad2f")] assert session["endTime"] > session["startTime"] and session["duration"] > 0 + assert graph.nodes[("Session", "213dad2f")]["startTime"] == session["startTime"] + assert graph.nodes[("Session", "213dad2f")]["duration"] == session["duration"] assert sorted(t for t, _ in graph2.nodes) == types @@ -465,13 +476,14 @@ def test_a2a_tool_started_is_a_handoff_not_a_tool() -> None: graph.apply(normalizer.process(started, kind="event")) # no Tool/ToolCall; the target is an unattached Agent named tool.name - assert list(graph.nodes) == [("Agent", "handoff_agent")] - assert not graph.edges + assert sorted(t for t, _ in graph.nodes) == ["Agent", "MAS", "Session"] + assert [i for t, i in graph.nodes if t == "Agent"] == ["handoff_agent"] + assert not any(e["edge_type"] == "executesAgent" for e in graph.edges.values()) assert normalizer.process(started, kind="event") == StreamDelta() # an Agent of that name already there is used, not duplicated graph.apply(normalizer.process(_tool_event("tool.completed", **a2a, **{"tool.output": "out"}), kind="event")) - assert list(graph.nodes) == [("Agent", "handoff_agent")] + assert [i for t, i in graph.nodes if t == "Agent"] == ["handoff_agent"] # the target's own events then use that node (keyed on the name) and connect it node = _node_event("topology.node.started", **{"agent.name": "handoff_agent", "agent.input": "q"}) @@ -555,6 +567,8 @@ def test_a2a_agent_card_creates_unattached_agent_then_events_connect_it() -> Non assert _edges(late) == edges_before # a card without a description leaves the existing one alone nameless = json.dumps({"name": "schedule_agent"}) + late.apply(late_normalizer.process(_card_event(nameless), kind="event")) + assert late.nodes[("Agent", span)]["description"] == "Updated." assert late_normalizer.process(_card_event(nameless), kind="event") == StreamDelta() for bad in ("not json", "[]", "{}"): @@ -708,16 +722,17 @@ def test_a2a_message_received_keeps_the_message_on_the_recipients_call() -> None normalizer = StreamNormalizer(graph) received = _a2a_message_event("a2a.message.received", "2026-10-15T14:50:47.020000+00:00") - # the recipient's call is not there yet: the same nodes as sent, but no State + # Retain the pending message without inventing a recipient Agent or AgentCall. graph.apply(normalizer.process(received, kind="event")) - assert "State" not in {t for t, _ in graph.nodes} - assert "Schedule Agent" in {i for t, i in graph.nodes if t == "Agent"} + assert sorted(t for t, _ in graph.nodes) == ["MAS", "Session", "State"] + pending = next(n for (t, _), n in graph.nodes.items() if t == "State") + assert pending["rtNormalization"]["receiverName"] == "Schedule Agent" # once the recipient's own node.started has created its call, the message is kept on it node = _node_event("topology.node.started", **{"agent.name": "Schedule Agent", "agent.input": "q"}) node["SpanId"] = "4163c385d3710aac" graph.apply(normalizer.process(node, kind="event")) - graph.apply(normalizer.process(received, kind="event")) + assert normalizer.process(received, kind="event") == StreamDelta() receiver_call = next( n["id"] for (t, _), n in graph.nodes.items() if t == "AgentCall" and n["spanId"] == "4163c385d3710aac" ) @@ -1002,7 +1017,8 @@ def test_edge_updated_bridges_the_handoff_between_two_agent_calls() -> None: second["LogAttributes"]["event.time"] = "2026-10-15T14:50:48.900000+00:00" for event in (first, first_done, second): graph.apply(normalizer.process(event, kind="event")) - assert "ProcessingCall" not in {t for t, _ in graph.nodes} + # Timestamp fallback can already bridge siblings; explicit hints preserve provenance. + assert "ProcessingCall" in {t for t, _ in graph.nodes} # the source is found by its span id, the target by its name only handoff = _edge_updated_event("aaaa", "worker") @@ -1010,9 +1026,19 @@ def test_edge_updated_bridges_the_handoff_between_two_agent_calls() -> None: assert "ProcessingCall" in {t for t, _ in graph.nodes} assert "Processing" in {t for t, _ in graph.nodes} assert any(e["edge_type"] == "hasProcessingCall" for e in graph.edges.values()) + target_call = next( + n for (t, _), n in graph.nodes.items() if t == "AgentCall" and n["spanId"] == "bbbb" + ) + assert target_call["handoffSourceSpanIds"] == ["aaaa"] assert normalizer.process(handoff, kind="event") == StreamDelta() # a target that is no agent with a call (an MCP server) matches nothing - assert normalizer.process(_edge_updated_event("aaaa", "debug"), kind="event") == StreamDelta() - # nor does an unknown source span - assert normalizer.process(_edge_updated_event("nope", "worker"), kind="event") == StreamDelta() + graph.apply(normalizer.process(_edge_updated_event("aaaa", "debug"), kind="event")) + assert ("Agent", "debug") not in graph.nodes + # Unknown sources are retained for late arrival, not turned into executions. + graph.apply(normalizer.process(_edge_updated_event("nope", "worker"), kind="event")) + hints = graph.nodes[("Session", "213dad2f")]["rtNormalization"]["handoffs"] + assert {"source": "nope", "target": "worker"} in hints + assert not any( + n["spanId"] == "nope" for (t, _), n in graph.nodes.items() if t == "AgentCall" + ) diff --git a/backend/norm/tests/test_pipeline_functional.py b/backend/norm/tests/test_pipeline_functional.py index 0906b97..20c92f0 100644 --- a/backend/norm/tests/test_pipeline_functional.py +++ b/backend/norm/tests/test_pipeline_functional.py @@ -11,9 +11,8 @@ ``opentelemetry-instrumentation-langchain``, which observe-sdk activates by default whenever langchain/langgraph is installed -- OTel GenAI-semconv names like ``execute_task ``/``invoke_agent LangGraph`` that don't match any -dispatched span-name suffix and are silently ignored by design: the current -architecture only ever builds nodes from a span whose name it recognizes, it -never tries to account for every row in the trace. +dispatched span-name suffix. They do not become ontology executions, but their +compact ancestry can still be used to connect recognized child calls to agents. """ from __future__ import annotations diff --git a/backend/norm/tests/test_reconciliation.py b/backend/norm/tests/test_reconciliation.py new file mode 100644 index 0000000..79a2183 --- /dev/null +++ b/backend/norm/tests/test_reconciliation.py @@ -0,0 +1,1060 @@ +# Copyright (c) 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 + +import itertools +import json +import random + +import pytest + +from norm import InMemoryGraph, StreamDelta, StreamNormalizer, normalize + + +def process_record(graph, record, kind): + delta = StreamNormalizer(graph).process(record, kind) + graph.apply(delta) + return delta + + +def span(name, span_id, **attrs): + return { + "SpanName": name, + "SpanId": span_id, + "Timestamp": "2026-11-10 00:00:00.000000000", + "Duration": 2_000_000_000, + "SpanAttributes": {"application_id": "app", "session.id": "app_s1", **attrs}, + } + + +def event(name, span_id="", time="2026-11-10T00:00:01+00:00", service="app", **attrs): + return { + "ServiceName": service, + "SpanId": span_id, + "LogAttributes": { + "event.name": name, + "event.time": time, + "application_id": service, + "session.id": "app_s1", + **attrs, + }, + } + + +def replay(records): + graph = InMemoryGraph() + for record, kind in records: + process_record(graph, record, kind) + return graph + + +def nodes(graph, kind): + return [n for (t, _), n in graph.nodes.items() if t == kind] + + +def stored_copy(graph): + restored = InMemoryGraph() + restored.nodes = {key: json.loads(json.dumps(value)) for key, value in graph.nodes.items()} + restored.edges = {key: json.loads(json.dumps(value)) for key, value in graph.edges.items()} + restored.span_contexts = dict(graph.span_contexts) + return restored + + +@pytest.mark.parametrize("explicit_owner", [False, True]) +def test_mcp_transport_ancestry_backfills_tool_ownership_in_every_order(explicit_owner): + agent = span("planner.agent", "agent", agent_id="planner") + client = span( + "MCP send tools/call search", + "client", + **({"ioa_observe.agent.span_id": "agent"} if explicit_owner else {}), + ) + client["ParentSpanId"] = "framework" if explicit_owner else "agent" + server = span("tools/call search", "server") + server["SpanAttributes"] = {} + server["ParentSpanId"] = "client" + tool = span( + "search.tool", + "tool", + **{"application_id": "tool-service", "ioa_observe.entity.name": "search"}, + ) + tool["ParentSpanId"] = "server" + records = [agent, client, server, tool] + for record in records: + record["TraceId"] = "trace" + expected_nodes, expected_edges = normalize(records) + expected_nodes = {(n["node_type"], n["id"]): n for n in expected_nodes} + expected_edges = {(e["edge_type"], e["from_id"], e["to_id"]): e for e in expected_edges} + for order in itertools.permutations(records): + graph = InMemoryGraph() + for record in order: + process_record(graph, record, "span") + graph = stored_copy(graph) + call = nodes(graph, "ToolCall")[0] + owner = nodes(graph, "AgentCall")[0] + assert ("hasToolCall", owner["id"], call["id"]) in graph.edges + assert ("usesTool", "planner", "search") in graph.edges + assert [mas["id"] for mas in nodes(graph, "MAS")] == ["app"] + assert graph.nodes == expected_nodes + assert { + key: {k: v for k, v in edge.items() if k not in ("from_type", "to_type")} + for key, edge in graph.edges.items() + } == expected_edges + for record in records: + assert process_record(graph, record, "span") == StreamDelta() + + +def test_transport_ancestry_does_not_cross_trace_or_session(): + agent = span("planner.agent", "agent", agent_id="planner") + agent["TraceId"] = "trace" + client = span("tools/call.mcp", "client", **{"ioa_observe.agent.span_id": "agent"}) + client["TraceId"] = "other-trace" + tool = span("search.tool", "tool", **{"ioa_observe.entity.name": "search"}) + tool.update(TraceId="trace", ParentSpanId="client") + graph = replay((record, "span") for record in (agent, client, tool)) + assert not any(edge == "hasToolCall" for edge, _, _ in graph.edges) + client["TraceId"] = "trace" + client["SpanAttributes"]["session.id"] = "app_other" + process_record(graph, client, "span") + assert not any(edge == "hasToolCall" for edge, _, _ in graph.edges) + client["SpanAttributes"]["session.id"] = "app_s1" + process_record(graph, client, "span") + owner = nodes(graph, "AgentCall")[0] + call = nodes(graph, "ToolCall")[0] + assert ("hasToolCall", owner["id"], call["id"]) in graph.edges + + +def test_transport_ancestry_cycle_remains_unowned(): + first = span("tools/call.mcp", "one") + first["ParentSpanId"] = "two" + second = span("tools/call search", "two") + second["ParentSpanId"] = "one" + tool = span("search.tool", "tool", **{"ioa_observe.entity.name": "search"}) + tool["ParentSpanId"] = "one" + graph = replay((record, "span") for record in (tool, first, second)) + assert len(nodes(graph, "ToolCall")) == 1 + assert not any(edge == "hasToolCall" for edge, _, _ in graph.edges) + + +def test_sessionless_transport_is_private_context_not_an_execution(): + record = { + "TraceId": "trace", + "SpanId": "server", + "ParentSpanId": "client", + "SpanName": "tools/call search", + "ServiceName": "tool-service", + "SpanAttributes": {}, + } + graph = InMemoryGraph() + delta = process_record(graph, record, "span") + assert not delta.nodes and not delta.edges + assert not graph.nodes and not graph.edges + assert len(delta.span_contexts) == 1 + graph = stored_copy(graph) + assert graph.load_span_contexts("trace")[0].session_id == "" + assert process_record(graph, record, "span") == StreamDelta() + + +@pytest.mark.parametrize("include_transport_span", [False, True]) +@pytest.mark.parametrize("include_message_input", [False, True]) +def test_mcp_received_event_supplies_tool_parent_ownership_in_every_order( + include_transport_span, include_message_input +): + agent = span("planner.agent", "agent", agent_id="planner") + tool = span("search.tool", "tool", **{"ioa_observe.entity.name": "search"}) + tool["ParentSpanId"] = "server" + received = event( + "mcp.message.received", + "server", + **{ + "source.agent": "planner", + "source.agent.span.id": "agent", + "source.agent.trace.id": "trace", + "operation.name": "tools/call", + "message.id": "request", + "message.input": '{"name": "search", "arguments": {}}', + }, + ) + if not include_message_input: + received["LogAttributes"].pop("message.input") + records = [(agent, "span"), (tool, "span"), (received, "event")] + if include_transport_span: + server = span("tools/call search", "server") + server["SpanAttributes"] = {} + server["ParentSpanId"] = "client" + records.append((server, "span")) + for record, _ in records: + record["TraceId"] = "trace" + for order in itertools.permutations(records): + graph = InMemoryGraph() + for record, kind in order: + process_record(graph, record, kind) + graph = stored_copy(graph) + owner = nodes(graph, "AgentCall")[0] + assert len(nodes(graph, "ToolCall")) == (2 if include_message_input else 1) + for call in nodes(graph, "ToolCall"): + assert ("hasToolCall", owner["id"], call["id"]) in graph.edges + context = graph.span_contexts[("trace", "server")] + assert context.owner_span_id == "agent" + assert context.parent_span_id == ("client" if include_transport_span else "") + for record, kind in records: + assert process_record(graph, record, kind) == StreamDelta() + graph.span_contexts.clear() + for key in list(graph.edges): + if key[0] == "hasToolCall": + del graph.edges[key] + process_record(graph, received, "event") + for call in nodes(graph, "ToolCall"): + assert ("hasToolCall", owner["id"], call["id"]) in graph.edges + + +def test_mcp_received_event_does_not_infer_an_owner_in_another_trace(): + agent = span("planner.agent", "agent", agent_id="planner") + tool = span("search.tool", "tool", **{"ioa_observe.entity.name": "search"}) + tool["ParentSpanId"] = "server" + received = event( + "mcp.message.received", + "server", + **{ + "source.agent": "planner", + "source.agent.span.id": "agent", + "source.agent.trace.id": "other-trace", + "message.id": "request", + "message.input": '{"name": "search", "arguments": {}}', + }, + ) + for record in (agent, tool, received): + record["TraceId"] = "trace" + graph = replay([(agent, "span"), (received, "event"), (tool, "span")]) + call = next(n for n in nodes(graph, "ToolCall") if n["spanId"] == "tool") + assert not any( + edge == "hasToolCall" and target == call["id"] for edge, _, target in graph.edges + ) + + +def test_explicit_transport_owner_takes_precedence_over_outer_agent_parent(): + inner = span("inner.agent", "inner", agent_id="inner") + outer = span("outer.agent", "outer", agent_id="outer") + client = span("tools/call.mcp", "client", **{"ioa_observe.agent.span_id": "inner"}) + client["ParentSpanId"] = "outer" + tool = span("search.tool", "tool", **{"ioa_observe.entity.name": "search"}) + tool["ParentSpanId"] = "client" + for order in ((tool, client, outer, inner), (outer, inner, client, tool)): + graph = replay((record, "span") for record in order) + call = nodes(graph, "ToolCall")[0] + parents = [ + source + for edge, source, target in graph.edges + if edge == "hasToolCall" and target == call["id"] + ] + assert len(parents) == 1 + assert graph.nodes[("AgentCall", parents[0])]["spanId"] == "inner" + + +def test_replaying_legacy_completed_spans_upgrades_missing_ancestry(): + agent = span("planner.agent", "agent", agent_id="planner") + tool = span("search.tool", "tool", **{"ioa_observe.entity.name": "search"}) + tool["ParentSpanId"] = "server" + server = span( + "MCP server tools/call search", "server", **{"ioa_observe.agent.span_id": "agent"} + ) + for record in (agent, tool, server): + record["TraceId"] = "trace" + graph = replay((record, "span") for record in (tool, agent)) + graph.span_contexts.clear() + for node in graph.nodes.values(): + node.get("rtNormalization", {}).pop("traceId", None) + graph = stored_copy(graph) + for record in (tool, agent, server): + process_record(graph, record, "span") + call = nodes(graph, "ToolCall")[0] + owner = nodes(graph, "AgentCall")[0] + assert call["rtNormalization"]["traceId"] == "trace" + assert ("hasToolCall", owner["id"], call["id"]) in graph.edges + assert ("usesTool", "planner", "search") in graph.edges + + +def test_session_completion_spans_keep_latest_end_and_redeliver_without_changes(): + start = span("session.start", "start", ioa_start_time=1000) + latest = span("session.end", "latest", **{"session.ended_at": 1010}) + early = span("session.end", "early", **{"session.ended_at": 1005}) + for endings in ((latest, early), (early, latest)): + graph = replay((record, "span") for record in (start, *endings)) + assert nodes(graph, "Session")[0]["endTime"] == 1010 + for record in (latest, early): + assert process_record(graph, record, "span") == StreamDelta() + + +def test_agent_and_tool_with_same_id_keep_distinct_capability_relationships(): + agent = span("schedule.agent", "agent", agent_id="schedule") + tool = span( + "schedule.tool", + "tool", + **{"ioa_observe.entity.name": "schedule", "ioa_observe.agent.span_id": "agent"}, + ) + graph = replay((record, "span") for record in (tool, agent)) + process_record(graph, span("execute_tool schedule", "transport"), "span") + assert ("Agent", "schedule") in graph.nodes + assert ("Tool", "schedule") in graph.nodes + assert ("usesTool", "schedule", "schedule") in graph.edges + edge = graph.edges[("usesTool", "schedule", "schedule")] + assert (edge["from_type"], edge["to_type"]) == ("Agent", "Tool") + owner = nodes(graph, "AgentCall")[0] + call = nodes(graph, "ToolCall")[0] + assert graph.edges[("executes", owner["id"], "schedule")]["to_type"] == "Agent" + assert graph.edges[("executes", call["id"], "schedule")]["to_type"] == "Tool" + + +@pytest.mark.parametrize("order", [(0, 1, 2, 3), (3, 2, 1, 0), (2, 0, 3, 1)]) +def test_batch_and_stream_share_full_graph_payloads(order): + records = [ + span("planner.agent", "agent", agent_id="planner"), + span( + "model.chat", + "chat", + **{ + "ioa_observe.agent.span_id": "agent", + "gen_ai.provider.name": "test", + "gen_ai.request.model": "model", + }, + ), + span( + "search.tool", + "tool", + **{ + "ioa_observe.agent.span_id": "agent", + "ioa_observe.entity.name": "search", + }, + ), + span( + "workflow.graph", + "graph", + **{ + "gen_ai.ioa.graph": json.dumps( + {"nodes": {"planner": {}, "router": {}, "search": {}}} + ) + }, + ), + ] + ordered = [records[index] for index in order] + batch_nodes, batch_edges = normalize(ordered) + graph = replay((record, "span") for record in ordered) + assert graph.nodes == {(node["node_type"], node["id"]): node for node in batch_nodes} + assert { + key: {k: v for k, v in edge.items() if k not in ("from_type", "to_type")} + for key, edge in graph.edges.items() + } == {(edge["edge_type"], edge["from_id"], edge["to_id"]): edge for edge in batch_edges} + assert [agent["id"] for agent in nodes(graph, "Agent")] == ["planner"] + + +def test_legacy_worker_metadata_survives_reload_and_completed_span_enrichment(): + started = event( + "tool.started", + "tool", + **{ + "tool.name": "search", + "tool.call.id": "natural-id", + "ioa_observe.agent.span_id": "agent", + "tool.input": "provisional", + }, + ) + graph = stored_copy(replay([(started, "event")])) + completed = span( + "search.tool", + "tool", + **{ + "gen_ai.tool.call.id": "natural-id", + "ioa_observe.entity.name": "search", + "ioa_observe.entity.input": "question", + "ioa_observe.entity.output": "answer", + "ioa_observe.agent.span_id": "agent", + }, + ) + process_record(graph, completed, "span") + graph = stored_copy(graph) + process_record(graph, span("planner.agent", "agent", agent_id="planner"), "span") + call = nodes(graph, "ToolCall")[0] + assert call["id"] == "natural-id" + assert call["rtNormalization"]["spanComplete"] is True + assert "inputPriority" not in call["rtNormalization"] + assert len(nodes(graph, "ToolCall")) == 1 + agent = nodes(graph, "AgentCall")[0] + assert ("hasToolCall", agent["id"], call["id"]) in graph.edges + assert process_record(graph, completed, "span") == StreamDelta() + + +def test_legacy_handoff_property_is_preserved_without_new_metadata(): + target = span( + "target.agent", + "target-span", + **{ + "agent_id": "target", + "ioa_start_time": 1001, + "ioa_observe.entity.input": "request", + "ioa_observe.entity.output": "reply", + "ioa_observe.handoff.source.span_ids": json.dumps(["source-span"]), + }, + ) + graph = stored_copy(replay([(target, "span")])) + for node in graph.nodes.values(): + node.pop("rtNormalization", None) + process_record( + graph, + span( + "source.agent", + "source-span", + **{ + "agent_id": "source", + "ioa_start_time": 1000, + "ioa_observe.entity.output": "handoff", + }, + ), + "span", + ) + call = next(n for n in nodes(graph, "AgentCall") if n["spanId"] == "target-span") + assert call["handoffSourceSpanIds"] == ["source-span"] + assert len(nodes(graph, "ProcessingCall")) == 1 + assert len(nodes(graph, "AgentCall")) == 2 + + +@pytest.mark.parametrize("reverse", [False, True]) +def test_cards_and_discovered_tools_preserve_application_names_with_underscores(reverse): + records = [ + event( + "a2a.agent_card", + "card", + service="tool_service", + **{ + "session.id": "origin_mas_s1", + "capability.content": json.dumps({"name": "Planner"}), + }, + ), + event( + "mcp.tools.discovered", + "discovery", + service="tool_service", + **{ + "session.id": "origin_mas_s1", + "ioa_observe.agent.span_id": "agent", + "capability.content": json.dumps({"tools": [{"name": "search"}]}), + }, + ), + event( + "topology.node.started", + "agent", + service="origin_mas", + **{"session.id": "origin_mas_s1", "agent.name": "Planner"}, + ), + ] + graph = replay((record, "event") for record in (reversed(records) if reverse else records)) + assert [mas["id"] for mas in nodes(graph, "MAS")] == ["origin_mas"] + assert [session["id"] for session in nodes(graph, "Session")] == ["s1"] + assert ("usesTool", "Planner", "search") in graph.edges + assert ("belongsToMAS", "Planner", "origin_mas") in graph.edges + + +def test_container_final_state_follows_last_ending_child(): + long = span( + "long.agent", + "long", + **{"agent_id": "long", "ioa_start_time": 1000, "ioa_observe.entity.output": "last"}, + ) + long["Duration"] = 10_000_000_000 + short = span( + "short.agent", + "short", + **{"agent_id": "short", "ioa_start_time": 1001, "ioa_observe.entity.output": "first"}, + ) + for records in ((long, short), (short, long)): + graph = replay((record, "span") for record in records) + for cls in ("Session", "MASCall"): + container = nodes(graph, cls)[0] + assert container["startTime"] == 1000 + assert container["endTime"] == 1010 + final = next( + edge["to_id"] + for edge in graph.edges.values() + if edge["edge_type"] == "hasFinalState" and edge["from_id"] == container["id"] + ) + assert graph.nodes[("State", final)]["content"] == "last" + + +def test_invalid_graph_node_shape_fails_in_batch_and_stream(): + record = span("workflow.graph", "graph", **{"gen_ai.ioa.graph": '{"nodes":[]}'}) + with pytest.raises(ValueError, match="nodes must be an object"): + normalize([record]) + with pytest.raises(ValueError, match="nodes must be an object"): + StreamNormalizer(InMemoryGraph()).process(record) + + +def test_native_session_states_survive_until_child_boundaries_are_usable(): + graph = replay([(event("topology.session.started"), "event")]) + initial = next( + edge["to_id"] + for edge in graph.edges.values() + if edge["edge_type"] == "hasInitialState" and edge["from_id"] == "s1" + ) + process_record( + graph, event("topology.node.started", "agent", **{"agent.name": "planner"}), "event" + ) + assert ("hasInitialState", "s1", initial) in graph.edges + process_record( + graph, + event("topology.session.completed", time="2026-11-10T00:00:04+00:00"), + "event", + ) + assert any( + edge["edge_type"] == "hasFinalState" and edge["from_id"] == "s1" + for edge in graph.edges.values() + ) + process_record( + graph, + event( + "topology.node.completed", + "agent", + time="2026-11-10T00:00:03+00:00", + **{"agent.name": "planner", "agent.output": "done"}, + ), + "event", + ) + final = next( + edge["to_id"] + for edge in graph.edges.values() + if edge["edge_type"] == "hasFinalState" and edge["from_id"] == "s1" + ) + assert graph.nodes[("State", final)]["content"] == "done" + + +def test_pending_handoff_metadata_is_independent_of_arrival_order(): + records = [ + event( + "topology.edge.updated", + "edge", + **{"source.agent.span.id": source, "target.agent": target}, + ) + for source, target in (("one", "two"), ("two", "three")) + ] + first = replay((record, "event") for record in records) + second = replay((record, "event") for record in reversed(records)) + assert first.nodes == second.nodes + assert first.edges == second.edges + assert not nodes(first, "Agent") + + +def test_capability_ownership_is_independent_of_span_arrival_order(): + records = [ + span("planner.agent", "agent", agent_id="planner"), + span( + "model.chat", + "chat", + **{ + "agent_id": "planner", + "ioa_observe.agent.span_id": "agent", + "gen_ai.provider.name": "test", + "gen_ai.request.model": "model", + }, + ), + span( + "search.tool", + "tool", + **{ + "agent_id": "planner", + "ioa_observe.agent.span_id": "agent", + "ioa_observe.entity.name": "search", + }, + ), + ] + for order in itertools.permutations(records): + graph = replay((r, "span") for r in order) + agent = nodes(graph, "AgentCall")[0] + for node_type, edge_type in ( + ("LLMCall", "hasLLMCall"), + ("ToolCall", "hasToolCall"), + ): + assert ( + edge_type, + agent["id"], + nodes(graph, node_type)[0]["id"], + ) in graph.edges + assert process_record(graph, order[-1], "span") == StreamDelta() + + +@pytest.mark.parametrize("event_first", [False, True]) +def test_agent_events_and_spans_share_one_complete_call(event_first): + started = event("topology.node.started", "agent", **{"agent.name": "Planner"}) + completed = span( + "planner.agent", + "agent", + **{ + "agent_id": "planner-id", + "ioa_observe.entity.name": "Planner", + "ioa_observe.entity.input": "question", + "ioa_observe.entity.output": "answer", + }, + ) + records = [(started, "event"), (completed, "span")] + graph = replay(records if event_first else reversed(records)) + assert len(nodes(graph, "Agent")) == len(nodes(graph, "AgentCall")) == 1 + call = nodes(graph, "AgentCall")[0] + assert call["duration"] == 2000 + final = graph.edges[ + ( + "hasFinalState", + call["id"], + next( + e["to_id"] + for e in graph.edges.values() + if e["edge_type"] == "hasFinalState" and e["from_id"] == call["id"] + ), + ) + ] + assert graph.nodes[("State", final["to_id"])]["content"] == "answer" + assert process_record(graph, completed, "span") == StreamDelta() + + +@pytest.mark.parametrize("server_first", [False, True]) +def test_mcp_participant_does_not_create_another_mas(server_first): + started = event("topology.node.started", "agent", **{"agent.name": "Planner"}) + message = event( + "mcp.message.received", + "server", + service="tool-server", + **{ + "source.agent": "Planner", + "source.agent.span.id": "agent", + "target.agent": "debug", + "message.id": "request-1", + "message.input": '{"name":"search","arguments":{}}', + }, + ) + records = [(started, "event"), (message, "event")] + graph = replay(records if not server_first else reversed(records)) + assert [n["id"] for n in nodes(graph, "MAS")] == ["app"] + assert len(nodes(graph, "Session")) == len(nodes(graph, "Agent")) == 1 + assert {e["to_id"] for e in graph.edges.values() if e["edge_type"] == "executesSession"} == { + "app" + } + + +@pytest.mark.parametrize("graph_first", [False, True]) +def test_graph_metadata_declares_only_observed_agents(graph_first): + graph_span = span( + "workflow.graph", + "graph", + **{ + "gen_ai.ioa.graph": json.dumps( + { + "nodes": {n: {"id": n, "name": n} for n in ("planner", "router", "tools")}, + } + ) + }, + ) + agent_span = span("planner.agent", "agent", agent_id="planner") + records = [(graph_span, "span"), (agent_span, "span")] + graph = replay(records if graph_first else reversed(records)) + assert [n["name"] for n in nodes(graph, "Agent")] == ["planner"] + assert nodes(graph, "Agent")[0]["declared"] is True + assert process_record(graph, graph_span, "span") == StreamDelta() + + +@pytest.mark.parametrize("edge_first", [False, True]) +def test_explicit_handoff_is_backfilled_without_assigning_a_second_agent(edge_first): + handoff = event( + "topology.edge.updated", + "transport", + **{ + "source.agent": "Planner", + "source.agent.span.id": "source", + "target.agent": "Worker", + "topology.edge.kind": "a2a_message", + "network.protocol.name": "a2a", + }, + ) + records = [ + ( + event("topology.node.started", "source", **{"agent.name": "Planner"}), + "event", + ), + ( + event( + "topology.node.completed", + "source", + **{ + "agent.name": "Planner", + "agent.output": "plan", + }, + ), + "event", + ), + ( + event( + "topology.node.started", + "target", + **{ + "agent.name": "Worker", + "agent.input": "execute", + }, + ), + "event", + ), + ] + graph = replay( + ([(handoff, "event")] + records) if edge_first else (records + [(handoff, "event")]) + ) + assert nodes(graph, "ProcessingCall") + for call in nodes(graph, "AgentCall"): + assert ( + sum( + e["edge_type"] == "executesAgent" and e["from_id"] == call["id"] + for e in graph.edges.values() + ) + == 1 + ) + assert process_record(graph, handoff, "event") == StreamDelta() + + +def test_presented_tools_and_a2a_completion_are_retained(): + graph = replay( + [ + ( + event("topology.node.started", "agent", **{"agent.name": "Planner"}), + "event", + ), + ( + event( + "llm.tools.presented", + "agent", + **{ + "capability.content": json.dumps( + {"tools": [{"type": "function", "function": {"name": "search"}}]} + ), + }, + ), + "event", + ), + ( + event( + "a2a.message.completed", + "transport", + **{ + "source.agent": "Planner", + "source.agent.span.id": "agent", + "target.agent": "Worker", + "message.id": "m1", + "message.output": "reply", + }, + ), + "event", + ), + ] + ) + assert any(n["name"] == "search" for n in nodes(graph, "Tool")) + assert any(n["content"] == "reply" for n in nodes(graph, "State")) + + +@pytest.mark.parametrize("message_kind", ["received", "completed"]) +def test_unresolved_a2a_messages_stay_connected_until_their_owner_arrives(message_kind): + message = event( + f"a2a.message.{message_kind}", + "transport", + **{ + "source.agent": "Sender", + "source.agent.span.id": "sender", + "target.agent": "Worker Display Name", + "message.id": "message", + "message.input": "request", + "message.output": "response", + }, + ) + unrelated = event( + "topology.node.completed", + "other", + **{"agent.name": "worker_runtime", "agent.input": "input", "agent.output": "output"}, + ) + owner = event( + "topology.node.started", + "receiver" if message_kind == "received" else "sender", + **{"agent.name": "Worker Display Name" if message_kind == "received" else "Sender"}, + ) + for order in ((message,), *itertools.permutations((message, unrelated))): + graph = replay((record, "event") for record in order) + state = next( + n + for n in nodes(graph, "State") + if n["content"] == ("request" if message_kind == "received" else "response") + ) + assert ("hasState", "s1", state["id"]) in graph.edges + assert not any( + edge == "hasState" and target == state["id"] and source != "s1" + for edge, source, target in graph.edges + ) + assert process_record(graph, message, "event") == StreamDelta() + graph = stored_copy(graph) + process_record(graph, owner, "event") + call = next(n for n in nodes(graph, "AgentCall") if n["spanId"] == owner["SpanId"]) + assert ("hasState", call["id"], state["id"]) in graph.edges + assert ("hasState", "s1", state["id"]) not in graph.edges + assert process_record(graph, message, "event") == StreamDelta() + + +@pytest.mark.parametrize("call_kind", ["llm", "tool"]) +def test_completion_before_start_preserves_input_and_corrects_timing(call_kind): + attrs = { + "agent.name": "Planner", + "ioa_observe.agent.span_id": "agent", + "llm.name": "test/model", + "tool.name": "search", + } + records = [ + (event("topology.node.started", "agent", **{"agent.name": "Planner"}), "event"), + ( + event( + f"{call_kind}.completed", + "child", + time="2026-11-10T00:00:02+00:00", + **{**attrs, f"{call_kind}.output": "answer"}, + ), + "event", + ), + ( + event( + f"{call_kind}.started", + "child", + **{**attrs, f"{call_kind}.input": "question"}, + ), + "event", + ), + ] + graph = replay(records) + call = nodes(graph, "LLMCall" if call_kind == "llm" else "ToolCall")[0] + assert call["duration"] == 1000 + states = { + e["edge_type"]: graph.nodes[("State", e["to_id"])]["content"] + for e in graph.edges.values() + if e["from_id"] == call["id"] and e["edge_type"] in ("hasInitialState", "hasFinalState") + } + assert states == {"hasInitialState": "question", "hasFinalState": "answer"} + + +def test_session_envelope_is_corrected_when_parent_span_arrives_last(): + graph = replay( + [ + ( + event("topology.node.started", "agent", **{"agent.name": "Planner"}), + "event", + ), + ( + span( + "planner.agent", + "agent", + **{ + "agent_id": "planner", + "ioa_observe.entity.name": "Planner", + }, + ), + "span", + ), + ] + ) + call = nodes(graph, "AgentCall")[0] + for kind in ("MASCall", "Session"): + container = nodes(graph, kind)[0] + assert container["startTime"] <= call["startTime"] + assert container["endTime"] >= call["endTime"] + + +def test_mixed_records_converge_and_redelivery_is_a_noop(): + records = [ + (event("topology.session.started"), "event"), + (event("topology.node.started", "agent", **{"agent.name": "Planner"}), "event"), + ( + event( + "llm.started", + "chat", + **{ + "agent.name": "Planner", + "ioa_observe.agent.span_id": "agent", + "llm.name": "test/model", + "llm.input": "question", + }, + ), + "event", + ), + ( + event( + "llm.completed", + "chat", + time="2026-11-10T00:00:02+00:00", + **{ + "agent.name": "Planner", + "ioa_observe.agent.span_id": "agent", + "llm.name": "test/model", + "llm.output": "answer", + }, + ), + "event", + ), + ( + event( + "mcp.message.received", + "server", + service="tool-server", + **{ + "source.agent": "Planner", + "source.agent.span.id": "agent", + "target.agent": "debug", + "message.id": "request-1", + "message.input": '{"name":"search","arguments":{}}', + }, + ), + "event", + ), + ( + event( + "tool.started", + "tool", + **{ + "tool.name": "search", + "tool.call.id": "request-1", + "ioa_observe.agent.span_id": "agent", + "tool.input": "query", + }, + ), + "event", + ), + ( + span( + "planner.agent", + "agent", + **{ + "agent_id": "planner", + "ioa_observe.entity.name": "Planner", + "ioa_observe.entity.input": "question", + "ioa_observe.entity.output": "answer", + }, + ), + "span", + ), + ] + expected = replay(records) + for seed in range(10): + order = random.Random(seed).sample(records, len(records)) + graph = replay(order) + assert graph.nodes == expected.nodes + assert graph.edges == expected.edges + for record, kind in records: + assert process_record(graph, record, kind) == StreamDelta() + + +def test_top_level_agent_id_does_not_duplicate_the_agent_named_by_events(): + completed = span("planner.agent", "agent", **{"ioa_observe.entity.name": "Planner"}) + completed["AgentId"] = "planner-runtime-id" + graph = replay( + [ + ( + event("topology.node.started", "agent", **{"agent.name": "Planner"}), + "event", + ), + (completed, "span"), + ] + ) + assert len(nodes(graph, "Agent")) == len(nodes(graph, "AgentCall")) == 1 + + +def test_presented_tools_enrich_existing_tool_definitions(): + graph = replay( + [ + (event("tool.started", "tool", **{"tool.name": "search"}), "event"), + ( + event( + "llm.tools.presented", + "agent", + **{ + "capability.content": json.dumps( + { + "tools": [ + { + "type": "function", + "function": { + "name": "search", + "description": "Search documents", + }, + } + ] + } + ), + }, + ), + "event", + ), + ] + ) + assert nodes(graph, "Tool")[0]["description"] == "Search documents" + + +def test_presented_tool_ownership_is_backfilled_when_agent_arrives_later(): + graph = replay( + [ + ( + event( + "llm.tools.presented", + "agent", + **{ + "capability.content": '{"tools":[{"function":{"name":"search"}}]}', + }, + ), + "event", + ), + ( + event("topology.node.started", "agent", **{"agent.name": "Planner"}), + "event", + ), + ] + ) + agent = nodes(graph, "Agent")[0] + assert ("usesTool", agent["id"], "search") in graph.edges + + +@pytest.mark.parametrize("event_first", [False, True]) +@pytest.mark.parametrize("call_kind,native_id", [("llm", True), ("tool", True), ("tool", False)]) +def test_completed_capability_spans_enrich_events_without_duplicate_calls( + event_first, + call_kind, + native_id, +): + started = event( + f"{call_kind}.started", + "child", + **{ + "agent.name": "Planner", + "ioa_observe.agent.span_id": "agent", + "llm.name": "test/model", + "tool.name": "search", + "tool.call.id": "request-1", + }, + ) + completed = span( + "model.chat" if call_kind == "llm" else "search.tool", + "child", + **{ + "agent_id": "Planner", + "ioa_observe.agent.span_id": "agent", + "gen_ai.provider.name": "test", + "gen_ai.request.model": "model", + "gen_ai.usage.input_tokens": "42", + "gen_ai.tool.call.id": "request-1", + "ioa_observe.entity.name": "search", + "ioa_observe.entity.output": "answer", + "gen_ai.output.messages": "answer", + }, + ) + if not native_id: + completed["SpanAttributes"].pop("gen_ai.tool.call.id") + records = [(started, "event"), (completed, "span")] + graph = replay(records if event_first else reversed(records)) + calls = nodes(graph, "LLMCall" if call_kind == "llm" else "ToolCall") + assert len(calls) == 1 + assert calls[0]["duration"] == 2000 + if call_kind == "llm": + assert calls[0]["promptTokenCount"] == 42 + else: + assert calls[0]["id"] == "request-1" + assert process_record(graph, completed, "span") == StreamDelta() diff --git a/backend/workers/rt-norm-worker/README.md b/backend/workers/rt-norm-worker/README.md index 84d3d0f..e36164a 100644 --- a/backend/workers/rt-norm-worker/README.md +++ b/backend/workers/rt-norm-worker/README.md @@ -3,11 +3,60 @@ Standalone worker (no `worker-base`) that normalizes OTel **spans** and **logs**, one record at a time, into the Neo4j knowledge graph, and calls back the URL of every rule a log record matches. -Nothing is kept between records: each is normalized against the graph as stored. What each log -event creates is in [`backend/norm/docs/events.md`](../../norm/docs/events.md). +Each record is normalized against the stored graph and compact trace context held in process +memory. The shared handlers are described in +[`backend/norm/docs/events.md`](../../norm/docs/events.md). Identity, lifecycle and ownership +reconciliation live in shared `norm`; the worker provides ingestion, persistence and callbacks. All commands run from `backend/`. +## Reconciliation + +- Child LLM/tool calls retain their owning agent span id. Ownership and capability-use edges + are backfilled when the agent arrives, including when its span finishes after its children. + Parent chains can cross transport/framework spans, including sessionless MCP server spans. + Private in-memory trace context preserves these links across arrival orders within the + lifetime of one worker store, but does not survive restarts. +- Lifecycle events create provisional calls. Completed spans enrich those same calls with + authoritative timing, content and metrics; an event sharing the span id does not suppress + the span. Late started events also correct event-only calls' input and start time. +- A propagated `_` identifies the originating MAS. A different + participating service name (for example an MCP server) does not create another MAS. +- Graph metadata is retained on the MAS, but only graph nodes matched to observed agents are + declared as Agents. Routers and tool-serving graph nodes are not assumed to be agents. +- Topology handoff hints are retained until both agent calls exist. They connect execution + trajectories, not a second `executesAgent` target on the sender. MCP/tool edges do not + manufacture recipient Agents. +- `llm.tools.presented` retains offered tools and links them to the owning agent when known. + A2A incoming messages and completion replies are retained as message States; they do not + replace an agent's own final State. + +Compact reconciliation hints and processed event ids are persisted as `rtNormalization` +properties on the relevant graph nodes. No raw telemetry replay buffer is stored. A new +worker instance can continue reconciliation using these graph hints, but must rebuild any +required transport ancestry by replaying the relevant spans/events. Redelivered +events/completed spans are idempotent once context has been rebuilt. Both Neo4j and the +dry-run store accept earlier start-time corrections. + +These guarantees also apply to direct users of `norm.StreamNormalizer`. Batch span +normalization uses the same identity, ownership, graph-role and trajectory reconciliation. +This is not a migration or cleanup of previously misclassified graph data. Use a fresh graph +for an end-to-end comparison; the worker does not delete unrelated existing MAS/Agent nodes. +Ontology/SHACL checks are diagnostic and are not bypassed or changed by this worker. + +Private ancestry is cached in memory by trace and span id on the `Neo4jStore` instance. +The CLI and HTTP receiver reuse that instance across records/requests; a fresh store starts +with empty context, including a separate file-ingestion command. No `_NormalizationTrace` +nodes are read or written. Existing nodes created by older worker versions are left untouched; +exclude them from views until explicitly cleaned up. + +Restarting the worker loses this context, and multiple workers do not share it. Route records +for a trace to the same worker/store. To repair missing ownership after a restart, replay the +relevant spans and MCP events into that same instance. Already persisted KG relationships are +not removed merely because the cache is empty. The cache currently has no eviction or expiry, +so its memory usage grows with the number of spans processed during the store's lifetime. +No raw telemetry payloads are cached, and older duplicate MAS/Agent identities are not removed. + ## Setup Neo4j must be running. The worker reads its connection from environment variables (there is no diff --git a/backend/workers/rt-norm-worker/src/rt_norm_worker/store.py b/backend/workers/rt-norm-worker/src/rt_norm_worker/store.py index dcde9a6..e4ec8e5 100644 --- a/backend/workers/rt-norm-worker/src/rt_norm_worker/store.py +++ b/backend/workers/rt-norm-worker/src/rt_norm_worker/store.py @@ -3,20 +3,17 @@ """Writes one span's KG delta straight to Neo4j, updating nodes in place.""" -import logging import json +import logging from collections.abc import Sequence from typing import Any -from norm import StreamDelta +from norm import StreamDelta, SpanContext logger = logging.getLogger(__name__) # Node types merged on `name` (unique constraint) instead of `id`; mirrors scope.client.dal. NAME_KEYED_NODE_TYPES = frozenset({"LLM"}) -# Set when a node is first created and never changed afterwards (endTime is not: it advances -# as a container, e.g. an AgentCall or Session, completes). -IMMUTABLE_PROPS = frozenset({"startTime"}) _NON_PROPS = frozenset({"node_type"}) _EDGE_NON_PROPS = frozenset({"edge_type", "from_id", "from_type", "to_id", "to_type"}) @@ -49,10 +46,11 @@ def _node(label: str, props: dict[str, Any]) -> dict[str, Any]: class Neo4jStore: - """Neo4j implementation of norm's ``GraphReader``, plus writer for its deltas.""" + """Neo4j graph reader/writer with store-local, in-memory trace context.""" def __init__(self, connector: Any): self._db = connector + self._span_contexts: dict[str, dict[str, SpanContext]] = {} def load( self, session_id: str, node_ids: Sequence[str] @@ -61,7 +59,7 @@ def load( params = {"sid": session_id, "ids": list(node_ids)} nodes: dict[tuple[str, str], dict[str, Any]] = {} for row in self._db.execute( - "MATCH (n) WHERE n.sessionId = $sid OR n.id IN $ids " + "MATCH (n) WHERE ($sid <> '' AND n.sessionId = $sid) OR n.id IN $ids " "OR (n:Agent AND n.name IN $ids) " "RETURN labels(n)[0] AS label, properties(n) AS props", params, @@ -72,16 +70,28 @@ def load( edges: list[dict[str, Any]] = [] for row in self._db.execute( "MATCH (a)-[r]->(b) " - "WHERE a.sessionId = $sid OR b.sessionId = $sid " + "WHERE ($sid <> '' AND (a.sessionId = $sid OR b.sessionId = $sid)) " "OR (a:Agent AND a.name IN $ids) OR (b:Agent AND b.name IN $ids) " "OR (a.id IN $ids AND b.id IN $ids) " - "RETURN type(r) AS edge_type, a.id AS from_id, b.id AS to_id, " + "RETURN type(r) AS edge_type, properties(r) AS props, a.id AS from_id, b.id AS to_id, " "labels(a)[0] AS from_label, properties(a) AS from_props, " "labels(b)[0] AS to_label, properties(b) AS to_props", params, ): - edges.append({"edge_type": row["edge_type"], "from_id": row["from_id"], "to_id": row["to_id"]}) - for label, props in ((row["from_label"], row["from_props"]), (row["to_label"], row["to_props"])): + edges.append( + { + **{k: _decode(v) for k, v in row.get("props", {}).items()}, + "edge_type": row["edge_type"], + "from_id": row["from_id"], + "to_id": row["to_id"], + "from_type": row["from_label"], + "to_type": row["to_label"], + } + ) + for label, props in ( + (row["from_label"], row["from_props"]), + (row["to_label"], row["to_props"]), + ): node = _node(label, props) nodes.setdefault((node["node_type"], node["id"]), node) return list(nodes.values()), edges @@ -96,16 +106,19 @@ def apply(self, delta: StreamDelta) -> None: self._upsert_node(node) for edge in delta.edges: self._upsert_edge(edge) + for context in delta.span_contexts: + self._span_contexts.setdefault(context.trace_id, {})[context.span_id] = context + + def load_span_contexts(self, trace_id: str) -> list[SpanContext]: + return list(self._span_contexts.get(trace_id, {}).values()) def _upsert_node(self, node: dict[str, Any]) -> None: node_type = _safe(str(node["node_type"])) props = _props(node, _NON_PROPS) - on_match = {k: v for k, v in props.items() if k not in IMMUTABLE_PROPS} key = "name" if node_type in NAME_KEYED_NODE_TYPES else "id" self._db.execute_command( - f"MERGE (n:{node_type} {{{key}: $key}}) " - "ON CREATE SET n += $props ON MATCH SET n += $on_match", - {"key": props[key], "props": props, "on_match": on_match}, + f"MERGE (n:{node_type} {{{key}: $key}}) SET n += $props", + {"key": props[key], "props": props}, ) def _upsert_edge(self, edge: dict[str, Any]) -> None: diff --git a/backend/workers/rt-norm-worker/src/rt_norm_worker/worker.py b/backend/workers/rt-norm-worker/src/rt_norm_worker/worker.py index c221df8..c1af343 100644 --- a/backend/workers/rt-norm-worker/src/rt_norm_worker/worker.py +++ b/backend/workers/rt-norm-worker/src/rt_norm_worker/worker.py @@ -24,7 +24,7 @@ def read_records(path: Path = DEFAULT_TRACES_PATH) -> Iterator[dict[str, Any]]: class Graph(GraphReader, Protocol): - """Where the KG lives: read what a span needs, write back what changed.""" + """Read the relevant graph and apply deltas, including corrected earlier timestamps.""" def apply(self, delta: Any) -> None: ... diff --git a/backend/workers/rt-norm-worker/tests/test_server.py b/backend/workers/rt-norm-worker/tests/test_server.py index 608e5c1..6c94265 100644 --- a/backend/workers/rt-norm-worker/tests/test_server.py +++ b/backend/workers/rt-norm-worker/tests/test_server.py @@ -6,6 +6,7 @@ import threading import urllib.error import urllib.request +from copy import deepcopy from norm import InMemoryGraph @@ -22,7 +23,9 @@ def _attrs(**kwargs: str) -> list[dict]: LOGS = { "resourceLogs": [ { - "resource": {"attributes": _attrs(**{"service.name": "noa-trip-planner-mas"})}, + "resource": { + "attributes": _attrs(**{"service.name": "noa-trip-planner-mas"}) + }, "scopeLogs": [ { "scope": {"name": "ioa_observe.runtime_events"}, @@ -84,13 +87,24 @@ def test_otlp_flattening_matches_the_collector_export_rows() -> None: assert span["SpanAttributes"] == {"n": "3", "ok": "true"} assert span["Timestamp"].endswith(".000000000") and span["Timestamp"][:4] == "2026" [log] = list(otlp.logs(LOGS)) - assert log["Body"] == "topology.session.started" and log["ServiceName"] == "noa-trip-planner-mas" + assert ( + log["Body"] == "topology.session.started" + and log["ServiceName"] == "noa-trip-planner-mas" + ) assert log["LogAttributes"]["session.id"] == SESSION -def _post(port: int, path: str, body: bytes, content_type: str = "application/json", **headers: str): +def _post( + port: int, + path: str, + body: bytes, + content_type: str = "application/json", + **headers: str, +): request = urllib.request.Request( - f"http://127.0.0.1:{port}{path}", data=body, headers={"Content-Type": content_type, **headers} + f"http://127.0.0.1:{port}{path}", + data=body, + headers={"Content-Type": content_type, **headers}, ) try: with urllib.request.urlopen(request) as response: @@ -107,13 +121,83 @@ def test_server_normalizes_logs_pushed_by_the_collector() -> None: port = server.server_address[1] try: assert _post(port, "/v1/logs", json.dumps(LOGS).encode()) == 200 - assert ("Session", "3aefd85c") in graph.nodes and ("MAS", "noa-trip-planner-mas") in graph.nodes + assert ("Session", "3aefd85c") in graph.nodes and ( + "MAS", + "noa-trip-planner-mas", + ) in graph.nodes # gzip, as the collector sends by default - assert _post(port, "/v1/logs", gzip.compress(json.dumps(LOGS).encode()), **{"Content-Encoding": "gzip"}) == 200 + assert ( + _post( + port, + "/v1/logs", + gzip.compress(json.dumps(LOGS).encode()), + **{"Content-Encoding": "gzip"}, + ) + == 200 + ) assert _post(port, "/v1/traces", json.dumps(TRACES).encode()) == 200 - assert _post(port, "/v1/logs", b"{", ) == 400 - assert _post(port, "/v1/logs", b"\x00", content_type="application/x-protobuf") == 415 + assert ( + _post( + port, + "/v1/logs", + b"{", + ) + == 400 + ) + assert ( + _post(port, "/v1/logs", b"\x00", content_type="application/x-protobuf") + == 415 + ) assert _post(port, "/v1/nope", b"{}") == 404 finally: server.shutdown() server.server_close() + + +def test_server_enriches_event_placeholders_with_completed_spans() -> None: + graph = InMemoryGraph() + logs = deepcopy(LOGS) + record = logs["resourceLogs"][0]["scopeLogs"][0]["logRecords"][0] + record["body"]["stringValue"] = "topology.node.started" + record["attributes"] = _attrs( + **{ + "event.name": "topology.node.started", + "event.time": "2026-11-06T16:00:00+00:00", + "session.id": "svc_s1", + "application_id": "svc", + "agent.name": "Planner", + } + ) + traces = deepcopy(TRACES) + span = traces["resourceSpans"][0]["scopeSpans"][0]["spans"][0] + span["spanId"] = record["spanId"] + span["name"] = "planner.agent" + span["attributes"] = _attrs( + **{ + "session.id": "svc_s1", + "application_id": "svc", + "agent_id": "planner-runtime", + "ioa_observe.entity.name": "Planner", + "ioa_observe.entity.output": "answer", + } + ) + server = make_server(graph, "127.0.0.1", 0) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + port = server.server_address[1] + assert _post(port, "/v1/logs", json.dumps(logs).encode()) == 200 + assert _post(port, "/v1/traces", json.dumps(traces).encode()) == 200 + calls = [n for (t, _), n in graph.nodes.items() if t == "AgentCall"] + assert len(calls) == 1 + assert calls[0]["duration"] == 1000 + final = next( + e["to_id"] + for e in graph.edges.values() + if e["edge_type"] == "hasFinalState" and e["from_id"] == calls[0]["id"] + ) + assert graph.nodes[("State", final)]["content"] == "answer" + finally: + server.shutdown() + server.server_close() + thread.join() diff --git a/backend/workers/rt-norm-worker/tests/test_store.py b/backend/workers/rt-norm-worker/tests/test_store.py new file mode 100644 index 0000000..700967e --- /dev/null +++ b/backend/workers/rt-norm-worker/tests/test_store.py @@ -0,0 +1,170 @@ +# Copyright (c) 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 + +from norm import InMemoryGraph, StreamDelta, SpanContext, StreamNormalizer + +from rt_norm_worker.store import Neo4jStore + + +class PopulatedDB: + def __init__(self): + self.commands = [] + self.reads = [] + + def execute(self, query, params): + self.reads.append((query, params)) + if "RETURN labels(n)" in query: + return [ + { + "label": "AgentCall", + "props": { + "id": "call", + "sessionId": "s1", + "rtNormalization": '__json__:{"spanComplete":true}', + }, + } + ] + return [ + { + "edge_type": "executesAgent", + "props": {"evidence": '__json__:{"source":"span"}'}, + "from_id": "call", + "from_label": "AgentCall", + "from_props": {"id": "call", "sessionId": "s1"}, + "to_id": "planner", + "to_label": "Agent", + "to_props": {"id": "planner", "name": "Planner"}, + } + ] + + def execute_command(self, command, params): + self.commands.append((command, params)) + + +def test_load_decodes_reconciliation_metadata_and_loads_edge_endpoints(): + db = PopulatedDB() + nodes, edges = Neo4jStore(db).load("s1", ["planner"]) + by_id = {n["id"]: n for n in nodes} + assert by_id["call"]["rtNormalization"] == {"spanComplete": True} + assert by_id["planner"] == { + "node_type": "Agent", + "id": "planner", + "name": "Planner", + } + assert edges == [ + { + "edge_type": "executesAgent", + "from_id": "call", + "to_id": "planner", + "from_type": "AgentCall", + "to_type": "Agent", + "evidence": {"source": "span"}, + } + ] + assert db.reads[0][1] == {"sid": "s1", "ids": ["planner"]} + assert "properties(r) AS props" in db.reads[1][0] + + +def test_apply_removes_superseded_edges_and_nodes_before_writing_replacements(): + db = PopulatedDB() + edge = { + "edge_type": "hasInitialState", + "from_id": "call", + "to_id": "old", + "from_type": "AgentCall", + "to_type": "State", + } + Neo4jStore(db).apply( + StreamDelta( + removed_edges=[edge], + removed_nodes=[{"node_type": "State", "id": "old"}], + nodes=[{"node_type": "State", "id": "new", "content": "question"}], + edges=[{**edge, "to_id": "new"}], + ) + ) + queries = [q for q, _ in db.commands] + assert queries[0].endswith("DELETE r") + assert queries[1].endswith("DETACH DELETE n") + assert queries[2].startswith("MERGE (n:State") + assert queries[3].startswith("MERGE (a:AgentCall") + assert db.commands[0][1] == {"from_id": "call", "to_id": "old"} + assert db.commands[1][1] == {"id": "old"} + assert db.commands[3][1]["to_id"] == "new" + + +def test_memory_store_applies_earlier_start_time_corrections(): + graph = InMemoryGraph() + node = {"node_type": "Session", "id": "s1", "startTime": 20.0, "endTime": 30.0} + graph.apply(StreamDelta(nodes=[node])) + graph.apply(StreamDelta(nodes=[{**node, "startTime": 10.0, "duration": 20000.0}])) + assert graph.nodes[("Session", "s1")]["startTime"] == 10.0 + + +def test_private_trace_context_stays_in_memory_and_is_scoped_to_store_lifetime(): + db = PopulatedDB() + store = Neo4jStore(db) + sessionless = SpanContext("trace", "server", "client", "", "") + client = SpanContext("trace", "client", "", "agent", "session") + other_trace = SpanContext("other", "server", "", "other-agent", "other-session") + store.apply(StreamDelta(span_contexts=[sessionless])) + store.apply(StreamDelta(span_contexts=[client, other_trace])) + assert set(store.load_span_contexts("trace")) == {sessionless, client} + assert store.load_span_contexts("other") == [other_trace] + assert store.load_span_contexts("different") == [] + store.apply(StreamDelta(span_contexts=[sessionless])) + assert set(store.load_span_contexts("trace")) == {sessionless, client} + enriched = SpanContext("trace", "server", "client", "agent", "session") + store.apply(StreamDelta(span_contexts=[enriched])) + assert set(store.load_span_contexts("trace")) == {enriched, client} + assert store.load_span_contexts("other") == [other_trace] + assert Neo4jStore(db).load_span_contexts("trace") == [] + assert db.commands == [] + assert db.reads == [] + + +def test_late_sessionless_transport_backfills_ownership_using_store_memory(monkeypatch): + db = PopulatedDB() + store = Neo4jStore(db) + graph = InMemoryGraph() + monkeypatch.setattr(store, "load", graph.load) + common = { + "TraceId": "trace", + "Timestamp": "2026-11-10 00:00:00.000000000", + "Duration": 1_000_000_000, + "SpanAttributes": {"application_id": "app", "session.id": "app_s1"}, + } + records = [ + { + **common, + "SpanName": "planner.agent", + "SpanId": "agent", + "SpanAttributes": {**common["SpanAttributes"], "agent_id": "planner"}, + }, + { + **common, + "SpanName": "search.tool", + "SpanId": "tool", + "ParentSpanId": "server", + "SpanAttributes": {**common["SpanAttributes"], "ioa_observe.entity.name": "search"}, + }, + { + **common, + "SpanName": "tools/call search", + "SpanId": "server", + "ParentSpanId": "agent", + "SpanAttributes": {}, + }, + ] + for record in records: + delta = StreamNormalizer(store).process(record, "span") + store.apply(delta) + graph.apply(delta) + graph.span_contexts.clear() + owner = next(n for (kind, _), n in graph.nodes.items() if kind == "AgentCall") + call = next(n for (kind, _), n in graph.nodes.items() if kind == "ToolCall") + assert ("hasToolCall", owner["id"], call["id"]) in graph.edges + assert store.load_span_contexts("trace") + assert db.reads == [] + assert not any("_NormalizationTrace" in query for query, _ in db.commands) + for record in records: + assert StreamNormalizer(store).process(record, "span") == StreamDelta() diff --git a/backend/workers/rt-norm-worker/tests/test_worker.py b/backend/workers/rt-norm-worker/tests/test_worker.py index 64b8733..4f8aae2 100644 --- a/backend/workers/rt-norm-worker/tests/test_worker.py +++ b/backend/workers/rt-norm-worker/tests/test_worker.py @@ -51,13 +51,13 @@ def test_neo4j_store_writes_delta(tmp_path): assert any("MERGE (a)-[r:" in c for c, _ in db.commands) -def test_existing_nodes_keep_start_time_but_update_end_time(): +def test_existing_nodes_accept_corrected_start_and_end_time(): db = FakeDB() Neo4jStore(db)._upsert_node( {"node_type": "AgentCall", "id": "x", "startTime": 1.0, "endTime": 2.0, "name": "n"} ) _, params = db.commands[0] assert params["props"]["startTime"] == 1.0 - assert "startTime" not in params["on_match"] - assert params["on_match"]["endTime"] == 2.0 - assert params["on_match"]["name"] == "n" + assert params["props"]["endTime"] == 2.0 + assert params["props"]["name"] == "n" + assert "SET n += $props" in db.commands[0][0]