diff --git a/README.md b/README.md index e43fc43..d089dff 100644 --- a/README.md +++ b/README.md @@ -11,16 +11,18 @@ Latest milestone: [v0.5.0 — third demo and three-demo structure](https://githu - [telemetry-window-demo](#telemetry-window-demo) - [ai-assisted-detection-demo](demos/ai-assisted-detection-demo/README.md) - [rule-evaluation-and-dedup-demo](demos/rule-evaluation-and-dedup-demo/README.md) +- [config-change-investigation-demo](demos/config-change-investigation-demo/README.md) | Demo | Input | Deterministic core | LLM role | Main artifacts | Guardrails / non-goals | | --- | --- | --- | --- | --- | --- | | [telemetry-window-demo](#telemetry-window-demo) | JSONL / CSV events | Windows
Features
Alert thresholds | None | `features.csv`
`alerts.csv`
`summary.json`
3 PNG plots | MVP only
No realtime
No case management | | [ai-assisted-detection-demo](demos/ai-assisted-detection-demo/README.md) | JSONL auth / web / process | Normalize
Rules
Grouping
ATT&CK mapping | JSON-only case drafting | `rule_hits.json`
`case_bundles.json`
`case_summaries.json`
`case_report.md`
`audit_traces.jsonl` | Human verification required
No autonomous response
No final verdict | | [rule-evaluation-and-dedup-demo](demos/rule-evaluation-and-dedup-demo/README.md) | JSON raw rule hits | Scope resolution
Cooldown grouping
Suppression reasoning | None | `rule_hits_before_dedup.json`
`rule_hits_after_dedup.json`
`dedup_explanations.json`
`dedup_report.md` | No realtime
No dashboard
No AI stage | +| [config-change-investigation-demo](demos/config-change-investigation-demo/README.md) | JSONL config changes
Policy denials
Follow-on events | Normalize
Risky-change rules
Bounded correlation | None | `change_events_normalized.json`
`investigation_hits.json`
`investigation_summary.json`
`investigation_report.md` | No realtime
No dashboard
No AI stage | ## What This Repo Is -`telemetry-lab` is a small portfolio repository for telemetry analytics and constrained detection-oriented workflows. It is organized as three local, file-based demos that are reproducible from committed sample data and intentionally scoped for public review rather than production use. +`telemetry-lab` is a small portfolio repository for telemetry analytics and constrained detection-oriented workflows. It is organized as four local, file-based demos that are reproducible from committed sample data and intentionally scoped for public review rather than production use. ### telemetry-window-demo @@ -32,7 +34,11 @@ Latest milestone: [v0.5.0 — third demo and three-demo structure](https://githu ### rule-evaluation-and-dedup-demo -`rule-evaluation-and-dedup-demo` starts from raw rule hits and makes cooldown behavior legible. It shows which hits were kept, which were suppressed, how scope was resolved, and why repeated hits collapsed into fewer retained alerts. +`rule-evaluation-and-dedup-demo` starts from raw rule hits and makes cooldown behavior legible. It shows which hits were kept, which were suppressed, how scope was resolved, and why repeated hits collapsed into fewer retained alerts. + +### config-change-investigation-demo + +`config-change-investigation-demo` follows risky configuration changes into bounded follow-on evidence such as policy denials and service signals. It stays deterministic, file-based, and review-oriented, with no added AI stage. ## Quick Run @@ -44,7 +50,8 @@ python -m telemetry_window_demo.cli run --config configs/default.yaml Other demo entrypoints: - `python -m telemetry_window_demo.cli run-ai-demo` -- `python -m telemetry_window_demo.cli run-rule-dedup-demo` +- `python -m telemetry_window_demo.cli run-rule-dedup-demo` +- `python -m telemetry_window_demo.cli run-config-change-demo` That command reads `data/raw/sample_events.jsonl` and regenerates: @@ -109,6 +116,7 @@ Cooldown behavior: ## Repo Guide - [`demos/rule-evaluation-and-dedup-demo/README.md`](demos/rule-evaluation-and-dedup-demo/README.md) explains the third demo and links its committed before/after dedup artifacts +- [`demos/config-change-investigation-demo/README.md`](demos/config-change-investigation-demo/README.md) explains the config-change investigation demo and its committed artifacts - [`docs/sample-output.md`](docs/sample-output.md) summarizes the committed sample artifacts - [`docs/roadmap.md`](docs/roadmap.md) sketches the next demo directions - [`data/processed/summary.json`](data/processed/summary.json) captures the default run in machine-readable form diff --git a/demos/config-change-investigation-demo/README.md b/demos/config-change-investigation-demo/README.md new file mode 100644 index 0000000..e1c9bc6 --- /dev/null +++ b/demos/config-change-investigation-demo/README.md @@ -0,0 +1,78 @@ +# Config-Change Investigation Demo + +This demo is part of `telemetry-lab` and stays intentionally small, local, and reviewer-friendly. + +It focuses on deterministic investigation logic for risky configuration changes and nearby evidence. There is no new AI stage in this demo. + +## Purpose + +The goal is to make one compact config-change investigation path legible from committed sample data. + +The demo starts from configuration changes, policy denials, and follow-on telemetry, then: + +- normalizes the inputs into shared internal records +- applies deterministic risky-change rules +- attaches nearby supporting evidence using bounded time and shared-system correlation +- writes machine-readable summaries and a short reviewer-facing report + +## Quick Start + +From the repository root: + +```bash +python -m pip install -e . +python -m telemetry_window_demo.cli run-config-change-demo +``` + +Generated artifacts are written to `demos/config-change-investigation-demo/artifacts/`. + +## Demo Inputs + +- config changes: `data/raw/config_changes.jsonl` +- policy denials: `data/raw/policy_denials.jsonl` +- follow-on events: `data/raw/follow_on_events.jsonl` +- investigation config: `config/investigation.yaml` + +The bundled sample includes: + +- one risky MFA-related change with nearby denials and follow-on signals +- one risky public-bind change with nearby denials and service events +- one benign config change that should not trigger an investigation +- one risky break-glass change with no nearby supporting evidence inside the bounded window + +## Deterministic Correlation + +This demo uses a bounded correlation window after each triggering config change. + +Evidence is attached only when: + +1. `target_system` matches the triggering change +2. the evidence timestamp falls within the configured correlation window after the change + +## Expected Artifacts + +- `artifacts/change_events_normalized.json` +- `artifacts/investigation_hits.json` +- `artifacts/investigation_summary.json` +- `artifacts/investigation_report.md` + +## Artifact Semantics + +- `change_events_normalized.json`: normalized config changes before any rule match is applied +- `investigation_hits.json`: full investigation records, including the triggering change and attached evidence +- `investigation_summary.json`: reduced machine-readable summaries for each investigation +- `investigation_report.md`: a short reviewer report showing the trigger, evidence counts, and bounded-correlation explanation + +## Reviewer Walkthrough + +1. Open `change_events_normalized.json` and identify the risky config keys and values. +2. Open `investigation_hits.json` and verify which changes became investigations and which evidence records were attached. +3. Open `investigation_summary.json` and confirm the final summaries stay deterministic and bounded. +4. Open `investigation_report.md` and verify that a risky change with no nearby evidence remains explicit rather than silently discarded. + +## Limitations + +- synthetic sample data only +- no realtime ingestion or service deployment +- bounded correlation by system and time only +- no model-generated reasoning or autonomous response diff --git a/demos/config-change-investigation-demo/artifacts/change_events_normalized.json b/demos/config-change-investigation-demo/artifacts/change_events_normalized.json new file mode 100644 index 0000000..71e7e67 --- /dev/null +++ b/demos/config-change-investigation-demo/artifacts/change_events_normalized.json @@ -0,0 +1,46 @@ +[ + { + "change_id": "cfg-001", + "timestamp": "2026-03-22T09:00:00Z", + "actor": "ops-admin", + "target_system": "identity-proxy", + "config_key": "disable_admin_mfa", + "old_value": "false", + "new_value": "true", + "change_result": "success", + "change_ticket": "CHG-1001" + }, + { + "change_id": "cfg-002", + "timestamp": "2026-03-22T09:20:00Z", + "actor": "deploy-bot", + "target_system": "payments-api", + "config_key": "public_bind_cidr", + "old_value": "10.20.0.0/24", + "new_value": "0.0.0.0/0", + "change_result": "success", + "change_ticket": "CHG-1002" + }, + { + "change_id": "cfg-003", + "timestamp": "2026-03-22T09:40:00Z", + "actor": "deploy-bot", + "target_system": "payments-api", + "config_key": "log_level", + "old_value": "info", + "new_value": "debug", + "change_result": "success", + "change_ticket": "CHG-1003" + }, + { + "change_id": "cfg-004", + "timestamp": "2026-03-22T10:00:00Z", + "actor": "sre-admin", + "target_system": "vault-gateway", + "config_key": "break_glass_mode", + "old_value": "disabled", + "new_value": "enabled", + "change_result": "success", + "change_ticket": "CHG-1004" + } +] diff --git a/demos/config-change-investigation-demo/artifacts/investigation_hits.json b/demos/config-change-investigation-demo/artifacts/investigation_hits.json new file mode 100644 index 0000000..be38f8e --- /dev/null +++ b/demos/config-change-investigation-demo/artifacts/investigation_hits.json @@ -0,0 +1,142 @@ +[ + { + "investigation_id": "CCI-001", + "severity": "critical", + "rule_id": "cfg_disable_admin_mfa", + "target_system": "identity-proxy", + "actor": "ops-admin", + "triggering_change": { + "change_id": "cfg-001", + "timestamp": "2026-03-22T09:00:00Z", + "actor": "ops-admin", + "target_system": "identity-proxy", + "config_key": "disable_admin_mfa", + "old_value": "false", + "new_value": "true", + "change_result": "success", + "change_ticket": "CHG-1001" + }, + "trigger_reason": "Admin MFA was disabled on a protected system.", + "correlation_window_minutes": 15, + "bounded_correlation_reason": "Attached evidence shares target_system 'identity-proxy' and falls within 15 minutes after the triggering change.", + "attached_policy_denials": [ + { + "denial_id": "den-001", + "timestamp": "2026-03-22T09:04:00Z", + "actor": "ops-admin", + "target_system": "identity-proxy", + "policy_name": "admin-login-guard", + "decision": "denied", + "reason": "MFA policy blocked admin login after configuration drift." + }, + { + "denial_id": "den-002", + "timestamp": "2026-03-22T09:09:00Z", + "actor": "service-account", + "target_system": "identity-proxy", + "policy_name": "token-exchange-guard", + "decision": "denied", + "reason": "Token exchange blocked after admin-auth policy divergence." + } + ], + "attached_follow_on_events": [ + { + "event_id": "fo-001", + "timestamp": "2026-03-22T09:05:00Z", + "target_system": "identity-proxy", + "event_type": "auth_fail_burst", + "details": "5 privileged login failures from 203.0.113.24 after the config change." + }, + { + "event_id": "fo-002", + "timestamp": "2026-03-22T09:11:00Z", + "target_system": "identity-proxy", + "event_type": "service_restart", + "details": "identity-proxy restarted after an auth-policy reload." + } + ], + "evidence_counts": { + "policy_denials": 2, + "follow_on_events": 2 + } + }, + { + "investigation_id": "CCI-002", + "severity": "high", + "rule_id": "cfg_public_bind_cidr", + "target_system": "payments-api", + "actor": "deploy-bot", + "triggering_change": { + "change_id": "cfg-002", + "timestamp": "2026-03-22T09:20:00Z", + "actor": "deploy-bot", + "target_system": "payments-api", + "config_key": "public_bind_cidr", + "old_value": "10.20.0.0/24", + "new_value": "0.0.0.0/0", + "change_result": "success", + "change_ticket": "CHG-1002" + }, + "trigger_reason": "Public bind CIDR was expanded to all addresses.", + "correlation_window_minutes": 15, + "bounded_correlation_reason": "Attached evidence shares target_system 'payments-api' and falls within 15 minutes after the triggering change.", + "attached_policy_denials": [ + { + "denial_id": "den-003", + "timestamp": "2026-03-22T09:23:00Z", + "actor": "deploy-bot", + "target_system": "payments-api", + "policy_name": "public-exposure-guard", + "decision": "denied", + "reason": "Public bind CIDR exceeded the approved network range." + } + ], + "attached_follow_on_events": [ + { + "event_id": "fo-003", + "timestamp": "2026-03-22T09:26:00Z", + "target_system": "payments-api", + "event_type": "service_restart", + "details": "payments-api restarted after listener rebind." + }, + { + "event_id": "fo-004", + "timestamp": "2026-03-22T09:31:00Z", + "target_system": "payments-api", + "event_type": "edge_warning", + "details": "Edge listener observed requests from the newly public CIDR." + } + ], + "evidence_counts": { + "policy_denials": 1, + "follow_on_events": 2 + } + }, + { + "investigation_id": "CCI-003", + "severity": "high", + "rule_id": "cfg_break_glass_mode", + "target_system": "vault-gateway", + "actor": "sre-admin", + "triggering_change": { + "change_id": "cfg-004", + "timestamp": "2026-03-22T10:00:00Z", + "actor": "sre-admin", + "target_system": "vault-gateway", + "config_key": "break_glass_mode", + "old_value": "disabled", + "new_value": "enabled", + "change_result": "success", + "change_ticket": "CHG-1004" + }, + "trigger_reason": "Break-glass mode was enabled on a sensitive service.", + "correlation_window_minutes": 15, + "bounded_correlation_reason": "Attached evidence shares target_system 'vault-gateway' and falls within 15 minutes after the triggering change.", + "attached_policy_denials": [], + "attached_follow_on_events": [], + "evidence_counts": { + "policy_denials": 0, + "follow_on_events": 0 + } + } +] diff --git a/demos/config-change-investigation-demo/artifacts/investigation_report.md b/demos/config-change-investigation-demo/artifacts/investigation_report.md new file mode 100644 index 0000000..4cb021f --- /dev/null +++ b/demos/config-change-investigation-demo/artifacts/investigation_report.md @@ -0,0 +1,58 @@ +# Config-Change Investigation Demo Report + +This deterministic demo correlates risky configuration changes with bounded follow-on evidence. +It does not use an LLM and does not produce autonomous response actions. + +## Run Summary + +- normalized_change_events: 4 +- risky_change_hits: 3 +- investigations: 3 +- correlation_window_minutes: 15 + +## CCI-001 + +- Severity: critical +- Target system: identity-proxy +- Triggering change: cfg-001 (disable_admin_mfa -> true) +- Trigger reason: Admin MFA was disabled on a protected system. +- Attached policy denials: 2 +- Attached follow-on events: 2 +- Bounded correlation: Attached evidence shares target_system 'identity-proxy' and falls within 15 minutes after the triggering change. + +Policy denials: +- den-001: admin-login-guard -> MFA policy blocked admin login after configuration drift. +- den-002: token-exchange-guard -> Token exchange blocked after admin-auth policy divergence. + +Follow-on events: +- fo-001: auth_fail_burst -> 5 privileged login failures from 203.0.113.24 after the config change. +- fo-002: service_restart -> identity-proxy restarted after an auth-policy reload. + +## CCI-002 + +- Severity: high +- Target system: payments-api +- Triggering change: cfg-002 (public_bind_cidr -> 0.0.0.0/0) +- Trigger reason: Public bind CIDR was expanded to all addresses. +- Attached policy denials: 1 +- Attached follow-on events: 2 +- Bounded correlation: Attached evidence shares target_system 'payments-api' and falls within 15 minutes after the triggering change. + +Policy denials: +- den-003: public-exposure-guard -> Public bind CIDR exceeded the approved network range. + +Follow-on events: +- fo-003: service_restart -> payments-api restarted after listener rebind. +- fo-004: edge_warning -> Edge listener observed requests from the newly public CIDR. + +## CCI-003 + +- Severity: high +- Target system: vault-gateway +- Triggering change: cfg-004 (break_glass_mode -> enabled) +- Trigger reason: Break-glass mode was enabled on a sensitive service. +- Attached policy denials: 0 +- Attached follow-on events: 0 +- Bounded correlation: Attached evidence shares target_system 'vault-gateway' and falls within 15 minutes after the triggering change. + +No nearby supporting evidence fell inside the bounded correlation window. diff --git a/demos/config-change-investigation-demo/artifacts/investigation_summary.json b/demos/config-change-investigation-demo/artifacts/investigation_summary.json new file mode 100644 index 0000000..c03968d --- /dev/null +++ b/demos/config-change-investigation-demo/artifacts/investigation_summary.json @@ -0,0 +1,38 @@ +[ + { + "investigation_id": "CCI-001", + "severity": "critical", + "target_system": "identity-proxy", + "triggering_change_id": "cfg-001", + "summary": "disable_admin_mfa changed from false to true on identity-proxy, followed by 2 policy denials and 2 follow-on events within 15 minutes.", + "evidence_counts": { + "policy_denials": 2, + "follow_on_events": 2 + }, + "bounded_correlation_reason": "Attached evidence shares target_system 'identity-proxy' and falls within 15 minutes after the triggering change." + }, + { + "investigation_id": "CCI-002", + "severity": "high", + "target_system": "payments-api", + "triggering_change_id": "cfg-002", + "summary": "public_bind_cidr changed from 10.20.0.0/24 to 0.0.0.0/0 on payments-api, followed by 1 policy denials and 2 follow-on events within 15 minutes.", + "evidence_counts": { + "policy_denials": 1, + "follow_on_events": 2 + }, + "bounded_correlation_reason": "Attached evidence shares target_system 'payments-api' and falls within 15 minutes after the triggering change." + }, + { + "investigation_id": "CCI-003", + "severity": "high", + "target_system": "vault-gateway", + "triggering_change_id": "cfg-004", + "summary": "break_glass_mode changed from disabled to enabled on vault-gateway, followed by 0 policy denials and 0 follow-on events within 15 minutes.", + "evidence_counts": { + "policy_denials": 0, + "follow_on_events": 0 + }, + "bounded_correlation_reason": "Attached evidence shares target_system 'vault-gateway' and falls within 15 minutes after the triggering change." + } +] diff --git a/demos/config-change-investigation-demo/config/investigation.yaml b/demos/config-change-investigation-demo/config/investigation.yaml new file mode 100644 index 0000000..64b1c17 --- /dev/null +++ b/demos/config-change-investigation-demo/config/investigation.yaml @@ -0,0 +1,25 @@ +input_paths: + config_changes: data/raw/config_changes.jsonl + policy_denials: data/raw/policy_denials.jsonl + follow_on_events: data/raw/follow_on_events.jsonl +artifacts_dir: artifacts +correlation_minutes: 15 +rules: + - rule_id: cfg_disable_admin_mfa + config_key: disable_admin_mfa + severity: critical + risky_values: + - "true" + reason: Admin MFA was disabled on a protected system. + - rule_id: cfg_public_bind_cidr + config_key: public_bind_cidr + severity: high + risky_values: + - 0.0.0.0/0 + reason: Public bind CIDR was expanded to all addresses. + - rule_id: cfg_break_glass_mode + config_key: break_glass_mode + severity: high + risky_values: + - enabled + reason: Break-glass mode was enabled on a sensitive service. diff --git a/demos/config-change-investigation-demo/data/raw/config_changes.jsonl b/demos/config-change-investigation-demo/data/raw/config_changes.jsonl new file mode 100644 index 0000000..2c88aee --- /dev/null +++ b/demos/config-change-investigation-demo/data/raw/config_changes.jsonl @@ -0,0 +1,4 @@ +{"change_id":"cfg-001","timestamp":"2026-03-22T09:00:00Z","actor":"ops-admin","target_system":"identity-proxy","config_key":"disable_admin_mfa","old_value":"false","new_value":"true","change_result":"success","change_ticket":"CHG-1001"} +{"change_id":"cfg-002","timestamp":"2026-03-22T09:20:00Z","actor":"deploy-bot","target_system":"payments-api","config_key":"public_bind_cidr","old_value":"10.20.0.0/24","new_value":"0.0.0.0/0","change_result":"success","change_ticket":"CHG-1002"} +{"change_id":"cfg-003","timestamp":"2026-03-22T09:40:00Z","actor":"deploy-bot","target_system":"payments-api","config_key":"log_level","old_value":"info","new_value":"debug","change_result":"success","change_ticket":"CHG-1003"} +{"change_id":"cfg-004","timestamp":"2026-03-22T10:00:00Z","actor":"sre-admin","target_system":"vault-gateway","config_key":"break_glass_mode","old_value":"disabled","new_value":"enabled","change_result":"success","change_ticket":"CHG-1004"} diff --git a/demos/config-change-investigation-demo/data/raw/follow_on_events.jsonl b/demos/config-change-investigation-demo/data/raw/follow_on_events.jsonl new file mode 100644 index 0000000..904a419 --- /dev/null +++ b/demos/config-change-investigation-demo/data/raw/follow_on_events.jsonl @@ -0,0 +1,5 @@ +{"event_id":"fo-001","timestamp":"2026-03-22T09:05:00Z","target_system":"identity-proxy","event_type":"auth_fail_burst","details":"5 privileged login failures from 203.0.113.24 after the config change."} +{"event_id":"fo-002","timestamp":"2026-03-22T09:11:00Z","target_system":"identity-proxy","event_type":"service_restart","details":"identity-proxy restarted after an auth-policy reload."} +{"event_id":"fo-003","timestamp":"2026-03-22T09:26:00Z","target_system":"payments-api","event_type":"service_restart","details":"payments-api restarted after listener rebind."} +{"event_id":"fo-004","timestamp":"2026-03-22T09:31:00Z","target_system":"payments-api","event_type":"edge_warning","details":"Edge listener observed requests from the newly public CIDR."} +{"event_id":"fo-005","timestamp":"2026-03-22T10:30:00Z","target_system":"vault-gateway","event_type":"access_request","details":"A break-glass token was requested outside the bounded correlation window."} diff --git a/demos/config-change-investigation-demo/data/raw/policy_denials.jsonl b/demos/config-change-investigation-demo/data/raw/policy_denials.jsonl new file mode 100644 index 0000000..0955864 --- /dev/null +++ b/demos/config-change-investigation-demo/data/raw/policy_denials.jsonl @@ -0,0 +1,4 @@ +{"denial_id":"den-001","timestamp":"2026-03-22T09:04:00Z","actor":"ops-admin","target_system":"identity-proxy","policy_name":"admin-login-guard","decision":"denied","reason":"MFA policy blocked admin login after configuration drift."} +{"denial_id":"den-002","timestamp":"2026-03-22T09:09:00Z","actor":"service-account","target_system":"identity-proxy","policy_name":"token-exchange-guard","decision":"denied","reason":"Token exchange blocked after admin-auth policy divergence."} +{"denial_id":"den-003","timestamp":"2026-03-22T09:23:00Z","actor":"deploy-bot","target_system":"payments-api","policy_name":"public-exposure-guard","decision":"denied","reason":"Public bind CIDR exceeded the approved network range."} +{"denial_id":"den-004","timestamp":"2026-03-22T09:55:00Z","actor":"ops-admin","target_system":"payments-api","policy_name":"runtime-policy","decision":"denied","reason":"Runtime policy denied a test action outside the investigation window."} diff --git a/docs/roadmap.md b/docs/roadmap.md index f2c7519..bbcc581 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,23 +1,24 @@ -# Roadmap - -This repository is intentionally small, so the next steps should be new demos that make the existing telemetry pipeline easier to understand rather than a broad platform build-out. - +# Roadmap + +This repository is intentionally small, so the next steps should be new demos that make the existing telemetry pipeline easier to understand rather than a broad platform build-out. + Recently added: - [rule-evaluation-and-dedup-demo](../demos/rule-evaluation-and-dedup-demo/README.md) now shows raw rule hits, retained alerts, and suppression reasons side by side. - -## 1. Auth/Login Anomaly Triage Demo - -Goal: -Add a demo that walks from bursty login failures into follow-on signals such as source spread, eventual success, or repeated target concentration. - -Why it helps the portfolio: -This strengthens the repo's analyst-facing story. It shows how simple window features and rule output can support a concrete triage narrative instead of stopping at generic alert generation. - -## 2. Config-Change Investigation Demo +- [config-change-investigation-demo](../demos/config-change-investigation-demo/README.md) now shows risky configuration changes, bounded evidence attachment, and deterministic investigation summaries. + +## 1. Auth/Login Anomaly Triage Demo + +Goal: +Add a demo that walks from bursty login failures into follow-on signals such as source spread, eventual success, or repeated target concentration. + +Why it helps the portfolio: +This strengthens the repo's analyst-facing story. It shows how simple window features and rule output can support a concrete triage narrative instead of stopping at generic alert generation. + +## 2. Config-Change Drift Follow-Up Demo Goal: -Add a compact scenario centered on risky configuration changes, follow-on policy denials, and a short machine-readable investigation summary. +Add a compact follow-up scenario centered on repeated config drift, rollback attempts, and evidence that the remediation path actually reduced nearby denials or noisy follow-on signals. Why it helps the portfolio: -This broadens the repo beyond auth-only behavior while staying inside the same local, file-based pipeline. It gives the project a second clear demo narrative that is still easy to explain from committed sample data. +This would build on the current config-change investigation demo without changing the repo's local, file-based character. It would show not just the initial risky change, but also how deterministic evidence can support a short remediation narrative. diff --git a/src/telemetry_window_demo/cli.py b/src/telemetry_window_demo/cli.py index a3f52a7..77e5f9e 100644 --- a/src/telemetry_window_demo/cli.py +++ b/src/telemetry_window_demo/cli.py @@ -1,253 +1,277 @@ -from __future__ import annotations - -import argparse -from pathlib import Path -from typing import Any - -from .features import compute_window_features -from .io import ( - format_timestamp, - load_alert_table, - load_config, - load_events, - load_feature_table, - resolve_config_path, - write_json, - write_table, -) -from .preprocess import normalize_events -from .rules import apply_rules -from .visualize import plot_outputs -from .windowing import build_windows - - -def main() -> None: - parser = build_parser() - args = parser.parse_args() - args.func(args) - - -def build_parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - prog="telemetry-window-demo", - description="Windowed telemetry analytics on timestamped event streams.", - ) - subparsers = parser.add_subparsers(dest="command", required=True) - - run_parser = subparsers.add_parser("run", help="Run the full telemetry pipeline.") - run_parser.add_argument("--config", required=True, help="Path to a YAML config file.") - run_parser.set_defaults(func=run_command) - - summarize_parser = subparsers.add_parser( - "summarize", - help="Summarize an input event file.", - ) - summarize_parser.add_argument("--input", required=True, help="Path to .jsonl or .csv.") - summarize_parser.set_defaults(func=summarize_command) - - plot_parser = subparsers.add_parser("plot", help="Render plots from CSV outputs.") - plot_parser.add_argument("--features", required=True, help="Path to features.csv.") - plot_parser.add_argument("--alerts", help="Path to alerts.csv.") - plot_parser.add_argument( - "--output-dir", - default="data/processed", - help="Directory where plot images will be written.", - ) - plot_parser.set_defaults(func=plot_command) - - run_ai_demo_parser = subparsers.add_parser( - "run-ai-demo", - help="Run the constrained AI-assisted detection demo with JSON-only summarization.", - ) - run_ai_demo_parser.add_argument( - "--demo-root", - help="Path to demos/ai-assisted-detection-demo.", - ) - run_ai_demo_parser.set_defaults(func=run_ai_demo_command) - +from __future__ import annotations + +import argparse +from pathlib import Path +from typing import Any + +from .features import compute_window_features +from .io import ( + format_timestamp, + load_alert_table, + load_config, + load_events, + load_feature_table, + resolve_config_path, + write_json, + write_table, +) +from .preprocess import normalize_events +from .rules import apply_rules +from .visualize import plot_outputs +from .windowing import build_windows + + +def main() -> None: + parser = build_parser() + args = parser.parse_args() + args.func(args) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + prog="telemetry-window-demo", + description="Windowed telemetry analytics on timestamped event streams.", + ) + subparsers = parser.add_subparsers(dest="command", required=True) + + run_parser = subparsers.add_parser("run", help="Run the full telemetry pipeline.") + run_parser.add_argument("--config", required=True, help="Path to a YAML config file.") + run_parser.set_defaults(func=run_command) + + summarize_parser = subparsers.add_parser( + "summarize", + help="Summarize an input event file.", + ) + summarize_parser.add_argument("--input", required=True, help="Path to .jsonl or .csv.") + summarize_parser.set_defaults(func=summarize_command) + + plot_parser = subparsers.add_parser("plot", help="Render plots from CSV outputs.") + plot_parser.add_argument("--features", required=True, help="Path to features.csv.") + plot_parser.add_argument("--alerts", help="Path to alerts.csv.") + plot_parser.add_argument( + "--output-dir", + default="data/processed", + help="Directory where plot images will be written.", + ) + plot_parser.set_defaults(func=plot_command) + + run_ai_demo_parser = subparsers.add_parser( + "run-ai-demo", + help="Run the constrained AI-assisted detection demo with JSON-only summarization.", + ) + run_ai_demo_parser.add_argument( + "--demo-root", + help="Path to demos/ai-assisted-detection-demo.", + ) + run_ai_demo_parser.set_defaults(func=run_ai_demo_command) + run_rule_dedup_demo_parser = subparsers.add_parser( "run-rule-dedup-demo", help="Run the rule evaluation and dedup demo with suppression explanations.", ) - run_rule_dedup_demo_parser.add_argument( - "--demo-root", - help="Path to demos/rule-evaluation-and-dedup-demo.", + run_rule_dedup_demo_parser.add_argument( + "--demo-root", + help="Path to demos/rule-evaluation-and-dedup-demo.", ) run_rule_dedup_demo_parser.set_defaults(func=run_rule_dedup_demo_command) - return parser - - -def run_command(args: argparse.Namespace) -> None: - config_path = Path(args.config).resolve() - config = load_config(config_path) - time_config = config.get("time", {}) - feature_config = config.get("features", {}) - rules_config = config.get("rules") or {} - input_path = resolve_config_path(config_path, config["input_path"]) - output_dir = resolve_config_path(config_path, config.get("output_dir", "data/processed")) - - events = load_events(input_path) - normalized = normalize_events( - events, - timestamp_col=time_config.get("timestamp_col", "timestamp"), - error_statuses=feature_config.get("error_statuses"), - high_severity_levels=feature_config.get("severity_levels"), - ) - windows = build_windows( - normalized, - timestamp_col=time_config.get("timestamp_col", "timestamp"), - window_size_seconds=int(time_config.get("window_size_seconds", 60)), - step_size_seconds=int(time_config.get("step_size_seconds", 10)), - ) - features = compute_window_features( - normalized, - windows, - count_event_types=feature_config.get("count_event_types"), - ) - alerts = apply_rules(features, rules_config) - cooldown_seconds = int(rules_config.get("cooldown_seconds", 0)) - - feature_path = write_table(features, output_dir / "features.csv") - alert_path = write_table(alerts, output_dir / "alerts.csv") - plot_paths = plot_outputs(features, alerts, output_dir) - summary_path = output_dir / "summary.json" - summary = _build_run_summary( - input_path=input_path, - output_dir=output_dir, - normalized=normalized, - windows=windows, - features=features, - alerts=alerts, - cooldown_seconds=cooldown_seconds, - feature_path=feature_path, - alert_path=alert_path, - summary_path=summary_path, - plot_paths=plot_paths, + run_config_change_demo_parser = subparsers.add_parser( + "run-config-change-demo", + help="Run the config-change investigation demo with deterministic correlation.", ) - write_json(summary, summary_path) - - print(f"[OK] Loaded {len(normalized)} events") - print(f"[OK] Generated {len(features)} windows") - print(f"[OK] Computed {max(len(features.columns) - 2, 0)} features per window") - print(f"[OK] Triggered {len(alerts)} alerts") - print(f"[OK] Saved {feature_path.name}, {alert_path.name}") - print(f"[OK] Saved plots to {_display_path(output_dir)}") - for plot_path in plot_paths: - print(f" - {plot_path.name}") - - -def summarize_command(args: argparse.Namespace) -> None: - events = normalize_events(load_events(args.input)) - min_time = format_timestamp(events["timestamp"].min()) - max_time = format_timestamp(events["timestamp"].max()) - top_event_types = events["event_type"].value_counts().head(5).to_dict() - overall_error_rate = float(events["is_error"].mean()) if not events.empty else 0.0 - - print(f"events: {len(events)}") - print(f"time_range: {min_time} -> {max_time}") - print(f"unique_sources: {events['source'].nunique()}") - print(f"unique_targets: {events['target'].nunique()}") - print(f"overall_error_rate: {overall_error_rate:.2f}") - print(f"top_event_types: {top_event_types}") - - -def plot_command(args: argparse.Namespace) -> None: - features = load_feature_table(args.features) - alerts = ( - load_alert_table(args.alerts) - if args.alerts - else load_alert_table(Path(args.features).with_name("alerts.csv")) + run_config_change_demo_parser.add_argument( + "--demo-root", + help="Path to demos/config-change-investigation-demo.", ) - plot_paths = plot_outputs(features, alerts, args.output_dir) - print(f"[OK] Saved plots to {_display_path(Path(args.output_dir).resolve())}") - for plot_path in plot_paths: - print(f" - {plot_path.name}") - - -def run_ai_demo_command(args: argparse.Namespace) -> None: - from .ai_assisted_detection_demo import default_demo_root, run_demo + run_config_change_demo_parser.set_defaults(func=run_config_change_demo_command) + + return parser + + +def run_command(args: argparse.Namespace) -> None: + config_path = Path(args.config).resolve() + config = load_config(config_path) + time_config = config.get("time", {}) + feature_config = config.get("features", {}) + rules_config = config.get("rules") or {} + input_path = resolve_config_path(config_path, config["input_path"]) + output_dir = resolve_config_path(config_path, config.get("output_dir", "data/processed")) + + events = load_events(input_path) + normalized = normalize_events( + events, + timestamp_col=time_config.get("timestamp_col", "timestamp"), + error_statuses=feature_config.get("error_statuses"), + high_severity_levels=feature_config.get("severity_levels"), + ) + windows = build_windows( + normalized, + timestamp_col=time_config.get("timestamp_col", "timestamp"), + window_size_seconds=int(time_config.get("window_size_seconds", 60)), + step_size_seconds=int(time_config.get("step_size_seconds", 10)), + ) + features = compute_window_features( + normalized, + windows, + count_event_types=feature_config.get("count_event_types"), + ) + alerts = apply_rules(features, rules_config) + cooldown_seconds = int(rules_config.get("cooldown_seconds", 0)) + + feature_path = write_table(features, output_dir / "features.csv") + alert_path = write_table(alerts, output_dir / "alerts.csv") + plot_paths = plot_outputs(features, alerts, output_dir) + summary_path = output_dir / "summary.json" + summary = _build_run_summary( + input_path=input_path, + output_dir=output_dir, + normalized=normalized, + windows=windows, + features=features, + alerts=alerts, + cooldown_seconds=cooldown_seconds, + feature_path=feature_path, + alert_path=alert_path, + summary_path=summary_path, + plot_paths=plot_paths, + ) + write_json(summary, summary_path) + + print(f"[OK] Loaded {len(normalized)} events") + print(f"[OK] Generated {len(features)} windows") + print(f"[OK] Computed {max(len(features.columns) - 2, 0)} features per window") + print(f"[OK] Triggered {len(alerts)} alerts") + print(f"[OK] Saved {feature_path.name}, {alert_path.name}") + print(f"[OK] Saved plots to {_display_path(output_dir)}") + for plot_path in plot_paths: + print(f" - {plot_path.name}") + + +def summarize_command(args: argparse.Namespace) -> None: + events = normalize_events(load_events(args.input)) + min_time = format_timestamp(events["timestamp"].min()) + max_time = format_timestamp(events["timestamp"].max()) + top_event_types = events["event_type"].value_counts().head(5).to_dict() + overall_error_rate = float(events["is_error"].mean()) if not events.empty else 0.0 + + print(f"events: {len(events)}") + print(f"time_range: {min_time} -> {max_time}") + print(f"unique_sources: {events['source'].nunique()}") + print(f"unique_targets: {events['target'].nunique()}") + print(f"overall_error_rate: {overall_error_rate:.2f}") + print(f"top_event_types: {top_event_types}") + + +def plot_command(args: argparse.Namespace) -> None: + features = load_feature_table(args.features) + alerts = ( + load_alert_table(args.alerts) + if args.alerts + else load_alert_table(Path(args.features).with_name("alerts.csv")) + ) + plot_paths = plot_outputs(features, alerts, args.output_dir) + print(f"[OK] Saved plots to {_display_path(Path(args.output_dir).resolve())}") + for plot_path in plot_paths: + print(f" - {plot_path.name}") + + +def run_ai_demo_command(args: argparse.Namespace) -> None: + from .ai_assisted_detection_demo import default_demo_root, run_demo + + demo_root = Path(args.demo_root).resolve() if args.demo_root else default_demo_root() + result = run_demo(demo_root=demo_root) + + print(f"[OK] Loaded {result['raw_event_count']} raw events") + print(f"[OK] Normalized {result['normalized_event_count']} events") + print(f"[OK] Triggered {result['rule_hit_count']} rule hits") + print(f"[OK] Built {result['case_count']} cases") + print(f"[OK] Validated {result['summary_count']} JSON summaries") + print(f"[OK] Rejected {result['rejected_summary_count']} summaries") + print(f"[OK] Wrote {result['audit_record_count']} audit records") + print(f"[OK] Saved artifacts to {_display_path(result['artifacts_dir'])}") + for name, path in result["artifacts"].items(): + print(f" - {name}: {_display_path(path)}") + + +def run_rule_dedup_demo_command(args: argparse.Namespace) -> None: + from .rule_evaluation_and_dedup_demo import default_demo_root, run_demo demo_root = Path(args.demo_root).resolve() if args.demo_root else default_demo_root() - result = run_demo(demo_root=demo_root) - - print(f"[OK] Loaded {result['raw_event_count']} raw events") - print(f"[OK] Normalized {result['normalized_event_count']} events") - print(f"[OK] Triggered {result['rule_hit_count']} rule hits") - print(f"[OK] Built {result['case_count']} cases") - print(f"[OK] Validated {result['summary_count']} JSON summaries") - print(f"[OK] Rejected {result['rejected_summary_count']} summaries") - print(f"[OK] Wrote {result['audit_record_count']} audit records") + result = run_demo(demo_root=demo_root) + + print(f"[OK] Loaded {result['raw_hit_count']} raw rule hits") + print(f"[OK] Retained {result['retained_alert_count']} alerts") + print(f"[OK] Suppressed {result['suppressed_hit_count']} repeated hits") + print(f"[OK] Evaluated {result['group_count']} rule/scope groups") print(f"[OK] Saved artifacts to {_display_path(result['artifacts_dir'])}") for name, path in result["artifacts"].items(): print(f" - {name}: {_display_path(path)}") -def run_rule_dedup_demo_command(args: argparse.Namespace) -> None: - from .rule_evaluation_and_dedup_demo import default_demo_root, run_demo +def run_config_change_demo_command(args: argparse.Namespace) -> None: + from .config_change_investigation_demo import default_demo_root, run_demo demo_root = Path(args.demo_root).resolve() if args.demo_root else default_demo_root() result = run_demo(demo_root=demo_root) - print(f"[OK] Loaded {result['raw_hit_count']} raw rule hits") - print(f"[OK] Retained {result['retained_alert_count']} alerts") - print(f"[OK] Suppressed {result['suppressed_hit_count']} repeated hits") - print(f"[OK] Evaluated {result['group_count']} rule/scope groups") + print(f"[OK] Loaded {result['change_event_count']} config changes") + print(f"[OK] Flagged {result['risky_change_count']} risky changes") + print(f"[OK] Built {result['investigation_count']} investigations") print(f"[OK] Saved artifacts to {_display_path(result['artifacts_dir'])}") for name, path in result["artifacts"].items(): print(f" - {name}: {_display_path(path)}") - - -def _display_path(path: Path) -> str: - cwd = Path.cwd().resolve() - resolved = path.resolve() - try: - return resolved.relative_to(cwd).as_posix() - except ValueError: - return resolved.as_posix() - - -def _build_run_summary( - input_path: Path, - output_dir: Path, - normalized: Any, - windows: list[Any], - features: Any, - alerts: Any, - cooldown_seconds: int, - feature_path: Path, - alert_path: Path, - summary_path: Path, - plot_paths: list[Path], -) -> dict[str, object]: - if alerts.empty: - rule_counts: dict[str, int] = {} - else: - rule_counts = { - str(rule_name): int(count) - for rule_name, count in alerts["rule_name"].value_counts().sort_index().items() - } - - artifact_paths = [ - feature_path, - alert_path, - summary_path, - *plot_paths, - ] - - return { - "input_path": _display_path(input_path), - "output_dir": _display_path(output_dir), - "normalized_event_count": int(len(normalized)), - "window_count": int(len(windows)), - "feature_row_count": int(len(features)), - "alert_count": int(len(alerts)), - "triggered_rule_names": sorted(rule_counts), - "triggered_rule_counts": rule_counts, - "cooldown_seconds": int(cooldown_seconds), - "generated_artifacts": [_display_path(path) for path in artifact_paths], - } - - -if __name__ == "__main__": - main() + + +def _display_path(path: Path) -> str: + cwd = Path.cwd().resolve() + resolved = path.resolve() + try: + return resolved.relative_to(cwd).as_posix() + except ValueError: + return resolved.as_posix() + + +def _build_run_summary( + input_path: Path, + output_dir: Path, + normalized: Any, + windows: list[Any], + features: Any, + alerts: Any, + cooldown_seconds: int, + feature_path: Path, + alert_path: Path, + summary_path: Path, + plot_paths: list[Path], +) -> dict[str, object]: + if alerts.empty: + rule_counts: dict[str, int] = {} + else: + rule_counts = { + str(rule_name): int(count) + for rule_name, count in alerts["rule_name"].value_counts().sort_index().items() + } + + artifact_paths = [ + feature_path, + alert_path, + summary_path, + *plot_paths, + ] + + return { + "input_path": _display_path(input_path), + "output_dir": _display_path(output_dir), + "normalized_event_count": int(len(normalized)), + "window_count": int(len(windows)), + "feature_row_count": int(len(features)), + "alert_count": int(len(alerts)), + "triggered_rule_names": sorted(rule_counts), + "triggered_rule_counts": rule_counts, + "cooldown_seconds": int(cooldown_seconds), + "generated_artifacts": [_display_path(path) for path in artifact_paths], + } + + +if __name__ == "__main__": + main() diff --git a/src/telemetry_window_demo/config_change_investigation_demo/__init__.py b/src/telemetry_window_demo/config_change_investigation_demo/__init__.py new file mode 100644 index 0000000..275940a --- /dev/null +++ b/src/telemetry_window_demo/config_change_investigation_demo/__init__.py @@ -0,0 +1,5 @@ +"""Config-change investigation demo pipeline.""" + +from .pipeline import default_demo_root, run_demo + +__all__ = ["default_demo_root", "run_demo"] diff --git a/src/telemetry_window_demo/config_change_investigation_demo/pipeline.py b/src/telemetry_window_demo/config_change_investigation_demo/pipeline.py new file mode 100644 index 0000000..7cd756a --- /dev/null +++ b/src/telemetry_window_demo/config_change_investigation_demo/pipeline.py @@ -0,0 +1,499 @@ +from __future__ import annotations + +import json +from collections.abc import Mapping, Sequence +from datetime import UTC, datetime, timedelta +from pathlib import Path +from typing import Any + +import yaml + +SEVERITY_ORDER = {"low": 1, "medium": 2, "high": 3, "critical": 4} +CHANGE_REQUIRED_FIELDS = ( + "change_id", + "timestamp", + "actor", + "target_system", + "config_key", + "old_value", + "new_value", + "change_result", +) +DENIAL_REQUIRED_FIELDS = ( + "denial_id", + "timestamp", + "actor", + "target_system", + "policy_name", + "decision", + "reason", +) +FOLLOW_ON_REQUIRED_FIELDS = ( + "event_id", + "timestamp", + "target_system", + "event_type", + "details", +) + + +def default_demo_root() -> Path: + return Path(__file__).resolve().parents[3] / "demos" / "config-change-investigation-demo" + + +def run_demo( + demo_root: Path | None = None, + artifacts_dir: Path | None = None, +) -> dict[str, Any]: + demo_root = Path(demo_root or default_demo_root()).resolve() + config = load_yaml(demo_root / "config" / "investigation.yaml") + input_paths = config.get("input_paths", {}) + artifacts_dir = Path( + artifacts_dir + or resolve_demo_path(demo_root, str(config.get("artifacts_dir", "artifacts"))) + ).resolve() + artifacts_dir.mkdir(parents=True, exist_ok=True) + + config_changes = normalize_config_changes( + load_jsonl(resolve_demo_path(demo_root, str(input_paths["config_changes"]))) + ) + policy_denials = normalize_policy_denials( + load_jsonl(resolve_demo_path(demo_root, str(input_paths["policy_denials"]))) + ) + follow_on_events = normalize_follow_on_events( + load_jsonl(resolve_demo_path(demo_root, str(input_paths["follow_on_events"]))) + ) + + rule_hits = evaluate_risky_config_changes(config_changes, config.get("rules", [])) + investigations = build_investigations( + rule_hits, + policy_denials, + follow_on_events, + correlation_minutes=int(config.get("correlation_minutes", 15)), + ) + summary = build_investigation_summary( + investigations, + correlation_minutes=int(config.get("correlation_minutes", 15)), + ) + report_text = build_investigation_report( + config_changes=config_changes, + rule_hits=rule_hits, + investigations=investigations, + correlation_minutes=int(config.get("correlation_minutes", 15)), + ) + + paths = { + "change_events_normalized": write_json( + config_changes, + artifacts_dir / "change_events_normalized.json", + ), + "investigation_hits": write_json( + investigations, + artifacts_dir / "investigation_hits.json", + ), + "investigation_summary": write_json( + summary, + artifacts_dir / "investigation_summary.json", + ), + "investigation_report": write_text( + report_text, + artifacts_dir / "investigation_report.md", + ), + } + + return { + "demo_root": demo_root, + "artifacts_dir": artifacts_dir, + "change_event_count": len(config_changes), + "risky_change_count": len(rule_hits), + "investigation_count": len(investigations), + "artifacts": paths, + } + + +def load_yaml(path: Path) -> dict[str, Any]: + with path.open("r", encoding="utf-8") as handle: + payload = yaml.safe_load(handle) or {} + if not isinstance(payload, dict): + raise ValueError("YAML config must deserialize into a mapping.") + return payload + + +def load_jsonl(path: Path) -> list[dict[str, Any]]: + records: list[dict[str, Any]] = [] + with path.open("r", encoding="utf-8") as handle: + for line_number, line in enumerate(handle, start=1): + raw = line.strip() + if not raw: + continue + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise ValueError(f"Invalid JSONL at line {line_number} in {path}") from exc + if not isinstance(payload, dict): + raise ValueError("Expected JSON object records in JSONL input.") + records.append(payload) + return records + + +def resolve_demo_path(demo_root: Path, value: str) -> Path: + candidate = Path(value) + if candidate.is_absolute(): + return candidate + return (demo_root / candidate).resolve() + + +def normalize_config_changes(raw_events: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + seen_ids: set[str] = set() + for index, raw_event in enumerate(raw_events, start=1): + for field in CHANGE_REQUIRED_FIELDS: + value = raw_event.get(field) + if not isinstance(value, str) or not value.strip(): + raise ValueError( + f"Config change {index} is missing required string field '{field}'." + ) + + change_id = str(raw_event["change_id"]).strip() + if change_id in seen_ids: + raise ValueError(f"Duplicate change_id found in sample input: {change_id}") + seen_ids.add(change_id) + + normalized.append( + { + "change_id": change_id, + "timestamp": parse_timestamp(str(raw_event["timestamp"])), + "actor": str(raw_event["actor"]).strip(), + "target_system": str(raw_event["target_system"]).strip(), + "config_key": str(raw_event["config_key"]).strip(), + "old_value": str(raw_event["old_value"]).strip(), + "new_value": str(raw_event["new_value"]).strip(), + "change_result": str(raw_event["change_result"]).strip().lower(), + "change_ticket": normalize_optional_text(raw_event.get("change_ticket")), + } + ) + + return sorted( + normalized, + key=lambda event: (format_timestamp(event["timestamp"]), event["change_id"]), + ) + + +def normalize_policy_denials(raw_events: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + seen_ids: set[str] = set() + for index, raw_event in enumerate(raw_events, start=1): + for field in DENIAL_REQUIRED_FIELDS: + value = raw_event.get(field) + if not isinstance(value, str) or not value.strip(): + raise ValueError( + f"Policy denial {index} is missing required string field '{field}'." + ) + + denial_id = str(raw_event["denial_id"]).strip() + if denial_id in seen_ids: + raise ValueError(f"Duplicate denial_id found in sample input: {denial_id}") + seen_ids.add(denial_id) + + normalized.append( + { + "denial_id": denial_id, + "timestamp": parse_timestamp(str(raw_event["timestamp"])), + "actor": str(raw_event["actor"]).strip(), + "target_system": str(raw_event["target_system"]).strip(), + "policy_name": str(raw_event["policy_name"]).strip(), + "decision": str(raw_event["decision"]).strip().lower(), + "reason": str(raw_event["reason"]).strip(), + } + ) + + return sorted( + normalized, + key=lambda event: (format_timestamp(event["timestamp"]), event["denial_id"]), + ) + + +def normalize_follow_on_events(raw_events: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]: + normalized: list[dict[str, Any]] = [] + seen_ids: set[str] = set() + for index, raw_event in enumerate(raw_events, start=1): + for field in FOLLOW_ON_REQUIRED_FIELDS: + value = raw_event.get(field) + if not isinstance(value, str) or not value.strip(): + raise ValueError( + f"Follow-on event {index} is missing required string field '{field}'." + ) + + event_id = str(raw_event["event_id"]).strip() + if event_id in seen_ids: + raise ValueError(f"Duplicate event_id found in sample input: {event_id}") + seen_ids.add(event_id) + + normalized.append( + { + "event_id": event_id, + "timestamp": parse_timestamp(str(raw_event["timestamp"])), + "target_system": str(raw_event["target_system"]).strip(), + "event_type": str(raw_event["event_type"]).strip(), + "details": str(raw_event["details"]).strip(), + } + ) + + return sorted( + normalized, + key=lambda event: (format_timestamp(event["timestamp"]), event["event_id"]), + ) + + +def evaluate_risky_config_changes( + config_changes: Sequence[Mapping[str, Any]], + rules: Sequence[Mapping[str, Any]], +) -> list[dict[str, Any]]: + validated_rules = validate_rules(rules) + hits: list[dict[str, Any]] = [] + for change in config_changes: + if str(change["change_result"]) != "success": + continue + for rule in validated_rules: + if str(change["config_key"]) != str(rule["config_key"]): + continue + if str(change["new_value"]).lower() not in rule["risky_values"]: + continue + hits.append( + { + "investigation_id": "", + "rule_id": str(rule["rule_id"]), + "severity": str(rule["severity"]), + "reason": str(rule["reason"]), + "change_event": dict(change), + } + ) + hits.sort( + key=lambda hit: ( + format_timestamp(hit["change_event"]["timestamp"]), + str(hit["rule_id"]), + str(hit["change_event"]["change_id"]), + ) + ) + for index, hit in enumerate(hits, start=1): + hit["investigation_id"] = f"CCI-{index:03d}" + return hits + + +def validate_rules(rules: Sequence[Mapping[str, Any]]) -> list[dict[str, Any]]: + validated: list[dict[str, Any]] = [] + for index, rule in enumerate(rules, start=1): + if not isinstance(rule, Mapping): + raise ValueError(f"Rule {index} must be a mapping.") + for field in ("rule_id", "config_key", "severity", "reason"): + value = rule.get(field) + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"Rule {index} is missing required string field '{field}'.") + risky_values = rule.get("risky_values") + if not isinstance(risky_values, list) or not risky_values: + raise ValueError(f"Rule {index} must define a non-empty risky_values list.") + if str(rule["severity"]).strip().lower() not in SEVERITY_ORDER: + raise ValueError(f"Rule {index} uses unsupported severity '{rule['severity']}'.") + validated.append( + { + "rule_id": str(rule["rule_id"]).strip(), + "config_key": str(rule["config_key"]).strip(), + "severity": str(rule["severity"]).strip().lower(), + "reason": str(rule["reason"]).strip(), + "risky_values": [str(value).strip().lower() for value in risky_values], + } + ) + return validated + + +def build_investigations( + rule_hits: Sequence[Mapping[str, Any]], + policy_denials: Sequence[Mapping[str, Any]], + follow_on_events: Sequence[Mapping[str, Any]], + correlation_minutes: int, +) -> list[dict[str, Any]]: + investigations: list[dict[str, Any]] = [] + correlation_window = timedelta(minutes=correlation_minutes) + + for hit in rule_hits: + change_event = hit["change_event"] + change_time = change_event["timestamp"] + window_end = change_time + correlation_window + target_system = str(change_event["target_system"]) + + attached_denials = [ + dict(denial) + for denial in policy_denials + if str(denial["target_system"]) == target_system + and change_time <= denial["timestamp"] <= window_end + ] + attached_follow_on = [ + dict(event) + for event in follow_on_events + if str(event["target_system"]) == target_system + and change_time <= event["timestamp"] <= window_end + ] + + investigations.append( + { + "investigation_id": str(hit["investigation_id"]), + "severity": str(hit["severity"]), + "rule_id": str(hit["rule_id"]), + "target_system": target_system, + "actor": str(change_event["actor"]), + "triggering_change": dict(change_event), + "trigger_reason": str(hit["reason"]), + "correlation_window_minutes": correlation_minutes, + "bounded_correlation_reason": ( + f"Attached evidence shares target_system '{target_system}' and falls within " + f"{correlation_minutes} minutes after the triggering change." + ), + "attached_policy_denials": attached_denials, + "attached_follow_on_events": attached_follow_on, + "evidence_counts": { + "policy_denials": len(attached_denials), + "follow_on_events": len(attached_follow_on), + }, + } + ) + + return investigations + + +def build_investigation_summary( + investigations: Sequence[Mapping[str, Any]], + correlation_minutes: int, +) -> list[dict[str, Any]]: + summary: list[dict[str, Any]] = [] + for investigation in investigations: + change = investigation["triggering_change"] + counts = investigation["evidence_counts"] + summary.append( + { + "investigation_id": str(investigation["investigation_id"]), + "severity": str(investigation["severity"]), + "target_system": str(investigation["target_system"]), + "triggering_change_id": str(change["change_id"]), + "summary": ( + f"{change['config_key']} changed from {change['old_value']} to " + f"{change['new_value']} on {change['target_system']}, followed by " + f"{counts['policy_denials']} policy denials and " + f"{counts['follow_on_events']} follow-on events within " + f"{correlation_minutes} minutes." + ), + "evidence_counts": dict(counts), + "bounded_correlation_reason": str(investigation["bounded_correlation_reason"]), + } + ) + return summary + + +def build_investigation_report( + config_changes: Sequence[Mapping[str, Any]], + rule_hits: Sequence[Mapping[str, Any]], + investigations: Sequence[Mapping[str, Any]], + correlation_minutes: int, +) -> str: + lines = [ + "# Config-Change Investigation Demo Report", + "", + "This deterministic demo correlates risky configuration changes with bounded follow-on evidence.", + "It does not use an LLM and does not produce autonomous response actions.", + "", + "## Run Summary", + "", + f"- normalized_change_events: {len(config_changes)}", + f"- risky_change_hits: {len(rule_hits)}", + f"- investigations: {len(investigations)}", + f"- correlation_window_minutes: {correlation_minutes}", + "", + ] + + if not investigations: + lines.append("No investigations were generated from the current sample.") + return "\n".join(lines).rstrip() + "\n" + + for investigation in investigations: + change = investigation["triggering_change"] + counts = investigation["evidence_counts"] + lines.extend( + [ + f"## {investigation['investigation_id']}", + "", + f"- Severity: {investigation['severity']}", + f"- Target system: {investigation['target_system']}", + f"- Triggering change: {change['change_id']} ({change['config_key']} -> {change['new_value']})", + f"- Trigger reason: {investigation['trigger_reason']}", + f"- Attached policy denials: {counts['policy_denials']}", + f"- Attached follow-on events: {counts['follow_on_events']}", + f"- Bounded correlation: {investigation['bounded_correlation_reason']}", + "", + ] + ) + + if investigation["attached_policy_denials"]: + lines.append("Policy denials:") + for denial in investigation["attached_policy_denials"]: + lines.append( + f"- {denial['denial_id']}: {denial['policy_name']} -> {denial['reason']}" + ) + lines.append("") + + if investigation["attached_follow_on_events"]: + lines.append("Follow-on events:") + for event in investigation["attached_follow_on_events"]: + lines.append( + f"- {event['event_id']}: {event['event_type']} -> {event['details']}" + ) + lines.append("") + + if not investigation["attached_policy_denials"] and not investigation["attached_follow_on_events"]: + lines.append( + "No nearby supporting evidence fell inside the bounded correlation window." + ) + lines.append("") + + return "\n".join(lines).rstrip() + "\n" + + +def normalize_optional_text(value: Any) -> str | None: + if value is None: + return None + text = str(value).strip() + return text or None + + +def parse_timestamp(raw_value: str) -> datetime: + return datetime.fromisoformat(raw_value.replace("Z", "+00:00")).astimezone(UTC) + + +def format_timestamp(value: Any) -> str: + timestamp = value if isinstance(value, datetime) else parse_timestamp(str(value)) + return timestamp.astimezone(UTC).isoformat().replace("+00:00", "Z") + + +def write_json(payload: Any, path: Path) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(serialize_record(payload), indent=2) + "\n", + encoding="utf-8", + ) + return path + + +def write_text(content: str, path: Path) -> Path: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content, encoding="utf-8") + return path + + +def serialize_record(value: Any) -> Any: + if isinstance(value, datetime): + return format_timestamp(value) + if isinstance(value, Path): + return value.as_posix() + if isinstance(value, dict): + return {key: serialize_record(item) for key, item in value.items()} + if isinstance(value, list): + return [serialize_record(item) for item in value] + return value diff --git a/tests/test_config_change_investigation_demo.py b/tests/test_config_change_investigation_demo.py new file mode 100644 index 0000000..9defee7 --- /dev/null +++ b/tests/test_config_change_investigation_demo.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +import json +from pathlib import Path + +from telemetry_window_demo.config_change_investigation_demo import default_demo_root, run_demo +from telemetry_window_demo.config_change_investigation_demo.pipeline import ( + build_investigations, + evaluate_risky_config_changes, + load_jsonl, + load_yaml, + normalize_config_changes, + normalize_follow_on_events, + normalize_policy_denials, +) + + +def _load_demo_inputs(): + demo_root = default_demo_root() + config = load_yaml(demo_root / "config" / "investigation.yaml") + config_changes = normalize_config_changes( + load_jsonl(demo_root / "data" / "raw" / "config_changes.jsonl") + ) + policy_denials = normalize_policy_denials( + load_jsonl(demo_root / "data" / "raw" / "policy_denials.jsonl") + ) + follow_on_events = normalize_follow_on_events( + load_jsonl(demo_root / "data" / "raw" / "follow_on_events.jsonl") + ) + return demo_root, config, config_changes, policy_denials, follow_on_events + + +def _load_json_file(path: Path): + return json.loads(path.read_text(encoding="utf-8")) + + +def test_normalize_config_changes_is_sorted_and_complete() -> None: + _, _, config_changes, _, _ = _load_demo_inputs() + + assert [change["change_id"] for change in config_changes] == [ + "cfg-001", + "cfg-002", + "cfg-003", + "cfg-004", + ] + assert config_changes[0]["target_system"] == "identity-proxy" + assert config_changes[1]["config_key"] == "public_bind_cidr" + + +def test_evaluate_risky_config_changes_flags_expected_changes() -> None: + _, config, config_changes, _, _ = _load_demo_inputs() + hits = evaluate_risky_config_changes(config_changes, config["rules"]) + + assert [hit["change_event"]["change_id"] for hit in hits] == [ + "cfg-001", + "cfg-002", + "cfg-004", + ] + assert [hit["severity"] for hit in hits] == ["critical", "high", "high"] + + +def test_build_investigations_uses_bounded_system_and_time_correlation() -> None: + _, config, config_changes, policy_denials, follow_on_events = _load_demo_inputs() + hits = evaluate_risky_config_changes(config_changes, config["rules"]) + investigations = build_investigations( + hits, + policy_denials, + follow_on_events, + correlation_minutes=int(config["correlation_minutes"]), + ) + + identity = next(item for item in investigations if item["investigation_id"] == "CCI-001") + payments = next(item for item in investigations if item["investigation_id"] == "CCI-002") + vault = next(item for item in investigations if item["investigation_id"] == "CCI-003") + + assert identity["evidence_counts"] == {"policy_denials": 2, "follow_on_events": 2} + assert payments["evidence_counts"] == {"policy_denials": 1, "follow_on_events": 2} + assert vault["evidence_counts"] == {"policy_denials": 0, "follow_on_events": 0} + + assert all( + denial["target_system"] == "payments-api" + for denial in payments["attached_policy_denials"] + ) + assert all( + event["event_id"] != "fo-005" for event in vault["attached_follow_on_events"] + ) + + +def test_run_demo_is_deterministic_and_matches_committed_artifacts(tmp_path) -> None: + demo_root, _, _, _, _ = _load_demo_inputs() + first_dir = tmp_path / "run-one" + second_dir = tmp_path / "run-two" + + first_result = run_demo(demo_root=demo_root, artifacts_dir=first_dir) + second_result = run_demo(demo_root=demo_root, artifacts_dir=second_dir) + + assert first_result["change_event_count"] == 4 + assert first_result["risky_change_count"] == 3 + assert first_result["investigation_count"] == 3 + assert second_result["investigation_count"] == first_result["investigation_count"] + + for name in ( + "change_events_normalized.json", + "investigation_hits.json", + "investigation_summary.json", + ): + expected = _load_json_file(demo_root / "artifacts" / name) + first = _load_json_file(first_dir / name) + second = _load_json_file(second_dir / name) + assert first == expected + assert second == expected + + expected_report = ( + demo_root / "artifacts" / "investigation_report.md" + ).read_text(encoding="utf-8") + assert (first_dir / "investigation_report.md").read_text(encoding="utf-8") == expected_report + assert (second_dir / "investigation_report.md").read_text(encoding="utf-8") == expected_report