+
+ Every SQL statement the simulator and dashboard run against CedarDB, in
+ one place. The interesting bit isn't any individual query — it's that
+ all of these run concurrently against the same instance:
+ N writer goroutines are CopyFrom-ing into events
+ at up to ~500 K events/sec (≈ 3 TB/day uncompressed) while four
+ different read paths (active-alerts queue, live event stream, customer
+ drill-down, ingest stats) are joining across the same hot table and the
+ normalized dimensions at 200 ms – 2 s refresh cadences. The whole pitch
+ is collapsing the BigQuery + Pub/Sub-plus-KV split onto one database
+ with real joins and no replication lag.
+
+ Tuning the cadences. Every "refresh N s" pill below is
+ an env-var-driven default — set HG_ALERTS_REFRESH,
+ HG_DRILLDOWN_REFRESH, HG_STATS_REFRESH,
+ HG_STORAGE_REFRESH, HG_SSE_INTERVAL
+ (on the server container), or HG_STORAGE_SAMPLER_INTERVAL
+ (on the simulator container) to any Go duration string (e.g.
+ "500ms", "2s", "30s") to dial
+ the read pressure CedarDB has to serve next to the ingest stream.
+
1. Hot write path2 queries — drives the firehose
+ +
+ The firehose. -writers producer goroutines partition
+ the device fleet, build row batches, and push them onto a buffered
+ channel. A single ingestor goroutine drains the
+ channel, coalesces up to 10 000 rows (or every 50 ms, whichever
+ comes first), and runs one binary CopyFrom. This
+ sidesteps CedarDB's "cannot start bulk operation until previous
+ bulk operation has become globally visible" error (SQLSTATE
+ 40P01) that fires when more than one goroutine COPYs in parallel,
+ while still keeping the binary protocol's ~3× wire and parsing
+ advantage over multi-row INSERT under simple-query mode.
+ event_id is an atomic.Int64 seeded from
+ MAX(event_id) at startup, so producers share the id
+ space without locking — the column is plain BIGINT,
+ not BIGSERIAL (CedarDB rejects COPY for sequence-
+ defaulted columns with "unable to cast from void to bigint").
+
-- One round trip per ingestor flush (≤ 10K rows, or every 50ms).
+COPY events (
+ event_id, device_id, household_id, ts,
+ kind, severity, value, battery_pct, rssi_dbm
+) FROM STDIN BINARY;
+-- ...one binary tuple per buffered event...+ On startup, the simulator runs one read to seed the id counter: +
+SELECT COALESCE(MAX(event_id), 0) FROM events;Insert one alert
+ +
+ Called inline from emitTick whenever EvaluateAlert
+ in rules.go promotes a triggered/tamper event to an
+ operator-actionable alert. status starts 'active'; the
+ background resolveAlertsLoop later marks it resolved or
+ false_alarm.
+
INSERT INTO alerts (
+ household_id,
+ triggered_event_id,
+ raised_at,
+ severity,
+ status,
+ detail
+)
+VALUES ($1, 0, now(), $2, 'active', $3);2. Dashboard reads9 queries — the demo story
+ +
+ The top panel of the dashboard. Joins alerts × households × plans
+ × regions so the SLA countdown
+ (plan.sla_seconds − age) and the dispatch-center attribution
+ are computed by the database, not the app. Ordered by severity DESC
+ then raised_at ASC — escalations first, then oldest-of-equal-severity.
+
SELECT a.alert_id,
+ a.severity,
+ a.detail,
+ EXTRACT(EPOCH FROM (now() - a.raised_at))::int AS age_s,
+ p.sla_seconds,
+ p.sla_seconds
+ - EXTRACT(EPOCH FROM (now() - a.raised_at))::int AS sla_remaining,
+ h.household_id,
+ h.address_hash,
+ p.name AS plan_name,
+ r.name AS region_name,
+ r.dispatch_center
+FROM alerts a
+JOIN households h ON h.household_id = a.household_id
+JOIN plans p ON p.plan_id = h.plan_id
+JOIN regions r ON r.region_id = h.region_id
+WHERE a.status = 'active'
+ORDER BY a.severity DESC, a.raised_at ASC
+LIMIT 25;
+ Bottom-left panel, the ticker tape. Joins events × devices ×
+ device_types × households × regions so each row renders with a
+ human-readable device code, location and region. The
+ WHERE e.kind > 0 filter drops heartbeats and
+ rides the events_kind_ts_idx index.
+
SELECT e.event_id,
+ e.ts,
+ e.household_id,
+ h.address_hash,
+ dt.code,
+ d.location,
+ e.kind,
+ e.severity,
+ COALESCE(e.battery_pct, -1),
+ r.name
+FROM events e
+JOIN devices d ON d.device_id = e.device_id
+JOIN device_types dt ON dt.device_type_id = d.device_type_id
+JOIN households h ON h.household_id = e.household_id
+JOIN regions r ON r.region_id = h.region_id
+WHERE e.kind > 0
+ORDER BY e.ts DESC
+LIMIT 25;+ Bottom-right panel auto-rotates to whichever household has the + highest-severity currently-active alert. Severity DESC then raised_at + ASC mirrors the operator's actual triage order. +
+SELECT a.household_id
+FROM alerts a
+WHERE a.status = 'active'
+ORDER BY a.severity DESC, a.raised_at ASC
+LIMIT 1;+ Runs only when the first query found no active alerts — picks the + chattiest household in the last 5 minutes so the drill-down panel never + sits empty during a demo. A small but real aggregate over the hot + table. +
+SELECT household_id
+FROM events
+WHERE ts > now() - interval '5 minutes'
+GROUP BY household_id
+ORDER BY COUNT(*) DESC
+LIMIT 1;+ Pulls the household's plan + region + dispatch center to render the + drill-down panel header (plan name, SLA seconds, armed state, region + DC). +
+SELECT h.address_hash,
+ h.armed,
+ p.name,
+ p.sla_seconds,
+ r.name,
+ r.dispatch_center
+FROM households h
+JOIN plans p ON p.plan_id = h.plan_id
+JOIN regions r ON r.region_id = h.region_id
+WHERE h.household_id = $1;
+ The "what's going on at this household right now" panel. Indexed on
+ events (household_id, ts DESC) so the LIMIT 20 returns
+ instantly even with the table at billion-row scale.
+
SELECT e.ts,
+ dt.code,
+ d.location,
+ e.kind,
+ e.severity,
+ COALESCE(e.battery_pct, -1)
+FROM events e
+JOIN devices d ON d.device_id = e.device_id
+JOIN device_types dt ON dt.device_type_id = d.device_type_id
+WHERE e.household_id = $1
+ORDER BY e.ts DESC
+LIMIT 20;
+ The two most recent storage_samples rows answer both
+ "how many rows are in events" and "how fast are they growing": the
+ table is just (sampled_at, uncompressed_bytes) pairs
+ the simulator writes every HG_STORAGE_SAMPLER_INTERVAL
+ from its in-process atomic counter, so the delta-over-time gives
+ you events/sec without ever touching events. Original
+ iterations of this query did COUNT(*) FILTER (WHERE ts >
+ now() - interval '1 second') on events directly — but
+ without an events (ts DESC) index that's a full table
+ scan, which at billion-row scale takes long enough to make the
+ footer freeze. (The published rows_per_sec is a
+ sample-interval average rather than a strict 1-second window.)
+
SELECT
+ (SELECT uncompressed_bytes FROM storage_samples
+ ORDER BY sampled_at DESC LIMIT 1) AS latest_bytes,
+ (SELECT sampled_at FROM storage_samples
+ ORDER BY sampled_at DESC LIMIT 1) AS latest_ts,
+ (SELECT uncompressed_bytes FROM storage_samples
+ WHERE sampled_at < (SELECT MAX(sampled_at) FROM storage_samples)
+ ORDER BY sampled_at DESC LIMIT 1) AS prior_bytes,
+ (SELECT sampled_at FROM storage_samples
+ WHERE sampled_at < (SELECT MAX(sampled_at) FROM storage_samples)
+ ORDER BY sampled_at DESC LIMIT 1) AS prior_ts;
+-- total_events = latest_bytes / 48
+-- rows_per_sec = (latest_bytes - prior_bytes) / dt / 48+ Same shape as the events counter but over the alerts table. +
+SELECT
+ COUNT(*) FILTER (WHERE status = 'active') AS active,
+ COUNT(*) AS total
+FROM alerts;
+ Feeds the Storage Growth gauge. Picks the freshest sample and, for
+ each window, the freshest sample at or before the
+ window-start cutoff — handles uneven sampler timing without lying
+ about the elapsed seconds (the rate is computed in Go from the two
+ timestamps the query returns, not assumed). Reads only the
+ storage_samples table; the underlying compression view
+ is hit once every 5 s by the sampler, not here.
+
SELECT
+ (SELECT uncompressed_bytes FROM storage_samples
+ ORDER BY sampled_at DESC LIMIT 1) AS latest_bytes,
+ (SELECT sampled_at FROM storage_samples
+ ORDER BY sampled_at DESC LIMIT 1) AS latest_ts,
+
+ (SELECT uncompressed_bytes FROM storage_samples
+ WHERE sampled_at <= now() - interval '1 minute'
+ ORDER BY sampled_at DESC LIMIT 1) AS bytes_1m,
+ (SELECT sampled_at FROM storage_samples
+ WHERE sampled_at <= now() - interval '1 minute'
+ ORDER BY sampled_at DESC LIMIT 1) AS ts_1m,
+
+ (SELECT uncompressed_bytes FROM storage_samples
+ WHERE sampled_at <= now() - interval '5 minutes'
+ ORDER BY sampled_at DESC LIMIT 1) AS bytes_5m,
+ (SELECT sampled_at FROM storage_samples
+ WHERE sampled_at <= now() - interval '5 minutes'
+ ORDER BY sampled_at DESC LIMIT 1) AS ts_5m,
+
+ (SELECT uncompressed_bytes FROM storage_samples
+ WHERE sampled_at <= now() - interval '15 minutes'
+ ORDER BY sampled_at DESC LIMIT 1) AS bytes_15m,
+ (SELECT sampled_at FROM storage_samples
+ WHERE sampled_at <= now() - interval '15 minutes'
+ ORDER BY sampled_at DESC LIMIT 1) AS ts_15m;3. Background maintenance4 queries — keep the demo dynamic
+ ++ Severity 1–2 alerts age out as false_alarm. Without this the operator + queue would just grow forever during a long-running demo. +
+UPDATE alerts
+SET status = 'false_alarm',
+ resolved_at = now(),
+ resolution_ms = (EXTRACT(EPOCH FROM (now() - raised_at)) * 1000)::int
+WHERE alert_id IN (
+ SELECT alert_id
+ FROM alerts
+ WHERE status = 'active' AND severity <= 2
+ ORDER BY raised_at ASC
+ LIMIT 2000
+);+ Severity 3+ alerts only get cleared after they've sat in the queue + for > 20 seconds — gives the operator panel something to look at + before the simulated dispatch fires. +
+UPDATE alerts
+SET status = 'resolved',
+ resolved_at = now(),
+ resolution_ms = (EXTRACT(EPOCH FROM (now() - raised_at)) * 1000)::int
+WHERE alert_id IN (
+ SELECT alert_id
+ FROM alerts
+ WHERE status = 'active' AND severity >= 3
+ AND raised_at < now() - interval '20 seconds'
+ ORDER BY raised_at ASC
+ LIMIT 600
+);
+ Flips ~5 % of households armed/disarmed each round, sent as a single
+ pgx.Batch. Keeps the rules engine outputs varying — door
+ opens flip between "routine" and "alarmable" without restarting the
+ sim.
+
UPDATE households
+SET armed = $1
+WHERE household_id = $2;
+ The size value is 48 × s.eventID.Load() — the in-process
+ atomic counter the producers increment per generated row, no DB
+ query needed. Earlier iterations polled
+ cedardb_compression_info (laggy because it only
+ reflects rows packed into column-store blocks) and then
+ COUNT(*) FROM events (exact but increasingly expensive
+ as the table grows past a billion rows, and a likely cause of write
+ stalls because the long scan held an MVCC snapshot). The atomic
+ counter is exact for rows that producers tried to write —
+ diverges from the table count only by the channel queue plus the
+ in-flight COPY batch (≤ ~160 K rows ≈ 8 MB at the configured
+ backpressure ceiling). sampled_at is the table's PK
+ and is unique at the 5 s sampler cadence (microsecond resolution).
+
INSERT INTO storage_samples (sampled_at, uncompressed_bytes)
+VALUES (now(), $1);4. Startup & bootstrap5 queries — once per cold start
+ +
+ Diagnostic log only — we always call ApplySchema anyway
+ because the file is idempotent (CREATE TABLE IF NOT EXISTS on
+ everything).
+
SELECT (
+ SELECT COUNT(*) FROM information_schema.tables
+ WHERE table_name = 'households'
+) = 1;
+ The embedded schema.sql is split on ; and each
+ statement is run individually under simple-query protocol — CedarDB
+ accepts DDL through that path but silently no-ops some of it under
+ extended Parse/Bind/Execute, so we force the simpler path.
+
CREATE TABLE IF NOT EXISTS plans (
+ plan_id INTEGER PRIMARY KEY,
+ name TEXT NOT NULL,
+ monthly_price_usd NUMERIC(8, 2) NOT NULL,
+ sla_seconds INTEGER NOT NULL
+);
+-- ... regions, device_types, households, devices, events, alerts ...
+-- ... plus the four supporting indexes (trimmed for the high-rate demo —
+-- every secondary index on events is write amplification at 500 K/s):
+CREATE INDEX IF NOT EXISTS events_household_ts_idx ON events (household_id, ts DESC);
+CREATE INDEX IF NOT EXISTS events_kind_ts_idx ON events (kind, ts DESC);
+CREATE INDEX IF NOT EXISTS alerts_status_raised_idx
+ ON alerts (status, raised_at DESC);
+CREATE INDEX IF NOT EXISTS alerts_household_raised_idx
+ ON alerts (household_id, raised_at DESC);
+ The three small static dimension tables are populated from
+ catalog.go using INSERT ... ON CONFLICT DO UPDATE
+ so the simulator is safe to restart against a hot database. All three
+ upserts ride a single pgx.Batch.
+
-- plans
+INSERT INTO plans (plan_id, name, monthly_price_usd, sla_seconds)
+VALUES ($1, $2, $3, $4)
+ON CONFLICT (plan_id) DO UPDATE SET
+ name = EXCLUDED.name,
+ monthly_price_usd = EXCLUDED.monthly_price_usd,
+ sla_seconds = EXCLUDED.sla_seconds;
+
+-- regions
+INSERT INTO regions (region_id, name, dispatch_center, timezone)
+VALUES ($1, $2, $3, $4)
+ON CONFLICT (region_id) DO UPDATE SET
+ name = EXCLUDED.name,
+ dispatch_center = EXCLUDED.dispatch_center,
+ timezone = EXCLUDED.timezone;
+
+-- device_types
+INSERT INTO device_types (device_type_id, code, name, default_severity)
+VALUES ($1, $2, $3, $4)
+ON CONFLICT (device_type_id) DO UPDATE SET
+ code = EXCLUDED.code,
+ name = EXCLUDED.name,
+ default_severity = EXCLUDED.default_severity;Upsert household fleet
+ +
+ Synthesised fleet from generateFleet(). Run as batches of
+ 1 000 INSERTs per pgx.Batch.SendBatch — simpler and almost
+ as fast as a staging-table COPY at this scale, idempotent on restart.
+
INSERT INTO households (
+ household_id, plan_id, region_id, address_hash, armed
+)
+VALUES ($1, $2, $3, $4, $5)
+ON CONFLICT (household_id) DO UPDATE SET
+ plan_id = EXCLUDED.plan_id,
+ region_id = EXCLUDED.region_id,
+ armed = EXCLUDED.armed;Upsert device fleet
+ +
+ Same shape as the households upsert. ~10 devices per household by
+ default; total fleet is whatever -households ×
+ -devices-per-household lands on.
+
INSERT INTO devices (
+ device_id, household_id, device_type_id, location, last_battery_pct
+)
+VALUES ($1, $2, $3, $4, $5)
+ON CONFLICT (device_id) DO UPDATE SET
+ household_id = EXCLUDED.household_id,
+ device_type_id = EXCLUDED.device_type_id,
+ location = EXCLUDED.location,
+ last_battery_pct = EXCLUDED.last_battery_pct;5. Schema reset1 query · destructive · behind a flag
+ +
+ Only runs when the simulator is started with -reset-schema.
+ Dropped in reverse-FK order so the CASCADE never has anything real to
+ do, then ApplySchema rebuilds.
+
DROP TABLE IF EXISTS storage_samples CASCADE;
+DROP TABLE IF EXISTS alerts CASCADE;
+DROP TABLE IF EXISTS events CASCADE;
+DROP TABLE IF EXISTS devices CASCADE;
+DROP TABLE IF EXISTS households CASCADE;
+DROP TABLE IF EXISTS device_types CASCADE;
+DROP TABLE IF EXISTS regions CASCADE;
+DROP TABLE IF EXISTS plans CASCADE;