Skip to content

Add a Postgres event log and live agent events #84

Open

Description

@jacobprall

Goal

Turn the factory into a complete reference for an agentic app on Render, with one source of truth and agents you can watch, starting with the agent where it runs today, in the Sandbox:

  • Postgres holds an append-only event log. Projections (tables derived from the log in the same transaction) answer every "what is true now" question: who owns an issue, which run is active, which Sandbox it holds, what an agent run cost.
  • GitHub keeps repository content and the artifacts people read. The status comment becomes a view rendered from Postgres, not a lock.
  • Agents run on Pi in the Sandbox, beside the checkout. The runner reports events on its output, and the Workflows host relays them into the log as they arrive. Every agent run streams live over server-sent events (SSE), per agent run or per issue and pull request.

The design borrows from render-agents (event log, projections, live deltas over NOTIFY, the runner-as-sidecar idea) and keeps the factory's own strengths: the run kernel, typed submit, validated publishing, and the egress allow-list from #82.

Plan

Step Work Depends on Outcome
1 Postgres event log and projections none The log with per-stream versions and ordered reads, an event catalog, projections, migrations at startup, PGlite tests
2 Run coordination on Postgres 1 Claims, cancellation, and Sandbox tracking move off the status comment
3 Live agent events 1 Every agent run recorded in the log, token deltas, SSE per agent run and per subject
4 Live relay from the Sandbox 3 The runner's events reach the log while it works, through the Workflows host
5 Pi session driver (optional) 4 Pi 1.5.2 through createAgentSession

Steps 2 and 3 can run in parallel after step 1.

Deferred

  • Workflow placement: Pi in a Workflows task with tools in the Sandbox, and an allow-list without model hosts. Held until the agent in the Sandbox is solid; the latency and rate-limit spike results below are kept for it.
  • agentTask() and a placement demo: a generic agent subtask and a label-triggered side-by-side comparison. Waits for the workflow placement.
  • Spec approval and artifact markers in Postgres: removes little GitHub reading for the state it adds.

Design

Decisions

  • Postgres is required and is the source of truth for factory state. Every state change is an event; current state is a projection updated in the same transaction. Both services apply migrations at startup under an advisory lock.
  • Ordered reads. Each stream has a version with a unique key, and appends can require the version they expect. Single-stream readers page by that version; per-subject readers page by transaction and read only below the oldest running transaction, so a late commit can never be skipped.
  • Nothing reads a GitHub comment to decide ownership. A claim is an event on the subject's stream, fenced by the stream's version. A new run takes ownership first and then stops the previous run, so the previous run is cut off even if cancelling it fails. The status comment and factory:working are synced from Postgres, best effort.
  • Only the Workflows host and the gateway write to Postgres. Nothing in a Sandbox gets a database credential, an endpoint to post to, or a token, and the Sandbox allow-list doesn't change.
  • Runner output is untrusted. The host turns it into assistant.*, tool.*, and usage events for the run's own agent run, and nothing else. Outcomes, costs, and ownership come from the host's own reading of the root-owned result file.
  • Token deltas are streamed, never stored: NOTIFY on a live channel plus an in-memory cache of in-progress messages per gateway instance, as in render-agents.
  • Every agent run streams, including existing workflows through runAgent. The status comment shows the subject's stream path.
  • Pi stays on today's driver until the live relay lands. The upgrade to createAgentSession comes last; its spikes passed.
  • SSE only, with a bearer token for readers.
  • No fallbacks for state written before the upgrade: let runs finish before deploying.

Endpoints (gateway)

  • GET /agent-runs/:id/stream: one agent run; event IDs are stream versions; closes after agent_run.finished.
  • GET /subjects/:owner/:repo/:kind/:number/stream: everything on an issue or pull request (claims, task runs, agent runs); stays open.

Both send the stored backlog after Last-Event-ID, then the text so far of in-progress messages, then live events.

Spike results (November 16, all passed)

  1. Disconnected commands. A Sandbox command kept running for 420 silent seconds after its client disconnected, with no gaps in its output, and the Sandbox stayed running past its 300-second idle timeout. tail -c +offset -f --pid resumed at exactly the next line. The live relay can drop and resume without losing or repeating output.
  2. Ordered reads. On real Postgres, paging by seq alone lost a late-committing event; the (tx, seq) watermark read delivered it. Racing appends to one stream version conflict instead of skipping a version.
  3. PGlite. xid8, the watermark query, unique conflicts, advisory locks, and LISTEN/NOTIFY (delivered on commit) all work, so npm test needs no setup. It runs one transaction at a time, so race tests run against real Postgres in CI.
  4. Workflows to Postgres. Render documents that Workflows can reach a database's internal URL; the Blueprint with fromDatabase on both services validates against the CLI and the published schema. npm run doctor confirms it live on the first deploy.
  5. Pi 1.5.2. A Proxy over ModelRuntime.streamSimple enforces the turn limit exactly as today. With discovery turned off, a checkout's AGENTS.md, skills, and .pi/extensions/ never load (a control run with the flags off loaded and ran them). The runner still bundles into one file (14.5 MB, up from 6.6 MB) that runs in a Sandbox.
  6. Remote tool latency (for the deferred workflow placement). About 0.3 seconds per command, read, or listing and 0.5 per write; 130 back-to-back commands ran at 246 a minute with no rate limiting. The factory's one-request-a-second client pacing would dominate if tools went remote.

What this unlocks

Seeing agents work

  • Follow any agent run, or everything happening on an issue, live with curl -N: tokens as the model writes them, each tool call and result, claims and task runs. Today nothing is visible until a run ends.
  • Resume after a disconnect; join mid-message and get the text so far.
  • Every run, tool call, and final message stays queryable afterwards, not only in log lines.

Answering questions with one query

  • Cost per issue, per agent, and per model.
  • Which runs are active, what each is doing, and which Sandbox it holds.
  • How a subject reached its current state: who triggered what, which run superseded which, who cancelled.
  • Turn counts, failure codes, and timings across runs, for tuning prompts, turn limits, and models.

Coordination that doesn't depend on GitHub comments

  • Ownership is one row read instead of listing and parsing comments before every write.
  • Safer supersession: the previous run is fenced immediately.
  • Fewer GitHub API calls and less rate-limit pressure.
  • Editing or deleting the status comment can't confuse ownership, and a GitHub outage can't fail a run.
  • Redelivered webhooks are recorded once, keyed by delivery ID.

Security

  • Nothing new enters the Sandbox: no credential, endpoint, or allow-list entry.
  • Runner output can't change outcomes, costs, or ownership.
  • After the Pi upgrade, no repository file loads as a prompt, skill, or extension.

Platform

  • One set of patterns (event log, projections, live deltas) shared with render-agents.
  • A full Render stack in one Blueprint: web service, Workflows, Sandboxes, and Postgres.
  • Pi 1.5.2 on its SDK, matching render-agents.

Next steps it makes possible (not in scope)

  • The workflow placement and the placement demo (deferred above).
  • A web dashboard on the SSE endpoints and projections.
  • Budgets and spend alerts per repository or issue from agent_runs.
  • Evals and regression tracking by replaying and comparing recorded runs.
  • Notifications (Slack and others) from the log.
  • Triggers beyond GitHub (Linear, Slack, schedules) feeding the same coordination.

Trade-offs and risks

  • Postgres becomes required infrastructure. Free databases expire after 30 days, so the Blueprint uses a paid instance, which the Deploy to Render button will create.
  • The gateway, which today holds only the webhook secret, gains a database credential that can write factory state. A compromised gateway can forge events.
  • Stream readers share one bearer token, and events can quote repository content and tool output.
  • Ownership is still check-then-act against GitHub writes; the window is narrower, not closed. Publishing still refuses to overwrite commits the factory didn't make.
  • A task Render cancels without running finally leaves its claim held until the next run or /factory cancel resolves it, as today.
  • The Pi upgrade roughly doubles the runner bundle, uploaded once per snapshot.

Not planned

The render-agents reconciler, sandbox generations, credential brokering (tracked in #45), chat with follow-up and steer messages, approvals, agent-to-agent messaging, and its web UI. Render Key Value for delta replay.

Activity

  1. added
    enhancementNew feature or request
    bothNeeds a person for part of it; an agent can do the code
    on Nov 17, 2026
  2. changed the title [-]Add a Postgres event log, live agent events, and a choice of where Pi runs[/-] [+]Add a Postgres event log and live agent events[/+] on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bothNeeds a person for part of it; an agent can do the codeenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions