Skip to content

feat(breg): add read-only external views over Adapter projections - #2634

Open
jeremi wants to merge 62 commits into
mainfrom
feat/breg-external-views
Open

jeremi wants to merge 62 commits into
mainfrom
feat/breg-external-views

Conversation

@jeremi

@jeremi jeremi commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Pull Request

Summary

Adds read-only external views to BReg. A registry can show and filter
selected facts another institution keeps, through an Adapter projection,
while it stores only its own facts and the identifiers that link them.
Owning area: BReg (registry-breg, registry-bregctl), with the Adapter
projection contract it reads through.

  • Views and links. A package declares external record types,
    projections (live or indexed, one or many) and views whose links read
    through them. Indexed views list, filter, sort and count over a kept
    copy; live views read on demand.
  • Authority changes rebind; operational changes do not. A projection's
    binding and grant fingerprints cover the pack contract, the operation
    and the allowed fields, and a changed cache lifetime starts a fresh
    epoch. Renewing a grant cutoff, replacing a CA bundle, moving an
    endpoint or changing acquisition limits keeps the cache. Each projection
    is checked on its own, so a mismatch takes only its views offline.
  • Deployment configuration is the only cutoff. Removing a binding and
    restarting stops acquisition; its views answer source.unavailable.
    bregctl external-projection withdraw purges retained values but does
    not stop acquisition. Backups exclude projection data, so a restore
    starts with no source values, as on a fresh install.
  • One cache lifetime, which a deployment may shorten but never extend.
    A successful source response marked no-store never enters the store
    and removes the previous snapshot. A no-store on an error, a redirect
    or the challenge that starts a credential refresh changes nothing, so a
    refresh that fails keeps the previous snapshot. A credential failure
    during an acquisition fails the projection, even when a route step
    catches it.
  • Operator commands: bregctl external-projection inspect|withdraw|rotate,
    doctor checks with the grant horizon, and monitoring codes.
  • Link keys. A record with no link key is covered, with no outside
    value, and is never sent to the source. A malformed key (over 512 bytes,
    surrounding white space, a control character) is never sent either: that
    record alone reads as having no outside value, the refresh and live reads
    count and log it as invalid-key, and lists, filters, sorts and counts
    keep serving the other records. Writing a malformed key to a field an
    external view links from is refused as request.invalid on create,
    patch, action, proposal, batch and import writes; proposals and action
    input are refused when submitted, before anything is stored. The same
    value in a field no view links from is accepted.
  • Decimals from an external source are served as JSON strings on live
    and indexed paths. Integers keep their exact digits; other numbers are
    spelled without an exponent, so one very small or very large value no
    longer fails its batch.
  • Continuations. A cursor is issued only for a sort value the next page
    accepts. A page whose last row carries a sort value too long to resume
    from answers source.unavailable with no nextCursor, after the earlier
    pages were served.
  • Scale. Refresh and label pages are bounded to their continuation,
    and indexed reads look up each root's snapshot by primary key. Scale and
    recovery runners live under products/breg/loadtest/external-views/.

Evidence

Run locally against a disposable PostgreSQL database.

On the branch head:

Claim Command Result
Formatting cargo fmt --all -- --check pass
Lints cargo clippy -p registry-breg --all-targets -- -D warnings, again with --features postgres-test; cargo clippy -p registry-adapter-engine --all-targets -- -D warnings pass (3 runs)
Adapter engine cargo test -p registry-adapter-engine 408 passed
BReg library tests against PostgreSQL, including external projection maintenance cargo test -p registry-breg --lib --features postgres-test,tooling,schema 763 passed, 1 ignored
BReg write paths and external views cargo test -p registry-breg --features postgres-test --test postgres_mutation; the same with --test external_views 20 passed; 8 passed
Product contracts, including SEC-206 traceability products/breg/scripts/check-contracts.sh; products/adapter/scripts/check-contracts.sh pass. On macOS the webhook receiver test needs the FIPS crypto library on the loader path, so the BReg script's Python steps ran again with it; the Adapter corpus holds 191 members, including the adapter.pack.projection-invalid member this push adds
Load-test runners python3 -B -m unittest test_runners test_scale in products/breg/loadtest/external-views/ pass

On an earlier head of this branch, before the last review round; CI runs
them on this head:

Claim Command Result
Full BReg PostgreSQL suite products/breg/scripts/test-postgres.sh 1,236 passed, 0 failed, 9 ignored (opt-in: installed Node/Python packages, 5 perf benchmarks, generated consoles, the geojson example)
bregctl cargo clippy -p registry-bregctl --all-targets -- -D warnings; cargo test -p registry-bregctl --test cli; cargo test -p registry-bregctl --lib pass; 186 passed; 629 passed (the lib run needs a short TMPDIR on macOS for one socket path test)
Docs site pages changed here npm run check in docs/site pass (markdownlint 0, Vale 0 errors, astro check 0 errors, built links pass)

Tests covering the review fixes:

  • link keys: a_root_without_a_link_key_is_covered_and_reads_absent and
    the invalid-key refresh and live-read tests in external_runtime.rs;
    invalid_label_selectors_are_unavailable_without_failing_the_rest_of_the_label_batch;
    an_unlinked_root_reads_as_a_complete_empty_related_collection;
  • link keys on write: a_malformed_link_key_is_refused_when_a_patch_proposal_is_prepared,
    a_malformed_link_key_is_refused_when_a_create_proposal_is_prepared,
    a_malformed_link_key_is_refused_when_an_action_builds_its_effect_document,
    and the_write_path_refuses_a_malformed_link_key_on_the_linked_field_only
    (red when the write-time check is removed);
  • decimals: exponent_range_decimals_are_served_as_plain_strings;
  • continuations: a_cursor_is_issued_only_for_a_sort_value_it_will_accept;
  • acquisition: a_caught_failure_of_the_initial_authorization_fails_the_acquisition,
    a_replayed_response_that_is_not_data_cannot_forbid_retention,
    a_202_marked_no_store_keeps_the_previous_snapshot (each red when the
    guarded line is reverted);
  • bounded pages and index use: the dense-scope EXPLAIN tests in
    external_label_acquisition.rs and
    an_indexed_read_looks_up_each_root_snapshot_by_its_primary_key;
  • operator codes and readiness: external_projection_failures_each_carry_one_code,
    a_refused_runtime_configuration_is_reported_as_status_reports_it,
    external_projection_operations_report_database_prerequisites_with_the_shared_codes,
    a_moved_activation_is_reported_as_not_ready.

The planner-shape tests depend on PostgreSQL planner behaviour (default
selectivity, collation match); comments in the tests say so. They forbid
a plain Sort and bound rows read, and allow an Incremental Sort.

Notes

  • Malformed link keys: a key stored before its field was linked by a
    view keeps reading as no outside value until it is corrected; the
    refresh report and logs give the count. A proposal stored before the
    link carries its key to Apply, where the write is refused as
    request.invalid. Drafts validate inputs, not entity fields, so a draft
    accepts a malformed key until it is submitted. A mutation in an action
    hook's answer is refused as hook.proposal.mutation_refused, and
    history-migration SQL steps write without the check; reads guard those
    rows. An invalid label key still leaves only that related row without
    its label.
  • Decimal precision: a source decimal is read as a double, so a value
    with more than about 15 significant digits is rounded before BReg serves
    it. BReg does not accept decimal strings from a source.
  • $skiptoken length: a cursor payload may reach 8 KiB, which is about
    11 KiB once encrypted and base64url-encoded in the URL. A proxy with an
    8 KiB request-line limit refuses such a next page. Only sorts over long
    text values reach that size; a deployment that sorts on them should
    raise the proxy limit.
  • no-store on an unaccepted 2xx: egress honours no-store on any 2xx,
    including a status the pack does not accept, so such a response removes
    the previous snapshot. This fails closed (data the source asked not to
    keep is dropped), and is left as it is.
  • CodeQL: three high rust/cleartext-storage-database alerts point into
    #[cfg(test)] PostgreSQL test modules (external_view_query.rs,
    external_projection.rs, external_label_acquisition.rs); they are
    test fixtures, not runtime storage, and are to be dismissed as used in
    tests.
  • Adapter corpus: the registered adapter.pack.projection-invalid
    code had no bad-authoring corpus member, so the Adapter contracts failed
    at their last step. This push adds the member (an exact projection
    whose selector and record key name different schemes) and its README
    row.
  • Rust policy: the failing check on this PR is fixed by the dependency
    update in Support Studio review, action dates, and workflow timing #2630.
  • Overlap with Support Studio review, action dates, and workflow timing #2630 on 11 files (Cargo.lock, BReg API and model,
    registry-bregctl/src/lib.rs, the CLI docs catalogue, the security
    matrix and traceability files, the authoring schema). Whichever merges
    second rebases.
  • Compatibility: bregctl external-projection reports a refused
    runtime file with the same codes and exit status (1) as bregctl status.
    external-projection.unavailable is now the doctor warning only.
  • Backups: the backup page shows the four --exclude-table-data
    options for packages with indexed projections; restore assumes them.
  • Follow-ups filed: bregctl dev: a detached supervisor can inherit another start's output pipe, hanging parallel installed-dev tests #2631 (dev supervisor pipe inheritance hang),
    test(breg): the source successor test and frozen_rhai_package leak a container, a volume and a temp dir every run #2632 (test leaks), evidencectl dev: reading a state file refuses a valid file renamed over between lookup and open #2633 (evidencectl state read false refusal).
  • Found, not in this PR: products/breg/scripts/test-postgres.sh does
    not run registry-breg --lib, so the external projection maintenance
    PostgreSQL tests run only in CI, and they need the tooling feature
    (with postgres-test alone the filter matches nothing). Clippy with
    postgres-test,tooling,schema --all-targets fails on a type-complexity
    lint in crates/registry-breg/tests/postgres_migration.rs, code already
    on main.

DCO

  • Every commit includes a Signed-off-by trailer.
  • I reviewed the submitted changes and am responsible for the contribution.

Baseline of the external record projections candidate: BReg serves
read-only views over records an Adapter projects from an external
source, with grants checked on every read, sealed page cursors and a
retained projection store. Review fixes follow in separate commits.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A cached external value now lives for the authored cacheTtlSeconds,
which a deployment binding may only shorten, and is purged when it
expires. A separate retention window, refresh interval and purge
deadline added configuration without adding protection, so they are
gone from the binding, the store and the source facts.

The deployment allowedSelectors list is also removed. Local row
authority already decides which roots may be acquired.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Scope fingerprints now cover only the reviewed source contract, its
operation and the allowed fields, so a renewed grant cutoff, a CA change or
a connection edit no longer discards the cache. Each projection binds on
its own: a contract mismatch rotates to a fresh epoch, and a projection
that still cannot bind disables only its views with an unavailable
Problem that names it. Startup and doctor report unavailable projections
with the reason.

Deployment configuration is the only acquisition cutoff: withdrawal purges
data and a binding still configured binds again at the next start.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The bregctl generate target list gained external-source-facts with external views; the published reference test still expected the earlier list.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Operators maintained a projection cache scope through a psql script that
took digests, epochs and activation IDs copied by hand. bregctl now reads
the runtime configuration, computes the binding and grant digests and the
effective cache lifetime as startup does, and acts on one indexed
projection under the configured migration role. Withdraw and rotate take
the registry lock, check the active package, activation and managed
catalog in the changing transaction, and are audited under
breg-external-projection-audit/v1. Inspect reads the scope and fixed
counters only.

The scope script is removed; the recovery drill and the operator guide use
the bregctl commands.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Incomplete indexed coverage now returns an external_view.incomplete
Problem instead of a bare status body. Scope lock and database failures
on the read path answer source.unavailable rather than incomplete
coverage or an invalid cursor; only a withdrawn or rebound projection
keeps the detail fallback and cursor refusal.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
An unavailable, incomplete or unbindable refresh for one key no longer
removes the previous snapshot or bumps the scope generation. The snapshot
stays readable until its own expiry; only explicit denial or no-store
removes it, so a transient source failure no longer turns covered indexed
reads into incomplete coverage.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
… refreshes

The scope generation now moves only on rotation, withdrawal and contract
reset, so a reader's continuation survives concurrent publication. A
label sweep confirms that the exact parent snapshots supplying its
selectors are still current before each source call, instead of stopping
whenever any parent root refreshed. Publication still rechecks current
membership in the same statement. A database failure while rechecking
the parent is reported as unavailable instead of abandoning the page.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The test-only helper publishes without the root eligibility statement
that runtime publication always wraps around the store function. Its
name now says so, so a reader cannot mistake it for the guarded path.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A refresh receipt now stays live for the time left before the page
deadline plus a ten second margin, instead of a fixed two minutes. A
page that runs longer than the old lease but inside its own deadline
still publishes, and a short page no longer holds receipts live longer
than it can use them. The store refuses leases outside one second to
one hour.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A refresh page published every key in one transaction, holding the
projection scope row from its first publication to commit. Indexed reads
share that row, so they waited for the whole page and fell into their
lock timeout. Each outcome now commits on its own, after rechecking the
projection epoch and, for label sweeps, the parent generation.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
An HTTP continuation under another principal or another row boundary
claim answers query.cursor_invalid, and the original caller continues.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A date's text output follows the session DateStyle, so a cursor issued under one style failed to continue under another. The cursor now carries the ISO 8601 form, which the continuation parses under any style.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The cursor binding used the operator's Rust Debug form, which changes with a rename. It now uses the contract names equals and in, as query bindings do.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A dense eligible scope already pages in primary-key order without a sort, and a selective row boundary pages faster with a bounded sort over its few rows than with a forced primary-key walk. The page query no longer turns sorting off; a test pins the dense-scope plan.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ding

A source reply marked no-store refuses an indexed acquisition through the bound source, and the refresh records it as a retention refusal, which removes the previous snapshot.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
No caller ever built the no-store variant; the Adapter refuses a no-store reply before a snapshot exists. The store keeps its check that retention covers the source expiry.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Operators could not tell why external projection data went stale or
stayed stored: a failed purge pass logged no cause, refresh outcomes were
not logged, and a failed live acquisition was discarded.

Each purge pass now logs the rows it removed per projection and reason
(expired, replaced, withdrawn), a failed pass logs its failure category,
and a pass that reaches its batch bound warns. Each refresh page logs its
published count and its failed keys per reason, a refresh that fails as
a whole logs its reason, and a failed live acquisition logs its reason
instead of being dropped. No line carries a source key, selector, value
or secret.

A startup check also reports, for each projection a server would serve,
the source grant expiry and the observation time of the newest snapshot
its active epoch holds.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
bregctl doctor now adds an external-projection.status advisory for each
projection a server would serve. It names when the source grant expires
and, for an indexed projection, when the newest snapshot it holds was
observed, so an operator sees a lapsing grant or a stalled refresh sweep
before readers do.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…itted

The storage guide told operators to monitor purge failures and backlog
against an erasure margin that no setting defines. It now states the purge
worker's actual bounds and lists the log lines, metric and doctor
advisory BReg emits for refresh and purge, with their reason categories.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
`dev start` polls state.json while its supervisor saves it by renaming a
successor over the path. A poll that opened the earlier file and then looked
the path up again saw the successor's inode and refused the start with
"local state changed while opening it". The read now checks the opened
descriptor itself: owner-only, regular, and at most one link, where zero
links means a writer replaced the path after the open and the bytes are the
complete earlier version. The pre-open path check and NOFOLLOW stay.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Both retention workers and the external projection purge classify PostgreSQL failures with it, so it belongs with the PostgreSQL kernel rather than in the hook dispatcher.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Four library modules include the harness file, each with its own counter, so two copies could build the same role name from one process id and clock reading. A random suffix needs no state shared between the copies.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…undary

The refresh page query joins the source view to the base table on the record id and bounded only the view. PostgreSQL does not carry that inequality across the join, so the merge join scanned the base table from its first record on every page and paging cost grew with the square of the scope. In the dense test the base table scan read 9555 rows plus 497 filtered for a page of 101 after the midpoint; with the bound repeated on the base table it reads about one page.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The live, no-store path had no end-to-end HTTP test. Read a live link
through the router with a granted reader and a recorded source exchange:
the answer comes from the source, nothing lands in the external
projection tables or any other table, and reads the grant or row
boundary does not cover are refused as problems before reaching the
source.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…itoring

Operators had no row for the retention worker an indexed external
projection starts, and no entry for the two doctor codes it reports.
Describe the worker's series, its pass signals and bound, the
external-projection.status advisory, and the unavailable warning, in
line with the storage guide's monitoring advice.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
A root whose link field is empty or unset names no outside subject, but it
counted as uncovered, so every list, filter, order and count over its view
answered incomplete, and a live read of it answered unavailable. It now
counts as covered with its external fields absent and is never sent to the
source.

A link key the store or the source contract refuses made the refresh page
fail as a whole, so the sweep could not move past it. Each key is now
checked before acquisition: an invalid key counts as unavailable for its
root, is logged as invalid-key with a count and no value, and the rest of
the page refreshes. The view stays incomplete until the key is corrected.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T14:36:34.140705Z 3a4459d New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b12fb60c29

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

That resolver would take the authorised local reference and return a source-scoped handle
before Adapter acquisition. The current checker refuses unknown binding configuration.

Write access to a link's `fromField` chooses which outside subject BReg fetches. Whoever can set that field on a root decides which subject the source is asked for under the deployment's source grant, and which subject's values readers of that root then see. Grant write access to a link field only to identities trusted to assert that link, and review changes to it as you would review a disclosure.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the new operator obligations to the BReg security model

Indexed external views introduce new disclosure and storage obligations: writers of fromField choose the outside subject disclosed, projection tables must be excluded from logical backups, and source-grant withdrawal must be reflected in deployment configuration. These are documented here, but products/breg/SECURITY-MODEL.md remains unchanged and neither describes nor links these obligations. Update that central security model in this change, as explicitly required by products/breg/AGENTS.md, so operators following its backup and authorization guidance do not miss the new requirements.

AGENTS.md reference: products/breg/AGENTS.md:L210-L211

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in this PR. The BReg security model now lists the three external view obligations: grant write access to a link's fromField only to identities trusted to assert that link, keep the projection tables out of logical backups, and withdraw a source grant in deployment configuration. Each links to the storage contract.

Comment on lines +229 to +230
ExternalFieldType::Decimal => {
json!({"type":"string","pattern":"^-?[0-9]+(\\.[0-9]+)?$"})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Align decimal responses with their generated OpenAPI schema

For any external field declared as decimal, the generated response schema requires a JSON string, but external_source::typed_value accepts only JSON numbers and neither response path converts them: live reads return the source fields directly, while indexed reads preserve the JSONB values through project_fields. Consequently, successful decimal responses violate the advertised contract and fail validation or deserialization in schema-derived consumers. Normalize decimal values to the declared string representation, or change the schema consistently with the intended wire format.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Decimal external fields are now served as JSON strings on the live and indexed paths, matching the generated schema, so a schema-derived client reads them without losing precision. Integers keep their exact digits; other numbers are spelled without an exponent, so a very small or very large value no longer fails its batch. A source decimal is read as a double, so more than about 15 significant digits is rounded; the storage contract states that bound.

Comment on lines +561 to +565
ExternalFieldType::String => value
.as_str()
.filter(|s| s.len() <= 4096)
.map(str::to_owned)
.ok_or_else(invalid)?,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow admitted sort values to round-trip through continuations

When a sortable external string exceeds 4096 bytes, acquisition and storage can accept it because external string fields have no corresponding length ceiling. A first sorted page ending on, for example, a 5000-byte value can successfully issue a cursor, but continuation_sql subsequently passes that cursor's sort value through scalar, where this filter rejects it; the HTTP layer then returns source.unavailable instead of the next page. Separate caller-filter limits from internal continuation validation, or enforce a compatible bound before accepting sortable source values.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A continuation now resumes from a sort value of up to 6144 bytes, separately from the caller-filter limit. The server issues a cursor only for a sort value the next page accepts: when the last row of a page carries a longer value, or its cursor would exceed the 8 KiB payload, that page answers source.unavailable with no nextCursor, after the earlier pages were served. The storage contract says so.

Comment on lines +3542 to +3544
let successful = matches!(&reply, PhysicalSendReply::Ordinary(Ok(reply))
if (200..300).contains(&reply.status) && reply.status != 202);
projection.observe_response(successful);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve successful OAuth credential-refresh retries

For a projection using OAuth2, a source 401 followed by a successful automatic credential refresh still makes the entire acquisition fail. This observer permanently sets ProjectionHttpState.failed before AuthPlan::on_upstream_status selects RetryWithFreshCredential; the existing retry branch refreshes the credential and repeats the send, but ProjectionProgram::execute rejects the final successful result because the failure flag remains set. Record terminal send failures after credential recovery, rather than poisoning the acquisition for a 401 that the engine successfully handles.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. A 401 that starts a credential refresh is no longer recorded as a failure; the retried send is observed instead, so a refresh that succeeds keeps the acquisition. In the other direction, any credential failure during a projection acquisition, including a token endpoint failure that a route step catches, now fails the projection, so partial data is never stored as complete.

Comment on lines +395 to +399
if http.failed.load(Ordering::Relaxed)
|| http.successful.load(Ordering::Relaxed) == previous_successes
{
return Err(ProjectionError::Unavailable);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor observed no-store restrictions before returning route errors

When an indexed refresh receives Cache-Control: no-store together with an unsuccessful response, or the route fails after receiving that header, these early returns classify the acquisition as Unavailable before checking the recorded retention restriction. BReg maps that error to RefreshFailure::Unavailable, which deliberately preserves the previous cached snapshot, whereas ProjectionError::NoStore maps to the retention-refusal path that removes it. Check the trusted no-store flag before the HTTP-success and route-result checks so a source's explicit storage restriction is not lost merely because the same refresh also failed.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly applied. A no-store on a successful data response now wins over any later failure in the same acquisition, so a route failure after that header still removes the previous snapshot.

We did not apply it to unsuccessful responses. The storage contract keeps the previous snapshot when a refresh fails because the source is unavailable, and RFC 9111 applies no-store only to the response that carries it. Many sources mark every error response no-store (Registry Stack's own problem responses do), so honouring it there would purge the kept snapshot on every transient 503, 429 or token rejection. A no-store on an error, a redirect or the 401 that starts a credential refresh now changes nothing, and the storage contract states the rule.

…boundary

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…ntions lint

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…e them on write

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…feature

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…efresh

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…o-store win over failures

A 401 answered by a successful OAuth credential refresh no longer poisons the acquisition, and a no-store response now yields NoStore even when the status or route failed.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…in the load-test runners

Resolves the CodeQL uninitialized-local and unused-import alerts without changing what the runners do.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tion on credential failures

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…pt snapshot

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tion builds its record

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…of failing the batch

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…accepts

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…isition

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
The corpus gate requires every registered authoring code to have a member; adapter.pack.projection-invalid had none, so the Adapter contract script failed once the conventions lint passed.

Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a4459da23

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

let mut parts = digits.split('.');
let integer = parts.next().unwrap_or("");
let fraction = parts.next();
if value.len() > 128

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow admitted decimal sort values to resume pagination

For an indexed sortable decimal such as 1e-200, the new decimals_to_wire normalization stores a valid 202-byte plain decimal string. A sorted first page ending on that value successfully issues a cursor because sort_value_resumable accepts up to 6144 bytes, but continuation_sql subsequently calls bounded_scalar, where this unconditional 128-byte decimal limit rejects it. The next page therefore returns source.unavailable despite unchanged, complete coverage. Apply a continuation-compatible decimal bound, or enforce the same restriction before admitting source values and issuing cursors.

Useful? React with 👍 / 👎.

Comment on lines +453 to +455
|| !self
.fields_schema()
.validate(&Value::Object(returned.clone()))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate field subsets against a projected source schema

When an otherwise accepted fieldsSchema declares multiple required properties, a live view granting only a subset of those fields can never return a present record. check_request accepts the narrower requested set, and BReg passes only the profile's readable fields, but this validation applies the entire schema, so omitted, unrequested required properties cause InvalidResult. Returning those properties instead also fails the exact requested-field check above. Validate against a schema projected to the requested fields, or reject incompatible schemas during binding, rather than accepting a binding whose narrower live views always answer unavailable.

Useful? React with 👍 / 👎.


fn audit(fixture: &Activated) -> RegistryAudit {
fixture.database.audit(
AuditProfile::production_from_secret_bytes(vec![0x4e; 32].into())
Comment on lines +483 to +486
include!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/support/postgres_harness.rs"
));
Comment on lines +880 to +883
include!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/tests/support/postgres_harness.rs"
));

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants