Skip to content

fix(export-dynamic): verify registry artifacts before skipping workspace (RHDHBUGS-3635) - #160

Open
gashcrumb wants to merge 5 commits into
redhat-developer:mainfrom
gashcrumb:fix/RHDHBUGS-3635-verify-registry-artifacts
Open

gashcrumb wants to merge 5 commits into
redhat-developer:mainfrom
gashcrumb:fix/RHDHBUGS-3635-verify-registry-artifacts

Conversation

@gashcrumb

Copy link
Copy Markdown
Member

Summary

Fixes RHDHBUGS-3635

When determining whether to skip a workspace because its last commit is an ancestor of the last published commit, verify that all expected container images for the workspace actually exist in the registry and contain valid, non-empty dynamic package metadata (io.backstage.dynamic-packages).

If any expected artifact is missing from the registry or has invalid/empty dynamic package metadata (such as from a cancelled publish or export failure), do not skip the workspace — proceed with exporting and publishing to self-heal the missing/broken artifact.

@gashcrumb
gashcrumb requested a review from davidfestal August 26, 2026 12:57
Comment thread export-dynamic/export-dynamic.sh Resolved
@gashcrumb
gashcrumb force-pushed the fix/RHDHBUGS-3635-verify-registry-artifacts branch 3 times, most recently from 74b9cbc to 164415b Compare September 22, 2026 15:28
Comment thread export-dynamic/action.yaml Resolved
Comment thread export-dynamic/export-dynamic.sh Resolved
Comment thread export-dynamic/export-dynamic.sh Resolved
@jonkoops
jonkoops requested a review from nickboldt October 6, 2026 15:10
nickboldt
nickboldt previously approved these changes Oct 6, 2026
@nickboldt

Copy link
Copy Markdown
Member

to validate this is working we should run it against the broken scorecard images?

@gashcrumb

Copy link
Copy Markdown
Member Author

Great suggestion! We investigated this and tested against those images:

Background & Quay vs GHCR

  1. Where the broken images were: The broken scorecard images with empty io.backstage.dynamic-packages annotations were downstream Konflux builds in Quay (quay.io/rhdh/..., from RHDHBUGS-3834), caused by an empty pipeline parameter. The upstream scorecard artifacts that export-dynamic publishes to GHCR were never broken and have always had valid annotations.
  2. Image Indexes vs Manifests: Quay/Konflux builds are multi-architecture OCI image indexes (application/vnd.oci.image.index.v1+json). On an image index, the io.backstage.dynamic-packages annotation is stamped on the child platform manifests rather than the root index.

Updates in this PR

We updated the verification check (export-dynamic/verify-registry-artifacts.sh) so that when skopeo inspect --raw returns an image index / manifest list without root annotations, it resolves .manifests[0].digest and inspects the child platform manifest.

Test Results

Tested against both Quay multi-arch images and GHCR single manifests:

  1. Broken Quay scorecard image (:2.0.0--1.1.1 from RHDHBUGS-3834):
    • Inspects index $\rightarrow$ resolves child manifest $\rightarrow$ catches empty annotation "" $\rightarrow$ Correctly rejected (skipWorkspace=false, workspace cannot be skipped).
  2. Fixed Quay scorecard image (:2.0.0--1.1.2):
    • Inspects index $\rightarrow$ resolves child manifest $\rightarrow$ verifies valid base64 annotation $\rightarrow$ Correctly accepted.
  3. Valid GHCR image (bs_1.54.9__1.4.0):
    • Single manifest with annotation $\rightarrow$ Correctly accepted.
  4. Non-existent tag / missing annotation:
    • Correctly rejected.

We have also added an automated test suite (export-dynamic/verify-registry-artifacts.test.sh) covering offline mock fixtures (valid manifests, empty annotations, missing annotations, empty arrays, index dereferencing) and live registry checks, wired directly into .github/workflows/test-pack-dist-dynamic.yaml so CI validates this on every PR.

…ace (RHDHBUGS-3635)

Assisted-By: opencode
Signed-off-by: Stan Lewis <gashcrumb@gmail.com>

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
Assisted-By: opencode
Signed-off-by: Stan Lewis <gashcrumb@gmail.com>

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
…ification

- Write WORKSPACE_SKIPPED_UNCHANGED_SINCE=false to GITHUB_OUTPUT before
  exiting when skopeo is missing, so the action step captures the output
  even though the shell step runs with `|| true`; downstream steps then
  correctly treat the workspace as non-skipped instead of green-lighting
  a broken artifact.
- Remove the stray reference to a `push-container-image` input in the
  action.yaml description for `last-publish-commit`; that flag is
  hardcoded in the step env, not an action input.
- Correct the README claim that the script 'fails with an error' (true
  when invoked directly, not when called through the action) to describe
  the actual observable outcome via the action.
- Add a README note explaining that a `versions.json` bump on a release
  branch causes all workspaces to be re-exported on the next run because
  the tag prefix changes; skopeo then finds no images under the new
  prefix and the skip is bypassed for all of them.

Assisted-By: opencode

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
…ion test suite

- Support multi-platform OCI image indexes and manifest lists in verify_registry_artifact by dereferencing child manifests when top-level annotations are not present.
- Extract verification logic into export-dynamic/verify-registry-artifacts.sh following the pack-dist-dynamic pattern.
- Add comprehensive test suite in export-dynamic/verify-registry-artifacts.test.sh and wire into test-pack-dist-dynamic workflow.

Assisted-By: opencode
Signed-off-by: Stan Lewis <gashcrumb@gmail.com>

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
…fact

Merge collapsible if statements in verify-registry-artifacts.sh to resolve SonarCloud S1066.

Assisted-By: opencode
Signed-off-by: Stan Lewis <gashcrumb@gmail.com>

rh-pre-commit.version: 2.4.0
rh-pre-commit.check-secrets: ENABLED
@gashcrumb
gashcrumb force-pushed the fix/RHDHBUGS-3635-verify-registry-artifacts branch from 00c2bcc to 81a2a1e Compare November 4, 2026 15:07
@gashcrumb

Copy link
Copy Markdown
Member Author

Rebased onto main at f2f7098. The conflicts with #182 are resolved and the PR is mergeable again.

How the conflicts were resolved

Every #182 hunk is byte-identical to main; the diff against main is purely additive.

Relationship to #182 — complementary, not superseded

Worth stating explicitly now that both are in tree, since the two gates sound similar:

#182 validate-plugin-payload.sh this PR verify-registry-artifacts.sh
Inspects local artifact, via containers-storage: / docker-daemon: and an oci-archive layer check remote image, via skopeo inspect --raw docker://...
Runs after export and package, before push before deciding to skip, with no export or push
Answers "is what I just built non-empty?" "does a usable image already exist, so may this workspace be skipped?"

#182 cannot cover this PR's cases: it never contacts a registry, so it cannot see a tag that was never pushed (failed sibling run, registry GC, manual deletion), a tag-prefix bump where an ancestor check says "unchanged" but the new tags do not exist yet, or images published before #182 landed.

The one genuinely shared piece is the annotation-emptiness predicate. It is kept here deliberately — it is the only recovery path for images pushed before #182. If we would rather have a single gate, this is the piece to drop, keeping only the existence probe; say so and I will follow up.

Verification on the rebased branch

validate-plugin-payload.test.sh (17 cases) and verify-registry-artifacts.test.sh (offline plus live registry) both pass, shellcheck -x is clean on the touched scripts, and actionlint is clean on the changed workflow.

Two follow-ups this raises, not addressed here

  1. The skip path duplicates the command -v skopeo guard rather than calling _require_skopeo, because it must also emit WORKSPACE_SKIPPED_UNCHANGED_SINCE=false before exiting. Worth unifying.
  2. feat(export-dynamic): reject hollow OCI payloads before push #182 removed a || true that previously swallowed the non-zero exit when skopeo is missing, so that path now fails the action with an empty failed-exports. The README wording still holds, but downstream always() steps are what actually matter now.

@sonarqubecloud

sonarqubecloud Bot commented Nov 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants