Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/publish-gemara.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@
# Trusted CI publisher for the target namespace. Without that binding the hub
# returns 403; adding a GitHub secret will not fix it.
#
# Staging: grcli pushes each bundle to staging/<namespace>/<name>
# and the hub copies it into <namespace>/<name> when the sync accepts it. A
# grcli-version from before the staging cutover gets 401 UNAUTHORIZED at its
# first blob push against a cut-over hub; set grcli-version to a release that
# pushes to staging.
#
# --- Call it from a catalog repo ---
# name: Publish catalog
# on:
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,22 @@
Notable changes to `grcli`. This project is pre-1.0; while on `v0.x`, a breaking
change bumps the minor version.

## Unreleased

### Changed

- **Breaking:** `publish` pushes every bundle and its signature to the staging
repository `staging/<namespace>/<name>` and then syncs `<namespace>/<name>`
(the `staging` package of grc-store-protocol v0.28.1 names the repository). There is no fallback to pushing
the served repository, so this release cannot publish to a hub from before
the staging cutover, and older releases cannot publish to a hub after it
(their first blob push gets `401 UNAUTHORIZED`).
- The hub client's timeout rises from 60 seconds to 3 minutes, above the
hub's 120-second catalog sync deadline.
- A refused push names the staging repository and the namespace you need to
own, and a refused sync shows the hub's error code and says whether to
publish again.

## [0.1.0] - Unreleased

First release of `grcli` under `github.com/gemaraproj/grcli`, published as a
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ a user command.)
`hub` (`/v1/bundles/sync`, `/v2/token`, discovery) · `registry` (OCI packing via `oras-go`) ·
`sign` (cosign shell-out, Sigstore bundle) · `provenance` (SLSA v1.0 predicate) · `source` (load/merge/verify YAML) · `refs` ·
`digest` (SHA256) · `cache` (immutable-tag disk cache — v2 stores the whole bundle: files +
`bundle.json`). Imports `grc-store-protocol` (discovery, syncapi, registrytoken, spdx).
`bundle.json`). Imports `grc-store-protocol` (discovery, syncapi, registrytoken, spdx, staging, apierror).
`publish` pushes to `staging.Repository(<ns>/<name>)` and syncs the served name, with no fallback.
**Auth is no longer here**: `internal/auth` (device flow, credential store, token resolution, GHA
OIDC) was deleted in favour of `github.com/gemaraproj/grc-store-clientkit/auth`, shared with
privateer-sdk. `cmd.grcliApp` (`cmd/root.go`) is the per-tool identity that keeps grcli's own
Expand Down
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,18 @@ single `-`, trims leading/trailing `-`, `_`, and `.`, and lowercases the
result. So a bundle with `author.id: TAG-SC` and `id: cnsc` publishes to
`tag-sc/cnsc` — regardless of which namespace you are a member of.

**The bytes go to staging first.** `publish` always pushes the bundle and its
signature to the staging repository `staging/<namespace>/<name>`, then calls
`POST /v1/bundles/sync` naming `<namespace>/<name>`. The hub checks the staged
bundle and copies an accepted one into `<namespace>/<name>`; nobody pushes a
served repository directly. The hub deletes the staged upload
once it decides, and staging removes anything left there after about 24 hours,
so a failed publish is fixed by running `grcli publish` again. There is no
fallback: a hub from before the staging cutover gives no push grant on
`staging/...`, and the push fails with `401` naming the staging repository.
Older grcli releases push the served repository directly, which a cut-over hub
refuses with a `401 UNAUTHORIZED` at the first blob push; upgrade grcli.

**This is the usual cause of a 403 on publish.** Authorization is per
*namespace* (the part before the `/`), so you must own — or hold a trusted
publisher binding for — the namespace the metadata names, not the one you
Expand Down
55 changes: 45 additions & 10 deletions cmd/publish.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"errors"
"fmt"
"io"
"net/http"
"os"
"regexp"
"strings"
Expand All @@ -16,7 +17,10 @@ import (
"github.com/spf13/cobra"
"github.com/spf13/viper"

"oras.land/oras-go/v2/registry/remote/errcode"

"github.com/revanite-io/grc-store-protocol/spdx"
"github.com/revanite-io/grc-store-protocol/staging"

"github.com/gemaraproj/grc-store-clientkit/auth"
"github.com/gemaraproj/grcli/internal/digest"
Expand Down Expand Up @@ -49,9 +53,10 @@ func newPublishCmd(v *viper.Viper) *cobra.Command {
Use: "publish [file...]",
Short: "Bundle one Gemara artifact with provenance and push it to grc.store",
Long: `Loads the file(s) describing a single artifact, attaches a SLSA-shaped
provenance record, packs an OCI bundle, pushes it to the configured
registry, optionally signs with cosign, and notifies the hub via
POST /v1/bundles/sync.
provenance record, packs an OCI bundle, pushes it to the staging
repository staging/<namespace>/<name> on the hub's registry, optionally
signs it, and asks the hub to check it and publish it to
<namespace>/<name> via POST /v1/bundles/sync.

Files can be supplied as positional arguments (grcli publish a.yaml
b.yaml) or via -f / --file. The two forms are mutually exclusive —
Expand Down Expand Up @@ -106,7 +111,8 @@ need to set a secret.`,
// and artifact metadata defaults are merged.
type publishTarget struct {
registryHost string
repository string
repository string // the served repository: provenance, version check, sync
pushRepo string // staging.Repository(repository): where the bytes go
tag string
dryRun bool
output string
Expand Down Expand Up @@ -175,7 +181,7 @@ func runPublish(cmd *cobra.Command, v *viper.Viper, positional []string) error {
// The registry rejects unauthenticated writes. Mint a repo-scoped
// push token from the hub and export it so both the oras push and
// the cosign signature push authenticate.
if err := authenticatePush(ctx, v, target.repository); err != nil {
if err := authenticatePush(ctx, v, target.pushRepo); err != nil {
return err
}
}
Expand Down Expand Up @@ -212,7 +218,8 @@ func runPublish(cmd *cobra.Command, v *viper.Viper, positional []string) error {
}

return signAndNotify(ctx, v, signContext{
repository: target.repository,
repository: target.pushRepo,
served: target.repository,
tag: target.tag,
reference: result.Reference,
registryHost: target.registryHost,
Expand All @@ -223,7 +230,8 @@ func runPublish(cmd *cobra.Command, v *viper.Viper, positional []string) error {

// signContext carries the push coordinates the sign + notify step needs.
type signContext struct {
repository string
repository string // the staging repository the bundle and signature go to
served string // the served repository the sync names
tag string
reference string // <registry>/<repository>:<tag>, bare host
registryHost string // scheme-prefixed oras dial target
Expand Down Expand Up @@ -284,9 +292,21 @@ func resolveTarget(ctx context.Context, v *viper.Viper, loaded *source.Loaded) (
registryHost = d.RegistryURL
}

// Every publish goes to the staging form of the served repository and
// the sync names the served one; there is no fallback to pushing the
// served repository.
var pushRepo string
if !dryRun {
var ok bool
if pushRepo, ok = staging.Repository(repository); !ok {
return publishTarget{}, fmt.Errorf("repository %q must be <namespace>/<name> with no empty segment, and %q is not a namespace you can publish to", repository, staging.Segment)
}
}

target := publishTarget{
registryHost: registryHost,
repository: repository,
pushRepo: pushRepo,
tag: tag,
dryRun: dryRun,
output: v.GetString(flagOutput),
Expand All @@ -311,14 +331,29 @@ func pushBundle(ctx context.Context, target publishTarget, in registry.PackInput
result.Reference, result.ManifestDigest, result.BodyDigest, artifactType, artifactID)
return result, nil
}
result, err := registry.PushRemote(ctx, target.registryHost, target.repository, target.tag, in)
result, err := registry.PushRemote(ctx, target.registryHost, target.pushRepo, target.tag, in)
if err != nil {
return nil, fmt.Errorf("push: %w", err)
return nil, pushError(target, err)
}
fmt.Fprintf(out, "pushed %s\n manifest digest: %s\n", result.Reference, result.ManifestDigest)
return result, nil
}

// pushError explains a registry refusal. The registry token grants push on
// a staging repository only to the namespace's owner, and only on a hub past
// the staging cutover; anything else gets pull only and zot answers the
// first blob push with 401.
func pushError(target publishTarget, err error) error {
var resp *errcode.ErrorResponse
if errors.As(err, &resp) && (resp.StatusCode == http.StatusUnauthorized || resp.StatusCode == http.StatusForbidden) {
ns, _, _ := strings.Cut(target.repository, "/")
return fmt.Errorf("push: the registry refused the push to staging repository %s (%d): "+
"you need to own namespace %q (or be its trusted CI publisher), and the hub must run the staging cutover; "+
"a hub that predates it does not accept staging pushes: %w", target.pushRepo, resp.StatusCode, ns, err)
}
return fmt.Errorf("push to %s: %w", target.pushRepo, err)
}

// signAndNotify runs the optional cosign step and the hub sync call.
// Status lines go to os.Stdout rather than a passed-in writer because
// the cosign subprocess inside sign.Sign writes to os.Stdout/os.Stderr
Expand Down Expand Up @@ -352,7 +387,7 @@ func signAndNotify(ctx context.Context, v *viper.Viper, sc signContext) error {
if err != nil {
return err
}
syncResp, err := hub.New(hubURL, token).Sync(ctx, sc.repository, sc.tag)
syncResp, err := hub.New(hubURL, token).Sync(ctx, sc.served, sc.tag)
if err != nil {
return fmt.Errorf("hub sync: %w", err)
}
Expand Down
36 changes: 36 additions & 0 deletions cmd/publish_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,18 @@ package cmd
import (
"bytes"
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"

"github.com/gemaraproj/grc-store-clientkit/auth"
"github.com/spf13/viper"
"github.com/stretchr/testify/require"
"oras.land/oras-go/v2/registry/remote/errcode"

"github.com/gemaraproj/grcli/internal/registry"
"github.com/gemaraproj/grcli/internal/source"
Expand Down Expand Up @@ -54,6 +58,7 @@ func TestResolveTarget(t *testing.T) {
wantTarget: publishTarget{
registryHost: "registry.example",
repository: "my-team/my-policy",
pushRepo: "staging/my-team/my-policy",
tag: "1.2.3",
output: "grcli-out",
},
Expand All @@ -68,6 +73,7 @@ func TestResolveTarget(t *testing.T) {
wantTarget: publishTarget{
registryHost: "registry.example",
repository: "custom/repo",
pushRepo: "staging/custom/repo",
tag: "1.2.3",
output: "grcli-out",
},
Expand All @@ -87,6 +93,19 @@ func TestResolveTarget(t *testing.T) {
output: "/tmp/out",
},
},
{
// A staging name is not a served repository: name the served one.
name: "repository-in-staging-refused",
flags: map[string]any{flagURL: srv.URL, flagRepository: "staging/my-team/my-policy"},
loaded: loadedFull,
wantErrSub: "is not a namespace you can publish to",
},
{
name: "single-segment-repository-refused",
flags: map[string]any{flagURL: srv.URL, flagRepository: "solo"},
loaded: loadedFull,
wantErrSub: `repository "solo" must be <namespace>/<name>`,
},
{
name: "missing-url-when-not-dry-run",
flags: map[string]any{},
Expand Down Expand Up @@ -304,3 +323,20 @@ func TestApplyVersionFlag(t *testing.T) {
})
}
}

// TestPushError checks that a registry 401/403 on the staging push names the
// staging repository, the namespace and the cutover, and that other errors
// still name where the push went.
func TestPushError(t *testing.T) {
target := publishTarget{repository: "acme/ctl", pushRepo: "staging/acme/ctl"}
for _, status := range []int{http.StatusUnauthorized, http.StatusForbidden} {
err := pushError(target, fmt.Errorf("packing bundle: %w", &errcode.ErrorResponse{Method: http.MethodPost, StatusCode: status}))
for _, want := range []string{"staging/acme/ctl", `"acme"`, "staging cutover", strconv.Itoa(status)} {
require.Contains(t, err.Error(), want)
}
var resp *errcode.ErrorResponse
require.ErrorAs(t, err, &resp, "the registry error stays wrapped")
}
err := pushError(target, errors.New("connection refused"))
require.Equal(t, "push to staging/acme/ctl: connection refused", err.Error())
}
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ require (
github.com/gemaraproj/go-gemara v0.9.2
github.com/opencontainers/go-digest v1.0.0
github.com/opencontainers/image-spec v1.1.1
github.com/revanite-io/grc-store-protocol v0.8.0
github.com/revanite-io/grc-store-protocol v0.28.1
github.com/sigstore/sigstore-go v1.3.0
github.com/spf13/cobra v1.10.2
github.com/spf13/viper v1.21.0
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -278,8 +278,8 @@ github.com/prometheus/common v0.67.5 h1:pIgK94WWlQt1WLwAC5j2ynLaBRDiinoAb86HZHTU
github.com/prometheus/common v0.67.5/go.mod h1:SjE/0MzDEEAyrdr5Gqc6G+sXI67maCxzaT3A2+HqjUw=
github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc=
github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo=
github.com/revanite-io/grc-store-protocol v0.8.0 h1:ZlAm78+yzAHte23FsNNfLpuf0KiVeprwQr5OepfmE7o=
github.com/revanite-io/grc-store-protocol v0.8.0/go.mod h1:naeWSRGJ9dtyzndHdsA4jrENlSmSlIgLQDAKRSpPhwU=
github.com/revanite-io/grc-store-protocol v0.28.1 h1:qgQCQQIXbAyOh1A0SLrhBrDEpYDZmguPKRdKbV81J1w=
github.com/revanite-io/grc-store-protocol v0.28.1/go.mod h1:lsuyx03NQdd71b9FiXa8Auvz6Go8Mzde3gsOGUGJd9g=
github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
Expand Down
64 changes: 58 additions & 6 deletions internal/hub/hub.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ import (
"strings"
"time"

"github.com/revanite-io/grc-store-protocol/apierror"
"github.com/revanite-io/grc-store-protocol/staging"
"github.com/revanite-io/grc-store-protocol/syncapi"
)

Expand All @@ -35,12 +37,17 @@ type Client struct {
HTTP *http.Client
}

// requestTimeout bounds every hub call. It sits above catalog sync's
// 120-second write deadline on the hub, so a slow sync ends with the hub's
// answer rather than a client timeout.
const requestTimeout = 3 * time.Minute

// New returns a Client with sensible defaults.
func New(baseURL, token string) *Client {
return &Client{
BaseURL: strings.TrimRight(baseURL, "/"),
Token: token,
HTTP: &http.Client{Timeout: 60 * time.Second},
HTTP: &http.Client{Timeout: requestTimeout},
}
}

Expand Down Expand Up @@ -300,15 +307,60 @@ func (c *Client) Sync(ctx context.Context, repository, tag string) (*SyncRespons
return nil, fmt.Errorf("reading sync response from %s: %w", url, err)
}
if resp.StatusCode/100 != 2 {
// Same shape as VersionExists/GetCatalog default branches (URL +
// status + body snippet) so an operator chasing a 5xx on any hub
// endpoint sees the same diagnostic surface.
return nil, fmt.Errorf("hub sync %s returned %d: %s",
url, resp.StatusCode, strings.TrimSpace(string(rb)))
return nil, newSyncError(url, repository, tag, resp.StatusCode, rb)
}
out := &SyncResponse{}
if err := json.Unmarshal(rb, out); err != nil {
return nil, fmt.Errorf("decoding hub response: %w", err)
}
return out, nil
}

// SyncError is a refused POST /v1/bundles/sync. Code and Detail come from
// the hub's apierror envelope; Code is empty when the body was not one.
type SyncError struct {
URL string // the sync endpoint, for an operator chasing a 5xx
Repository, Tag string
Status int
Code, Detail string
}

func newSyncError(url, repository, tag string, status int, body []byte) *SyncError {
e := &SyncError{URL: url, Repository: repository, Tag: tag, Status: status}
var env apierror.Envelope
if json.Unmarshal(body, &env) == nil && env.Error != "" {
e.Code, e.Detail = env.Error, env.Detail
} else {
e.Detail = strings.TrimSpace(string(body))
}
return e
}

func (e *SyncError) Error() string {
what := fmt.Sprintf("%s refused %s:%s (%d", e.URL, e.Repository, e.Tag, e.Status)
if e.Code != "" {
what += " " + e.Code
}
what += ")"
if e.Detail != "" {
what += ": " + e.Detail
}
return what + ". " + e.hint()
}

// hint says what to do next. The hub deletes the staged upload on success
// and on every 4xx but a 429, and keeps it on a 429 or 5xx, so a refusal
// means publishing again, not syncing again.
func (e *SyncError) hint() string {
stagingRepo, _ := staging.Repository(e.Repository)
switch {
case e.Status == http.StatusUnauthorized:
return "Your hub login is missing or expired: run grcli login (in CI, check the job has id-token: write)."
case e.Code == "fetch_failed":
return fmt.Sprintf("The hub could not read the bundle from %s: the push did not finish, or it was pushed more than 24 hours ago and staging removed it. Run grcli publish again.", stagingRepo)
case e.Status == http.StatusTooManyRequests || e.Status >= 500:
return "Nothing was published; run grcli publish again in a moment."
default:
return "Nothing was published and the hub discarded the staged upload; fix the cause and run grcli publish again."
}
}
Loading
Loading