Skip to content

fix: Harden E2E tool downloads and fail cancel waits early - #3075

Open
chmouel wants to merge 4 commits into
tektoncd:mainfrom
chmouel:e2e-retry-gotestsum-fail-fast-cancel
Open

chmouel wants to merge 4 commits into
tektoncd:mainfrom
chmouel:e2e-retry-gotestsum-fail-fast-cancel

Conversation

@chmouel

@chmouel chmouel commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

📝 Description of the Change

The scheduled E2E run 42565470032 failed during a GitHub outage that slowed down and cut off downloads for clients that are not logged in. Most jobs died while installing their tools, before a single test ran. One more test waited ten minutes for something that was never going to happen.

This makes the E2E jobs hold up better when GitHub has a bad day:

  • The test runner tool is now downloaded with the job's own GitHub login, retried when the download fails or comes back incomplete, and saved in full before it is unpacked. Before, a partial download killed the job straight away.
  • The ko image builder is no longer downloaded from GitHub releases. It is installed from the Go package mirror instead, at a fixed version.
  • The tools every job needs (ko, minica, gotestsum, gosmee and snazy) are now kept in the GitHub Actions cache at fixed versions, so most runs reuse them and never need to download them from GitHub again.
  • The cancel tests now stop as soon as their pipeline finishes without being cancelled, and the failure message says which pipeline it was and how it ended. Before, they waited for the full ten minute timeout and only reported a generic timeout.

The image publishing workflow is unchanged.

🔗 Linked GitHub Issue

N/A

🧪 Testing Strategy

  • Unit tests
  • Integration tests
  • End-to-end tests
  • Manual testing
  • Not Applicable

🤖 AI Assistance

AI assistance can be used for various tasks, such as code generation,
documentation, or testing.

Please indicate whether you have used AI assistance
for this PR and provide details if applicable.

  • I have not used any AI assistance for this PR.
  • I have used AI assistance for this PR.

Important

Slop will be simply rejected, if you are using AI assistance you need to make sure you
understand the code generated and that it meets the project's standards. you
need at least know how to run the code and deploy it (if needed). See
startpaac to make it easy
to deploy and test your code changes.

If the majority of the code in this PR was generated by an AI, please add a Co-authored-by trailer to your commit message.
For example:

Co-authored-by: Claude noreply@anthropic.com

✅ Submitter Checklist

  • 📝 My commit messages are clear, informative, and follow the project's How to write a git commit message guide. The Gitlint linter ensures in CI it's properly validated
  • ✨ I have ensured my commit message prefix (e.g., fix:, feat:) matches the "Type of Change" I selected above.
  • ♽ I have run make test and make lint locally to check for and fix any
    issues. For an efficient workflow, I have considered installing
    pre-commit and running pre-commit install to
    automate these checks.
  • 📖 I have added or updated documentation for any user-facing changes.
  • 🧪 I have added sufficient unit tests for my code changes.
  • 🎁 I have added end-to-end tests where feasible. See README for more details.
  • 🔎 I have addressed any CI test flakiness or provided a clear reason to bypass it.
  • If adding a provider feature, I have filled in the following and updated the provider documentation:
    • GitHub App
    • GitHub Webhook
    • Gitea/Forgejo
    • GitLab
    • Bitbucket Cloud
    • Bitbucket Data Center

Copilot AI balanced review requested due to automatic review settings November 9, 2026 08:08
@pipelines-as-code

pipelines-as-code Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Paco Review

This PR adds a FailOnOtherTerminalReason option to the test wait helper so that cancel-oriented E2E tests fail fast when a PipelineRun finishes with an unexpected reason instead of polling until timeout. It also reworks the gotestsum install script to use the GitHub REST API for authenticated downloads, download to a temp file with retries, and exposes GH_TOKEN in the E2E workflow.

Paco found no new confirmed issues in the code reviewed.

Note

Take this AI review with a grain of salt and use it alongside tests and human review.

Reviewed commit: 779fa51. 0 verified findings; 0 summary-only.

@pipelines-as-code pipelines-as-code Bot added the paco/reviewing Paco is reviewing this pull request label Nov 9, 2026
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.99%. Comparing base (c20766e) to head (997162d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3075   +/-   ##
=======================================
  Coverage   91.98%   91.99%           
=======================================
  Files         167      167           
  Lines       12976    12976           
=======================================
+ Hits        11936    11937    +1     
+ Misses       1039     1038    -1     
  Partials        1        1           
Flag Coverage Δ
unit-tests 91.99% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pipelines-as-code pipelines-as-code Bot added paco/review-hard Paco review difficulty paco/clean Paco review found nothing to report and removed paco/reviewing Paco is reviewing this pull request labels Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The review identifies no active issues with the E2E reliability improvements.

No open issues 🎉

What changed in this PR

This PR improves E2E reliability by hardening gotestsum downloads against interrupted responses and shortening cancellation-test failures when a PipelineRun finishes with an unexpected reason.

Changes:

  • Downloads gotestsum to a temporary file with retries and uses authenticated GitHub API downloads when a token and jq are available.
  • Adds opt-in fail-fast behavior for unexpected terminal PipelineRun reasons, with unit coverage.
  • Enables fail-fast waits in the two single-run cancellation tests while preserving existing behavior elsewhere.
File Description
test/​pkg/​wait/​wait.go Adds an optional early error for unexpected terminal reasons.
test/​pkg/​wait/​wait_test.go Covers early failure and continued polling for running PipelineRuns.
test/​github_gitops_commands_prefix_test.go Enables fail-fast waiting in the GitHub custom-prefix cancellation test.
test/​gitea_gitops_commands_test.go Enables fail-fast waiting in the Gitea cancellation test.
hack/​install-gotestsum.sh Adds retryable file downloads and authenticated release-asset lookup.
.github/​workflows/​e2e.yaml Supplies GH_TOKEN to the E2E test step for tool downloads.

Review effort: Balanced

@chmouel chmouel changed the title fix: Harden E2E gotestsum download and fail cancel waits early fix: Harden E2E tool downloads and fail cancel waits early Nov 9, 2026
@chmouel
chmouel force-pushed the e2e-retry-gotestsum-fail-fast-cancel branch from b0a39a8 to ab9f819 Compare November 9, 2026 08:29
The E2E jobs download the gotestsum test runner from GitHub releases
before running any test. During GitHub incidents that throttle
anonymous traffic, this download can come back cut short, and the
script fed the partial file straight to tar, killing the job before a
single test ran.

The installer now saves the file first and retries the download when
it fails or is incomplete. When a token is available it fetches the
file through the GitHub API with that token, since the plain release
link ignores tokens and is treated as anonymous traffic. The E2E
workflow passes the job's own GitHub token to the test step so this
authenticated path is used in CI.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
The cancel E2E tests wait for their PipelineRun to be marked as
cancelled. When the cancel request was missed and the run simply
finished, the test kept waiting for the full ten minute timeout and
then reported only a generic timeout.

Waiting for a PipelineRun outcome can now opt in to stopping as soon
as a matching run ends with a different result, and the error names
the run and how it ended. The two cancel tests that watch a single
run use it. Tests that track several runs keep the previous behavior,
since other runs there are expected to finish differently.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
The E2E jobs used the setup-ko action to install the ko image builder.
That action downloads ko from GitHub releases as an anonymous client,
with no retry, and unpacks it while it is still downloading. During
GitHub incidents that throttle anonymous traffic, this broke most
E2E jobs before any test ran.

ko is now installed with the Go toolchain, which fetches it from the
Go module proxy. This avoids GitHub release downloads entirely and
pins ko to a known version.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Every E2E job downloaded the same helper tools again, several of them
from GitHub releases, which fail when GitHub throttles downloads.

The gosmee and snazy installs are now pinned to a release and kept in
the Actions cache. ko, minica and gotestsum share one cache keyed on
their versions, and minica is pinned instead of following its latest
version. Once cached, these tools are restored without contacting
GitHub releases at all.

Building ko from source adds time to every E2E job, and the shared Go
cache does not reliably keep it, since that cache is saved by
whichever workflow runs first and never updated afterwards.

The built ko binary is now cached on its own, keyed on the ko version,
so later runs restore it in seconds and only build it again when the
version changes.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
@chmouel
chmouel force-pushed the e2e-retry-gotestsum-fail-fast-cancel branch from ab9f819 to 997162d Compare November 9, 2026 08:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci e2e paco/clean Paco review found nothing to report paco/review-hard Paco review difficulty testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants