chore(ci): use yaml config file for e2e credentials by theakshaypant · Pull Request #3031 · tektoncd/pipelines-as-code

theakshaypant · 2026-10-16T11:13:41Z

📝 Description of the Change

Replace hardcoded provider env vars in the e2e workflow with a YAML
config file loaded from the E2E_CONFIG_YAML secret and exposed through
PAC_E2E_CONFIG.

Secrets still needed by shell steps (rate-limit check, startpaac,
gosmee, log scrubbing) stay as step-scoped env vars. Add GHE webhook
token, webhook secret and application ID to the config schema and
rename the Bitbucket Cloud secret to webhook_secret to match the env
var the tests read. Update the example config and README.

🔗 Linked GitHub Issue

N/A

🧪 Testing Strategy

🤖 AI Assistance

AI assistance can be used for various tasks, such as code generation,
documentation, or testing.

Please indicate whether you have used AI assistance
for this PR and provide details if applicable.

I have not used any AI assistance for this PR.
I have used AI assistance for this PR.

Important

Slop will be simply rejected, if you are using AI assistance you need to make sure you
understand the code generated and that it meets the project's standards. you
need at least know how to run the code and deploy it (if needed). See
startpaac to make it easy
to deploy and test your code changes.

If the majority of the code in this PR was generated by an AI, please add a Co-authored-by trailer to your commit message.
For example:

Co-authored-by: Claude noreply@anthropic.com

✅ Submitter Checklist

pipelines-as-code · 2026-10-16T11:14:25Z

Paco Review

The PR moves E2E test configuration from individual GitHub Actions environment variables/secrets into a single YAML config file secret (E2E_CONFIG_YAML). It also renames the BitbucketCloudConfig.Secret field to WebhookSecret (changing the YAML key from secret to webhook_secret) and adds new fields to GitHubEnterpriseConfig. The example config changes the GitHub api_url value from https://api.github.com to the bare host api.github.com. The env var name for the Bitbucket Cloud webhook secret also changed from TEST_BITBUCKET_CLOUD_SECRET to TEST_BITBUCKET_CLOUD_WEBHOOK_SECRET.

Paco found no new confirmed issues in the code reviewed.

Note

Take this AI review with a grain of salt and use it alongside tests and human review.

2 candidates not published

bbcloud-yaml-key-rename: reject. The YAML key rename from secret to webhook_secret is an intentional change. The example file, the env var tag, and the struct field were all updated together in a coordinated manner. The Bitbucket Cloud setup code (test/pkg/bitbucketcloud/setup.go) reads the webhook secret via os.Getenv("TEST_BITBUCKET_CLOUD_WEBHOOK_SECRET"), which matches the new env tag. The old env var name was TEST_BITBUCKET_CLOUD_SECRET, so this is not just a YAML key rename but a deliberate alignment of field name, env var, and YAML key. Anyone deploying this PR would update the E2E_CONFIG_YAML secret at the same time (as documented in the updated README). This is an expected migration, not a silent regression introduced by the PR.
github-api-url-no-scheme: reject. The bare-host format for api_url was already the convention before this PR. The workflow file previously set TEST_GITHUB_API_URL: api.github.com (before line 76), and the rate-limit script in hack/gh-workflow-ci.sh explicitly prepends https:// (line 226: "https://${TEST_GITHUB_API_URL}"). The test/README.md already documented it as api.github.com without a scheme. The Go test setup code in test/pkg/github/setup.go and test/pkg/github/crd.go reads the value from os.Getenv and uses it as-is for the GitHub client, which handles bare hosts. The example file comment clarifies existing behavior rather than introducing a new incompatibility.

_{Reviewed commit: 18de39b. 0 verified findings; 0 summary-only.}

pipelines-as-code

Paco inline comments -- see the Paco Review summary comment for the overview.

pipelines-as-code · 2026-10-16T11:14:37Z

-	Secret        string `env:"TEST_BITBUCKET_CLOUD_SECRET"         json:"secret"         yaml:"secret"`
-	E2ERepository string `env:"TEST_BITBUCKET_CLOUD_E2E_REPOSITORY" json:"e2e_repository" yaml:"e2e_repository"`
+	APIURL        string `env:"TEST_BITBUCKET_CLOUD_API_URL"           json:"api_url"        yaml:"api_url"`
+	User          string `env:"TEST_BITBUCKET_CLOUD_USER"              json:"user"           yaml:"user"`


[HIGH] The old field Secret string (env tag TEST_BITBUCKET_CLOUD_SECRET) has been renamed to WebhookSecret string (env tag TEST_BITBUCKET_CLOUD_WEBHOOK_SECRET). Any existing call sites that reference cfg.BitbucketCloud.Secret will fail to compile, and any code that set TEST_BITBUCKET_CLOUD_SECRET in the environment will silently receive an empty string instead of the expected value. Confirm all consumers of this field have been updated to use WebhookSecret and that the old env-var name is no longer set anywhere in tests or CI scripts.

Copilot

🟡 Changes recommended

The migration introduces shell interpretation, missing credentials, older-branch compatibility failures, and incomplete log redaction.

4 open findings

Safely pass YAML secrets without shell evaluation · New
Restore rate-limit step environment variables · New
Preserve compatibility exports for older PR heads · New
Restore GHE relay URL secret for log redaction · New

What changed in this PR

Moves E2E test credentials into a YAML configuration supplied through a GitHub Secret.

Changes:

Replaces job-level credential variables with configuration-file loading.
Adds GitHub Enterprise fields and aligns Bitbucket Cloud webhook-secret configuration.
Retains dynamic test values as workflow environment variables.

File	Description
`test/pkg/configfile/config.go`	Updates provider configuration fields and mappings.
`test/e2e-config.yaml.example`	Documents the additional credential fields.
`.github/workflows/e2e.yaml`	Writes the secret-backed configuration and reduces environment declarations.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

        # Remove the old label check which is no longer relevant for triggering.
        if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || github.event_name == 'pull_request_target' }}
        env:
+          # Dynamic runtime values (override config file)


        if: ${{ always() }}
        env:
-          TEST_GITHUB_SECOND_SMEE_URL: ${{ secrets.TEST_GITHUB_SECOND_SMEE_URL }}
          TEST_GITHUB_SECOND_WEBHOOK_SMEE_URL: ${{ steps.gosmee-ghe-webhook-url.outputs.url }}


codecov · 2026-10-16T11:18:22Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.98%. Comparing base (c20766e) to head (66bfd14).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3031   +/-   ##
=======================================
  Coverage   91.98%   91.98%           
=======================================
  Files         167      167           
  Lines       12976    12976           
=======================================
  Hits        11936    11936           
  Misses       1039     1039           
  Partials        1        1

Flag	Coverage Δ
unit-tests	`91.98% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

theakshaypant · 2026-10-16T11:18:36Z

      - name: Run gosmee for second controller GHE App
        if: startsWith(matrix.provider, 'github_ghe') || matrix.provider == 'concurrency'
+        env:
+          TEST_GITHUB_SECOND_SMEE_URL: ${{ secrets.TEST_GITHUB_SECOND_SMEE_URL }}


Only this secret needs to be maintained separately in addition to the config file secret

Copilot

🟡 Changes recommended

Credential masking, secret references, older-PR compatibility, and GitLab overrides need correction before merging.

7 open issues

2 resolved since last review

_{Review effort: Balanced}

+          printf '%s\n' "$E2E_CONFIG_YAML" > /tmp/e2e-config.yaml
+          echo "PAC_E2E_CONFIG=/tmp/e2e-config.yaml" >> "$GITHUB_ENV"


+          TEST_GITHUB_TOKEN: ${{ secrets.TEST_GITHUB_TOKEN }}
+          TEST_GITHUB_API_URL: ${{ secrets.TEST_GITHUB_API_URL }}


-          VARS_TEST_GITHUB_SECOND_APPLICATION_ID: ${{ vars.TEST_GITHUB_SECOND_APPLICATION_ID }}
+          APPLICATION_ID: ${{ vars.APPLICATION_ID }}
+          APP_PRIVATE_KEY: ${{ secrets.APP_PRIVATE_KEY }}
+          TEST_EL_WEBHOOK_SECRET: ${{ secrets.TEST_EL_WEBHOOK_SECRET }}


+        run: |
+          umask 077
+          printf '%s\n' "$E2E_CONFIG_YAML" > /tmp/e2e-config.yaml
+          echo "PAC_E2E_CONFIG=/tmp/e2e-config.yaml" >> "$GITHUB_ENV"


+        run: |
+          umask 077
+          printf '%s\n' "$E2E_CONFIG_YAML" > /tmp/e2e-config.yaml
+          echo "PAC_E2E_CONFIG=/tmp/e2e-config.yaml" >> "$GITHUB_ENV"


chmouel · 2026-10-23T10:44:36Z

this is fine by me if it workeds :D probably want to keep the secret so we can revert if needed

theakshaypant · 2026-10-23T11:26:12Z

this is fine by me if it workeds :D probably want to keep the secret so we can revert if needed

Still need to test it so kept it in draft.
Was trying to remove all other variables/secrets from e2e.yaml but became a bigger change. Will focus on this next

chmouel · 2026-11-09T08:37:35Z

/paco review

Replace hardcoded provider env vars in the e2e workflow with a YAML config file loaded from the E2E_CONFIG_YAML secret and exposed through PAC_E2E_CONFIG. Secrets still needed by shell steps (rate-limit check, startpaac, gosmee, log scrubbing) stay as step-scoped env vars. Add GHE webhook token, webhook secret and application ID to the config schema and rename the Bitbucket Cloud secret to webhook_secret to match the env var the tests read. Update the example config and README. Signed-off-by: Akshay Pant <akpant@redhat.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chmouel · 2026-11-09T08:38:59Z

/paco review

The e2e workflow still needed eight separate secrets and two variables next to the E2E_CONFIG_YAML secret, for the GitHub App keys, the relay URLs, the webhook secret and the cleanup job tokens. Keeping the same value in two places made it easy for them to drift apart. The GitHub App IDs, private keys and relay URL now live in the config file too. Each workflow step reads the values it needs from the file and exports them for that step only. The cleanup job reads its tokens from the same file instead of the old GH_APPS_TOKEN secret. The step that writes the file now also hides every token, secret, password, key and relay URL in the logs one by one, because GitHub only hides a multi-line secret line by line. The cluster setup also passes the GitHub Enterprise host name from the file again. Without it, the second controller did not trust that host until a GitHub App event happened to arrive first, so webhook tests could fail at random.

chmouel · 2026-11-09T13:42:22Z

I tested this on my fork with a single E2E_CONFIG_YAML secret and no other e2e secrets or variables. All 9 e2e jobs passed:
https://github.com/chmouel/pipelines-as-code/actions/runs/42605041783

I pushed one commit on top of yours with what I needed to get there:

The GitHub App IDs, private keys and smee URL now come from the config file too, so the workflow no longer needs the separate APP_PRIVATE_KEY, PYSMEE_URL, TEST_GITHUB_SECOND_*, TEST_EL_WEBHOOK_SECRET and TEST_GITHUB_TOKEN secrets. Each step reads what it needs with yq.
The step that writes the file masks every token, secret, password, key and smee URL separately. GitHub only masks a multi-line secret line by line, so a bare token printed by a test or the controller would otherwise show up in the logs. I grepped the job logs of the fork run for every value and found none in clear.
The cleanup job reads its tokens from the config file instead of GH_APPS_TOKEN.
startpaac gets PAC_GITHUB_SECOND_HOSTNAME from the config again. Without it the GHE controller had an empty trusted-provider-hostnames and github_ghe_3 failed whenever a webhook test ran before any GitHub App event.
The config test now covers the new GHE fields and the Bitbucket Cloud webhook_secret.

Before merging we need to create E2E_CONFIG_YAML on this repo. After the merge, open PRs have to rebase, since their branches won't read the new GHE and Bitbucket fields. Once a nightly run passes, we can delete the old secrets.

Follow-ups: pipelines-as-code/startpaac#35 to let startpaac read this file directly, and #3076 to clean up the env var naming.

chmouel · 2026-11-09T15:03:49Z

Also tested on upstream. I set the E2E_CONFIG_YAML secret on tektoncd/pipelines-as-code, pushed this branch to a temporary branch here and dispatched the e2e workflow on it, so it ran this PR's workflow with the new secret. All 9 jobs passed:
https://github.com/tektoncd/pipelines-as-code/actions/runs/42628749818

The first attempt had 6 failures because the hook.pipelinesascode.com relay was down for a VM update between 14:06 and 14:13 UTC. Webhook events were lost and those tests timed out. All 6 jobs passed on the rerun. I also checked the logs of both attempts for every token, secret, password, key and smee URL in the config, and none of them showed up unmasked.

Merging now. Open PRs need to rebase on main after this so their e2e runs pick up the new config fields. The old secrets stay in place until the release branches are moved over too (backports to release-v0.52.x, release-v0.50.x and release-v0.48.x, each tested on its own branch first).

Copilot AI balanced review requested due to automatic review settings October 16, 2026 11:13

Copilot started reviewing on behalf of theakshaypant October 16, 2026 11:14

View session

pipelines-as-code Bot added paco/review-moderate Paco review difficulty security-review Flagged as security-sensitive by Paco labels Oct 16, 2026

pipelines-as-code Bot reviewed Oct 16, 2026

View reviewed changes

theakshaypant force-pushed the chore/ci-use-config-file branch from e16c6ca to 47eb242 Compare October 16, 2026 11:15

Copilot AI reviewed Oct 16, 2026

View reviewed changes

theakshaypant commented Oct 16, 2026

View reviewed changes

theakshaypant marked this pull request as draft October 16, 2026 11:19

theakshaypant force-pushed the chore/ci-use-config-file branch from 47eb242 to ac937ec Compare October 21, 2026 05:13

theakshaypant requested a balanced review from Copilot October 21, 2026 05:14

Copilot started reviewing on behalf of theakshaypant October 21, 2026 05:15

View session

Copilot AI reviewed Oct 21, 2026

View reviewed changes

chmouel force-pushed the chore/ci-use-config-file branch from ac937ec to 18de39b Compare November 9, 2026 08:38

pipelines-as-code Bot added paco/reviewing Paco is reviewing this pull request paco/clean Paco review found nothing to report and removed paco/reviewing Paco is reviewing this pull request labels Nov 9, 2026

This was referenced Nov 9, 2026

Read GitHub App credentials from the PAC e2e config file pipelines-as-code/startpaac#35

Open

Use consistent names for end-to-end test settings #3076

Open

chmouel marked this pull request as ready for review November 9, 2026 15:03

chmouel merged commit c5b1b65 into tektoncd:main Nov 9, 2026
42 of 49 checks passed

		printf '%s\n' "$E2E_CONFIG_YAML" > /tmp/e2e-config.yaml
		echo "PAC_E2E_CONFIG=/tmp/e2e-config.yaml" >> "$GITHUB_ENV"

		TEST_GITHUB_TOKEN: ${{ secrets.TEST_GITHUB_TOKEN }}
		TEST_GITHUB_API_URL: ${{ secrets.TEST_GITHUB_API_URL }}

Conversation

theakshaypant commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description of the Change

🔗 Linked GitHub Issue

🧪 Testing Strategy

🤖 AI Assistance

✅ Submitter Checklist

Uh oh!

pipelines-as-code Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Paco Review

Uh oh!

pipelines-as-code Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pipelines-as-code Bot Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

theakshaypant Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

chmouel commented Oct 23, 2026

Uh oh!

theakshaypant commented Oct 23, 2026

Uh oh!

chmouel commented Nov 9, 2026

Uh oh!

chmouel commented Nov 9, 2026

Uh oh!

chmouel commented Nov 9, 2026

Uh oh!

chmouel commented Nov 9, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

theakshaypant commented Oct 16, 2026 •

edited

Loading

pipelines-as-code Bot commented Oct 16, 2026 •

edited

Loading

codecov Bot commented Oct 16, 2026 •

edited

Loading