Skip to content

cmdiotest: pin current TUI behaviour ahead of the Charm v2 migration - #7734

Open
pietern wants to merge 3 commits into
pietern/selftest-tui-coveragefrom
pietern/termtest-pins
Open

pietern wants to merge 3 commits into
pietern/selftest-tui-coveragefrom
pietern/termtest-pins

Conversation

@pietern

@pietern pietern commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

This PR pins current Charm v1 TUI behaviour, so the Charm v2 migration shows its exact user-visible changes as golden diffs.

Changes

Test infrastructure only. No production behaviour changes.

Two behaviour-neutral seams for tests, both with zero-value defaults that keep production behaviour:

  • cmdio.NewTestIOWithOptions takes templates, the color environment and redirected streams. The color-from-environment lookup moves into colorFromEnv, shared by NewIO and the test path.
  • tableview.RunWithOptions takes an input and an initial window size. Run calls it with zero options.

termtest capabilities (all opt-in; existing goldens unchanged):

  • Styled goldens with inline markers ({fg=4}?{/}, {bold}, {fg=@208}, {fg=#8fb3dc}), covering SGR colors, bold, faint, italic, underline styles (4:n, 21), blink, reverse, strikethrough and overline. Colon and semicolon color forms are decoded.
  • Mode log (GoldenModes): every set and reset is recorded, so a repeated write shows as a diff. Covers cursor visibility, bracketed paste, alternate screen, mouse, kitty keyboard, modifyOtherKeys and DECST8C.
  • Query log (GoldenQueries): DECRQM, kitty flags, OSC 10/11, CPR, DA1/DA2, XTVERSION. A Responder answers them after a delay.
  • Leftover reports input left unread on exit. It closes the harness's end of the input pipe and reads to EOF, so it needs no platform-specific reads. Its tests are skipped on Windows.
  • Input helpers: TypeBurst, Paste, key constants for Alt and Shift+Enter forms, and Settle, which waits until output is quiet.
  • WaitForScreen, WaitForOrExit, WaitForReplies and WaitForExit.
  • Options: Env, ColorFromEnv, stream redirection, templates, ScreenRows (non-positive values use the default), MapNewline and Normalize (applied to screen snapshots only).
  • LogRawOnFailure logs the raw output stream when a test fails.

Pinned areas (libs/cmdio/cmdiotest/pin_*_test.go, 103 new goldens under testdata/):

  • Keys: burst input, Alt+Enter, Alt+Left, Shift+Enter (kitty and modifyOtherKeys forms), kitty Esc, bracketed paste with newlines.
  • Colors: the swatch, select, prompt and spinner across TERM (unset, xterm-256color, screen, vt100, dumb), CI, CLICOLOR=0, NO_COLOR, tmux with truecolor and COLORTERM.
  • Terminal modes on exit for prompt, select, pager and spinner.
  • Queries and leftover replies on a quick quit, an immediate terminal answer, and an unsolicited OSC 11 reply.
  • Pager: paging on space, Enter draining, errors on the first and later pages, empty and one-page lists, and spinner frames during the first page.
  • Tableview search input (Alt+f and paste).
  • Bytes LogString writes to a redirected stderr. LogString itself writes its bytes unchanged. The SGR is stripped by the color helpers and the renderer, and the goldens show the effect.

The goldens are the reference. When the v2 swap lands these tests fail, and the golden diffs show the exact change. Regenerate with -update only for an intended change.

Known flake: TestPagerPin_SpaceLoadsNextPage once showed the second page over the first. It has not recurred since the test waits for output to settle before pressing space. On failure the test logs the raw output stream for diagnosis.

Waits that gate a golden fail the test on timeout. Two waits are deliberately tolerant, and say so at the call site: the spinner-frame count (the count is the pinned value) and the select's second cursor hide (a version that hides once must still reach its golden).

No .nextchanges fragment: this is test-only.

Why

The v2 migration changes key decoding, color detection, the bytes written when a TUI starts and exits, and paging/search behaviour. The existing tests run against pipes and an emulator, so they cannot show what changes for a user. Pinning the current v1 behaviour first makes each intended change a reviewable golden diff.

Some v1 behaviour pinned here is accidental (for example, Alt+Enter submits, and a late OSC 11 reply is typed into the prompt). The pins record it so the migration decides on each change explicitly rather than inheriting it silently.

Tests

  • go build ./..., go vet (linux, darwin, windows) and golangci-lint on ./libs/cmdio/... and ./libs/tableview/... pass. gofmt is clean.
  • TERM=xterm-256color go test -race -count=10 ./libs/cmdio/... passes.
  • Flakiness: after the termtest change, 3 rounds of 8 parallel -race -run Pin -count=10 and 1 round of the same under 8 CPU-hog processes: zero failures. One full-package -race -count=3 run failed once on TestSelectBaseline_EscKey (a key typed just before a snapshot was read before its echo). It has not reproduced: four further cmdiotest-only runs and a full ./libs/cmdio/... run passed, and the test passes 10 of 10 in isolation. It does not use Settle, Leftover or WaitForInputRead.
  • go test on ./libs/cmdio/cmdiotest/ passes with TERM=dumb and with TERM unset. go test ./cmd/auth/... ./libs/tableview/... passes.
  • go test ./acceptance/... passes.
  • The testdata diff against the base contains only added files.

This pull request and its description were written by Isaac.

NewTestIOWithOptions lets tests set the templates, color environment and
redirected streams that NewTestIO leaves off. The color environment
lookup moves into colorFromEnv so NewIO and the test path share it.

tableview.RunWithOptions lets tests supply input and an initial window
size. Run calls it with zero options, so production behaviour is
unchanged.

Both seams are behaviour-neutral when given their zero values.

Co-authored-by: Isaac <no-reply@databricks.com>
@pietern
pietern requested review from a team as code owners November 17, 2026 21:50

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 58b99e3

Run: 44343716565

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 14:58
💚​ aws-windows-latest-4core-16gb 1 274 17 12:39
💚​ azure linux-2core-8gb 1 271 19 16:51
💚​ azure-windows-latest-4core-16gb 1 273 17 11:27
💚​ gcp linux-2core-8gb 1 272 19 16:01
💚​ gcp-windows-latest-4core-16gb 1 274 17 14:03
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
6:36 azure linux-2core-8gb TestFilerReadWrite/workspace_files
5:28 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
5:23 azure linux-2core-8gb TestFilerReadWrite/workspace_files_extensions
5:20 gcp linux-2core-8gb TestFilerWorkspaceNotebook/scalaNb.scala
5:17 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
4:53 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
4:52 gcp-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
4:48 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files
4:36 gcp-windows-latest-4core-16gb TestSyncFullFileSync
4:21 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
3:48 aws linux-2core-8gb TestFilerReadWrite/workspace_files
3:47 gcp-windows-latest-4core-16gb TestFilerReadWrite/workspace_files
3:47 azure-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
3:36 azure-windows-latest-4core-16gb TestAccept
3:32 gcp-windows-latest-4core-16gb TestAccept
3:29 aws-windows-latest-4core-16gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_python
3:25 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
3:11 aws linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
3:05 azure linux-2core-8gb TestExportDir
3:02 azure-windows-latest-4core-16gb TestLock
3:02 gcp linux-2core-8gb TestFilerReadWrite/workspace_files
2:57 azure linux-2core-8gb TestLock
2:57 gcp linux-2core-8gb TestLockUnlockWithAllowsLockFileNotExist
2:52 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:50 gcp linux-2core-8gb TestSyncIncrementalFileSync
2:49 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsDelete
2:48 gcp-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
2:45 gcp-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:44 aws-windows-latest-4core-16gb TestAccept
2:41 gcp-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions
2:39 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:34 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:33 aws-windows-latest-4core-16gb TestExportDir
2:29 azure-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:27 aws linux-2core-8gb TestWorkspaceFilesExtensions_ExportFormatIsPreserved/jupyter_python
2:24 azure-windows-latest-4core-16gb TestSyncFullFileSync
2:23 azure-windows-latest-4core-16gb TestFilerReadDir/workspace_files
2:23 aws-windows-latest-4core-16gb TestFilerReadWrite/workspace_files_extensions
2:22 gcp linux-2core-8gb TestFilerWorkspaceNotebook/sqlNb.sql
2:22 gcp linux-2core-8gb TestFilerWorkspaceNotebook/sqlJupyterNotebook.ipynb
2:18 azure linux-2core-8gb TestSyncFullFileSync
2:18 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:17 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:13 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/scalaNb.scala
2:12 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat
2:08 azure-windows-latest-4core-16gb TestSyncIncrementalFileSync
2:06 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:06 azure-windows-latest-4core-16gb TestSyncIncrementalSyncPythonNotebookToFile
2:04 aws linux-2core-8gb TestFilerReadWrite/workspace_files_extensions
2:04 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/rNb.r

@pietern

pietern commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

…put helpers

Opt-in capabilities for pinning what a command writes to a terminal:

- GoldenStyled renders SGR as inline markers ({fg=4}?{/}, {bold}).
- GoldenModes logs mode changes and the modes still active on exit,
  covering cursor visibility, bracketed paste, alternate screen, mouse,
  kitty keyboard and modifyOtherKeys.
- GoldenQueries logs terminal queries (DECRQM, kitty flags, OSC 11, CPR,
  DA1/DA2, XTVERSION). Responder answers them after a delay.
- Leftover reports input bytes unread at exit (Unix only).
- TypeBurst, Paste, Settle, WaitForScreen and WaitForOrExit cover input
  and timing that the existing helpers could not express.
- Options gains Env, ColorFromEnv, stream redirection, templates,
  screen rows and newline mapping.

Existing goldens and default behaviour are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@pietern
pietern force-pushed the pietern/termtest-pins branch from d3620d1 to 7f68a1a Compare November 18, 2026 13:07
Adds pin tests that record what the Charm v1 prompt, select, spinner,
pager, tableview and LogString do today: keys (burst input, Alt and
Shift+Enter, kitty Esc, paste), colors across TERM, CI, CLICOLOR and
NO_COLOR, terminal modes on exit, terminal queries and leftover replies,
pager paging and spinner frames, tableview search input, and stderr
styling when a stream is redirected.

The goldens under testdata/ are the reference. When the v2 swap lands,
these tests fail and the golden diffs show the exact user-visible
change. Regenerate with -update only for an intended change.

Co-authored-by: Isaac <no-reply@databricks.com>
@pietern
pietern force-pushed the pietern/termtest-pins branch from 7f68a1a to 58b99e3 Compare November 18, 2026 16:07
tm := runPinPager(t, it)
termtest.GoldenText(t, "01-first-page-loading", observeSpinnerFrames(tm, 2, 3*time.Second))
tm.Type(termtest.KeyEnter)
before := len(tm.Raw())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With WaitForInputRead gone, nothing makes sure the program reads Enter before the first page lands. Type only writes to the pipe, and the spinner keeps ticking while the fetch is in flight, so its next frame satisfies waitForOutputPast whether or not Enter was read.

If the key is read late, startDrain takes the at-prompt branch (pager.go:229, starts a new fetch) instead of the one this test is named for (pager.go:226, drainAll recorded during an in-flight fetch). Both branches print all 120 rows, and onlyPagerRows strips the prompt remnants, so the goldens match either way and the test can't tell which branch it covered.

There's no portable way to see that the key was read, and nothing on screen changes when drainAll is set mid-fetch. So I'd drop the two wait lines and waitForOutputPast (this is its only caller), and name the test after what it does guarantee, e.g. TestPagerPin_EnterDrainsRemainingPages. A short comment could say the key may reach the pager during the fetch or at the prompt, and that both drain. The testdata/ directory would move with the rename. Or is the in-flight branch worth keeping a Unix-only read barrier for?

@pietern

pietern commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 58b99e3. Each finding was verified against the code.

Not ready yet: part of the automated review did not complete.

This PR adds test infrastructure to pin current Charm v1 TUI behavior ahead of a v2 migration, so the swap surfaces its changes as golden diffs. Production changes are minimal and behavior-neutral: an unexported colorFromEnv helper, a NewTestIOWithOptions test constructor, and a RunWithOptions entry point for tableview. The test harness extensions (styled/modes/queries goldens, responder, leftover capture) are well-documented and the 103 golden files serve their stated sentinel purpose. After tracing every candidate claim against the code, no defects survive.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants