Skip to content

Scroll full-screen TUI with mouse wheel on paired phones - #310

Merged
johannesjo merged 3 commits into
mainfrom
ccr-51034d92-71dhul
Oct 11, 2026
Merged

johannesjo merged 3 commits into
mainfrom
ccr-51034d92-71dhul

Conversation

@johannesjo

Copy link
Copy Markdown
Owner

Adds support for scrolling full-screen terminal UIs (like vim or Claude Code's fullscreen mode) on paired phones by converting touch drags into mouse wheel reports that the TUI can read, keeping its transcript out of the terminal history.

Changes

Android app:

  • Added AgentWheel class implementing NestedScrollConnection to convert vertical drag gestures into mouse wheel turns for full-screen agents
  • Modified TerminalText to apply nested scroll handling and track line height for wheel calculations
  • Added TerminalScreen.alternateScreen property to detect when a full-screen program is active
  • Updated AgentScreen to pass wheel scroll callback when in full-screen mode
  • Added unit tests for AgentWheel behavior

Web/Electron:

  • Added ScrollCommand protocol message type for wheel turns over a specific cell
  • Implemented scrollAgent() function in ws.ts to send scroll commands from the phone
  • Added scrollAgent() handler in pty.ts that converts wheel turns to mouse reports via the terminal's query responder
  • Implemented wheelReport() in TerminalQueryResponder to generate SGR or default mouse encoding reports based on the program's active mouse tracking mode
  • Modified AgentDetail to detect full-screen mode and route scrolls to scrollAgent() instead of terminal history scrolling
  • Added validation and tests for scroll message parsing and handling

Key implementation details:

  • Wheel reports are only sent when the program is actively reading mouse input (not in X10 mode or when disabled)
  • Scroll messages are rate-limited to 100 lines per message to avoid overwhelming the PTY
  • Full-screen detection uses xterm's alternate screen buffer state
  • Mouse encoding (SGR vs default) is tracked by monitoring DECSET mode changes
  • Scrolls over full-screen UIs bypass the terminal history, allowing the program to manage its own scrollback

https://claude.ai/code/session_017VZQeYWdHG5Mpbg1FEyvZk

Claude Code's fullscreen renderer runs in the alternate screen, which has
no terminal history. The desktop pane scrolls it by sending mouse-wheel
reports to the agent; the phone only scrolled its local xterm buffer, so
it showed a single page.

A swipe over an alternate-screen terminal now sends a `scroll` message.
The server's mirror tracks the mouse encoding the agent asked for and
writes matching wheel reports, only while the agent reads the wheel.
Scrolling bypasses draft tracking so it never holds back peer prompts.
Like view-size, it needs a paired phone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017VZQeYWdHG5Mpbg1FEyvZk
The Android terminal had the same single-page limit as the phone web UI
in Claude Code's fullscreen mode, and it showed the shell's main-screen
history above the alternate screen, which the desktop never shows.

The alternate screen now shows alone, as in xterm. While it is up on a
paired phone, a vertical drag the terminal view cannot follow turns into
the server's `scroll` message, so the agent scrolls its own transcript.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017VZQeYWdHG5Mpbg1FEyvZk
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-11T11:27:14.441965Z 690f891 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 690f89140f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

if (mouseEncoding === 'default') {
// Each value is one byte offset by 32, so cells past 223 cannot be reported.
if (x > 223 || y > 223) return null;
report = `\x1b[M${String.fromCharCode(button + 32, x + 32, y + 32)}`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve binary bytes in default mouse reports

When a TUI uses the legacy/default mouse encoding and the target column or row is 96–223, x + 32 or y + 32 is at least 128. This constructs those protocol bytes as Unicode characters, after which scrollAgent passes the string to a PTY configured with node-pty's default UTF-8 encoding, expanding each required single byte into a multibyte sequence and making wheel events over that part of a wide/tall terminal unparsable. Xterm deliberately emits default mouse reports through its binary channel, while node-pty accepts a Buffer; return binary data here or reject coordinates above 95. Xterm default encoder, node-pty encoding contract.

Useful? React with 👍 / 👎.

Default-encoded mouse reports carry cell positions as raw bytes. Written
to the PTY as a string, a column or row of 96 or more became two UTF-8
bytes, and the program read a broken report followed by a stray key.

The desktop pane never sends those reports at all: xterm emits them as
binary events, which the renderer does not forward. Match it and send
wheel turns only to programs that asked for SGR reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017VZQeYWdHG5Mpbg1FEyvZk
@johannesjo
johannesjo merged commit ea0b553 into main Oct 11, 2026
5 checks passed
@johannesjo
johannesjo deleted the ccr-51034d92-71dhul branch October 11, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants