Skip to content

fix(vision): align screenshots with controller coordinates - #40

Merged
KhazixW2 merged 2 commits into
MaaXYZ:mainfrom
1204244136:codex/fix-screenshot-coordinates
Oct 1, 2026
Merged

KhazixW2 merged 2 commits into
MaaXYZ:mainfrom
1204244136:codex/fix-screenshot-coordinates

Conversation

@1204244136

@1204244136 1204244136 commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Default both controllers to a 720-pixel short side, preserve screenshot dimensions unless resizing is explicit, and remove duplicate OCR ROI offsets.

Add configurable controller resolution, optional screenshot coordinate transforms, migration documentation, and native regression tests using synthetic images.

Sourcery 摘要

在统一且可配置的坐标系下对齐控制器截图、OCR 和输入操作,同时除非请求调整大小,否则保留截图尺寸不变。

新功能:

  • 允许连接选择控制器截图的短边长度,或使用原生尺寸。
  • 提供可选的截图元数据,用于将裁剪或调整大小后的图像坐标映射回控制器坐标。

错误修复:

  • 将 OCR 区域和返回的框与控制器坐标系对齐,避免重复应用 ROI 偏移。
  • 默认保留截图尺寸,并确保截图、OCR、操作和 Pipeline 坐标保持一致。

增强功能:

  • 围绕可配置的默认 720 像素短边,统一 Win32 和 ADB 的坐标行为。
  • 记录坐标约定、缩放变换,以及模板和 Pipeline ROI 的兼容性要求。

文档:

  • 添加英文和中文文档,涵盖截图坐标约定、控制器分辨率设置,以及图像到控制器的坐标变换。

测试:

  • 添加原生回归测试,覆盖可配置分辨率、坐标一致的点击、裁剪和调整大小变换、四舍五入后的尺寸、Pipeline 缩放匹配、MCP 元数据,以及全局 OCR 框。
Original summary in English

Sourcery 摘要

统一控制器、截图、OCR、输入和 Pipeline 的坐标处理,同时使分辨率和图像到控制器的转换可配置。

新功能:

  • 允许 Win32 和 ADB 连接选择控制器截图的短边尺寸或原生尺寸。
  • 提供可选的截图元数据,用于将裁剪或缩放后的图像坐标映射回控制器坐标。

错误修复:

  • 将截图、OCR 区域和结果、输入操作以及 Pipeline 坐标统一到共享的控制器坐标系中。
  • 防止重复应用 OCR 区域偏移,并在未明确请求缩放时保留截图尺寸。

增强功能:

  • 在 Win32 和 ADB 控制器之间统一默认的 720 像素短边坐标基准。
  • 明确坐标、缩放、裁剪、模板和 Pipeline ROI 的兼容性要求。

文档:

  • 使用英文和中文记录截图坐标约定、可配置的控制器分辨率以及图像到控制器的转换。

测试:

  • 增加原生回归测试,覆盖可配置分辨率、坐标一致的输入、裁剪和缩放转换、四舍五入后的尺寸、Pipeline 缩放匹配、MCP 元数据以及全局 OCR 框。
Original summary in English

Summary by Sourcery

Unify controller, screenshot, OCR, input, and Pipeline coordinate handling while making resolution and image-to-controller transformations configurable.

New Features:

  • Allow Win32 and ADB connections to select a controller screenshot short side or native dimensions.
  • Provide optional screenshot metadata for mapping cropped or resized image coordinates back to controller coordinates.

Bug Fixes:

  • Align screenshots, OCR regions and results, input actions, and Pipeline coordinates to a shared controller coordinate system.
  • Prevent duplicate OCR region offsets and preserve screenshot dimensions unless resizing is explicitly requested.

Enhancements:

  • Standardize the default 720-pixel short-side coordinate basis across Win32 and ADB controllers.
  • Clarify coordinate, scaling, cropping, template, and Pipeline ROI compatibility requirements.

Documentation:

  • Document screenshot coordinate conventions, configurable controller resolutions, and image-to-controller transformations in English and Chinese.

Tests:

  • Add native regression coverage for configurable resolutions, coordinate-consistent input, crop and resize transforms, rounded dimensions, Pipeline scale matching, MCP metadata, and global OCR boxes.

Default both controllers to a 720-pixel short side, preserve screenshot dimensions unless resizing is explicit, and remove duplicate OCR ROI offsets.

Add configurable controller resolution, optional screenshot coordinate transforms, migration documentation, and native regression tests using synthetic images.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

您好——我发现了 1 个问题

AI 代理提示
请处理本次代码审查中的评论:

## 单条评论

### 评论 1
<location path="maa_mcp/vision.py" line_range="149" />
<code_context>
     region: Optional[Tuple[int, int, int, int]] = None,
-    resolution: Optional[int] = 720,
-) -> Optional[str]:
+    resolution: Optional[int] = None,
+    include_metadata: bool = False,
+) -> Optional[Union[str, dict]]:
</code_context>
<issue_to_address>
**issue (bug_risk):** 未经过验证的非整数 `resolution` 会传递到 `_resize_short_edge`;诸如 `1.5` 这样的值会使 `cv2.resize` 接收到浮点型输出尺寸,并引发底层 OpenCV `TypeError`,而不是返回文档中说明的失败结果或清晰的参数错误。

**触发条件:** MCP 调用方无视类型注解,提供浮点型分辨率时。

**建议修复:** 在处理前将 `resolution` 验证为精确的正整数(或 `None`),并抛出清晰的 `ValueError`。
</issue_to_address>

Sourcery 对开源项目免费——如果您喜欢我们的审查结果,请考虑分享 ✨
Original comment in English

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="maa_mcp/vision.py" line_range="149" />
<code_context>
     region: Optional[Tuple[int, int, int, int]] = None,
-    resolution: Optional[int] = 720,
-) -> Optional[str]:
+    resolution: Optional[int] = None,
+    include_metadata: bool = False,
+) -> Optional[Union[str, dict]]:
</code_context>
<issue_to_address>
**issue (bug_risk):** A non-integer `resolution` reaches `_resize_short_edge` without validation; values such as `1.5` make `cv2.resize` receive a floating-point output dimension and raise a low-level OpenCV `TypeError` instead of returning the documented failure result or a clear argument error.

**Triggers:** When an MCP caller supplies a floating-point resolution despite the type annotation.

**Suggested fix:** Validate `resolution` as an exact positive integer (or `None`) before processing and raise a clear `ValueError`.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread maa_mcp/vision.py Resolved
Reject non-integer, boolean, and non-positive resolutions with ValueError before controller access or OpenCV resizing. Cover invalid inputs and preserve the None passthrough mode.
@KhazixW2
KhazixW2 merged commit 3005b29 into MaaXYZ:main Oct 1, 2026
10 checks passed
KhazixW2 added a commit that referenced this pull request Nov 8, 2026
CI 发版时 GitHub Release 正文只有一段静态安装说明,看不出这个版本改了什么。本 PR 接入
git-cliff,让每次发版自动按 conventional commits 汇总本次变更。

## 改动

### `.github/cliff.toml`(新增)

参考 MAAGC 的配置改写,按 commit 前缀分组:

| 前缀 | 分组 |
|------|------|
| `feat` | ✨ 新功能 |
| `fix` | 🐛 Bug修复 |
| `docs` | 📚 文档 |
| `perf` | 🚀 性能优化 |
| `refactor` | 🚜 代码重构 |
| `test` / `style` / `chore` / `ci` / `remove` | 各自一组 |

- 正文条目带上 scope 与 GitHub 用户名(`commit.remote.username`)
- 首次贡献者单列「新贡献者 ❤️」一节,带 PR 链接
- 尾部附安装与使用命令
- 相比 MAAGC 增加了 `remove` 分组(MaaMCP 有一条 `remove:` commit)

### `.github/workflows/publish.yml`

新增 `changelog` job,`build` 改为 `needs: changelog`:

- `actions/checkout` 带 `fetch-depth: 0`(git-cliff 需要完整历史与 tag)与 `ref:
${{ inputs.tag || github.ref }}`(检出版本对应的 tag)
- `--current`:只处理「属于当前 tag 的提交」,即本次要发布的那一版
- 正式版额外加 `--tag-pattern
"^v[0-9]+[.][0-9]+[.][0-9]+$"`,把基线锁定在**上一个正式版**,这样 beta
期间的提交也会并入本次说明;预发布版不加该参数(否则输出为空)
- job 级 `permissions:` 只读(`contents: read` + `pull-requests:
read`)——git-cliff 拉 PR 元数据算贡献者统计需要 `pull-requests: read`
- 生成失败时退回静态安装说明(`continue-on-error` + fallback 步骤),不阻塞发版

顺带修掉旧写法的一个问题:原来 `body:` 用 YAML 双引号标量,换行被折叠掉,Release 正文里的代码块挤成一行(```sh
pip install ... ```)。新的 fallback 用 heredoc 写到 `$GITHUB_OUTPUT`,换行正常保留。

### `pyproject.toml`

`[project.urls]` 从已失效的 `MistEO/MaaMCP` 指向 `MAA-AI/MaaMCP`(PyPI 项目元数据)。

## 验证

- `actionlint 1.7.12` 对改后的 `publish.yml` 无告警
- `git-cliff 2.14.2`(action 默认版本)在真实仓库跑通,含**detached HEAD 于 tag** 的检出(与
CI 一致),v1.3.0 输出:

  ```
  ## 1.3.0 (2026-11-08)

  ### ✨ 新功能
- *(pipeline)* Run_pipeline/benchmark_node 支持单次 pipeline_override 与工具侧
timeout_seconds @qiyinxi
  - *(pipeline)* 支持 JSONC 语法(pipeline 文件与 interface.json)  @KhazixW2

  ### 🐛 Bug修复
  - *(jsonc)* 处理 Sourcery 复查指出的超大整数、内存占用与同名窗口问题  @KhazixW2
- *(vision)* Align screenshots with controller coordinates @1204244136
- *(vision)* Validate screenshot resolution before processing
@1204244136

  ## 新贡献者 ❤️
  * @1204244136 made their first contribution in [#40](...)
  * @qiyinxi made their first contribution in [#39](...)
  ```

- 预发布路径(不加 `--tag-pattern`)同样验证通过
- 确认 action 的 `run.sh` 会按 bash 规则拆分带引号的 `--tag-pattern` 正则,不会被截断
- **在真实 GitHub Actions 里端到端跑通**:用一个临时自检工作流复刻 `publish.yml` 的 job(同样的
`permissions:`、同样的 `--current`/`--tag-pattern` 参数),断言
  - 正文包含版本号标题 `## <version>`
  - 正文使用 cliff.toml 的自定义分组(`### 🐛 Bug修复`),而不是 git-cliff 的内置默认格式
  - fallback **没有**被触发

  该自检在 GitHub Actions 上通过后已删除(不属于本 PR 的最终改动)。

## 备注

本次同时把已发布的 v1.3.0 Release 正文补全了(原先是折叠成单行的静态说明)。

---

<details>
<summary>Original summary in English</summary>

## Sourcery summary

Automatically generate rich GitHub Release notes while ensuring the
publish workflow keeps working even if changelog generation fails.

New features:
- Generate GitHub Release notes automatically from Conventional Commits,
including categorized changes, contributor attribution, first-time
contributors, and installation and usage instructions.

Bug fixes:
- Preserve multiline formatting in the fallback Release body so
installation and usage code blocks render correctly.

Enhancements:
- Use release-specific commit ranges for stable and prerelease versions
while allowing publishing to continue with static fallback content if
changelog generation fails.

CI:
- Add a dedicated changelog job to the publishing workflow and pass its
output to the GitHub Release step.

Tests:
- Validate changelog generation for stable releases, including the
generated version entry and tagged checkout behavior.

Chores:
- Update project URLs to the MAA-AI GitHub organization.

</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants