宿主侧点击：modavd_touch_* 走模拟器控制台

CMakeLists.txt

@@ -28,13 +28,21 @@ if(NOT WIN32)
   endif()
 endif()
 
-set(MX_SOURCES src/core.c src/shm.c src/console.c src/log.c)
+set(MX_SOURCES src/core.c src/shm.c src/console.c src/console_command.c src/input.c src/log.c)
+
+if(NOT WIN32)
+  # POSIX: the per-handle input lock.
+  find_package(Threads REQUIRED)
+endif()
 
 function(mx_link_system target)
   if(WIN32)
     target_link_libraries(${target} PRIVATE ws2_32)
-  elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux")
-    target_link_libraries(${target} PRIVATE rt)
+  else()
+    target_link_libraries(${target} PRIVATE Threads::Threads)
+    if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+      target_link_libraries(${target} PRIVATE rt)
+    endif()
   endif()
 endfunction()
 
@@ -57,6 +65,11 @@ target_link_libraries(test_core PRIVATE mx_core)
 mx_link_system(test_core)
 add_test(NAME test_core COMMAND test_core)
 
+add_executable(test_input tests/test_input.c)
+target_link_libraries(test_input PRIVATE mx_core)
+mx_link_system(test_input)
+add_test(NAME test_input COMMAND test_input)
+
 if(NOT WIN32)
   add_executable(test_open_posix tests/test_open_posix.c)
   target_link_libraries(test_open_posix PRIVATE mx_core)

README.md

@@ -10,7 +10,7 @@
 
 ## 这是什么
 
-一个 C 库：从 [ModAVD](https://github.com/ModAVD/ModAVD) 实例直接取画面，不经 adb。给 MAA（MaaCore）、MaaFramework 这类自动化框架接入用。以后还会加宿主侧的点击和按键。
+一个 C 库：从 [ModAVD](https://github.com/ModAVD/ModAVD) 实例直接取画面、在宿主这边点击，不经 adb。给 MAA（MaaCore）、MaaFramework 这类自动化框架接入用。
 
 - 画面来自模拟器自己的共享内存：`screenrecord webrtc start`（或启动参数 `-share-vid`）之后，模拟器把每一帧写进一块按控制台端口命名的共享内存。
   - Windows 上叫 `SHM_videmulator<端口>`，Linux、macOS 上叫 `videmulator<端口>`；
@@ -21,12 +21,13 @@
   - 拷出一帧，换成调用方要的格式；
   - 遇到新帧写到一半时重拷；
   - 告诉调用方现在能不能用。
+- 点击走模拟器控制台的 `event send`，发标准的多点触控事件（槽位、跟踪编号、坐标、压力），到达模拟器自己的虚拟触屏：安卓看到的是屏幕上的手指，和在模拟器窗口里点一样，不是应用注入的事件，客体里也不用装东西。最多 10 根手指。
 - 库**不自己退回 adb**：出错时返回错误码，让调用方按自己的规则换别的办法。
 - 只依赖系统库：Windows 上是 `KERNEL32`、`WS2_32`，C 运行库静态链接；POSIX 上是 `shm_open`、`mmap` 和套接字。
 
 ## 状态
 
-- v0.1：只有截图。点击、按键、文字的函数已经声明，现在返回 `MODAVD_E_UNSUPPORTED`。
+- 截图（接口版本 1）和点击（接口版本 2，`MODAVD_CAP_TOUCH`）。按键、文字的函数已经声明，现在返回 `MODAVD_E_UNSUPPORTED`：控制台发的按键会进触屏设备、被丢掉，`event text` 会悄悄丢掉键盘映射里没有的字。
 - Windows：截图这套逻辑在 ModAVD 上已经用了一个多月，形式是下面的兼容外壳。
 - macOS（Apple Silicon）：11-09 有贡献者用官方 Emulator 37.2.13、API 35 ARM64 实测过截图。
 - Linux：能编，单元测试在 CI 上跑；还没有在模拟器上实测。
@@ -106,11 +107,12 @@ ctest --test-dir build --output-on-failure
 
 ## English
 
-ModAVDExtras is a small C library that reads the screen of a [ModAVD](https://github.com/ModAVD/ModAVD) instance directly, without adb, for automation frameworks such as MaaCore (MAA) and MaaFramework. Host-side touch and key input will follow.
+ModAVDExtras is a small C library that reads the screen of a [ModAVD](https://github.com/ModAVD/ModAVD) instance directly, without adb, for automation frameworks such as MaaCore (MAA) and MaaFramework, and touches it from the host.
 
 - Frames come from the emulator's own shared memory: after `screenrecord webrtc start` (or `-share-vid`), the emulator writes every frame to a region named after its console port: `SHM_videmulator<port>` on Windows, `videmulator<port>` elsewhere. The layout is a 24-byte header (width, height, fps, frame number, timestamp) followed by BGRA pixels.
 - The library finds the console port from an adb address, starts frame sharing through the console when it is off, copies a frame in the format the caller asks for (copying again if a new frame arrived meanwhile), and reports whether capture is usable. It never falls back to adb by itself.
-- Version 0.1 does screen capture only; the input functions are declared and return `MODAVD_E_UNSUPPORTED`. See [include/modavd_extras.h](include/modavd_extras.h).
+- Touch (API version 2, `MODAVD_CAP_TOUCH`) goes through the console's `event send` as standard multi-touch events (slot, tracking id, position, pressure) to the emulator's own virtual touch screen: Android sees a finger on the screen, as with a click on the emulator window, not an injected event, and nothing is installed in the guest. Up to 10 contacts.
+- Keys and text are declared and return `MODAVD_E_UNSUPPORTED`: keys sent through the console reach the touch screen device and are dropped, and `event text` silently drops characters missing from the keyboard map. See [include/modavd_extras.h](include/modavd_extras.h).
 - `external_renderer_ipc.dll` is a Windows-only compatibility shell. It exports the four capture functions that MaaCore looks up in MuMu's DLL, on top of the same code, so MaaCore versions that do not know ModAVD can still capture from it. AUTO-MAS uses it today. It will be retired once MaaCore can connect to ModAVD directly.
 - Status: Windows has used this capture code for over a month through the shell below; on Apple Silicon macOS a contributor tested capture with the official Emulator 37.2.13 and API 35 ARM64; Linux builds and passes the unit tests in CI but has not been tested with an emulator. From v0.1.2 the [releases](https://github.com/ModAVD/ModAVDExtras/releases) carry `modavd_extras.dll` (Windows x64), `libmodavd_extras.dylib` (macOS arm64) and `libmodavd_extras.so` (Linux x64), plus the Windows-only shell.
 - Build: `build.ps1` on Windows (Visual Studio 2022 Build Tools, CMake, Ninja); CMake on Linux and macOS. The unit tests need no emulator.

include/modavd_extras.h

@@ -1,5 +1,5 @@
 /*
- * ModAVDExtras: screen capture (and, later, input) for ModAVD instances,
+ * ModAVDExtras: screen capture and touch input for ModAVD instances,
  * for automation frameworks that want to skip adb.
  *
  * Copyright 2026 qiyinxi
@@ -37,8 +37,10 @@ extern "C" {
 #define MODAVD_API __attribute__((visibility("default")))
 #endif
 
-/* Bumped whenever a function or field is added. Nothing is ever removed. */
-#define MODAVD_API_VERSION 1
+/* Bumped whenever a function or field is added, or a declared one starts to work.
+ * Nothing is ever removed.
+ * 1: capture. 2: touch (modavd_touch_*). */
+#define MODAVD_API_VERSION 2
 
 typedef struct modavd_instance* modavd_handle;
 
@@ -57,7 +59,7 @@ enum {
 /* modavd_capabilities() */
 enum {
     MODAVD_CAP_SCREENCAP = 1u << 0,
-    MODAVD_CAP_TOUCH = 1u << 1, /* not yet */
+    MODAVD_CAP_TOUCH = 1u << 1, /* API version 2 */
     MODAVD_CAP_KEY = 1u << 2,   /* not yet */
     MODAVD_CAP_TEXT = 1u << 3,  /* not yet */
 };
@@ -154,14 +156,30 @@ MODAVD_API int modavd_console_port(modavd_handle h);      /* > 0, or MODAVD_E_IN
 MODAVD_API int modavd_capture(modavd_handle h, int display, uint32_t format, uint32_t flags, void* buf,
                               size_t buf_size, modavd_frame_info* info);
 
-/* ---- input: declared for the next version, MODAVD_E_UNSUPPORTED for now ---- */
+/* ---- touch (API version 2, MODAVD_CAP_TOUCH) ---- */
 
-/* Coordinates are display pixels; contact is 0-based (at most 10 at once). */
+/* Touches on the emulator's own touch screen, sent through its console: Android
+ * sees them as a finger on the screen, not as injected events. Needs nothing in
+ * the guest. The console session opens on the first call and reopens once if it
+ * broke (the emulator restarted).
+ *
+ * Coordinates are display pixels of the shared frame (clamped to it); contact is
+ * 0-based, at most 10 at once; pressure 0 = the default (what a mouse click on
+ * the emulator window uses). Only display 0.
+ * - touch_down on a contact that is already down moves it.
+ * - touch_move on a contact that is not down: MODAVD_E_INVALID.
+ * - touch_up on a contact that is not down sends nothing and returns MODAVD_OK.
+ * MODAVD_E_NOT_READY: no frame was published yet, so the display size is unknown.
+ * MODAVD_E_NOT_FOUND: the console does not answer. MODAVD_E_IO: auth failed or
+ * the console refused the command. */
 MODAVD_API int modavd_touch_down(modavd_handle h, int display, int contact, int x, int y, int pressure);
 MODAVD_API int modavd_touch_move(modavd_handle h, int display, int contact, int x, int y, int pressure);
 MODAVD_API int modavd_touch_up(modavd_handle h, int display, int contact);
-/* Lifts every contact this handle pressed. modavd_close() will do the same. */
+/* Lifts every contact this handle pressed. modavd_close() does the same. */
 MODAVD_API int modavd_touch_release_all(modavd_handle h);
+
+/* ---- keys and text: declared, MODAVD_E_UNSUPPORTED for now ---- */
+
 /* Android key code (KEYCODE_*); down = 1 press, 0 release. */
 MODAVD_API int modavd_key(modavd_handle h, int display, int android_keycode, int down);
 MODAVD_API int modavd_input_text(modavd_handle h, const char* utf8);

src/console.c

@@ -1,6 +1,7 @@
 /*
  * A minimal client for the emulator console (telnet-style text on 127.0.0.1).
- * Version 0 only uses it to start frame sharing; input will use it too.
+ * Used to start frame sharing (one-off connections) and to send touch input
+ * (a session kept open per handle).
  *
  * Copyright 2026 qiyinxi
  * SPDX-License-Identifier: LGPL-3.0-or-later
@@ -97,8 +98,9 @@ static int send_line(mx_socket s, const char* line)
     return send(s, cmd, n, MX_SEND_FLAGS) == n ? 0 : -1;
 }
 
-int mx_console_command(int port, const char* command, int timeout_ms)
+int mx_console_open(int port, int timeout_ms, mx_console* c)
 {
+    c->sock = (intptr_t)MX_BAD_SOCKET;
     char token[256];
     read_auth_token(token, sizeof(token));
 
@@ -108,55 +110,88 @@ int mx_console_command(int port, const char* command, int timeout_ms)
         return MODAVD_E_IO;
     }
 #endif
-    int rc = MODAVD_E_NOT_FOUND;
     mx_socket s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
-    if (s != MX_BAD_SOCKET) {
+    if (s == MX_BAD_SOCKET) {
 #ifdef _WIN32
-        DWORD tv = (DWORD)timeout_ms;
+        WSACleanup();
+#endif
+        return MODAVD_E_IO;
+    }
+#ifdef _WIN32
+    DWORD tv = (DWORD)timeout_ms;
 #else
-        struct timeval tv = { timeout_ms / 1000, (timeout_ms % 1000) * 1000 };
+    struct timeval tv = { timeout_ms / 1000, (timeout_ms % 1000) * 1000 };
 #endif
-        setsockopt(s, SOL_SOCKET, SO_RCVTIMEO, (const char*)&tv, sizeof(tv));
+    setsockopt(s, SOL_SOCKET, SO_RCVTIMEO, (const char*)&tv, sizeof(tv));
 #if defined(SO_NOSIGPIPE)
-        int one = 1;
-        setsockopt(s, SOL_SOCKET, SO_NOSIGPIPE, &one, sizeof(one));
+    int one = 1;
+    setsockopt(s, SOL_SOCKET, SO_NOSIGPIPE, &one, sizeof(one));
 #endif
-        struct sockaddr_in addr;
-        memset(&addr, 0, sizeof(addr));
-        addr.sin_family = AF_INET;
-        addr.sin_port = htons((unsigned short)port);
-        addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
-        char buf[4096];
-        if (connect(s, (struct sockaddr*)&addr, sizeof(addr)) == 0 && read_until_ok(s, buf, sizeof(buf)) == 1) {
-            int authed = 1;
-            if (token[0]) {
-                char auth[300];
-                snprintf(auth, sizeof(auth), "auth %s", token);
-                authed = send_line(s, auth) == 0 && read_until_ok(s, buf, sizeof(buf)) == 1;
-            }
-            if (!authed) {
-                mx_log(MODAVD_LOG_WARN, "console %d auth failed", port);
-                rc = MODAVD_E_IO;
-            }
-            else {
-                int ok = send_line(s, command) == 0 && read_until_ok(s, buf, sizeof(buf)) == 1;
-                mx_log(MODAVD_LOG_INFO, "console %d %s -> %s", port, command, ok ? "OK" : "fail");
-                rc = ok ? MODAVD_OK : MODAVD_E_IO;
-            }
-            send_line(s, "quit");
-        }
-        else {
+    struct sockaddr_in addr;
+    memset(&addr, 0, sizeof(addr));
+    addr.sin_family = AF_INET;
+    addr.sin_port = htons((unsigned short)port);
+    addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+    char buf[4096];
+    int rc = MODAVD_OK;
+    if (connect(s, (struct sockaddr*)&addr, sizeof(addr)) != 0 || read_until_ok(s, buf, sizeof(buf)) != 1) {
 #ifdef _WIN32
-            int err = WSAGetLastError();
+        int err = WSAGetLastError();
 #else
-            int err = 0;
+        int err = 0;
 #endif
-            mx_log(MODAVD_LOG_WARN, "console %d connect failed (%d)", port, err);
+        mx_log(MODAVD_LOG_WARN, "console %d connect failed (%d)", port, err);
+        rc = MODAVD_E_NOT_FOUND;
+    }
+    else if (token[0]) {
+        char auth[300];
+        snprintf(auth, sizeof(auth), "auth %s", token);
+        if (send_line(s, auth) != 0 || read_until_ok(s, buf, sizeof(buf)) != 1) {
+            mx_log(MODAVD_LOG_WARN, "console %d auth failed", port);
+            rc = MODAVD_E_IO;
         }
+    }
+    if (rc != MODAVD_OK) {
         mx_close_socket(s);
+#ifdef _WIN32
+        WSACleanup();
+#endif
+        return rc;
+    }
+    c->sock = (intptr_t)s;
+    return MODAVD_OK;
+}
+
+int mx_console_send(mx_console* c, const char* command)
+{
+    if (c->sock == (intptr_t)MX_BAD_SOCKET) {
+        return MODAVD_E_NOT_FOUND;
+    }
+    mx_socket s = (mx_socket)c->sock;
+    char buf[4096];
+    if (send_line(s, command) != 0) {
+        mx_console_close(c);
+        return MODAVD_E_NOT_FOUND;
+    }
+    int r = read_until_ok(s, buf, sizeof(buf));
+    if (r < 0) {
+        /* The emulator went away or stopped answering: the session is unusable. */
+        mx_console_close(c);
+        return MODAVD_E_NOT_FOUND;
+    }
+    return r == 1 ? MODAVD_OK : MODAVD_E_IO;
+}
+
+void mx_console_close(mx_console* c)
+{
+    if (c->sock == (intptr_t)MX_BAD_SOCKET) {
+        return;
     }
+    mx_socket s = (mx_socket)c->sock;
+    send_line(s, "quit");
+    mx_close_socket(s);
+    c->sock = (intptr_t)MX_BAD_SOCKET;
 #ifdef _WIN32
     WSACleanup();
 #endif
-    return rc;
 }

src/console_command.c

@@ -0,0 +1,22 @@
+/*
+ * One console command on its own connection (used to start frame sharing).
+ * Kept apart from console.c so that a test can replace it while the touch
+ * session functions still come from the library.
+ *
+ * Copyright 2026 qiyinxi
+ * SPDX-License-Identifier: LGPL-3.0-or-later
+ */
+#include "mx_internal.h"
+
+int mx_console_command(int port, const char* command, int timeout_ms)
+{
+    mx_console c;
+    int rc = mx_console_open(port, timeout_ms, &c);
+    if (rc != MODAVD_OK) {
+        return rc;
+    }
+    rc = mx_console_send(&c, command);
+    mx_log(MODAVD_LOG_INFO, "console %d %s -> %s", port, command, rc == MODAVD_OK ? "OK" : "fail");
+    mx_console_close(&c);
+    return rc == MODAVD_OK ? MODAVD_OK : MODAVD_E_IO;
+}

src/core.c

@@ -133,10 +133,14 @@ int modavd_open(const modavd_open_params* params, modavd_handle* out)
     }
 
     struct modavd_instance* inst = (struct modavd_instance*)calloc(1, sizeof(*inst));
-    if (!inst) {
+    struct mx_input* input = mx_input_create(port);
+    if (!inst || !input) {
+        free(inst);
+        mx_input_destroy(input);
         mx_shm_close(&shm);
         return MODAVD_E_IO;
     }
+    inst->input = input;
     inst->console_port = port;
     inst->simd = mx_cpu_simd();
     inst->shm = shm;
@@ -152,11 +156,13 @@ void modavd_close(modavd_handle h)
     if (!h) {
         return;
     }
+    /* Lifts the contacts this handle left down. */
+    mx_input_destroy(h->input);
     mx_shm_close(&h->shm);
     free(h);
 }
 
-uint32_t modavd_capabilities(modavd_handle h) { return h ? MODAVD_CAP_SCREENCAP : 0; }
+uint32_t modavd_capabilities(modavd_handle h) { return h ? MODAVD_CAP_SCREENCAP | MODAVD_CAP_TOUCH : 0; }
 
 int modavd_console_port(modavd_handle h) { return h ? h->console_port : MODAVD_E_INVALID; }
 
@@ -364,40 +370,3 @@ int modavd_capture(modavd_handle h, int display, uint32_t format, uint32_t flags
     }
     return rc;
 }
-
-/* Input arrives in the next version. */
-int modavd_touch_down(modavd_handle h, int display, int contact, int x, int y, int pressure)
-{
-    (void)h, (void)display, (void)contact, (void)x, (void)y, (void)pressure;
-    return MODAVD_E_UNSUPPORTED;
-}
-
-int modavd_touch_move(modavd_handle h, int display, int contact, int x, int y, int pressure)
-{
-    (void)h, (void)display, (void)contact, (void)x, (void)y, (void)pressure;
-    return MODAVD_E_UNSUPPORTED;
-}
-
-int modavd_touch_up(modavd_handle h, int display, int contact)
-{
-    (void)h, (void)display, (void)contact;
-    return MODAVD_E_UNSUPPORTED;
-}
-
-int modavd_touch_release_all(modavd_handle h)
-{
-    (void)h;
-    return MODAVD_E_UNSUPPORTED;
-}
-
-int modavd_key(modavd_handle h, int display, int android_keycode, int down)
-{
-    (void)h, (void)display, (void)android_keycode, (void)down;
-    return MODAVD_E_UNSUPPORTED;
-}
-
-int modavd_input_text(modavd_handle h, const char* utf8)
-{
-    (void)h, (void)utf8;
-    return MODAVD_E_UNSUPPORTED;
-}