Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/aws-device-farm-plugin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'agent-device': minor
---

Install AWS Device Farm support with `agent-device plugins add @agent-device/aws-device-farm`. Close existing sessions and stop the daemon for its state directory before reconnecting after upgrading.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This upgrade instruction is misleading for both current availability and embedded SDK hosts: the package is still unpublished, and embedded runtimes neither read the CLI plugin store nor support daemon stop. State the publication prerequisite, then give CLI hosts the store/restart path and embedded hosts the startAgentDevice({ plugins }) plus close/recreate path.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .changeset/aws-device-farm-plugin.md, line 5:

<comment>This upgrade instruction is misleading for both current availability and embedded SDK hosts: the package is still unpublished, and embedded runtimes neither read the CLI plugin store nor support `daemon stop`. State the publication prerequisite, then give CLI hosts the store/restart path and embedded hosts the `startAgentDevice({ plugins })` plus close/recreate path.</comment>

<file context>
@@ -0,0 +1,5 @@
+'agent-device': minor
+---
+
+Install AWS Device Farm support with `agent-device plugins add @agent-device/aws-device-farm`. Close existing sessions and stop the daemon for its state directory before reconnecting after upgrading.
</file context>
Suggested change
Install AWS Device Farm support with `agent-device plugins add @agent-device/aws-device-farm`. Close existing sessions and stop the daemon for its state directory before reconnecting after upgrading.
Release prerequisite: the plugin remains private and unavailable from npm until a usable package is published through the configured release workflow. CLI hosts can then run `agent-device plugins add @agent-device/aws-device-farm`, close existing sessions, and stop the matching daemon before reconnecting; embedded hosts supply the factory through `startAgentDevice({ plugins: [awsDeviceFarm()] })` and close/recreate the returned runtime instead.

1 change: 1 addition & 0 deletions .fallowrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
"tools/depgraph-view/app.js",
"scripts/check-provider-plugin.mjs",
"scripts/check-affected/command.ts",
"packages/provider-aws-device-farm/test/package-smoke.mjs",
"packages/provider-browserstack/test/package-smoke.mjs",
"packages/provider-browserstack/tsdown.config.ts",
"packages/provider-testmu/test/package-smoke.mjs",
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ The same session and evidence model works at every step: the agent explores the
| --- | --- | --- |
| Local | Trying commands and debugging apps on simulators, emulators, physical devices, macOS, and Linux. | Follow the Quick Start. |
| CI/CD | Automated pull request and merge validation with replay scripts and captured artifacts. | Try the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml). |
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack and TestMu AI through `agent-device plugins add`, AWS Device Farm, Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack, AWS Device Farm, and TestMu AI through `agent-device plugins add`, Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This parenthetical makes agent-device plugins add apply only to TestMu AI, leaving BrowserStack and AWS Device Farm's plugin requirement unclear. Say that all three providers require plugin installation.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At README.md, line 154:

<comment>This parenthetical makes `agent-device plugins add` apply only to TestMu AI, leaving BrowserStack and AWS Device Farm's plugin requirement unclear. Say that all three providers require plugin installation.</comment>

<file context>
@@ -151,7 +151,7 @@ The same session and evidence model works at every step: the agent explores the
 | Local | Trying commands and debugging apps on simulators, emulators, physical devices, macOS, and Linux. | Follow the Quick Start. |
 | CI/CD | Automated pull request and merge validation with replay scripts and captured artifacts. | Try the [EAS workflow template](https://github.com/callstackincubator/eas-agent-device/blob/main/.eas/workflows/agent-qa-mobile.yml). |
-| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack and TestMu AI through `agent-device plugins add`, AWS Device Farm, Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |
+| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack, AWS Device Farm, and TestMu AI through `agent-device plugins add`, Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |
 
 ## How it works
</file context>
Suggested change
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack, AWS Device Farm, and TestMu AI through `agent-device plugins add`, Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |
| Cloud / remote | Linux runners, managed devices, and remote jobs. | Set up a [remote proxy](https://agent-device.dev/docs/remote-proxy), connect a [device cloud](https://agent-device.dev/docs/device-clouds) (BrowserStack, AWS Device Farm, and TestMu AI, each installed with `agent-device plugins add`, or Limrun), or [contact Callstack](mailto:hello@callstack.com) for team QA. |


## How it works

Expand Down
1 change: 1 addition & 0 deletions examples/sdk/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
{ "path": "../../packages/provider-limrun" },
{ "path": "../../packages/provider-revyl" },
{ "path": "../../packages/provider-testmu" },
{ "path": "../../packages/provider-aws-device-farm" },
{ "path": "../../packages/provider-browserstack" },
{ "path": "../../packages/provider-webos" },
{ "path": "../../packages/provider-tizen" },
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -376,10 +376,10 @@
"yaml": "^2.9.0",
"yauzl": "^3.4.0",
"@agent-device/testmu": "workspace:*",
"@agent-device/aws-device-farm": "workspace:*",
"@agent-device/browserstack": "workspace:*",
"@agent-device/revyl": "workspace:*",
"@agent-device/webos": "workspace:*",
"@agent-device/tizen": "workspace:*",
"@agent-device/build-config": "workspace:*"
"@agent-device/tizen": "workspace:*"
}
}
2 changes: 1 addition & 1 deletion packages/command-surface/src/replay/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -319,7 +319,7 @@ export const testDaemonWriter: AsyncDaemonWriter = async (input, host, deps) =>
await import('@agent-device/replay-port/replay-backend-registry')
).withReplayBackend(
'maestro',
host.env,
host,
(backend) => collectSources(async () => backend),
deps?.resolveReplayBackend,
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ export async function resourceAdmissionWorld(
]
: []),
]),
input.host.env,
input.host,
{
adaptWebDriver: async () => {
throw new Error('Resource fixture cannot create providers');
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ async function fixture(storeEnv?: NodeJS.ProcessEnv) {
const discover = vi.fn(async () => []);
const registry = await startProviderRegistry(
await resolveProviderPlugins([]),
{},
{ env: {}, cwd: '/' },
{
assertReplaySelectionCurrent: storeEnv ? createPluginSelectionGuard([], storeEnv) : undefined,
adaptWebDriver: async () => {
Expand Down
3 changes: 3 additions & 0 deletions packages/maestro/src/plugin.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ const setupHost = {
env: {},
options: {},
clientVersion: '0.22.0-dev',
runHostCommand: async () => {
throw new Error('host command unused');
},
createError: () => {
throw new Error('setup must not use host services');
},
Expand Down
2 changes: 1 addition & 1 deletion packages/maestro/test/packed-runtime.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ const { default: maestroPlugin } = await import(
async function registry() {
const registry = await startProviderRegistry(
await resolveProviderPlugins([{ source: 'embedder', load: () => maestroPlugin() }]),
{},
{ env: {}, cwd: '/' },
{
adaptWebDriver: async () => {
throw new Error('not a provider');
Expand Down
2 changes: 1 addition & 1 deletion packages/plugins/src/connection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { storePluginCandidates } from './store.ts';
export type PluginConnectionCallbacks = Readonly<{
resolve(context: Omit<PluginConnectionContext, 'host'>): ReturnType<PluginConnection['resolve']>;
verify(
context: Omit<Pick<PluginConnectionContext, 'flags' | 'env' | 'host'>, 'host'>,
context: Omit<Pick<PluginConnectionContext, 'flags' | 'env' | 'cwd' | 'host'>, 'host'>,
): ReturnType<PluginConnection['verify']>;
}>;

Expand Down
147 changes: 147 additions & 0 deletions packages/plugins/src/host-command.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
import { expect, test } from 'vitest';
import { withCommandExecutorOverride } from '@agent-device/host-kit/command';
import { createPluginHost } from './host.ts';

test('plugin host runs commands with the AWS command bounds', async () => {
await withCommandExecutorOverride(
async (command, args, options) => {
expect(command).toBe('aws');
expect(args).toEqual(['devicefarm', 'get-project']);
expect(options.maxBuffer).toBe(10 * 1024 * 1024);
expect(options.timeoutMs).toBe(30_000);
return { stdout: '{"project":{}}', stderr: '', exitCode: 0 };
},
async () => {
await expect(
createPluginHost({ env: {}, cwd: '/' }, undefined).runHostCommand('aws', [
'devicefarm',
'get-project',
]),
).resolves.toEqual({ stdout: '{"project":{}}' });
},
);
});

test('commands use the supplied host and overlay request environment without ambient values', async () => {
const { mkdtempForTestSync } = await import('@agent-device/test-kit');
const { realpathSync } = await import('node:fs');
const { withCommandHost } = await import('@agent-device/host-kit/command');
const cwd = realpathSync(mkdtempForTestSync('plugin-command-host-'));
const host = { cwd, env: { PLUGIN_VALUE: 'host', HOST_ONLY: 'present' } };
const pluginHost = createPluginHost(host, undefined);
const args = [
'-e',
'process.stdout.write(JSON.stringify({cwd:process.cwd(), value:process.env.PLUGIN_VALUE, hostOnly:process.env.HOST_ONLY, ambient:process.env.PLUGIN_AMBIENT}))',
];
const previous = process.env.PLUGIN_AMBIENT;
process.env.PLUGIN_AMBIENT = 'ambient';
try {
const plain = await pluginHost.runHostCommand(process.execPath, args);
expect(JSON.parse(plain.stdout)).toEqual({ cwd, value: 'host', hostOnly: 'present' });
const scoped = await withCommandHost({ host, requestEnv: { PLUGIN_VALUE: 'request' } }, () =>
pluginHost.runHostCommand(process.execPath, args),
);
expect(JSON.parse(scoped.stdout)).toEqual({ cwd, value: 'request', hostOnly: 'present' });
} finally {
if (previous === undefined) delete process.env.PLUGIN_AMBIENT;
else process.env.PLUGIN_AMBIENT = previous;
}
});

test('connection resolve and verify use each call host independently', async () => {
const { mkdtempForTestSync } = await import('@agent-device/test-kit');
const { realpathSync } = await import('node:fs');
const { resolveProviderPlugins } = await import('./registry.ts');
const dirs = [
realpathSync(mkdtempForTestSync('plugin-resolve-')),
realpathSync(mkdtempForTestSync('plugin-verify-')),
];
const observed: unknown[] = [];
const run = async (host: import('./host.ts').ProviderPluginHost) => {
const result = await host.runHostCommand(process.execPath, [
'-e',
'process.stdout.write(JSON.stringify([process.cwd(), process.env.CONNECTION_VALUE]))',
]);
observed.push(JSON.parse(result.stdout));
};
const { connections } = await resolveProviderPlugins([
{
source: 'embedder',
load: () => ({
apiVersion: 2,
provider: 'example',
credentials: { variables: [] },
setup: () => ({}),
connection: {
capabilities: {
leaseKind: 'direct-device-provider',
requiresAppAttachment: false,
requiresRemoteDaemon: false,
supportsArtifacts: true,
supportsDeferredAppSelection: false,
supportsDirectPortReverse: false,
usesCloudWebDriverLease: true,
},
resolve: async ({ host }: { host: import('./host.ts').ProviderPluginHost }) => {
await run(host);
return { profile: { leaseProvider: 'example', platform: 'ios' } };
},
verify: async ({ host }: { host: import('./host.ts').ProviderPluginHost }) => {
await run(host);
return { provider: 'example', service: 'example', verificationMessage: 'ok' };
},
},
}),
},
]);
const connection = connections.get('example')!;
const flags = { json: false, help: false, version: false };
await connection.resolve({
flags,
stateDir: dirs[0]!,
cwd: dirs[0]!,
env: { CONNECTION_VALUE: 'resolve' },
});
await connection.verify({ flags, cwd: dirs[1]!, env: { CONNECTION_VALUE: 'verify' } });
expect(observed).toEqual([
[dirs[0], 'resolve'],
[dirs[1], 'verify'],
]);
});

test('provider registry setup receives its supplied command host', async () => {
const { mkdtempForTestSync } = await import('@agent-device/test-kit');
const { realpathSync } = await import('node:fs');
const { resolveProviderPlugins, startProviderRegistry } = await import('./registry.ts');
const cwd = realpathSync(mkdtempForTestSync('plugin-setup-host-'));
let observed: unknown;
const resolved = await resolveProviderPlugins([
{
source: 'embedder',
load: () => ({
apiVersion: 2,
provider: 'example',
credentials: { variables: [] },
setup: async (host: import('./host.ts').ProviderPluginHost) => {
const result = await host.runHostCommand(process.execPath, [
'-e',
'process.stdout.write(JSON.stringify([process.cwd(), process.env.SETUP_VALUE]))',
]);
observed = JSON.parse(result.stdout);
return { webDriver: { provider: 'example' } };
},
}),
},
]);
const registry = await startProviderRegistry(
resolved,
{ cwd, env: { SETUP_VALUE: 'supplied' } },
{
adaptWebDriver: async () => {
throw new Error('not needed for host assertion');
},
},
);
expect(observed).toEqual([cwd, 'supplied']);
await registry.close();
});
23 changes: 21 additions & 2 deletions packages/plugins/src/host.ts
Original file line number Diff line number Diff line change
@@ -1,21 +1,40 @@
import type { HostEnvironment } from '@agent-device/kernel/source-value';
import { AppError, type AppErrorCode, type AppErrorDetails } from '@agent-device/kernel/errors';
import { readVersion } from '@agent-device/host-kit/version';

export type HostCommandResult = Readonly<{ stdout: string }>;

export type RunHostCommand = (
command: string,
args: readonly string[],
) => Promise<HostCommandResult>;

export type ProviderPluginHost = Readonly<{
env: Readonly<Record<string, string | undefined>>;
options: Readonly<Record<string, unknown>>;
clientVersion: string;
createError(code: AppErrorCode, message: string, details?: AppErrorDetails): AppError;
runHostCommand: RunHostCommand;
}>;

export function createPluginHost(
env: NodeJS.ProcessEnv,
host: HostEnvironment,
options: Record<string, unknown> | undefined,
): ProviderPluginHost {
return Object.freeze({
env: Object.freeze({ ...env }),
env: Object.freeze({ ...host.env }),
options: Object.freeze({ ...options }),
clientVersion: readVersion(),
createError: (code, message, details) => new AppError(code, message, details),
runHostCommand: async (command, args) => {
const { runCmd } = await import('@agent-device/host-kit/command');
const { stdout } = await runCmd(command, [...args], {
cwd: host.cwd,
env: host.env,
maxBuffer: 10 * 1024 * 1024,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This maxBuffer setting does not bound runHostCommand: asynchronous runCmd ignores it and accumulates stdout/stderr without a limit. Add enforcement to the async executor (or use an async bounded command API) before relying on this 10 MiB cap for provider-controlled output.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/plugins/src/host.ts, line 34:

<comment>This `maxBuffer` setting does not bound `runHostCommand`: asynchronous `runCmd` ignores it and accumulates stdout/stderr without a limit. Add enforcement to the async executor (or use an async bounded command API) before relying on this 10 MiB cap for provider-controlled output.</comment>

<file context>
@@ -1,21 +1,40 @@
+      const { stdout } = await runCmd(command, [...args], {
+        cwd: host.cwd,
+        env: host.env,
+        maxBuffer: 10 * 1024 * 1024,
+        timeoutMs: 30_000,
+      });
</file context>

timeoutMs: 30_000,
});
return { stdout };
},
} satisfies ProviderPluginHost);
}
2 changes: 1 addition & 1 deletion packages/plugins/src/load.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ async function startStore(env: NodeJS.ProcessEnv, builtin: readonly string[] = [
...claimProviderIds(builtin),
...(await storePluginCandidates(env)),
]);
return await startProviderRegistry(resolved, env, FAKE_POLICY);
return await startProviderRegistry(resolved, { env, cwd: '/' }, FAKE_POLICY);
}

test('a store plugin loads its factory with the selection options and the host error constructor', async () => {
Expand Down
29 changes: 18 additions & 11 deletions packages/plugins/src/registry.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import type { HostEnvironment } from '@agent-device/kernel/source-value';
import { NATIVE_REPLAY_BACKEND } from '@agent-device/contracts/replay-execution';
import type {
ReplayExecution,
Expand Down Expand Up @@ -179,15 +180,15 @@ export async function resolveProviderPlugins(
*/
export async function startProviderRegistry(
resolved: ResolvedProviderPlugins,
env: NodeJS.ProcessEnv,
host: HostEnvironment,
policy: PluginRegistryPolicy,
scope: 'all' | 'replay-backends' = 'all',
): Promise<ProviderRegistry> {
for (const slot of resolved.slots) {
if (slot.failure !== undefined) continue;
try {
if (slot.replayPlugin) await setUpReplaySlot(slot, slot.replayPlugin, env);
else if (slot.plugin && scope === 'all') await setUpSlot(slot, slot.plugin, env, policy);
if (slot.replayPlugin) await setUpReplaySlot(slot, slot.replayPlugin, host);
else if (slot.plugin && scope === 'all') await setUpSlot(slot, slot.plugin, host, policy);
} catch (error) {
slot.failure = asPluginFailure(error, 'plugin_setup_failed');
slot.registration = undefined;
Expand Down Expand Up @@ -299,9 +300,9 @@ function claimProvider(slot: Slot, claimed: Set<string>): void {
async function setUpReplaySlot(
slot: Slot,
plugin: ReplayEnginePlugin,
env: NodeJS.ProcessEnv,
host: HostEnvironment,
): Promise<void> {
const result = await plugin.setup(createPluginHost(env, slot.candidate.hostOptions));
const result = await plugin.setup(createPluginHost(host, slot.candidate.hostOptions));
try {
slot.replaySetup = validateReplayEngineSetup(result, plugin.backend);
} catch (error) {
Expand Down Expand Up @@ -406,25 +407,31 @@ function withConnectionHost(
return {
capabilities: connection.capabilities,
resolve: (context) =>
connection.resolve({ ...context, host: createPluginHost(context.env, hostOptions) }),
connection.resolve({
...context,
host: createPluginHost({ env: context.env, cwd: context.cwd }, hostOptions),
}),
verify: (context) =>
connection.verify({ ...context, host: createPluginHost(context.env, hostOptions) }),
connection.verify({
...context,
host: createPluginHost({ env: context.env, cwd: context.cwd }, hostOptions),
}),
};
}

async function setUpSlot(
slot: Slot,
plugin: AgentDevicePlugin,
env: NodeJS.ProcessEnv,
host: HostEnvironment,
policy: PluginRegistryPolicy,
): Promise<void> {
const host = createPluginHost(env, slot.candidate.hostOptions);
const result = await plugin.setup(host);
const pluginHost = createPluginHost(host, slot.candidate.hostOptions);
const result = await plugin.setup(pluginHost);
try {
const { runtime, platformModule } = await buildRegistration(
plugin,
result,
host.clientVersion,
pluginHost.clientVersion,
policy,
);
const dispose = result.dispose?.bind(result);
Expand Down
Loading
Loading