feat: Maestro E2E functional test suite for Omi app#6283
feat: Maestro E2E functional test suite for Omi app#6283hniane1 wants to merge 4 commits intoBasedHardware:mainfrom
Conversation
…e#3857) - 10 Maestro flow files covering all core app functionality: - Login & Sign-in (Google OAuth) - Onboarding (name, permissions, device skip) - Conversations (list, detail, tabs, CRUD, folder filters) - Memories (view, create, edit, delete) - Chat (send message, AI response) - Apps/Plugins (marketplace browse, detail) - Settings (navigation, key sections) - Device Connection (BLE scan & pair) [device_required] - Recording & Transcription [device_required] - Logout (sign out, verify auth screen) - Tag system: 'core' (simulator-safe) vs 'device_required' (needs Omi HW) - Runner script with tag filtering, parallel-safe output, JUnit XML + Markdown reports - Unified test.sh entry point: --e2e flag for Maestro, default for unit/widget - Clean, DRY flow structure — no code duplication across flows - .gitignore updated to exclude generated reports
Greptile SummaryThis PR introduces a Maestro E2E functional test suite covering 10 core user flows (login, onboarding, conversations, memories, chat, apps, settings, device connection, recording, and logout) along with a Bash runner script and a unified There are several correctness issues that should be addressed before the suite is relied upon in CI:
Confidence Score: 2/5Not ready to merge — the test suite has multiple logic bugs that cause it to not correctly verify app behaviour or maintain clean test state. Three P1 bugs (global config never applied, meaningless chat assertion, screenshot cross-contamination) mean the suite would give false confidence when run. The flows themselves are well-structured and the runner script is a solid foundation, but these issues need to be fixed before the suite can be trusted. app/.maestro/scripts/run_all.sh (runner bugs) and app/.maestro/flows/05_chat.yaml (assertion and syntax issues) Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["bash test.sh --e2e --tags core"] --> B["run_all.sh"]
B --> C["collect_flows(tags)"]
C -->|"grep -A5 tags:"| D["Flow file list"]
D --> E{For each flow}
E --> F["maestro test flow.yaml\n--format junit\n--output report.xml"]
F -->|"exit 0"| G["PASS ✅"]
F -->|"exit ≠ 0"| H["FAIL ❌"]
G --> I["cp ~/.maestro/tests/*.png\nflow_output/\n⚠️ copies ALL screenshots"]
H --> I
I --> E
E -->|done| J["Generate report.md"]
J --> K[Exit 0 / 1]
subgraph "Never Used"
L["config/global.yaml\nonFlowStart: clearState\n❌ not passed via --config"]
end
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| maestro test "$flow_file" \ | ||
| $DEVICE_FLAG \ | ||
| --format junit \ | ||
| --output "$flow_output/report.xml" \ | ||
| > "$flow_output/stdout.log" 2>&1 |
There was a problem hiding this comment.
global.yaml config is never applied
The runner calls maestro test "$flow_file" directly without passing --config or referencing config/global.yaml. This means the onFlowStart: clearState directive defined in global.yaml never fires — each flow inherits the full app state from the previous one rather than starting from a known-clean state.
This has two consequences:
- The suite only works if run in strict sequence starting from flow 01 (login). If any earlier flow is skipped or fails mid-way, subsequent flows may start in an unexpected state.
- The
config/global.yamlfile is misleading — it documents anonFlowStarthook that is silently ignored at runtime.
To apply the global config, add --config "$MAESTRO_DIR/config/global.yaml" to the maestro test invocation:
maestro test "$flow_file" \
$DEVICE_FLAG \
--config "$MAESTRO_DIR/config/global.yaml" \
--format junit \
--output "$flow_output/report.xml" \
> "$flow_output/stdout.log" 2>&1| if [[ -d "$HOME/.maestro/tests" ]]; then | ||
| cp "$HOME/.maestro/tests"/*.png "$flow_output/" 2>/dev/null || true | ||
| fi |
There was a problem hiding this comment.
$HOME/.maestro/tests/*.png is the global Maestro screenshot directory shared by all test runs. Copying every *.png found there into each per-flow output directory means screenshots from earlier flows bleed into later flows' directories, making per-flow debugging confusing.
Restrict the copy to only screenshots generated after the flow started:
if [[ -d "$HOME/.maestro/tests" ]]; then
find "$HOME/.maestro/tests" -name "*.png" \
-newer "$flow_output/report.xml" \
-exec cp {} "$flow_output/" \; 2>/dev/null || true
fi
app/.maestro/scripts/run_all.sh
Outdated
| [[ -f "$flow_file" ]] || continue | ||
|
|
||
| local flow_tags | ||
| flow_tags=$(grep -A5 "^tags:" "$flow_file" | grep "^ *- " | sed 's/^ *- //' || true) |
There was a problem hiding this comment.
grep -A5 tag extraction is brittle
grep -A5 "^tags:" only captures the 5 lines immediately following the tags: key. If a future flow has more than ~4 tags, the trailing ones are silently dropped and tag-based filtering will produce incorrect results.
A safer extraction reads until the YAML block ends:
flow_tags=$(awk '/^tags:/{p=1; next} p && /^ *- /{gsub(/^ *- /,""); print} p && !/^ *- /{p=0}' "$flow_file" || true)
…robust tag parsing - P1: Pass --config global.yaml to maestro test so onFlowStart hooks fire - P1: Use find -newer to only copy screenshots from current flow run - P2: Replace brittle grep -A5 tag extraction with awk block parser
…syntax - P1: Replace text: '.*' (matches anything) with actual response keywords - P1: Fix pressKey optional: true to use block-mapping syntax - Wait for loading indicators to disappear before asserting response
2 participants
What
Adds a complete Maestro E2E functional test suite for the Omi Flutter mobile app. Closes #3857.
Why
The Omi app needs automated functional tests to detect regressions in core user flows — sign-in, conversations, recording, chat, etc. This suite lets a maintainer run tests on their machine with a single command and get a clear pass/fail report.
Test Coverage
core,authcore,onboardingcore,conversationscore,memoriescore,chatcore,appscore,settingsdevice_requireddevice_requiredcore,authHow to Use
After the run, check
app/.maestro/reports/report.mdfor the full Markdown report with:Design Decisions
coreflows run on simulator/emulator (no physical Omi device needed).device_requiredflows need Omi hardware nearby.config/global.yaml.app/scripts/test.sh --e2efor Maestro, default (no flags) for unit/widget tests.Files Changed
app/.maestro/— 10 Maestro flow files, runner script, config, READMEapp/scripts/test.sh— Unified test runnerapp/.gitignore— Exclude generated reports