Add MAUI iOS Inner Loop measurements for CI#5187
Draft
davidnguyen-tech wants to merge 29 commits intodotnet:mainfrom
Draft
Add MAUI iOS Inner Loop measurements for CI#5187davidnguyen-tech wants to merge 29 commits intodotnet:mainfrom
davidnguyen-tech wants to merge 29 commits intodotnet:mainfrom
Conversation
- Create iOSInnerLoopParser.cs: binlog parser for iOS inner loop build timings, extracting iOS-specific tasks (AOTCompile, Codesign, MTouch, etc.) and targets (_AOTCompile, _CodesignAppBundle, _CreateAppBundle, etc.) plus shared tasks (Csc, XamlC, LinkAssembliesNoShrink) - Wire into Startup.cs: add iOSInnerLoop to MetricType enum and map it to iOSInnerLoopParser in the parser switch expression - Fix Reporter.cs: guard against null/empty PERFLAB_BUILDTIMESTAMP to prevent ArgumentNullException on DateTime.Parse(null) when the env var is unset (falls back to DateTime.Now) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- const.py: Add IOSINNERLOOP constant and SCENARIO_NAMES mapping - ioshelper.py: New module with iOSHelper class for simulator and physical device management (boot, install, launch, terminate, uninstall, find bundle) - runner.py: Add iosinnerloop subparser, attribute assignment, and full execution branch (first build+deploy+launch, incremental loop with source toggling, binlog parsing, report aggregation, and Helix upload) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pre.py: Install maui-ios workload, create MAUI template (no-restore for Helix), strip non-iOS TFMs with flexible regex, inject MSBuild properties (AllowMissingPrunePackageData, UseSharedCompilation), copy merged NuGet.config for Helix-side restore, create modified source files for incremental edit loop, check Xcode compatibility. test.py: Thin entrypoint that builds TestTraits and invokes Runner. post.py: Uninstall app from simulator, shut down dotnet build server, clean directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Create the Helix machine setup script for MAUI iOS inner loop measurements. This script runs on the macOS Helix machine before test.py and handles: 1. DOTNET_ROOT/PATH configuration from the correlation payload SDK 2. Xcode selection — auto-detects highest versioned Xcode_*.app, matching the pattern used by maui_scenarios_ios.proj PreparePayloadWorkItem 3. iOS simulator runtime validation via xcrun simctl 4. Simulator device boot with graceful already-booted handling 5. maui-ios workload install using rollback file from pre.py, with --ignore-failed-sources for dead NuGet feeds 6. NuGet package restore with --ignore-failed-sources /p:NuGetAudit=false 7. Spotlight indexing disabled via mdutil to prevent file-lock errors Follows the same structure as the Android inner loop setup_helix.py: context dict pattern, step-by-step functions, structured logging to HELIX_WORKITEM_UPLOAD_ROOT for post-mortem debugging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Define the Helix .proj file for iOS inner loop measurements, modeled after the Android inner loop .proj and existing maui_scenarios_ios.proj patterns. Key design decisions: - Build on Helix machine (not build agent) because deploy requires a connected device/simulator. PreparePayloadWorkItem only creates the template and modified source files via pre.py. - Workload packs stripped from correlation payload (RemoveDotnetFromCorrelation Staging) and reinstalled on Helix machine by setup_helix.py. - Environment variables set via shell 'export' in PreCommands (not in Python) because os.environ changes don't persist across process boundaries. - No XHarness — iOS inner loop uses xcrun simctl directly. - Simulator-only for now; physical device support (ios-arm64, code signing) is structured as a future TODO pending runner.py device support. - 01:30 timeout to accommodate iOS build + workload install + NuGet restore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- sdk-perf-jobs.yml: Add Mono Debug job entry for maui_scenarios_ios_innerloop on osx-x64-ios-arm64 (Mac.iPhone.17.Perf queue) - run-performance-job.yml: Add maui_scenarios_ios_innerloop to the in() check so --runtime-flavor is forwarded to run_performance_job.py - run_performance_job.py: Add maui_scenarios_ios_innerloop to get_run_configurations() (CodegenType, RuntimeType, BuildConfig) and to the binlog copy block for PreparePayloadWorkItems artifacts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ioshelper.py: Add detect_connected_device() with auto-detection via xcrun devicectl (JSON + fallback text parsing), uninstall_app_physical, terminate_app_physical, close_physical_device, and cleanup() dispatch - runner.py: Add --device-type arg (simulator/device) to iosinnerloop subparser, auto-infer from RuntimeIdentifier, auto-detect device UDID, branch setup/install/startup/cleanup for physical vs simulator - setup_helix.py: Detect device type from IOS_RID env var, skip simulator boot for physical device, add detect_physical_device() for Helix - post.py: Handle physical device uninstall via devicectl with UDID auto-detection fallback - maui_scenarios_ios_innerloop.proj: Add physical device HelixWorkItem (conditioned on iOSRid=ios-arm64), pass IOS_RID to Pre/PostCommands, add --device-type arg to both simulator and device workitems Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e sort Fix 1 (Major): Replace non-existent 'devicectl terminate --bundle-id' with '--terminate-existing' flag on launch command. Make terminate_app_physical() a no-op with documentation explaining why. Fix 2 (Medium): Write devicectl JSON output to temp file instead of /dev/stdout, which mixes human-readable table and JSON. Applied in both ioshelper.py and setup_helix.py with proper temp file cleanup. Fix 3 (Medium): Add standard UUID pattern (8-4-4-4-12) to UDID regex in _detect_device_fallback() for CoreDevice UUID format compatibility. Fix 4 (Medium): Normalize MAUI template to always use Pages/ subdirectory in pre.py. If template puts MainPage files at root, move them to Pages/. Add explanatory comment in .proj documenting the coupling. Fix 5 (Minor): Use tuple-of-ints version sort for Xcode selection instead of string comparison (fixes 16.10 < 16.2 ordering bug). Fix 6 (Minor): Make simulator boot failure fatal with sys.exit(1). Add dynamic fallback to latest available iPhone simulator before failing. Fix 7 (Nit): Add missing trailing newline to runner.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace deprecated tempfile.mktemp() with tempfile.mkstemp() in both ioshelper.py and setup_helix.py to avoid TOCTOU race condition. - Fix unreachable fallback in detect_connected_device(): when devicectl exits non-zero (e.g., older Xcode without --json-output), call _detect_device_fallback() instead of returning None immediately. - Guard against missing JSON report in runner.py IOSINNERLOOP branch: Startup.cs only writes reports when PERFLAB_INLAB=1, so local runs would crash with FileNotFoundError. Now degrades gracefully with empty counters and a warning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Temporarily disable all other scenario jobs to speed up CI iteration while validating the new MAUI iOS Inner Loop scenario. This change should be reverted before merging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Capture dotnet build output instead of crashing on CalledProcessError - Create traces/ directory before first build - Fix setup_helix.py to write output.log (matches .proj expectation) - Improve error handling for build failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The dotnet build stdout/stderr wasn't appearing in Helix console logs, making it impossible to diagnose build failures. Explicitly capture and print build output through Python logging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build 2943141 hit the 90-minute timeout. iOS first build with AOT compilation can take 30+ minutes, plus 3 incremental iterations. Increasing to 2.5 hours to allow full completion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Helix machines have Xcode 26.2 but the iOS SDK requires 26.3. The minor version difference shouldn't affect build correctness, so bypass the check with ValidateXcodeVersion=false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mac.iPhone.17.Perf queue uses Intel x64 machines which need iossimulator-x64, not iossimulator-arm64. Add architecture detection in setup_helix.py and update default RID in .proj. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The traces upload directory already exists from the first build, causing copytree() to fail on subsequent iterations. Clear it before each parsetraces() call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The -v:n flag was added to debug build errors but produces excessive file copy logs. Default verbosity shows errors/warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add /p:MtouchLink=None to disable managed linker for Debug inner loop builds, avoiding MT0180 errors on machines without Xcode 26.3 - Add minimum Xcode version check in setup_helix.py for fast failure with clear diagnostics when machine has Xcode < 26.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove temporary ${{ if false }}: wrappers that disabled all jobs
except iOS inner loop during iterative CI debugging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…loop - Separate install from simulator/device setup in ioshelper.py - Capture install time for first and incremental deploys in runner.py - Add "Install Time" counter to both perf reports - Add CoreCLR Debug job entry in pipeline YAML - Add device (ios-arm64) job entries for both Mono and CoreCLR - Wire iOSRid env var through to MSBuild for device builds - [TEMP] Disable non-iOS-inner-loop jobs for CI validation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the dynamically-resolved manifest references SDK packs not yet propagated to NuGet feeds, fall back to installing without the rollback file. This avoids CI being blocked by transient feed propagation delays. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the rollback file references SDK packs not yet propagated to NuGet feeds, retry without the rollback file. Matches the fallback pattern already added to pre.py. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The simulator HelixWorkItem was unconditionally included, even when iOSRid=ios-arm64. This caused the simulator to receive device RID in _MSBuildArgs, producing ARM64 binaries that can't install on a simulator. Add Condition to exclude it from device jobs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…t.py Consolidate 12 duplicated simulator/device methods in ioshelper.py into a unified API (setup_device, install_app, measure_cold_startup, cleanup) that dispatches internally based on is_physical_device. Removes all if-is_physical dispatch branches from runner.py. Extract merge_build_deploy_and_startup and _make_counter to module-level helpers. Inline the incremental iteration loop (was a nested function with 10 parameters). Simplify post.py to reuse ioshelper instead of duplicating device detection. Extract inject_csproj_properties in pre.py. Net reduction: -232 lines across 4 files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ling Re-apply run_env_vars (including iOSRid) right before perf_send_to_helix() so the MSBuild .proj ItemGroup conditions can correctly exclude the simulator work item from device jobs. Add '|| exit $?' to setup_helix.py PreCommands so that when setup_helix exits non-zero (e.g., Xcode too old), the Helix shell stops instead of continuing to run test.py which would fail with a less clear error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Env var inheritance through msbuild.sh/tools.sh is unreliable for iOSRid. Add ios_rid field to PerfSendToHelixArgs and pass it as /p:iOSRid=<value> on the MSBuild command line so it reaches .proj evaluation deterministically. Also set it via set_environment_variables as belt-and-suspenders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mono_InnerLoop → Mono_InnerLoop_Simulator CoreCLR_InnerLoop → CoreCLR_InnerLoop_Simulator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace simctl (simulator) and devicectl (device) install/launch commands with mlaunch to match the real Visual Studio F5 developer experience: - Simulator: --launchsim combines install + launch (install_app returns 0) - Device: --installdev for install, --launchdev for launch - Device cleanup: --uninstalldevbundleid replaces devicectl uninstall - Simulator cleanup: unchanged (simctl terminate + uninstall) - Added _resolve_mlaunch() to find mlaunch from iOS SDK packs Device detection (devicectl) and simulator management (simctl boot/ terminate/uninstall) remain unchanged. The install_app/measure_cold_startup API is preserved so runner.py requires no changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Instead of making install_app() a no-op for simulator, use mlaunch --installsim to get a separate install measurement. measure_cold_startup() still uses --launchsim for launch timing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds MAUI iOS Inner Loop performance measurements to CI, supporting both iOS simulators and physical devices.
What's included:
Measurements:
Targets:
Based on: