Add MAUI Android inner loop deploy measurement scenario by davidnguyen-tech · Pull Request #5165 · dotnet/performance

davidnguyen-tech · 2026-03-19T11:09:00Z

Summary

Adds a new scenario that measures MAUI Android developer inner loop performance:

first build+deploy+startup time measurement
10 incremental build+deploy+startup time measurements
parse build&deploy times from .binglogs.
parse startups times from the Android activity manager output

The scenario is the first one in the repo to build the app on Helix.
We send the app source code as the Helix payload because of the nature of this scenario.

The incremental change is simulated by:

one-line change in a .cs file
one-line change in a .xml file

Sequential incremental inner loop measurements are implemented by switching between the updated and the original versions of the files.

Test targets

Device	Queue
Pixel 8	`Windows.11.Amd64.Pixel.Perf`
Galaxy A16	`Windows.11.Amd64.Galaxy.Lowend.Perf`
Android 36 Emulator	`Ubuntu.2204.Amd64.Android.36`

AzDO Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2941057&view=results

The Galaxy queue is flaky

davidnguyen-tech · 2026-03-20T22:55:37Z

/azp run

azure-pipelines · 2026-03-20T22:55:51Z

Azure Pipelines successfully started running 1 pipeline(s).

DrewScoggins · 2026-03-27T00:06:03Z

When this PR goes in, #5176, you will need to take the changes. I am changing how the auth works and it needs the update to upload.py to continue working.

Add a new C# parser that extracts build and deploy metrics from MSBuild binary logs (.binlog) for MAUI Android inner loop measurements. The parser captures: - Overall build duration (Publish Time) - Build task timings: Csc, XamlC, GenerateJavaStubs, D8, Javac, etc. - Build target timings: CoreCompile, _GenerateJavaStubs, _CompileToDalvik, etc. - Deploy task timings: FastDeploy, AndroidSignPackage, Aapt2Link - Deploy target timings: _Sign, _Upload, _DeployApk, _BuildApkFastDev Register the AndroidInnerLoop MetricType in Startup.cs so the parser can be selected via the measurement framework. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extend the shared test runner with a new ANDROIDINNERLOOP scenario type that orchestrates first deploy and incremental build+deploy+startup measurements for MAUI Android apps. Changes: - const.py: Add ANDROIDINNERLOOP constant and scenario name mapping - runner.py: Add argument parser and full ANDROIDINNERLOOP handler that performs first build+deploy, then N incremental iterations with source file toggling, binlog capture, startup time measurement via am start, and result upload to perflab - androidhelper.py: Add skip_install, screen_timeout_ms, skip_uninstall, and other parameters to AndroidHelper for inner loop reuse - startup.py: Fix copytree FileExistsError with dirs_exist_ok=True Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add the scenario-specific scripts for MAUI Android inner loop measurements: - pre.py: Bootstraps the .NET SDK, installs maui-android workload, restores NuGet packages, installs Android SDK dependencies (build tools, platform SDK, Java), and creates the MAUI test app with modified source files for incremental measurement - setup_helix.py: Helix-specific environment setup that discovers dotnet/SDK/Android/Java paths, installs workloads, and prepares the build environment on Helix agents - test.py: Entry point that invokes the shared test runner with ANDROIDINNERLOOP test type - post.py: Cleanup script that disables device animations, restores screen settings, and uninstalls the test APK using ADB Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add CI infrastructure to run inner loop measurements on Helix: - maui_scenarios_android_innerloop.proj: MSBuild project that defines Helix work items with per-platform PreCommands for environment setup, SDK discovery, workload installation, and test invocation - sdk-perf-jobs.yml: Add 6 inner loop job definitions covering Pixel 8, Galaxy A16, and Android 36 emulator queues, each with Mono and CoreCLR runtime configurations - build-machine-matrix.yml: Add ubuntu-x64-android-emulator build machine mapping to Ubuntu.2204.Amd64.Android.36 queue - run-performance-job.yml: Support androidinnerloop runtime flavor - run_performance_job.py: Extend maui_scenarios_android run_kind matching to include innerloop variant; copy binlogs to artifacts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add --screen-timeout-ms CLI argument (default 1800000 = 30 min) and ScreenTimeoutMs MSBuild property so the screen timeout can be tuned from the .proj file without code changes. Add LaunchState validation to AndroidHelper.measure_cold_startup(): if am start reports anything other than COLD (e.g. UNKNOWN when the screen is off), the method now throws with a clear diagnostic message suggesting to increase --screen-timeout-ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds a new Helix-driven scenario to measure MAUI Android “inner loop” performance by timing first build+deploy+startup and repeated incremental build+deploy+startup iterations, with binlog parsing for build/deploy breakdown and am start/logcat parsing for startup time.

Changes:

Introduces a new AndroidInnerLoop parser in the ScenarioMeasurement startup tool to extract target/task durations from .binlogs.
Adds a new androidinnerloop scenario flow in the Python runner, plus a dedicated mauiandroidinnerloop scenario directory (pre/setup/post scripts) and a new Helix project file.
Updates pipelines/job config to run the scenario on Pixel/Galaxy (Windows) and an Android emulator (Ubuntu), and fixes repeated trace directory uploads by allowing copytree into existing dirs.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/tools/ScenarioMeasurement/Util/Parsers/AndroidInnerLoopParser.cs	New binlog parser emitting build/deploy counters for the new scenario.
src/tools/ScenarioMeasurement/Startup/Startup.cs	Adds `AndroidInnerLoop` metric type wiring to the startup tool.
src/scenarios/shared/startup.py	Allows repeated trace uploads by using `copytree(..., dirs_exist_ok=True)`.
src/scenarios/shared/runner.py	Adds `androidinnerloop` subcommand and orchestrates build+deploy+startup iterations + aggregation/upload.
src/scenarios/shared/const.py	Adds `ANDROIDINNERLOOP` constant and scenario-name mapping.
src/scenarios/shared/androidhelper.py	Extends device setup options and adds cold-start measurement helper.
src/scenarios/mauiandroidinnerloop/test.py	New scenario entrypoint using shared runner.
src/scenarios/mauiandroidinnerloop/setup_helix.py	New Helix setup script (workloads, Android deps, adb readiness, restore).
src/scenarios/mauiandroidinnerloop/pre.py	Creates MAUI template payload + prepares file-edit toggles and NuGet config.
src/scenarios/mauiandroidinnerloop/post.py	Cleanup/uninstall/build-server shutdown for the new scenario.
scripts/run_performance_job.py	Adds new `run_kind` handling and binlog artifact copying for this scenario.
eng/pipelines/templates/run-performance-job.yml	Enables passing `--runtime-flavor` for the new run kind.
eng/pipelines/templates/build-machine-matrix.yml	Adds Ubuntu Android emulator queue to the build matrix (private builds).
eng/pipelines/sdk-perf-jobs.yml	Schedules the new scenario across Pixel/Galaxy/emulator for mono+coreclr Debug.
eng/performance/maui_scenarios_android_innerloop.proj	New Helix project defining payload prep + work items for device/emulator tracks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/tools/ScenarioMeasurement/Util/Parsers/AndroidInnerLoopParser.cs

src/scenarios/shared/runner.py

src/scenarios/mauiandroidinnerloop/post.py

eng/performance/maui_scenarios_android_innerloop.proj

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak · 2026-04-02T13:29:08Z

src/scenarios/mauiandroidinnerloop/pre.py

+def install_maui_android_workload(precommands: PreCommands):
+    '''
+    Install the maui-android workload (not the full 'maui' workload).
+    The full 'maui' workload includes iOS/macOS/Windows components that aren't
+    available on Linux. Since this scenario only needs Android, 'maui-android'
+    is sufficient and works on both Windows and Linux.
+    '''
+    # Why this is complex: we can't simply run `dotnet workload install maui-android`
+    # because that would install the latest public version, which may not match the
+    # SDK version being tested. Instead, we resolve the exact workload manifest version
+    # from the NuGet feed that matches our SDK, create a rollback file pinning that
+    # version, and install using --from-rollback-file.
+    logger.info("########## Installing maui-android workload ##########")
+
+    if precommands.has_workload:
+        logger.info("Skipping maui-android installation due to --has-workload=true")
+        return
+
+    feed = extract_latest_dotnet_feed_from_nuget_config(
+        path=os.path.join(get_repo_root_path(), "NuGet.config")
+    )
+    logger.info(f"Installing the latest maui-android workload from feed {feed}")
+
+    workload = "microsoft.net.sdk.android"
+    try:
+        packages = precommands.get_packages_for_sdk_from_feed(workload, feed)
+    except Exception as e:
+        logger.warning(f"Failed to get packages for {workload} from latest feed: {e}")
+        logger.info("Trying second latest feed as fallback")
+        fallback_feed = extract_latest_dotnet_feed_from_nuget_config(
+            path=os.path.join(get_repo_root_path(), "NuGet.config"),
+            offset=1
+        )
+        logger.info(f"Using fallback feed: {fallback_feed}")
+        packages = precommands.get_packages_for_sdk_from_feed(workload, fallback_feed)
+
+    # Filter to manifest packages only
+    pattern = r'Microsoft\.NET\.Sdk\..*\.Manifest\-\d+\.\d+\.\d+(\-(preview|rc|alpha)\.\d+)?$'
+    packages = [pkg for pkg in packages if re.match(pattern, pkg['id'])]
+    logger.info(f"After manifest pattern filtering, found {len(packages)} packages for {workload}")
+
+    # Extract SDK and .NET versions from package IDs
+    for package in packages:
+        match = re.search(r'Manifest-(.+)$', package["id"])
+        if not match:
+            raise Exception(f"Unable to find .NET SDK version in package ID: {package['id']}")
+        sdk_version = match.group(1)
+        package['sdk_version'] = sdk_version
+
+        ver_match = re.search(r'^\d+\.\d+', sdk_version)
+        if not ver_match:
+            raise Exception(f"Unable to find .NET version in SDK version '{sdk_version}'")
+        package['dotnet_version'] = ver_match.group(0)
+
+    # Keep only packages targeting the highest .NET version
+    dotnet_versions = [float(pkg['dotnet_version']) for pkg in packages]
+    highest = max(dotnet_versions)
+    packages = [pkg for pkg in packages if float(pkg['dotnet_version']) == highest]
+    logger.info(f"After .NET version filtering for {workload}: {len(packages)} packages (highest={highest})")
+
+    # Prefer non-preview packages
+    preview_pattern = r'\-(preview|rc|alpha)\.\d+$'
+    non_preview = [pkg for pkg in packages if not re.search(preview_pattern, pkg['id'])]
+    if non_preview:
+        packages = non_preview
+
+    # Sort by SDK version descending and take the latest
+    packages.sort(key=lambda x: x['sdk_version'], reverse=True)
+    if not packages:
+        raise Exception(f"No packages available for {workload} after filtering")
+
+    latest = packages[0]
+    logger.info(f"Latest package: ID={latest['id']}, Version={latest['latestVersion']}, SDK={latest['sdk_version']}")
+
+    # Create rollback file with only the android workload
+    rollback_value = f"{latest['latestVersion']}/{latest['sdk_version']}"
+    rollback_dict = {workload: rollback_value}
+    logger.info(f"Rollback dictionary: {rollback_dict}")
+    with open("rollback_maui.json", "w", encoding="utf-8") as f:
+        f.write(json.dumps(rollback_dict, indent=4))
+    logger.info("Created rollback_maui.json file")
+
+    # Install maui-android (not 'maui') — works on both Windows and Linux
+    precommands.install_workload('maui-android', ['--from-rollback-file', 'rollback_maui.json'])
+    logger.info("########## Finished installing maui-android workload ##########")


Do we need this custom logic over re-using

performance/src/scenarios/shared/mauisharedpython.py

Line 389 in 619a125

def install_latest_maui(

?

matouskozak · 2026-04-02T13:31:24Z

eng/pipelines/sdk-perf-jobs.yml

+    parameters:
+      jobTemplate: /eng/pipelines/templates/run-scenarios-job.yml
+      buildMachines:
+        - win-x64-android-arm64-pixel


Can we make it simpler and pass it like this?

Suggested change

- win-x64-android-arm64-pixel

- win-x64-android-arm64-pixel

- win-x64-android-arm64-galaxy

matouskozak · 2026-04-02T13:32:40Z

eng/performance/maui_scenarios_android_innerloop.proj

+    <_LinuxEnvVars>export DOTNET_ROOT=$HELIX_CORRELATION_PAYLOAD/dotnet;export DOTNET_CLI_TELEMETRY_OPTOUT=1;export DOTNET_MULTILEVEL_LOOKUP=0;export NUGET_PACKAGES=$HELIX_WORKITEM_ROOT/.packages;export ANDROID_HOME=$HELIX_WORKITEM_ROOT/android-sdk;export ANDROID_SDK_ROOT=$HELIX_WORKITEM_ROOT/android-sdk;export JAVA_HOME=$HELIX_WORKITEM_ROOT/jdk;export ANDROID_SERIAL=emulator-5554;export PATH=$HELIX_CORRELATION_PAYLOAD/dotnet:$HELIX_WORKITEM_ROOT/android-sdk/platform-tools:$HELIX_WORKITEM_ROOT/jdk/bin:$PATH</_LinuxEnvVars>
+  </PropertyGroup>
+
+  <!-- Windows device track: runs on Windows.11.Amd64.Pixel.Perf -->


I thought we run on both Pixel and Galaxy device, correct?

davidnguyen-tech changed the title ~~Add MAUI Android inner loop deploy measurement scenario~~ [WIP] Add MAUI Android inner loop deploy measurement scenario Mar 20, 2026

davidnguyen-tech changed the title ~~[WIP] Add MAUI Android inner loop deploy measurement scenario~~ [WIP] Add MAUI Android Inner Loop scenario Mar 30, 2026

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch 2 times, most recently from 0418d92 to 54f7a75 Compare March 31, 2026 15:18

davidnguyen-tech changed the title ~~[WIP] Add MAUI Android Inner Loop scenario~~ Add MAUI Android inner loop deploy measurement scenario Mar 31, 2026

davidnguyen-tech and others added 4 commits April 1, 2026 11:36

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch from 73918d8 to ff881bb Compare April 1, 2026 09:36

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch from ff881bb to 746709a Compare April 1, 2026 09:37

davidnguyen-tech marked this pull request as ready for review April 1, 2026 21:24

davidnguyen-tech requested review from DrewScoggins, LoopedBard3, kotlarmilos, matouskozak, simonrozsival and vitek-karas April 1, 2026 21:25

LoopedBard3 requested a review from Copilot April 1, 2026 21:34

Copilot started reviewing on behalf of LoopedBard3 April 1, 2026 21:35 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Address review notes

986c7bf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak reviewed Apr 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MAUI Android inner loop deploy measurement scenario#5165

Add MAUI Android inner loop deploy measurement scenario#5165
davidnguyen-tech wants to merge 6 commits intodotnet:mainfrom
davidnguyen-tech:feature/measure-maui-android-deploy

davidnguyen-tech commented Mar 19, 2026 •

edited

Loading

Uh oh!

davidnguyen-tech commented Mar 20, 2026

Uh oh!

azure-pipelines bot commented Mar 20, 2026

Uh oh!

DrewScoggins commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matouskozak Apr 2, 2026

Uh oh!

matouskozak Apr 2, 2026

Uh oh!

matouskozak Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	- win-x64-android-arm64-pixel
	- win-x64-android-arm64-pixel
	- win-x64-android-arm64-galaxy

Conversation

davidnguyen-tech commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test targets

AzDO Build

Uh oh!

davidnguyen-tech commented Mar 20, 2026

Uh oh!

azure-pipelines bot commented Mar 20, 2026

Uh oh!

DrewScoggins commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matouskozak Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

matouskozak Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

matouskozak Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davidnguyen-tech commented Mar 19, 2026 •

edited

Loading