-
Notifications
You must be signed in to change notification settings - Fork 295
Add MAUI Android inner loop deploy measurement scenario #5165
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
davidnguyen-tech
wants to merge
7
commits into
dotnet:main
Choose a base branch
from
davidnguyen-tech:feature/measure-maui-android-deploy
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
8123eed
Add AndroidInnerLoopParser for build/deploy binlog analysis
davidnguyen-tech f6c5869
Add ANDROIDINNERLOOP test type to Python test framework
davidnguyen-tech 7740568
Add MAUI Android inner loop scenario scripts
davidnguyen-tech 3f5e643
Wire MAUI Android inner loop into CI pipeline
davidnguyen-tech 746709a
Make screen timeout configurable and validate cold startup LaunchState
davidnguyen-tech 986c7bf
Address review notes
davidnguyen-tech 26ea3ea
Address review notes 2
davidnguyen-tech File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| <Project Sdk="Microsoft.DotNet.Helix.Sdk" DefaultTargets="Test"> | ||
|
|
||
| <Import Project="Scenarios.Common.props" /> | ||
| <!-- TODO: Scenarios.Common.props sets HelixResultsDestinationDir to a path under | ||
| CorrelationPayloadDirectory, which is cleaned up after SendToHelix. As a result, | ||
| this file's DownloadFilesFromResults artifacts (e.g., output.log) may not appear | ||
| in published pipeline artifacts. A future fix should override | ||
| HelixResultsDestinationDir to a path under ArtifactStagingDirectory that the | ||
| pipeline's "Gather logs" step publishes. --> | ||
|
|
||
| <PropertyGroup> | ||
| <IncludeXHarnessCli>true</IncludeXHarnessCli> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'mono'">/p:UseMonoRuntime=true</_MSBuildArgs> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr'">/p:UseMonoRuntime=false</_MSBuildArgs> | ||
|
|
||
| <!-- CoreCLR JIT (all inner loop runs use codeGenType=Default; AOT/R2R/NativeAOT are unused) --> | ||
| <_MSBuildArgs Condition="'$(RuntimeFlavor)' == 'coreclr' and '$(CodegenType)' == 'JIT'">$(_MSBuildArgs) /p:PublishReadyToRun=false /p:PublishReadyToRunComposite=false</_MSBuildArgs> | ||
|
|
||
| <!-- AndroidRid: defaults based on target OS but can be overridden. | ||
| Windows targets physical Pixel 8 (ARM64); Linux targets x86_64 emulator. --> | ||
| <AndroidRid Condition="'$(AndroidRid)' == '' and '$(TargetsWindows)' == 'true'">android-arm64</AndroidRid> | ||
| <AndroidRid Condition="'$(AndroidRid)' == '' and '$(TargetsWindows)' != 'true'">android-x64</AndroidRid> | ||
| <_MSBuildArgs>$(_MSBuildArgs) /p:RuntimeIdentifier=$(AndroidRid)</_MSBuildArgs> | ||
|
|
||
| <!-- TODO: https://github.com/dotnet/maui/issues/34706 --> | ||
| <_MSBuildArgs>$(_MSBuildArgs) /p:SupportedOSPlatformVersion=23</_MSBuildArgs> | ||
| <!-- Add TargetFrameworks override so dotnet build's implicit restore | ||
| only evaluates the android TFM, not multi-platform TFMs in the csproj. --> | ||
| <_MSBuildArgs Condition="!$(_MSBuildArgs.Contains('/p:TargetFrameworks='))">$(_MSBuildArgs) /p:TargetFrameworks=$(PERFLAB_Framework)-android</_MSBuildArgs> | ||
|
|
||
| <RunConfigsString>$(RuntimeFlavor)_$(CodegenType)</RunConfigsString> | ||
| <InnerLoopIterations Condition="'$(InnerLoopIterations)' == ''">10</InnerLoopIterations> | ||
| <!-- Screen timeout must be large enough so the display stays on for the entire scenario. | ||
| If the screen turns off mid-run, am start reports LaunchState: UNKNOWN and omits | ||
| TotalTime, which breaks cold startup measurement. --> | ||
| <ScreenTimeoutMs Condition="'$(ScreenTimeoutMs)' == ''">1800000</ScreenTimeoutMs> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- Remove .NET SDK packs from the Helix correlation payload to reduce transfer size. | ||
| The workload packs (~1-2 GB) are restored on the Helix machine during the run step, | ||
| so shipping them in the payload is unnecessary and slows down payload upload. --> | ||
| <Target Name="RemoveDotnetFromCorrelationStaging" BeforeTargets="BeforeTest"> | ||
| <Message Text="Removing Dotnet Packs from Correlation Staging" Importance="high" /> | ||
| <RemoveDir Directories="$(CorrelationPayloadDirectory)dotnet\packs" Condition="'$(TargetsWindows)' == 'true'" /> | ||
| <RemoveDir Directories="$(CorrelationPayloadDirectory)dotnet/packs" Condition="'$(TargetsWindows)' != 'true'" /> | ||
| </Target> | ||
|
|
||
| <ItemDefinitionGroup> | ||
| <HelixWorkItem> | ||
| <Timeout>01:00</Timeout> | ||
| </HelixWorkItem> | ||
| </ItemDefinitionGroup> | ||
|
|
||
| <ItemGroup> | ||
| <MAUIAndroidInnerLoopScenario Include="MAUI Android Inner Loop"> | ||
| <ScenarioDirectoryName>mauiandroidinnerloop</ScenarioDirectoryName> | ||
| <PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory> | ||
| </MAUIAndroidInnerLoopScenario> | ||
| </ItemGroup> | ||
|
|
||
| <!-- PreparePayloadWorkItem: create template and modified source on the build machine. | ||
| pre.py creates the MAUI template in app/, fixes .csproj TFMs, and prepares the | ||
| modified MainPage.xaml.cs in src/ for the incremental deploy simulation. --> | ||
| <ItemGroup> | ||
| <PreparePayloadWorkItem Include="@(MAUIAndroidInnerLoopScenario)"> | ||
| <Command>$(Python) pre.py default -f $(PERFLAB_Framework)</Command> | ||
| <WorkingDirectory>%(PreparePayloadWorkItem.PayloadDirectory)</WorkingDirectory> | ||
| </PreparePayloadWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <!-- Environment variables needed on Helix before setup_helix.py runs (so PATH includes adb). --> | ||
| <PropertyGroup> | ||
| <_WindowsEnvVars>set DOTNET_ROOT=%HELIX_CORRELATION_PAYLOAD%\dotnet;set DOTNET_CLI_TELEMETRY_OPTOUT=1;set DOTNET_MULTILEVEL_LOOKUP=0;set NUGET_PACKAGES=%HELIX_WORKITEM_ROOT%\.packages;set ANDROID_HOME=%HELIX_WORKITEM_ROOT%\android-sdk;set ANDROID_SDK_ROOT=%HELIX_WORKITEM_ROOT%\android-sdk;set JAVA_HOME=%HELIX_WORKITEM_ROOT%\jdk;set PATH=%HELIX_CORRELATION_PAYLOAD%\dotnet;;%HELIX_WORKITEM_ROOT%\android-sdk\platform-tools;;%HELIX_WORKITEM_ROOT%\jdk\bin;;%PATH%</_WindowsEnvVars> | ||
| <_LinuxEnvVars>export DOTNET_ROOT=$HELIX_CORRELATION_PAYLOAD/dotnet;export DOTNET_CLI_TELEMETRY_OPTOUT=1;export DOTNET_MULTILEVEL_LOOKUP=0;export NUGET_PACKAGES=$HELIX_WORKITEM_ROOT/.packages;export ANDROID_HOME=$HELIX_WORKITEM_ROOT/android-sdk;export ANDROID_SDK_ROOT=$HELIX_WORKITEM_ROOT/android-sdk;export JAVA_HOME=$HELIX_WORKITEM_ROOT/jdk;export ANDROID_SERIAL=emulator-5554;export PATH=$HELIX_CORRELATION_PAYLOAD/dotnet:$HELIX_WORKITEM_ROOT/android-sdk/platform-tools:$HELIX_WORKITEM_ROOT/jdk/bin:$PATH</_LinuxEnvVars> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup Condition="'$(TargetsWindows)' == 'true'"> | ||
| <HelixWorkItem Include="@(MAUIAndroidInnerLoopScenario -> 'Inner Loop Device - %(Identity)')"> | ||
| <PreCommands>$(_WindowsEnvVars);$(Python) setup_helix.py $(PERFLAB_Framework)-android "$(_MSBuildArgs)"</PreCommands> | ||
| <Command>$(Python) test.py androidinnerloop --csproj-path app/MauiAndroidInnerLoop.csproj --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" --package-name com.companyname.mauiandroidinnerloop -f $(PERFLAB_Framework)-android -c Debug --msbuild-args "$(_MSBuildArgs)" --scenario-name "%(Identity)" --inner-loop-iterations $(InnerLoopIterations) --screen-timeout-ms $(ScreenTimeoutMs) $(ScenarioArgs)</Command> | ||
| <PostCommands>$(Python) post.py</PostCommands> | ||
| <DownloadFilesFromResults>output.log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup Condition="'$(TargetsWindows)' != 'true'"> | ||
| <HelixWorkItem Include="@(MAUIAndroidInnerLoopScenario -> 'Inner Loop Emulator - %(Identity)')"> | ||
| <PreCommands>$(_LinuxEnvVars);$(Python) setup_helix.py $(PERFLAB_Framework)-android "$(_MSBuildArgs)"</PreCommands> | ||
| <Command>$(Python) test.py androidinnerloop --csproj-path app/MauiAndroidInnerLoop.csproj --edit-src "src/MainPage.xaml.cs;src/MainPage.xaml" --edit-dest "app/Pages/MainPage.xaml.cs;app/Pages/MainPage.xaml" --package-name com.companyname.mauiandroidinnerloop -f $(PERFLAB_Framework)-android -c Debug --msbuild-args "$(_MSBuildArgs)" --scenario-name "%(Identity)" --inner-loop-iterations $(InnerLoopIterations) --screen-timeout-ms $(ScreenTimeoutMs) $(ScenarioArgs)</Command> | ||
| <PostCommands>$(Python) post.py</PostCommands> | ||
| <DownloadFilesFromResults>output.log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <Import Project="PreparePayloadWorkItems.targets" /> | ||
|
|
||
| </Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| ''' | ||
| post cleanup script | ||
| ''' | ||
|
|
||
| import subprocess | ||
| import sys | ||
| import traceback | ||
| from performance.logger import setup_loggers, getLogger | ||
| from shared.postcommands import clean_directories | ||
| from shared.util import xharness_adb | ||
| from test import EXENAME | ||
|
|
||
| setup_loggers(True) | ||
| logger = getLogger(__name__) | ||
|
|
||
| try: | ||
| # Uninstall the app from the connected device so re-runs start from a clean state | ||
| package_name = f'com.companyname.{EXENAME.lower()}' | ||
| logger.info(f"Uninstalling {package_name} from device") | ||
| subprocess.run(xharness_adb() + ['uninstall', package_name], check=False) | ||
|
|
||
| logger.info("Shutting down dotnet build servers") | ||
| subprocess.run(['dotnet', 'build-server', 'shutdown'], check=False) | ||
|
|
||
| clean_directories() | ||
| except Exception as e: | ||
| logger.error(f"Post cleanup failed: {e}\n{traceback.format_exc()}") | ||
| sys.exit(1) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| ''' | ||
| pre-command: Set up a MAUI Android app for deploy measurement. | ||
| Creates the template (without restore) and prepares the modified file for incremental deploy. | ||
| NuGet packages are restored on the Helix machine, not shipped in the payload. | ||
| ''' | ||
| import os | ||
| import shutil | ||
| import sys | ||
| from performance.logger import setup_loggers, getLogger | ||
| from shared import const | ||
| from shared.mauisharedpython import install_latest_maui, MauiNuGetConfigContext | ||
| from shared.precommands import PreCommands | ||
| from test import EXENAME | ||
|
|
||
| setup_loggers(True) | ||
| logger = getLogger(__name__) | ||
| logger.info("Starting pre-command for MAUI Android deploy measurement") | ||
|
|
||
| precommands = PreCommands() | ||
|
|
||
| with MauiNuGetConfigContext(precommands.framework): | ||
| install_latest_maui( | ||
| precommands, | ||
| workloads=["microsoft.net.sdk.android"], | ||
| workload_name='maui-android', | ||
| ) | ||
| precommands.print_dotnet_info() | ||
|
|
||
| # Create template without restoring packages — packages will be restored | ||
| # on the Helix machine to avoid shipping ~1-2GB in the workitem payload. | ||
| precommands.new(template='maui', | ||
| output_dir=const.APPDIR, | ||
| bin_dir=const.BINDIR, | ||
| exename=EXENAME, | ||
| working_directory=sys.path[0], | ||
| no_restore=True, | ||
| extra_args=['-sc']) | ||
|
|
||
| # Copy the merged NuGet.config into the app directory. This file contains | ||
| # MAUI NuGet feed URLs added by MauiNuGetConfigContext. The Helix machine | ||
| # needs these feeds during restore, and we must copy before the context | ||
| # manager restores the original NuGet.config. | ||
| repo_root = os.path.normpath(os.path.join(sys.path[0], '..', '..', '..')) | ||
| repo_nuget_config = os.path.join(repo_root, 'NuGet.config') | ||
| app_nuget_config = os.path.join(const.APPDIR, 'NuGet.config') | ||
| shutil.copy2(repo_nuget_config, app_nuget_config) | ||
| logger.info(f"Copied merged NuGet.config from {repo_nuget_config} to {app_nuget_config}") | ||
|
|
||
| # Inject properties into the csproj so they apply to every command that | ||
| # targets this project (restore, build, install). | ||
| csproj_path = os.path.join(const.APPDIR, f'{EXENAME}.csproj') | ||
| with open(csproj_path, 'r') as f: | ||
| csproj_content = f.read() | ||
|
|
||
| logger.info(f"Original .csproj content:\n{csproj_content}") | ||
|
|
||
| injected_props = { | ||
| # Preview SDKs may lack prune-package-data files, causing NETSDK1226. | ||
| 'AllowMissingPrunePackageData': 'true', | ||
| # The perf repo globally disables the Roslyn compiler server to avoid | ||
| # BenchmarkDotNet file-locking issues. Re-enable it here to match real | ||
| # MAUI developer inner loop experience. | ||
| 'UseSharedCompilation': 'true', | ||
| } | ||
| csproj_modified = csproj_content | ||
| if '</PropertyGroup>' not in csproj_modified: | ||
| raise Exception( | ||
| f"Cannot inject properties into {csproj_path}: " | ||
| f"no <PropertyGroup> found in the generated template." | ||
| ) | ||
| for prop_name, prop_value in injected_props.items(): | ||
| if prop_name not in csproj_modified: | ||
| csproj_modified = csproj_modified.replace( | ||
| '</PropertyGroup>', | ||
| f' <{prop_name}>{prop_value}</{prop_name}>\n </PropertyGroup>', | ||
| 1 # only the first PropertyGroup | ||
| ) | ||
|
|
||
| with open(csproj_path, 'w') as f: | ||
| f.write(csproj_modified) | ||
|
|
||
| logger.info(f"Updated {csproj_path} with injected properties") | ||
| logger.info(f"Modified .csproj content:\n{csproj_modified}") | ||
|
|
||
| # Create modified source files in src/ for the incremental deploy simulation. | ||
| # The runner toggles between original and modified versions each iteration, | ||
| # exercising both the C# compiler (Csc) and XAML compiler (XamlC) paths. | ||
| src_dir = os.path.join(sys.path[0], const.SRCDIR) | ||
| os.makedirs(src_dir, exist_ok=True) | ||
|
|
||
| # --- Modified MainPage.xaml.cs: add a debug line in the constructor --- | ||
| cs_original = os.path.join(const.APPDIR, 'Pages', 'MainPage.xaml.cs') | ||
| cs_modified = os.path.join(src_dir, 'MainPage.xaml.cs') | ||
|
|
||
| with open(cs_original, 'r') as f: | ||
| cs_content = f.read() | ||
|
|
||
| cs_modified_content = cs_content.replace( | ||
| 'InitializeComponent();', | ||
| 'InitializeComponent();\n\t\tSystem.Diagnostics.Debug.WriteLine("incremental-touch");' | ||
| ) | ||
| if cs_modified_content == cs_content: | ||
| raise Exception("Could not find 'InitializeComponent();' in %s — template may have changed" % cs_original) | ||
|
|
||
| with open(cs_modified, 'w') as f: | ||
| f.write(cs_modified_content) | ||
| logger.info(f"Modified MainPage.xaml.cs written to {cs_modified}") | ||
|
|
||
| # --- Modified MainPage.xaml: change a label's text --- | ||
| xaml_original = os.path.join(const.APPDIR, 'Pages', 'MainPage.xaml') | ||
| xaml_modified = os.path.join(src_dir, 'MainPage.xaml') | ||
|
|
||
| with open(xaml_original, 'r') as f: | ||
| xaml_content = f.read() | ||
|
|
||
| xaml_modified_content = xaml_content.replace( | ||
| 'Text="Task Categories"', | ||
| 'Text="Task Categories (updated)"' | ||
| ) | ||
| if xaml_modified_content == xaml_content: | ||
| raise Exception("Could not find 'Text=\"Task Categories\"' in %s — template may have changed" % xaml_original) | ||
|
|
||
| with open(xaml_modified, 'w') as f: | ||
| f.write(xaml_modified_content) | ||
| logger.info(f"Modified MainPage.xaml written to {xaml_modified}") |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.