Add MAUI Android inner loop deploy measurement scenario by davidnguyen-tech · Pull Request #5165 · dotnet/performance

davidnguyen-tech · 2026-03-19T11:09:00Z

Summary

Adds a new scenario that measures MAUI Android developer inner loop performance:

first build+deploy+startup time measurement
10 incremental build+deploy+startup time measurements
parse build&deploy times from .binglogs.
parse startups times from the Android activity manager output

The scenario is the first one in the repo to build the app on Helix.
We send the app source code as the Helix payload because of the nature of this scenario.

The incremental change is simulated by:

one-line change in a .cs file
one-line change in a .xml file

Sequential incremental inner loop measurements are implemented by switching between the updated and the original versions of the files.

Test targets

Device	Queue
Pixel 8	`Windows.11.Amd64.Pixel.Perf`
Galaxy A16	`Windows.11.Amd64.Galaxy.Lowend.Perf`
Android 36 Emulator	`Ubuntu.2204.Amd64.Android.36`

AzDO Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2956892&view=results

davidnguyen-tech · 2026-03-20T22:55:37Z

/azp run

azure-pipelines · 2026-03-20T22:55:51Z

Azure Pipelines successfully started running 1 pipeline(s).

DrewScoggins · 2026-03-27T00:06:03Z

When this PR goes in, #5176, you will need to take the changes. I am changing how the auth works and it needs the update to upload.py to continue working.

Add a new C# parser that extracts build and deploy metrics from MSBuild binary logs (.binlog) for MAUI Android inner loop measurements. The parser captures: - Overall build duration (Publish Time) - Build task timings: Csc, XamlC, GenerateJavaStubs, D8, Javac, etc. - Build target timings: CoreCompile, _GenerateJavaStubs, _CompileToDalvik, etc. - Deploy task timings: FastDeploy, AndroidSignPackage, Aapt2Link - Deploy target timings: _Sign, _Upload, _DeployApk, _BuildApkFastDev Register the AndroidInnerLoop MetricType in Startup.cs so the parser can be selected via the measurement framework. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extend the shared test runner with a new ANDROIDINNERLOOP scenario type that orchestrates first deploy and incremental build+deploy+startup measurements for MAUI Android apps. Changes: - const.py: Add ANDROIDINNERLOOP constant and scenario name mapping - runner.py: Add argument parser and full ANDROIDINNERLOOP handler that performs first build+deploy, then N incremental iterations with source file toggling, binlog capture, startup time measurement via am start, and result upload to perflab - androidhelper.py: Add skip_install, screen_timeout_ms, skip_uninstall, and other parameters to AndroidHelper for inner loop reuse - startup.py: Fix copytree FileExistsError with dirs_exist_ok=True Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add the scenario-specific scripts for MAUI Android inner loop measurements: - pre.py: Bootstraps the .NET SDK, installs maui-android workload, restores NuGet packages, installs Android SDK dependencies (build tools, platform SDK, Java), and creates the MAUI test app with modified source files for incremental measurement - setup_helix.py: Helix-specific environment setup that discovers dotnet/SDK/Android/Java paths, installs workloads, and prepares the build environment on Helix agents - test.py: Entry point that invokes the shared test runner with ANDROIDINNERLOOP test type - post.py: Cleanup script that disables device animations, restores screen settings, and uninstalls the test APK using ADB Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add CI infrastructure to run inner loop measurements on Helix: - maui_scenarios_android_innerloop.proj: MSBuild project that defines Helix work items with per-platform PreCommands for environment setup, SDK discovery, workload installation, and test invocation - sdk-perf-jobs.yml: Add 6 inner loop job definitions covering Pixel 8, Galaxy A16, and Android 36 emulator queues, each with Mono and CoreCLR runtime configurations - build-machine-matrix.yml: Add ubuntu-x64-android-emulator build machine mapping to Ubuntu.2204.Amd64.Android.36 queue - run-performance-job.yml: Support androidinnerloop runtime flavor - run_performance_job.py: Extend maui_scenarios_android run_kind matching to include innerloop variant; copy binlogs to artifacts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add --screen-timeout-ms CLI argument (default 1800000 = 30 min) and ScreenTimeoutMs MSBuild property so the screen timeout can be tuned from the .proj file without code changes. Add LaunchState validation to AndroidHelper.measure_cold_startup(): if am start reports anything other than COLD (e.g. UNKNOWN when the screen is off), the method now throws with a clear diagnostic message suggesting to increase --screen-timeout-ms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds a new Helix-driven scenario to measure MAUI Android “inner loop” performance by timing first build+deploy+startup and repeated incremental build+deploy+startup iterations, with binlog parsing for build/deploy breakdown and am start/logcat parsing for startup time.

Changes:

Introduces a new AndroidInnerLoop parser in the ScenarioMeasurement startup tool to extract target/task durations from .binlogs.
Adds a new androidinnerloop scenario flow in the Python runner, plus a dedicated mauiandroidinnerloop scenario directory (pre/setup/post scripts) and a new Helix project file.
Updates pipelines/job config to run the scenario on Pixel/Galaxy (Windows) and an Android emulator (Ubuntu), and fixes repeated trace directory uploads by allowing copytree into existing dirs.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/tools/ScenarioMeasurement/Util/Parsers/AndroidInnerLoopParser.cs	New binlog parser emitting build/deploy counters for the new scenario.
src/tools/ScenarioMeasurement/Startup/Startup.cs	Adds `AndroidInnerLoop` metric type wiring to the startup tool.
src/scenarios/shared/startup.py	Allows repeated trace uploads by using `copytree(..., dirs_exist_ok=True)`.
src/scenarios/shared/runner.py	Adds `androidinnerloop` subcommand and orchestrates build+deploy+startup iterations + aggregation/upload.
src/scenarios/shared/const.py	Adds `ANDROIDINNERLOOP` constant and scenario-name mapping.
src/scenarios/shared/androidhelper.py	Extends device setup options and adds cold-start measurement helper.
src/scenarios/mauiandroidinnerloop/test.py	New scenario entrypoint using shared runner.
src/scenarios/mauiandroidinnerloop/setup_helix.py	New Helix setup script (workloads, Android deps, adb readiness, restore).
src/scenarios/mauiandroidinnerloop/pre.py	Creates MAUI template payload + prepares file-edit toggles and NuGet config.
src/scenarios/mauiandroidinnerloop/post.py	Cleanup/uninstall/build-server shutdown for the new scenario.
scripts/run_performance_job.py	Adds new `run_kind` handling and binlog artifact copying for this scenario.
eng/pipelines/templates/run-performance-job.yml	Enables passing `--runtime-flavor` for the new run kind.
eng/pipelines/templates/build-machine-matrix.yml	Adds Ubuntu Android emulator queue to the build matrix (private builds).
eng/pipelines/sdk-perf-jobs.yml	Schedules the new scenario across Pixel/Galaxy/emulator for mono+coreclr Debug.
eng/performance/maui_scenarios_android_innerloop.proj	New Helix project defining payload prep + work items for device/emulator tracks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Hooks the scenario into the canonical versions.json pipeline used by the other MAUI/.NET Android scenarios so SDK versions land in the Startup tool's Reporter (build.AdditionalData), and adds a single 'dotnet --info' call on Helix for triage of SDK/runtime mismatches between the build machine and the Helix machine. - pre.py: log rollback_maui.json contents after install_latest_maui so the workload version pinned for the build is visible in build logs. - setup_helix.py: run 'dotnet --info' on the Helix machine before the measured build so triage doesn't have to dig through binlogs to know what SDK was actually used. - shared/runner.py (ANDROIDINNERLOOP branch): require -c/--configuration and -f/--framework (the .proj already passes both). After the first Helix build, read SDK versions from the linked/ or apk-staging assets folder via get_sdk_versions, write them with versions_write_json, then versions_read_json_file_save_env so PERFLAB_DATA_* env vars feed Reporter.cs. Wrapped in try/except so version capture never regresses the measurement. Unlike the other scenarios this capture has to live in runner.py rather than pre.py because the scenario's first build is itself the measured build and runs on Helix, not on the build machine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

davidnguyen-tech · 2026-04-21T17:36:35Z

Hello @simonrozsival @jonathanpeppers , could you please check if I set up the MAUI SDK properly? Especially using the nightly builds & installing the workloads

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/scenarios/shared/runner.py:195

This error path prints the list of valid test types from shared.testtraits.testtypes, but that list currently does not include the newly added const.ANDROIDINNERLOOP. As a result the message is misleading and TestTraits won’t have an attribute for the new test type if any code later relies on it. Please add ANDROIDINNERLOOP to shared/testtraits.py:testtypes to keep the CLI’s supported subcommands consistent across the codebase.

        if not args.testtype:
            getLogger().error("Please specify a test type: %s. Type test.py <test type> -- help for more type-specific subcommands" % testtypes)
            sys.exit(1)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jonathanpeppers

Why does this use dotnet build -t:Install + separate am start instead of dotnet run? dotnet run is the gesture customers actually use for inner loop -- it builds, deploys, and launches in one command. You could simply time the dotnet run command itself to get the true end-to-end inner loop time, rather than parsing multiple binlogs.

RE: workloads

I don't see dotnet workload config --update-mode manifests being used, but it uses --from-rollback-file rollback_maui.json. Is there a log I can see, to just verify the right versions are installed?

- runner.py: mark csprojpath, edit-src, edit-dest, package-name as required=True so argparse surfaces missing args with a clear usage error instead of a late runtime exception. Drop the now-dead input validation block and the dead else-raise under editsrcs/editdests. - runner.py: discover the android-* RID folder under the build's obj tree instead of hardcoding android-arm64, so the emulator track (android-x64) captures SDK versions too. - startup.py: add copy_traces parameter (default True) to StartupWrapper.parsetraces. When False, skip the per-call TRACEDIR -> Helix upload-dir copy and perflab container upload. - runner.py (inner loop): pass copy_traces=False for the per-iteration parse calls. The final copy at end-of-run still runs, so trace uploads land once rather than O(N^2). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jonathanpeppers · 2026-04-21T17:53:36Z

You might need dotnet run -p:WaitForExit=false -bl.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

davidnguyen-tech · 2026-04-22T12:21:30Z

Why does this use dotnet build -t:Install + separate am start instead of dotnet run? dotnet run is the gesture customers actually use for inner loop -- it builds, deploys, and launches in one command. You could simply time the dotnet run command itself to get the true end-to-end inner loop time, rather than parsing multiple binlogs.

You might need dotnet run -p:WaitForExit=false -bl.

I mirrored what we do for Visual Studio's F5 flow, based on ClientTools.Platform. I'm open to migrating to dotnet run ... though, as it sounds simpler :)

I don't see dotnet workload config --update-mode manifests being used, but it uses --from-rollback-file rollback_maui.json. Is there a log I can see, to just verify the right versions are installed?

You can get to the logs through here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2957445&view=logs&j=690a1b7a-7282-5287-d42e-e3e8ecc170c3&t=cb92986b-5679-58ac-2ba2-5fb303127ef9&l=77

jonathanpeppers · 2026-04-22T13:56:42Z

Your workload install looks right:

Latest package: Microsoft.NET.Sdk.Android.Manifest-11.0.100-preview.4
Version=36.99.0-ci.main.112, SDK_Version=11.0.100-preview.4

davidnguyen-tech changed the title ~~Add MAUI Android inner loop deploy measurement scenario~~ [WIP] Add MAUI Android inner loop deploy measurement scenario Mar 20, 2026

davidnguyen-tech changed the title ~~[WIP] Add MAUI Android inner loop deploy measurement scenario~~ [WIP] Add MAUI Android Inner Loop scenario Mar 30, 2026

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch 2 times, most recently from 0418d92 to 54f7a75 Compare March 31, 2026 15:18

davidnguyen-tech changed the title ~~[WIP] Add MAUI Android Inner Loop scenario~~ Add MAUI Android inner loop deploy measurement scenario Mar 31, 2026

davidnguyen-tech and others added 4 commits April 1, 2026 11:36

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch from 73918d8 to ff881bb Compare April 1, 2026 09:36

davidnguyen-tech force-pushed the feature/measure-maui-android-deploy branch from ff881bb to 746709a Compare April 1, 2026 09:37

davidnguyen-tech marked this pull request as ready for review April 1, 2026 21:24

davidnguyen-tech requested review from DrewScoggins, LoopedBard3, kotlarmilos, matouskozak, simonrozsival and vitek-karas April 1, 2026 21:25

LoopedBard3 requested a review from Copilot April 1, 2026 21:34

Copilot started reviewing on behalf of LoopedBard3 April 1, 2026 21:35 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

Address review notes

986c7bf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

matouskozak reviewed Apr 2, 2026

View reviewed changes

Comment thread src/scenarios/mauiandroidinnerloop/pre.py Outdated

matouskozak reviewed Apr 2, 2026

View reviewed changes

Comment thread eng/pipelines/sdk-perf-jobs.yml

matouskozak reviewed Apr 2, 2026

View reviewed changes

Comment thread eng/performance/maui_scenarios_android_innerloop.proj Outdated

Address review notes 2

26ea3ea

This was referenced Apr 7, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

[android-arm64] The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#6408

Open

davidnguyen-tech marked this pull request as draft April 20, 2026 15:06

Copilot AI review requested due to automatic review settings April 21, 2026 17:31

Copilot started reviewing on behalf of davidnguyen-tech April 21, 2026 17:31 View session

davidnguyen-tech marked this pull request as ready for review April 21, 2026 17:32

davidnguyen-tech requested a review from jonathanpeppers April 21, 2026 17:32

Copilot AI reviewed Apr 21, 2026

View reviewed changes

Comment thread src/scenarios/shared/runner.py Outdated

Comment thread src/scenarios/shared/runner.py

Comment thread src/scenarios/shared/runner.py Outdated

jonathanpeppers reviewed Apr 21, 2026

View reviewed changes

Log rollback_maui.json contents on Helix machine

d73f5fb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 22, 2026 10:17

Copilot AI reviewed Apr 22, 2026

View reviewed changes

Comment thread src/scenarios/shared/mauisharedpython.py

Conversation

davidnguyen-tech commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test targets

AzDO Build

Uh oh!

davidnguyen-tech commented Mar 20, 2026

Uh oh!

azure-pipelines Bot commented Mar 20, 2026

Uh oh!

DrewScoggins commented Mar 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

davidnguyen-tech commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jonathanpeppers left a comment

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers commented Apr 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

davidnguyen-tech commented Apr 22, 2026

Uh oh!

jonathanpeppers commented Apr 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

davidnguyen-tech commented Mar 19, 2026 •

edited

Loading

davidnguyen-tech commented Apr 21, 2026 •

edited

Loading