Add test coverage for ResultsComparer and CLI helper paths by LoopedBard3 · Pull Request #5199 · dotnet/performance

LoopedBard3 · 2026-04-09T22:18:16Z

Summary

This expands automated coverage in the repo’s tooling-focused test surface without large production refactors.

What changed

Added a new ResultsComparer.Tests project covering:
- file and directory input handling
- JSON deserialization from file/stream
- stats aggregation and output behavior
- zip/tar.gz decompression flows
- duplicate-result/version-selection behavior
- CLI paths for invalid thresholds, missing inputs, and matrix output
Expanded BenchmarkDotNet.Extensions command-line option tests to cover:
- string parameter parsing
- boolean switch parsing
- missing/invalid integer argument paths
- absent-switch behavior
Improved Startup.Tests reliability by skipping ETW-dependent Windows tests when not running elevated, instead of aborting the run

Small correctness fixes found while adding coverage

The new tests exposed a few edge-case bugs, which are fixed in this PR:

CommandLineOptions.ParseAndRemoveStringsParameter now handles missing switches safely
ResultsComparer.Program now tolerates a missing optional --filter
TwoInputsComparer now treats EquivalenceTestConclusion.Base as a no-diff case
ResultsComparer.Data moniker parsing is now null-safe for additional archive path shapes

Validation

Validated the affected suites in the available environment:

ResultsComparer.Tests passing
targeted BenchmarkDotNet.Extensions command-line tests passing
Startup.Tests now skip environment-dependent cases cleanly

Note: test projects are kept at net11.0; full execution of those targets requires a local/CI environment with the .NET 11 SDK installed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Adds new unit tests and small robustness tweaks across tooling to improve confidence in ResultsComparer output/CLI parsing and reduce platform-specific test failures.

Changes:

Added a new ResultsComparer.Tests project with coverage for Program, Data, Helper, and Stats.
Updated ResultsComparer logic for filter handling, moniker parsing, and equivalence-test conclusions.
Improved platform gating in Startup.Tests and strengthened/nullability-corrected tests in the BenchmarkDotNet.Extensions harness.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/tools/ScenarioMeasurement/Startup.Tests/StartupTests.cs	Updates Windows/Linux platform detection and adds an admin-privilege gate for ETW-related tests.
src/tools/ResultsComparer/TwoInputsComparer.cs	Treats `EquivalenceTestConclusion.Base` as non-difference (like `Same`) when filtering results.
src/tools/ResultsComparer/ResultsComparer.sln	Adds the new `ResultsComparer.Tests` project and additional solution configurations.
src/tools/ResultsComparer/Properties/AssemblyInfo.cs	Exposes internals to `ResultsComparer.Tests` via `InternalsVisibleTo`.
src/tools/ResultsComparer/Program.cs	Makes filter parsing resilient to `null` filter arrays.
src/tools/ResultsComparer/Data.cs	Adjusts moniker detection (including empty-key guard and `Contains`-based matching for newer TFMs).
src/tools/ResultsComparer.Tests/StatsTests.cs	Adds unit tests for `Stats` formatting/aggregation and validation behavior.
src/tools/ResultsComparer.Tests/ResultsComparerTestData.cs	Adds a JSON test-data generator for BDN-like inputs.
src/tools/ResultsComparer.Tests/ResultsComparer.Tests.csproj	Introduces the new ResultsComparer unit test project.
src/tools/ResultsComparer.Tests/ProgramTests.cs	Adds CLI-level tests validating error output and “no diffs” behavior.
src/tools/ResultsComparer.Tests/HelperTests.cs	Adds tests for file discovery and JSON deserialization helpers.
src/tools/ResultsComparer.Tests/DataTests.cs	Adds tests for nested zip + tar.gz extraction and dedup/version preference logic.
src/tests/harness/BenchmarkDotNet.Extensions.Tests/PartitionFilterTests.cs	Makes parsed config nullable-aware and asserts non-null on success.
src/tests/harness/BenchmarkDotNet.Extensions.Tests/CommandLineOptionsTests.cs	Adds more test cases for parsing/removal helpers (strings/bool/int).
src/harness/BenchmarkDotNet.Extensions/CommandLineOptions.cs	Fixes parsing when the target string parameter switch is missing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-15T22:36:45Z

+    private static string InvokeProgram(string[] args)
+    {
+        using var writer = new StringWriter();
+        var originalOut = Console.Out;
+        Console.SetOut(writer);
+        try
+        {
+            Program.Main(args);
+            return writer.ToString();
+        }
+        finally
+        {
+            Console.SetOut(originalOut);
+        }


InvokeProgram calls ResultsComparer.Program.Main, which sets Thread.CurrentThread.CurrentCulture to InvariantCulture and does not restore it. Because tests may reuse threads, this can leak culture changes into subsequent tests and cause hard-to-diagnose failures. Consider capturing the current CurrentCulture/CurrentUICulture before invoking Main and restoring them in the finally block (alongside restoring Console.Out).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-20T20:35:50Z

+                var tarEntry = new UstarTarEntry(TarEntryType.RegularFile, entryName)
+                {
+                    DataStream = new MemoryStream(Encoding.UTF8.GetBytes(content))


In CreateTarGzArchive, each TarEntry's DataStream is set to a new MemoryStream but never disposed. Even though this is test-only code, it’s easy to avoid leaking disposables by creating the MemoryStream in a using scope and disposing it after TarWriter.WriteEntry returns.

Suggested change

var tarEntry = new UstarTarEntry(TarEntryType.RegularFile, entryName)

{

DataStream = new MemoryStream(Encoding.UTF8.GetBytes(content))

using var dataStream = new MemoryStream(Encoding.UTF8.GetBytes(content));

var tarEntry = new UstarTarEntry(TarEntryType.RegularFile, entryName)

{

DataStream = dataStream

LoopedBard3 and others added 2 commits April 9, 2026 11:34

Add coverage for ResultsComparer and CLI helpers

150611c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore test projects to net11.0

fe3c6bf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

LoopedBard3 requested a review from Copilot April 9, 2026 22:18

LoopedBard3 self-assigned this Apr 9, 2026

LoopedBard3 changed the title ~~Test all the things~~ Add test coverage for ResultsComparer and CLI helper paths Apr 9, 2026

Copilot started reviewing on behalf of LoopedBard3 April 9, 2026 22:18 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

Address PR review feedback

6e12ccf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

LoopedBard3 requested a review from DrewScoggins April 10, 2026 18:09

LoopedBard3 marked this pull request as ready for review April 10, 2026 18:11

LoopedBard3 marked this pull request as draft April 10, 2026 18:11

LoopedBard3 marked this pull request as ready for review April 13, 2026 20:01

Copilot AI review requested due to automatic review settings April 13, 2026 20:01

Copilot started reviewing on behalf of LoopedBard3 April 13, 2026 20:01 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

Comment thread src/tools/ResultsComparer/MultipleInputsComparer.cs Outdated

LoopedBard3 and others added 2 commits April 13, 2026 13:53

Fix Base classification in matrix comparisons

e942a40

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Run tooling tests in CI

a35c443

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 15, 2026 22:31

Copilot started reviewing on behalf of LoopedBard3 April 15, 2026 22:32 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

LoopedBard3 and others added 2 commits April 15, 2026 15:39

Skip tooling tests on main

83d3035

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Limit tooling tests to public non-main runs

d76ef41

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 15, 2026 22:40

Copilot started reviewing on behalf of LoopedBard3 April 15, 2026 22:40 View session

Copilot AI reviewed Apr 15, 2026

View reviewed changes

LoopedBard3 and others added 2 commits April 16, 2026 11:21

Fix brittle matrix comparison test

9cfdeb0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore culture after invoking ResultsComparer

2a79e4c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 16, 2026 19:15

Copilot started reviewing on behalf of LoopedBard3 April 16, 2026 19:15 View session

Copilot AI reviewed Apr 16, 2026

View reviewed changes

LoopedBard3 and others added 2 commits April 20, 2026 13:15

Bump Newtonsoft.Json for tooling tests

4d66ea4

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use net10 MicroBenchmarks in harness tests

70a7531

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 20, 2026 20:28

Copilot started reviewing on behalf of LoopedBard3 April 20, 2026 20:28 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

This was referenced Apr 20, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

[android-arm64] The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#6408

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test coverage for ResultsComparer and CLI helper paths#5199

Add test coverage for ResultsComparer and CLI helper paths#5199
LoopedBard3 wants to merge 11 commits intodotnet:mainfrom
LoopedBard3:TestAllTheThings

LoopedBard3 commented Apr 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

LoopedBard3 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Small correctness fixes found while adding coverage

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LoopedBard3 commented Apr 9, 2026 •

edited

Loading