[DRAFT] [Do not merge] Experiment with sharded SQL E2E PR tests

[mikaelweave]

Summary

• Adds opt-in runShardedE2eJobs and runShardedIntegrationTests modes to the SQL test job template.
• Enables sharded SQL E2E and SQL integration PR validation for Stu3, R4, and R5 SQL stages.
• Splits SQL E2E into normal parallel shards plus a terminal RequiresIsolation lane for async/global-state-sensitive tests.
• Splits SQL integration tests into General, SmartSearch, Search, Reindex, and SqlSpecific shards to measure whether the slow R4 SQL integration job can be reduced.
• Adds temporary [SmartSearchTiming] diagnostics to the SMART integration tests to capture setup, upsert, and per-search timings.
• Reuses SMART integration fixture/data setup once per FHIR version + data store so SmartOnFhir tests avoid reseeding the full dataset for every test method.
• Marks the 80K surrogate-id import stress test as ImportLongRunning so PR SQL E2E skips it while CI SQL E2E still runs it via explicit CI filters.

Experiment notes

This PR measures whether PR SQL validation wall-clock time improves when broad SQL test buckets are split by category/class filters, while keeping sensitive async-operation tests isolated rather than increasing in-process test parallelization.

E2E iteration findings

• Iteration 1 confirmed Azure Pipelines expanded the expected E2E shard jobs for all SQL versions.
• CustomSearch initially selected zero tests because custom search parameter tests are compiled as Search + IndexAndReindex, not CustomSearch; later iterations route those tests with an explicit SQL class filter and excludes them from Reindex.
• BulkDelete initially failed with concurrent 409 Conflict responses.
• Assembly-level E2E serialization alone did not remove BulkDelete conflicts.
• Moving BulkDelete out of the parallel shard set and running it after all other SQL E2E shards plus Reindex and BulkUpdate eliminated observed BulkDelete 409 Conflict failures for Stu3 and R4. R5 BulkDelete was blocked by an unrelated-looking R5 General 401 Unauthorized failure.
• Build 49548 showed R4 SQL Import taking 27m 38s of test time. The runner progress log showed GivenIncrementalLoad_80KSurrogateIds_BadRequestIsReturned active for roughly 23m, so that test is now tagged ImportLongRunning and excluded from the default PR SQL E2E filter.
• Build 49548 showed R4 SQL Reindex taking 20m 47s of test-task time because the first attempt failed 4 tests, then retry passed. Failures were search-parameter/reindex global-state sensitive: three jobs failed with Unable to sync search parameter cache, and cleanup logged a 409 conflict deleting a SearchParameter while a reindex job was ongoing.
• Build 49554/49562 showed Reindex passing cleanly after being moved later, which supports isolating async/global-state-sensitive E2E tests.

RequiresIsolation E2E experiment

• Added Categories.RequiresIsolation as an orthogonal scheduling trait for SQL E2E tests that mutate shared async/global state.
• Tagged SQL E2E BulkDeleteTests, BulkUpdateTests, and ReindexTests with RequiresIsolation.
• Standard sharded SQL E2E jobs now exclude Category=RequiresIsolation through mainCategoryFilter.
• The PR pipeline now runs one terminal sqlE2eTests_RequiresIsolation lane after all parallel E2E shards.
• Local R4 E2E discovery: terminal isolation filter selects 71 SQL tests (BulkUpdateTests=33, BulkDeleteTests=16, ReindexTests=22); all standard shards report isolationLeakCount=0.

SQL integration iteration

• Build 49526 showed R4 SQL integration taking ~42m total, with the test step taking ~35m including retry. R4 Cosmos integration took ~30m total.
• The long-pole evidence points to test execution, not report publishing; report/coverage aggregation was seconds.
• SmartSearch and SQL-specific tests are likely major contributors, with SmartSearch visible near the 20-25m mark in the R4 SQL log.
• SQL integration is split into five shards: General, SmartSearch, Search, Reindex, and SqlSpecific.
• Local discovery validated each shard for Stu3/R4/R5: 4 + 63 + 55 + 41 + 197 = 360 SQL-discoverable integration tests per version.

SMART timing diagnostics and fixture reuse

• Initial local R4 SQL SmartSearch replay showed InitializeAsync ran for every SMART test and repeatedly recreated the SQL fixture plus reseeded all SMART bundles/resources.
• SMART setup now uses a shared static context keyed by FHIR version + data store. The custom xUnit argument framework creates synthetic fixture instances, so the cached context is disposed at process exit rather than per class fixture instance.
• Mutation tests now use unique per-test resource IDs. Update tests first create the unique resource, then update that same ID, so they still exercise update behavior without mutating seeded shared resources.
• Local R4 SQL SmartSearch with pipeline-style coverage dropped from about 180s before fixture reuse to about 60-67s after fixture reuse.

Validation

• git diff --check
• dotnet build src\Microsoft.Health.Fhir.Tests.Common\Microsoft.Health.Fhir.Tests.Common.csproj -f net9.0 --configuration Release --no-restore
• dotnet build test\Microsoft.Health.Fhir.R4.Tests.E2E\Microsoft.Health.Fhir.R4.Tests.E2E.csproj -f net9.0 --configuration Release --no-restore /p:NETCoreSdkVersion=9.0.313
• dotnet build test\Microsoft.Health.Fhir.R4.Tests.Integration\Microsoft.Health.Fhir.R4.Tests.Integration.csproj -f net9.0
• dotnet build test\Microsoft.Health.Fhir.Stu3.Tests.Integration\Microsoft.Health.Fhir.Stu3.Tests.Integration.csproj -f net9.0
• dotnet build test\Microsoft.Health.Fhir.R5.Tests.Integration\Microsoft.Health.Fhir.R5.Tests.Integration.csproj -f net9.0
• dotnet <integration dll> --list-tests --filter ... for all SQL integration shard filters across Stu3/R4/R5.
• dotnet <R4 integration dll> --list-tests --filter "FullyQualifiedName!~CosmosDb&Category=SmartOnFhir" found 63 SQL SmartSearch tests.
• dotnet test test\Microsoft.Health.Fhir.R4.Tests.Integration\Microsoft.Health.Fhir.R4.Tests.Integration.csproj --configuration Release -f net9.0 -- --filter "FullyQualifiedName!~CosmosDb&Category=SmartOnFhir" --retry-failed-tests 3 --coverage --coverage-output-format cobertura --coverage-settings "<repo>\CodeCoverage.Mtp.settings.xml" --report-trx --output Detailed passed 63/63 in 60.1s test duration, 83.5s including build.
• R4 E2E discovery: FullyQualifiedName~SqlServer&Category=Import&Category!=ImportLongRunning does not list GivenIncrementalLoad_80KSurrogateIds_BadRequestIsReturned; FullyQualifiedName~SqlServer&Category=Import&Category=ImportLongRunning selects it.

[mikaelweave]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.32%. Comparing base (fdf6900) to head (cf00ab0).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5584      +/-   ##
==========================================
+ Coverage   77.28%   77.32%   +0.03%     
==========================================
  Files         995      996       +1     
  Lines       36467    36539      +72     
  Branches     5520     5539      +19     
==========================================
+ Hits        28184    28252      +68     
+ Misses       6924     6908      -16     
- Partials     1359     1379      +20     

see 21 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mikaelweave]

/azp run

[azure-pipelines]

Azure Pipelines failed to run 1 pipeline(s).