Test Suite Repair and Upgrade

[planetchili]

Collection of tests currently doesn't run on all machines for all team members. Causes include large datasets missing, dependency on machine configuration, overall brittleness due to non-deterministic systems, etc.

End goal is tests should run on any team member machine without exceptions, expected failures, sporadic failures, etc. Running tests should be 1 click and require minimal developer effort to determine true regression vs. false positives.

Tasks:

• Fix crash-flush log system tests
• Disable (temporarily) ETL logger feature tests
• Fix warm-up test race for all playback test groups
• Make paced-polled tests non-deterministic ( Deterministic Dynamic Poll During Playback #474 )
• Improve paced-polled visual analysis and golding tool
• Fix, retire, or rollup EndToEndTests group
• Fix RealtimeTests (machine dependency?)
• Upload large ETL test data to AuxData system minimized
• Analyze and diagnose divergence of OPM from service wrt minimized 8 (v2), 12 (cur, v1, v2)
• Disable swapchain pruning for OPM tests
• Exclude datasets from autogeneration of OPM tests
• Develop per-machine expectations store (not checked-in) for introspection etc. (hard machine dependence)
• Surface aux ETL backpressured playback tests individually via .h generation