Reject deleted or scaled-to-zero workloads and unschedulable pods in mitigation eval

[Nazmus Sakib (TheDeadcoder)]

Fixes #208

What this changes

1. Pods with no container statuses count as unhealthy. pods_unhealthy_reasons no longer skips them, unless the pod finished successfully. This matches KubeCtl._pod_is_ready_or_succeeded.
2. Eval remembers which workloads the app deployed. MitigationTask.record_deployed_workloads records every Deployment and StatefulSet and its replica count. Orchestrator.init_problem calls it right after app.deploy() and before inject_fault().
3. Eval fails if a recorded workload is gone or scaled to 0. wait_until_pods_healthy now checks this on every poll, inside the same settle window.
4. Four problems use the shared check. misconfig_app_hotel_res, wrong_bin_usage, redeploy_without_pv and scale_pod_social_net now call wait_until_pods_healthy. They keep their own extra checks. They also get the settle window from Add a bounded settle window to mitigation recovery eval #180.
5. The kafka problem checks its flag. kafka_queue_problems now also requires the kafkaQueueProblems flag to be off again. This uses a new OtelFaultInjector.get_flag_variant.
6. Two small helpers. KubeCtl.list_deployments and KubeCtl.list_stateful_sets.

Why the record is safe

The record stores each workload's configured replica count. Kubernetes sets this as soon as the app is deployed. So the record does not depend on pods being ready yet.

If a task never records its workloads, eval behaves exactly as before.

Tests

1. I added 7 unit tests in tests/orchestrator/test_mitigation_settle.py. They cover a pod stuck in Pending, a deleted workload, a workload scaled to 0, a restored workload, a workload deployed with 0 replicas, a task with no record, and the kafka flag on and off.
2. I replaced the old test that expected pods with no container statuses to be skipped.
3. I ran pytest tests --ignore=tests/integration before and after this change. The same 22 tests fail both times, because they need a live cluster. All new tests pass.

Live check

I ran all 14 mitigation problems on a kind cluster, before and after this change. After each action, I polled eval for 120 seconds, the same window AIOpsLab uses by default.

What happened after the fault	Eval passed before	Eval passed after
Broken Deployment scaled to 0	5 of 6	0 of 6
Broken Deployment deleted	5 of 6	0 of 6
Healthy Deployment scaled to 0	13 of 13	0 of 13
All pods restarted	4 of 14	2 of 14
No action for 300 seconds	4 of 14	2 of 14
recover_fault()	14 of 14	13 of 14

Notes on the table:

• The first two rows apply only to the 6 problems that leave a Deployment unhealthy after the fault.
• With no action, assign_to_non_existent_node and kafka_queue_problems now fail. redeploy_without_PV and misconfig_app_hotel_res still pass. These two are listed under "Not covered" below.
• After a restart, kafka_queue_problems now fails, because the flag is still on.
• The one recover_fault() that now fails is redeploy_without_PV. Its recover_fault() calls app.cleanup(), which deletes the application. So after it, the namespace has no Deployments at all. The old check passed because there were no pods left to check. The new check fails because every recorded Deployment is gone. This looks like a separate problem in that task's recovery, not in this change.

Not covered

1. redeploy_without_PV: data loss does not show in pod state.
2. misconfig_app_hotel_res: eval can still pass before the crash shows up.

[Nazmus Sakib (TheDeadcoder)]

Nazmus Sakib (Nazmus Sakib (@TheDeadcoder)) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree