Repository navigation
Elasticsearch restore does not restore the sts_k8s_logs data stream #126
Description
sts-backup elasticsearch restore never restores the sts_k8s_logs data stream, only its backing indices. This produces three failures.
1. Logs are orphaned after a successful restore. Since #40 (0.10.0), RestoreSnapshot is called with an explicit list of index names from restorableSnapshotIndices. The data stream name isn't in that list, so the .ds-sts_k8s_logs-* indices come back as ordinary hidden indices with no data stream. Seen restoring a 10-nonha backup into a fresh 150-ha instance (CLI 0.11.2, https://github.com/StackVista/stackstate/issues/882). The restore reported success, but there was no sts_k8s_logs data stream afterwards. Restoring {"indices":"sts_k8s_logs"} from the same snapshot repaired it.
2. A retry fails with 404. This is the original report, from verify-backup run 41391996239 (CLI 0.11.0). The first attempt stalled at 93/108 indices. Attempts 2 and 3 failed straight away:
Found 106 STS index(es) to delete
Rolling over datastream 'sts_k8s_logs'...
❌ Error: failed to rollover datastream: elasticsearch returned error: [404 Not Found] ... rollover target [sts_k8s_logs] does not exist
deleteAllSTSIndices rolls the data stream over whenever any .ds-sts_k8s_logs-* index is listed. Because of (1), those indices exist without the data stream.
3. Restore fails when the target is already ingesting logs. In that case the target has the data stream. deleteAllSTSIndices rolls it over, creating -000002, then deletes only the indices it listed beforehand, so the new write index survives. If the snapshot also contains a -000002, _restore fails with cannot restore index [.ds-sts_k8s_logs-<date>-000002] because an open index with same name already exists. Seen restoring a 150-ha backup into a 10-nonha instance (CLI 0.11.2, stackstate#882).
verify-backup doesn't catch (1) or (3). Its fresh instance receives no logs, and its Elasticsearch probe checks only topology events.
Manual workaround: DELETE _data_stream/sts_k8s_logs, restore {"indices":"sts_k8s_logs,sts_topology_events-*","include_global_state":false}, then sts-backup elasticsearch check-and-finalize --operation-id <snapshot>.
Done when
- the restore deletes the target's
sts_k8s_logsdata stream, if present, instead of rolling it over, and still deletes orphaned backing indices - the restore request includes the data stream name, so
sts_k8s_logsexists after restore with the snapshot's backing indices - regression tests cover: a target with no data stream, a target with orphaned backing indices, and a target whose live data stream has a write index that collides with the snapshot
Activity
[-]Elasticsearch restore retry fails: rollover of sts_k8s_logs 404s after a partial restore[/-][+]Elasticsearch restore does not restore the sts_k8s_logs data stream[/+]viliakov commentedon Nov 9, 2026
Verification gap tracked in https://github.com/StackVista/verify-backup/issues/22.
viliakov commentedon Nov 9, 2026
Design for the fix (verified against ES 8.19.16 source and experiments on sts-882-nonha). The full design, with tests and acceptance steps, is in mission-control
plans/backup-cli-126-design.md.Cleanup (replaces the rollover)
.ds-sts_k8s_logs-*index,DELETE _data_stream/sts_k8s_logs. A 404 means no data stream (orphans only), so carry on. A 200 means re-list the indices._restoreand print thecheck-and-finalize --finalize-onlyhint.Writers
sts_k8s_logs. The ES restore label already covers it in both profiles:receiverin non-HA,receiver-baseandreceiver-logsin HA.scale.ScaleDownwaits for pods matching the workload label, and pods never carry that label. So the wait returns while the pods are still terminating. Fix: wait on each scaled workload'sspec.selector. This applies to every datastore.Restore request
sts_k8s_logsto the explicit index list when the snapshot'sdata_streamscontains it. Keep the backing index names; that request is valid.sts_k8s_logsbacking indices but nosts_k8s_logsdata stream, fail before scale-down, with nothing touched. Chart SLM snapshots (sts*) always include the stream, so only hand-made snapshots hit this.partialis passed through unchanged.Unchanged
restorableSnapshotIndicesandcheck-and-finalize: the expected list stays concrete index names.The uncommitted
restore-logs-datastreamwork is kept as the base, plus the guard, the pod-wait fix and extra tests.Restore the sts_k8s_logs data stream with Elasticsearch restore (#131)