Use node image overlay during scale up

[jlebon]

(This isn't strictly an openshift/os change, but more how the node image is used.)

Starting in 4.19, we switched the installer to directly overlay the node image on top of the initial boot to avoid paying for another reboot (see openshift/installer#8742).

We could consider doing this as well for the scale up case where saving a reboot is also extremely valuable. With the bootimage skew work, we can add enforcements that this feature can only be turned on if the bootimage used is within some version, and that also limits CI coverage needed for this.

There would be some major caveats around this that users would have to accept before turning this on. Primarily the kernel mismatch. Again, I think skew limitations allow us to keep this testable, but any operator that has a tighter binding on the node image kernel than our skew check (or any drivers) wouldn't be usable on the first boot.

There's also soft-reboot nowadays of course. It'd be slower than the overlay approach but probably not by much. Worth looking at the timing. But we do still need to keep the overlay approach anyway for the live case (but ideally eventually extend bootc's soft-reboot feature to support this use case).

Probably simplest would be to move all the node pulling and overlaying logic out of the openshift/installer repo and into the base image where it can be more easily reused.

[travier]

It's unlikely that soft-reboot will work as in most cases, after a few weeks, the new image will have a new kernel and soft-reboot refuses to operate in this case. Doing a manual mount of the kernel modules like we do for the live case could be an option but I don't know if bootc folks will accept that and it's risky.

[sdodson]

Two things, I hadn't realized that the rebootless pivot kept the kernel from the boot image though I should've. I agree that in the situations where elastic scaling time is a priority we can forego with the caveats/limits you're putting forth. Perhaps a MachineConfig flag that determines whether or not kernel version convergence is required, it could be set on the pool level and perhaps only allowed for a pool that has boot image management enabled?

Have we looked at how we're chunking the images lately? It'd probably be good to take a look at that since the advent of layered builds and perhaps optimize specifically for z-stream updates and just assume that if folks care about scale up time they're leveraging boot image management to ensure that images are matched at least to the RHCOS X.Y release.

[jlebon]

Have we looked at how we're chunking the images lately?

Yeah, I've thought about passing the node image through chunkah though the gain there is probably going to be minimal given that the base RHCOS image itself was already well-split.

But to repeat something I've said elsewhere, we should definitely sanity-check/add CI checks verifying that the layer refs embedded in the bootimages are still getting good dedupe with the initial node image pull. We had issues with that initially that basically negated any potential gains there because of layer recompression in the node image pipeline.