Repository navigation
fix(wic): orphan warnings in initramfs ext4 fsck invocations on ro partitions - #701
Conversation
When in testing a wic image is used multiple times to initialize a test device it can happen that a stale journal i.e., one from system runs prior to flashing anew, gets applied. This can happen when ... - bmaptool is used to flash scarce wic images - the wic images don't populate the journal data region This fix stops using the lazy journal initialization so that the journal data gets part of the generated wic image, even though large parts of them are filled with `0` (and would therefore be left out normally). Signed-off-by: Harry Waschkeit <44188360+HarryWaschkeit@users.noreply.github.com>
JanZachmann
left a comment
There was a problem hiding this comment.
The approach works: with lazy_journal_init=0 the journal becomes part of the bmap, so any bmap-based flash overwrites the old journal. Two things before merge, see inline: the extra options replace wic's defaults (the inode count on data halves), and the result depends on FIEMAP on the build host.
Description: "scarce wic images" should be "sparse". Removing the journal from the read-only partitions is a separate change and could be its own commit.
21eca0a to
5c721b6
Compare
…pts` Signed-off-by: Harry Waschkeit <44188360+HarryWaschkeit@users.noreply.github.com>
…initramfs fsck output Signed-off-by: Harry Waschkeit <44188360+HarryWaschkeit@users.noreply.github.com>
JanZachmann
left a comment
There was a problem hiding this comment.
The PR title and description become the squash commit, and both describe the cause wrongly. The initramfs runs no fsck on rootA/rootB/factory (check_fs only runs for cert, etc, data and boot). EXT4-fs (...): orphan cleanup on readonly fs is a kernel info message at mount time: with the orphan_file feature, which the mke2fs.conf of the build enables by default, the kernel prints it on every read-only mount, also when there is no orphan. So nothing is found and nothing is left unfixed. The change removes that feature, together with the journal. Could you reword title and description along these lines? The fix(wic) scope will also be too narrow once the ext4 image gets the same options (see the rootB thread).
…ages Signed-off-by: Harry Waschkeit <44188360+HarryWaschkeit@users.noreply.github.com>
…options (#158) ## Summary - `write_partition` compares the partition file with the image in 4 KiB blocks, only where one of them holds data, and writes only the blocks that differ, instead of running `fallocate -d`. It also writes a block that is data in the partition file but a hole in the image, e.g. a zero block of a file that `e2cp` or `mcopy` wrote into free space. `fallocate`, `dd` and `sync` are removed from the Docker image. - The image is copied with `sparse::copy_file`, and a partition is read out of the image with the same sparse copy instead of `dd conv=sparse`: it finds the data with `SEEK_DATA`/`SEEK_HOLE`, keeps the holes and copies each data range in a loop. Decompressing an image keeps zero blocks as holes, reads a file made of several streams (e.g. from pbzip2) and ignores zero padding after the last one, and fails on a truncated or corrupt stream. A compressed image is decompressed straight into the temp dir, without a copy first. A symlinked image, e.g. the Yocto deploy link, is detected by its target, so a compressed one is no longer read as uncompressed. Images named `.gz` and `.bz2` are decompressed too; before, the output file had the name of the source, so creating it emptied the source. Compressing fails when the end of the stream cannot be written. - omnect-cli writes the bmap of `-b` itself: it maps the ranges of the input bmap, or the whole image without one, plus every block the image file stores as data, with a sha256 per range. No zeros are written into the image, so it stays sparse, and `bmaptool` is no longer needed at runtime: `-b` now works in the Docker image, and `bmap-tools` is removed from the Debian dependencies and the Docker image. The bmap is copied next to the image after the image was written back. - The input bmap is the file given with the new `--input-bmap` argument (requires `-b`), else `<image>.bmap` or the bmap of the uncompressed image next to it (e.g. `x.wic.bmap` for `x.wic.xz`, as the Yocto build creates it). It is only read with `-b`. The command fails if the input bmap is for an image of another size, if a range does not match its sha256 checksum, if the bmap has no range checksums, if the file is larger than 64 MiB or not UTF-8, or if `bmaptool` would find the input image itself as its bmap (an image named e.g. `x.bmap`). The new `--no-input-bmap` (requires `-b`) skips it, and the bmap then maps the whole image. The input bmap of an uncompressed image is checked before the image is copied. - The created bmap is named `<output image>.bmap` and written next to the output image, e.g. `x.wic.bmap` for `x.wic` and `x.wic.xz.bmap` for `x.wic.xz`, where `bmaptool` finds it. So with `-p` it no longer replaces the bmap of the input image, which stays unchanged. An image changed in place keeps the name of its bmap: the new bmap replaces the one `bmaptool` finds next to it, e.g. `x.wic.bmap` for `-i x.wic.xz -p xz`, so no bmap of the old image stays. - The output image is the input image, for a compressed image without the compression extension, plus the `-p` extension, as before. The new `-o`/`--output-image` writes it to another path, and the input image stays unchanged. With `-p`, the `-o` name must end in an extension of the compression that `bmaptool` reads (`.xz`, `.bz2`, `.gzip` or `.gz`); without `-p` it must not have a compression extension. An existing output image must be a regular file, so `-o /dev/sdX` or a directory is refused before the command runs. A command now fails instead of overwriting an existing file that is not the input image (a path given with `-o` is overwritten), writing a bmap that `bmaptool` would also use for an image that stays, i.e. `<name>` or `<name>.<compression>` next to it without another bmap that comes first, also named after the target of a symlinked bmap (e.g. `x.wic.bmap` for `-i x.wic.xz` without `-p`, also if no bmap exists yet and also with `--input-bmap` or `--no-input-bmap`, or for an in-place `-i x.wic.xz -p xz` while `x.wic` is next to it), or writing a bmap over the input or the output image (e.g. an image named `x.bmap`), or writing the output image over the input bmap, or to a path that `bmaptool` reads as the bmap of another image (e.g. `-i x.wic -o x.wic.bmap`), or writing an image without `-b` while `<output image>.bmap` is next to it, or for a compressed output image the bmap of the uncompressed image, which `bmaptool` also uses. That bmap would no longer match. So for several commands on one image, use `-b` on each. - The output image and the bmap are written to a new file in the same directory and renamed over the old one, so a failed write, e.g. on a full disk, no longer destroys an image that is changed in place. A symlink is followed, and the old file keeps its permissions. A symlink whose target does not exist yet is followed too. The directory needs write access and space for one more image, which is checked before the command runs; an existing output image in a directory the user cannot write is staged in TMPDIR and copied into it, so an in-place run works as before, and the new file belongs to the user who runs omnect-cli. The image is replaced before its bmap: if the image step fails, the old image keeps its bmap, and if the bmap step fails, the old bmap is removed, and the error names it if it stays. An output image that is a mount point, e.g. a file mounted into a container, cannot be replaced by a rename, so its new content is staged in TMPDIR and copied into it; a rename that fails with EBUSY, EXDEV or EPERM (a sticky dir like /tmp and a file of another user) also falls back to this copy; if that copy fails, the staged file is kept with its new bmap, the old bmap is removed, and the error names both. An old bmap that cannot be removed is named in the error too. With `-b` such a mounted image fails, because the bmap would be written next to the mount point and not next to the image. All temp dirs are created with mode 0700, because they hold images and keys. Ctrl-C, SIGTERM or SIGHUP removes every temp dir of a command before the process stops; a signal that is ignored, e.g. SIGHUP under `nohup`, is not handled. The handler starts with each image command, because as PID 1, e.g. in the Docker image, the process ignores a stop signal without a handler. As PID 1 the process then exits with 128 plus the signal number, because it ignores a signal it raises itself. An image command refuses a `TMPDIR` that contains `:` before it starts, because e2tools split their file arguments there. A container started with `--rm` removes a file kept in its `TMPDIR`, so the README asks for a `TMPDIR` mounted from the host for a mounted image. - `copy-from-image` no longer writes the image back. It only reads, so an error can no longer truncate the source, and for `x.wic.xz` it no longer leaves `x.wic` next to it. It reads an uncompressed image directly, without a temp copy, so it refuses an output file of `-f` that is the input image, also through a hard link, a symlink or an output dir, by device and inode. - The partition files are written to their own temp dir. Before, they were named `<num>.img` next to the image, so an image named e.g. `1.img` was overwritten by its partition 1. The temp dirs follow `TMPDIR`. - `set-device-certificate` and `set-edge-ca-certificate` write the new certificate and key to a temp dir instead of under fixed names next to the input image, so two runs on one base image no longer share them, a read-only image directory works, and no private key is left there. - Partitions of a GPT with 4096-byte sectors are read at the right offset; before, the offsets were always counted in 512-byte sectors. A partition that does not fit into the image, e.g. from a corrupt GPT, is refused, so its offset cannot overflow. - Known limitations, tracked as follow-up tasks: the rename replaces only the name it writes, so other hard links of the output image keep the old image. On a kernel before 5.8, which does not report mount points in statx, a mounted image file with `-b` is not detected, and the bmap is written next to the mount point. The created bmap maps every block that the image file stores as data, so for an image that is not sparse it maps the whole image, also with an input bmap. For a partition that does not start on a 4 KiB boundary, a block at the edge of each data run is written back, so the image is less sparse. In both cases the image content is correct. An uncompressed output image is copied twice, into TMPDIR and into the staged file. A stop signal between the image and the bmap step keeps the old bmap next to the new image. The output name checks do not know the archive extensions of `bmaptool` (`.tar.*`, `.zip`), nor `.zst`, `.lz4` and `.lzo` without `-p`. For a symlinked image, the bmap is looked up only next to the link name, so the bmap next to the target is not found and does not match after an in-place change. A bmap with fewer extensions than an image, e.g. `x.bmap` for `x.wic.xz`, is not checked as the bmap of an image that stays, because only `<name>` and `<name>.<compression>` count as images of `<name>.bmap`. - `copy-from-image` accepts an output file without a directory, e.g. `out.txt`, and checks all output dirs before it reads the image. - `actix-web` and `actix-http` are updated for RUSTSEC-2026-0446, RUSTSEC-2026-0447 and RUSTSEC-2026-0448. - **Breaking:** `--generate-bmap-file` is renamed to `--output-bmap`, so the two options read as a pair. The short option `-b` stays. With `-p` the bmap is `<output image>.bmap`, e.g. `x.wic.xz.bmap` where releases before wrote `x.wic.bmap`, unless the image is changed in place and a bmap exists next to it. `bmaptool` finds the new name for `x.wic.xz` by itself, but a script that passes `--bmap x.wic.bmap` must use the new name. The name conflicts above fail where they overwrote files before, or wrote a bmap that `bmaptool` then used for the wrong image: `-i x.wic.xz -b` without `-o` or `-p` now fails, because the new `x.wic.bmap` would not match `x.wic.xz`. `-p bzip2` writes `<image>.bz2` instead of `<image>.bzip2`, because `bmaptool` does not read `.bzip2`. An input named `.bzip2` is still read, but with `-p` the result is written next to it as `.bz2`, so a second run on the same input fails because that file exists. `-o` needs `.bz2`. - roxmltree is added to read the bmap, rustix for `SEEK_DATA`/`SEEK_HOLE` and statx, signal-hook to remove temp dirs on a stop signal (its `signal-hook-registry` was already in the tree through tokio), and libc for `sigaction` (already in the tree); libfs is removed. The test bmap `testfiles/image.wic.bmap` is now in git (`*.bmap` is in `.gitignore`), with checksums that match the test images. The version is bumped to 0.29.0, and ctutils to 0.4.5 for RUSTSEC-2026-0385. ## Reason A sparse image, e.g. one that xz decompressed into a file, or one after `fallocate -d`, has holes wherever the image is zero, and a bmap skips them. Some of these zeros must still be written: a bmap-based flash otherwise keeps what an earlier installation left there. An old ext4 journal can be replayed after a power cut and apply an old transaction to the new filesystem; old GPT entries break the entry array CRC; old FAT sectors and directory entries show up as used clusters or stale files. On an rpi4-64 image, mapping only the ext4 journals still drops 188 blocks of the Yocto bmap: GPT primary and backup entries, 69 blocks of boot, and 16–24 blocks on every ext4 partition. The Yocto bmap maps all of these ranges, because the build image holds real data there. Keeping its ranges mapped covers every format without knowing about file systems. From a sparse image alone, omnect-cli cannot tell such zero blocks from free space, so without an input bmap it maps the whole image. `bmaptool copy` reads a mapped range with plain reads, so a hole in such a range is written as zeros: the zeros only have to be in the bmap, not allocated in the image file. This replaces the image-side approach discussed in omnect/meta-omnect#701. With `default-features = false`, libfs made byte copies, so every image came back fully allocated, and the decompress wrote zero blocks as data. `fallocate -d` hid both. The sparse copy uses `SEEK_DATA`/`SEEK_HOLE` directly: `libfs::next_sparse_segments` seeks both files to the same offset, so it cannot read a partition out of the image, and `libfs::copy_file` copies each data segment with a single `copy_file_range` call, which moves at most 2 GiB, and ignores a short copy. All omnect partitions have a fixed size, so builds of one machine have the same image size, and the bmap of another build (e.g. of an older download next to a new image) passes a size check. The sha256 of each range tells the builds apart; `bmaptool copy` runs the same check on flash. omnect-cli derives every output name from the input name and finds the input bmap by name, so names can collide. omnect-cli runs in CI scripts, where nobody reads a warning, so each collision fails with an error that names the option to resolve it. A counter in the output name would not work: a script and the later `bmaptool copy` need a fixed name. Writing only changed blocks also fixes a gap of the old `dd conv=notrunc,sparse`: a block that changed to all zeros was skipped and kept its old content; now it is written because it differs. In the same way, `dd conv=sparse` turned every zero block of a partition into a hole, so a zero block of a copied file that landed in free space stayed a hole in the image, and a bmap flash kept the old content there. --------- Signed-off-by: Jan Zachmann <50990105+JanZachmann@users.noreply.github.com>
When read-only partitions get mounted during system start-up there are always orphan entries found that cannot be fixed.
Turning off journaling on read-only partition - which doesn't make sense anyway -, gets rid of these warnings..