Skip to content

Conversation

@pothos
Copy link
Member

@pothos pothos commented Dec 10, 2025

For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

This includes a systemd update to 258.2 to have a smaller diff for the backports.

How to use

Wait for verity cert fix for release builds. Update git refs before merging (drop mantle change here by updating it through the regular ref update PR).

Testing done

With mantle PR used here for GitHub Actions and on Jenkins

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)

  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc. (FROM_A=bincache FROM_B=bincache FILESONLY=1 CUTKERNEL=1 CUTSIGKEYPATH=1 FILE=… ./package-diff 4547.0.0 9999.0.101+kai-default-confext)

  • CI: http://localhost:8080/job/container/job/packages_all_arches/7135/cldsv/

@github-actions
Copy link

github-actions bot commented Dec 15, 2025

Test report for 4552.0.0+nightly-20251217-2100 / amd64 arm64

Platforms tested : qemu_uefi-amd64 qemu_update-amd64 qemu_uefi-arm64 qemu_update-arm64

🟢 ok bpf.execsnoop; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok bpf.local-gadget; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.basic; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.cloudinit.basic; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.cloudinit.multipart-mime; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.cloudinit.script; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid0.data; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid0.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid1.data; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.disk.raid1.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.etcd-member.discovery; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.etcd-member.etcdctlv3; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.etcd-member.v2-backup-restore; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.filesystem; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.flannel.udp; Succeeded: qemu_uefi-amd64 (1)

🟢 ok cl.flannel.vxlan; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.instantiated.enable-unit; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.kargs; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.luks; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.indirect; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.indirect.new; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.regular; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.regular.new; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.reuse; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.oem.wipe; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.partition_on_boot_disk; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.symlink; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.translation; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.btrfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.ext4root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.groups; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.once; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.sethostname; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.users; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v1.xfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.btrfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.ext4root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.users; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2.xfsroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2_1.ext4checkexisting; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2_1.swap; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.ignition.v2_1.vfat; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.install.cloudinit; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.internet; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.locksmith.cluster; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.initramfs.second-boot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.iptables; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.listeners; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.nftables; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.network.wireguard; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.omaha.ping; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.osreset.ignition-rerun; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.overlay.cleanup; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.swap_activation; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.sysext.boot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.sysext.fallbackdownload # SKIP; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tang.nonroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tang.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.toolbox.dnf-install; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.eventlog; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.nonroot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root-cryptenroll; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root-cryptenroll-pcr-noupdate; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.tpm.root-cryptenroll-pcr-withupdate; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.update.badverity; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.update.payload; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok cl.update.payload-boot-part-too-small; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok cl.update.reboot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.users.shells; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok cl.verity; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok confext.skiprefresh; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.auth.verify; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.groups; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.once; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.resource.local; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.resource.remote; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.resource.s3.versioned; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.security.tls; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.sethostname; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.ignition.systemd.enable-service; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.locksmith.reboot; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.locksmith.tls; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.selinux.boolean; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.selinux.enforce; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.tls.fetch-urls; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok coreos.update.badusr; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok devcontainer.docker; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok devcontainer.systemd-nspawn; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.btrfs-storage; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.containerd-restart; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.enable-service.sysext; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.lib-coreos-dockerd-compat; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.network-openbsd-nc; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.selinux; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok docker.userns; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok extra-test.[first_dual].cl.update.docker-btrfs-compat; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok extra-test.[first_dual].cl.update.payload; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok extra-test.[first_dual].cl.update.payload-boot-part-too-small; Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

🟢 ok kubeadm.v1.32.4.calico.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.32.4.cilium.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.32.4.flannel.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.33.0.calico.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.33.0.cilium.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.33.0.flannel.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.34.1.calico.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.34.1.cilium.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok kubeadm.v1.34.1.flannel.base; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok linux.nfs.v3; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok linux.nfs.v4; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok linux.ntp; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok misc.fips; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok packages; Succeeded: qemu_uefi-amd64 (4); qemu_uefi-arm64 (1); Failed: qemu_uefi-amd64 (1, 2, 3)

                Diagnostic output for qemu_uefi-amd64, run 3
    L1: " _packages/sys-block/open-iscsi (33.06s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: a2318d6c47ec: Download complete"
    L7: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L8: "cluster.go:125: 3d3086a1439f: Download complete"
    L9: "cluster.go:125: a2318d6c47ec: Pull complete"
    L10: "cluster.go:125: 3d3086a1439f: Pull complete"
    L11: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L12: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L13: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L14: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L15: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L16: " "
                Diagnostic output for qemu_uefi-amd64, run 2
    L1: " _packages/sys-block/open-iscsi (22.53s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: a2318d6c47ec: Verifying Checksum"
    L7: "cluster.go:125: a2318d6c47ec: Download complete"
    L8: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L9: "cluster.go:125: 3d3086a1439f: Download complete"
    L10: "cluster.go:125: a2318d6c47ec: Pull complete"
    L11: "cluster.go:125: 3d3086a1439f: Pull complete"
    L12: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L13: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L14: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L15: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L16: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L17: " "
                Diagnostic output for qemu_uefi-amd64, run 1
    L1: " _packages/sys-block/open-iscsi (32.82s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L7: "cluster.go:125: 3d3086a1439f: Download complete"
    L8: "cluster.go:125: a2318d6c47ec: Verifying Checksum"
    L9: "cluster.go:125: a2318d6c47ec: Download complete"
    L10: "cluster.go:125: a2318d6c47ec: Pull complete"
    L11: "cluster.go:125: 3d3086a1439f: Pull complete"
    L12: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L13: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L14: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L15: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L16: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L17: " "
    L18: "  "

🟢 ok sysext.custom-docker.sysext; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.custom-oem; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.disable-containerd; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.disable-docker; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok sysext.simple; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok systemd.journal.remote; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok systemd.journal.user; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

🟢 ok systemd.sysusers.gshadow; Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

@pothos
Copy link
Member Author

pothos commented Dec 16, 2025

I've added the upstream PR links to the commit message:
aef9fd6

@pothos
Copy link
Member Author

pothos commented Dec 16, 2025

For less disruption due to emerge version picking I've deleted the unused systemd ebuilds but can add them back and make sure that the mask setting is enough to use the wanted version.

Copy link
Contributor

@chewi chewi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume @danzatt will unrevert his changes along with the necessary fix before you merge this.

I haven't reviewed the upstream systemd changes, as I'll leave the experts to do that.

From 7f0d7d0eb24afe14cd62bfa78cb9e139b9f824d1

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
Signed-off-by: Kai Lueke <kailuke@microsoft.com>
Most sysexts need a daemon reload to get their services (re)started when
systemd-sysext (re)loads extensions after boot. Up to now this was done
through ensure-sysext.service but only when systemd-sysext.service was
restarted. A manual invocation of systemd-sysext did not get the
daemon reload and, e.g., Docker failed to start.
Since the daemon reload is done when at least one extension needs it
and we can say that this is likely true for most of the extensions we
build, we can unconditionally set it for now and see if we need to
make it conditional per sysext later, e.g., when one could only have
plain CLI extensions and no service extensions such as Docker/containerd
or Podman (and even NVIDIA/ZFS require services after load).

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
This reverts commit d28ece4.

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
Signed-off-by: Kai Lueke <kailuke@microsoft.com>
@pothos pothos force-pushed the kai/default-confext branch from cd18daa to 0ea4f07 Compare December 18, 2025 04:31
pothos added a commit to flatcar/init that referenced this pull request Dec 19, 2025
When the systemd-sysupdate.service runs it fails if there is no config.
Since we don't use it, it's expected to do nothing but this failure is
making problems when the service is started from the timer unit which is
now enabled by default. This service is also used by the sysext-bakery
update configs for drop-in steps that update sysupdate components. There
we let users add a noop config to prevent this service failure.
Since we run into this in Flatcar now and we anyway had users add this
manually, let's just add it directly to Flatcar.

## How to use

Needs new scripts PR. Should then be backported to Alpha and Beta.

## Testing done

Was tested in flatcar/scripts#3555
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants