architecture: 0.964 graphic: 0.932 performance: 0.816 files: 0.788 device: 0.787 boot: 0.772 permissions: 0.770 assembly: 0.750 PID: 0.713 register: 0.652 mistranslation: 0.637 arm: 0.604 user-level: 0.592 kernel: 0.591 peripherals: 0.558 socket: 0.544 semantic: 0.485 debug: 0.445 risc-v: 0.422 network: 0.386 vnc: 0.361 ppc: 0.353 hypervisor: 0.286 VMM: 0.235 TCG: 0.173 virtual: 0.168 x86: 0.150 i386: 0.136 KVM: 0.047 systemd-tmpfiles-setup-dev-early.service fails in emulated systemd-nspawn container Description of problem: When booting a fresh `debootstrap`ed Debian Trixie/testing rootfs with foreign architecture via `systemd-nspawn` and `qemu-user-static`, invoked via `systemd-binfmt`, the `systemd-tmpfiles-setup-dev-early.service` service within the guest fails, which leads to `/dev` not existing (respectively no default content), so that several other guest system components fail as well, like any console/shell access: ``` Starting systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully... systemd-tmpfiles-setup-dev-early.service: Failed to set up credentials: Invalid argument systemd-tmpfiles-setup-dev-early.service: Main process exited, code=exited, status=243/CREDENTIALS systemd-tmpfiles-setup-dev-early.service: Failed with result 'exit-code'. [FAILED] Failed to start systemd-tmpfiles-setup-dev-early.service - Create Static Device Nodes in /dev gracefully. See 'systemctl status systemd-tmpfiles-setup-dev-early.service' for details. Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev... systemd-tmpfiles-setup-dev.service: Failed to set up credentials: Invalid argument systemd-tmpfiles-setup-dev.service: Main process exited, code=exited, status=243/CREDENTIALS systemd-tmpfiles-setup-dev.service: Failed with result 'exit-code'. [FAILED] Failed to start systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev. See 'systemctl status systemd-tmpfiles-setup-dev.service' for details. ``` Steps to reproduce: 1. `apt install debootstrap systemd-container qemu-user-static` 2. `systemctl restart systemd-binfmt` 3. `mkdir rootfs` 4. `debootstrap --variant=minbase --include=systemd-sysv --arch=arm64 trixie ./rootfs 'https://deb.debian.org/debian'` 5. `systemd-nspawn -bD rootfs` Additional information: On Bookworm guest systems and/or without QEMU emulation, this works without issues, so I guess systemd recently started to use a certain syscall for the `ImportCredential=tmpfiles.*` method in systemd units, which is not supported by QEMU, probably similar to https://github.com/systemd/systemd/pull/28954? I hope it is fine to report it here. Always difficult to decide whether to report to the distribution (Debian) or one, and in case which, of the related projects, which do not work together. Debian Trixie currently ships `systemd 254.4-1` btw. I am not sure whether the issue was introduced with 253 or 254, since the linked issue prevented booting such containers on an earlier stage with 253, which was fixed in 254, which has the here reported issue.