Error when migrating s390x VM from QEMU 9.0 to 9.1: Unknown savevm section or instance 's390_css' Description of problem: I have been working on merging QEMU 9.1.1 (directly from Debian unstable), and I'm seeing this problem when trying to migrate an s390x VM from an Oracular host (which runs QEMU 9.0.2) to a Plucky host (which runs QEMU 9.1.1). The problem only happens on s390x (host and guest), and only when attempting to migrate from Oracular to Plucky. Migrations between Oracular guests work fine, as well as migrations between Plucky guests. This is the error I see after invoking `virsh migrate`: ``` error: internal error: QEMU unexpectedly closed the monitor (vm='kvmguest-jammy-normal'): 2024-11-27T21:13:43.745625Z qemu-system-s390x: Unknown savevm section or instance 's390_css' 0. Make sure that your current VM setup matches your saved VM setup, including any hotplugged devices 2024-11-27T21:13:43.746914Z qemu-system-s390x: load of migration failed: Invalid argument ``` Steps to reproduce: I only have one s390x machine available, so I am resorting to creating two LXD containers that are KVM-capable. One of the containers runs Oracular, the other runs Plucky. Please let me know if you would instructions on how to create such containers. Inside the Oracular container, using `uvt-kvm` to simplify the process of creating the VM: ``` # uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=s390x label=daily release=oracular # cat > guesttemplate.xml << _EOF_ hvm _EOF_ # uvt-kvm create --template /root/guesttemplate.xml --machine-type s390-ccw-virtio-9.0 --password=ubuntu --ssh-public-key-file /home/ubuntu/.ssh/authorized_keys kvmguest-oracular-upstream-cpu release=oracular arch=s390x label=daily ``` Wait a moment for the VM to boot, use `virsh list` to make sure it's running. Note that we force the machine type to be `s390-ccw-virtio-9.0`; this is necessary because Ubuntu overrides the default machine type with its own definition, and we want to make sure to use upstream's type here. Make sure you're running QEMU 9.1.1 at least on the Plucky container. Plucky currently ships with QEMU 9.0.2, which doesn't have the problem. If needed, my QEMU 9.1.1 build can be found at https://launchpad.net/~sergiodj/+archive/ubuntu/qemu. After everything is in place, try to migrate the machine: ``` # virsh migrate --unsafe --live kvmguest-oracular-upstream-cpu qemu+ssh://plucky-container-IP-here/system error: internal error: QEMU unexpectedly closed the monitor (vm='kvmguest-oracular-upstream-cpu'): 2024-11-29T22:28:21.417201Z qemu-system-s390x: Unknown savevm section or instance 's390_css' 0. Make sure that your current VM setup matches your saved VM setup, including any hotplugged devices 2024-11-29T22:28:21.417496Z qemu-system-s390x: load of migration failed: Invalid argument ``` Additional information: libvirt log from Oracular (QEMU 9.0.2): ``` LC_ALL=C \ [2/1817] PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin \ USER=root \ HOME=/var/lib/libvirt/qemu/domain-3-kvmguest-oracular-up \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-3-kvmguest-oracular-up/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-3-kvmguest-oracular-up/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-3-kvmguest-oracular-up/.config \ /usr/bin/qemu-system-s390x \ -name guest=kvmguest-oracular-upstream-cpu,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-3-kvmguest-oracular-up/master-key.aes"}' \ -machine s390-ccw-virtio-9.0,usb=off,dump-guest-core=off,memory-backend=s390.ram \ -accel kvm \ -cpu z13.2-base,aen=on,aefsi=on,diag318=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,cei=on,ap=on,gpereh=on,esop=on,ib=on,siif=on,ibs=on,apqi=on,apft=on,els=on,sief2=on,apqci=on,cte=on,ais=on,bpb=on,64bscao=on,ctop=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on,gsls=on \ -m size=524288k \ -object '{"qom-type":"memory-backend-ram","id":"s390.ram","size":536870912}' \ -overcommit mem-lock=off \ -smp 1,sockets=1,cores=1,threads=1 \ -uuid fa8bcf1a-8982-47ab-9766-ebbb695008e3 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=38,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device '{"driver":"virtio-serial-ccw","id":"virtio-serial0","devno":"fe.0.0003"}' \ -blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjQuMTA6czM5MHggMjAyNDExMjY=","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \ -blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/kvmguest-oracular-upstream-cpu.qcow","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \ -blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/kvmguest-oracular-upstream-cpu-ds.qcow","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \ -device '{"driver":"virtio-blk-ccw","devno":"fe.0.0001","drive":"libvirt-1-format","id":"virtio-disk1"}' \ -netdev '{"type":"tap","fd":"39","id":"hostnet0"}' \ -device '{"driver":"virtio-net-ccw","netdev":"hostnet0","id":"net0","mac":"52:54:00:d8:f0:5c","devno":"fe.0.0002"}' \ -chardev socket,id=charchannel0,fd=36,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -chardev pty,id=charconsole0 \ -device '{"driver":"sclpconsole","chardev":"charconsole0","id":"console0"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0004"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/3 (label charconsole0) 2024-11-28 20:56:00.522+0000: initiating migration 2024-11-28T20:56:01.114894Z qemu-system-s390x: Sibling indicated error 1 warning: old compression is deprecated; use multifd compression methods instead warning: old compression is deprecated; use multifd compression methods instead warning: old compression is deprecated; use multifd compression methods instead warning: block migration is deprecated; use blockdev-mirror with NBD instead ``` libvirt log from Plucky (QEMU 9.1.1): ``` LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin \ USER=root \ HOME=/var/lib/libvirt/qemu/domain-4-kvmguest-oracular-up \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-4-kvmguest-oracular-up/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-4-kvmguest-oracular-up/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-4-kvmguest-oracular-up/.config \ /usr/bin/qemu-system-s390x \ -name guest=kvmguest-oracular-upstream-cpu,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-4-kvmguest-oracular-up/master-key.aes"}' \ -machine s390-ccw-virtio-9.0,usb=off,dump-guest-core=off,memory-backend=s390.ram \ -accel kvm \ -cpu z13.2-base,aen=on,aefsi=on,diag318=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,cei=on,ap=on,gpereh=on,esop=on,ib=on,siif=on,ibs=on,apqi=on,apft=on,els=on,sief2=on,apqci=on,cte=on,ais=on,bpb=on,64bscao=on,ctop=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on,gsls=on \ -m size=524288k \ -object '{"qom-type":"memory-backend-ram","id":"s390.ram","size":536870912}' \ -overcommit mem-lock=off \ -smp 1,sockets=1,cores=1,threads=1 \ -uuid fa8bcf1a-8982-47ab-9766-ebbb695008e3 \ -display none \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=35,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ -device '{"driver":"virtio-serial-ccw","id":"virtio-serial0","devno":"fe.0.0003"}' \ -blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MjQuMTA6czM5MHggMjAyNDExMjY=","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \ -blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/kvmguest-oracular-upstream-cpu.qcow","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \ -blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/kvmguest-oracular-upstream-cpu-ds.qcow","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \ -device '{"driver":"virtio-blk-ccw","devno":"fe.0.0001","drive":"libvirt-1-format","id":"virtio-disk1"}' \ -netdev '{"type":"tap","fd":"36","id":"hostnet0"}' \ -device '{"driver":"virtio-net-ccw","netdev":"hostnet0","id":"net0","mac":"52:54:00:d8:f0:5c","devno":"fe.0.0002"}' \ -chardev socket,id=charchannel0,fd=34,server=on,wait=off \ -device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \ -chardev pty,id=charconsole0 \ -device '{"driver":"sclpconsole","chardev":"charconsole0","id":"console0"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -incoming defer \ -device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0004"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on char device redirected to /dev/pts/3 (label charconsole0) 2024-11-29T22:28:21.417201Z qemu-system-s390x: Unknown savevm section or instance 's390_css' 0. Make sure that your current VM setup matches your saved VM setup, including any hotplugged devices 2024-11-29T22:28:21.417496Z qemu-system-s390x: load of migration failed: Invalid argument ``` Domain XML: ```xml kvmguest-oracular-upstream-cpu fa8bcf1a-8982-47ab-9766-ebbb695008e3 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhWPh2Wfm2Ouh/W+H9IXJGFHfH4UVCB6+EBI0PwuDXR2Ocl4hTTSNPSX2LVS4MfVn9pgl5BK9MUVsMPfFjhmEhpNNt+rmaCelrDT8A7v/RoBY4IGEBFMhAkiwlI7pk3BrFoHEKtiijNLEWczdjMigZvhTs2amn8cUotFIsQSTpM7+7IX+m7clxfe6p59mVPjfMzBhwDG0GyV7CXdMpvsGlE2mPSacWWZ/baWIoFjKcmyQtTjSQleH1qSthI8rD5F7EyYd1Oa8Bo7vZ9j1/DPeGQRJPkebO81hPjm/1x1H5pTITIzARdNuBkM0yuDyqMQLP/u65WGinvXJYm20gEvMbiHGaT3il1QKKNEGmNGtY/SedRE8XQ58n090IBLz/3WJtjgQCY/SRgHUv7nMYYenmshvBfdue9kExJTjwWTRtT2R2UdkxS5UVye4vvDAY0DFuqX13wyvIeCU28MU+HpmnE31m9uXlVXXZxDuqGUBJ1PrDc4a40bvj9yTZTn9NEOs= root@localhost ssh-dss AAAAB3NzaC1kc3MAAACBAKqzgDKUGk6P/h6N5X4nJoHPr+MQzzXkotN8XEihvtWwvV1KYK+ioT69nA7ThEAZ6rPEjWPt7X4Sy6BcNd4j3kzlaXYLkrMJm3nohqbqQBDxCv8bhozy6HS/VDu95vrpnNFSiMRCfbBye0zyKfZsuRaPaKfHQ+8MnsBqSPxKajFrAAAAFQDuG3ZoanC+oZwMRYZ/am0vhfD+EwAAAIEAixSzoZr03kYZE+LcusyrasvZIqKF3P4M2vtzvFBPpPccFB5XoaqhWI4PvSxGYxZxlj9vRmSc8Yv56jdn8oDPIhFfgZVbDIkvpB2jQdb5VaRVWj7XwUcHB7B117Dr9qA6+6HJtBLRTDdTXzMQ+NhdFp42XCF+1qRefH9VPL9FoxwAAACAJa+u/YvaiGwT0DXBtTz4PgyFYmNHPvXBOVhDAw0likajBiuOdn8oL4KuWTafCq5ReDxXFaMML0OuT86+lSVt2naX0idyHjuSPkgmatozlpcz0kWYhuBl1B1sa3kr8xjDOUJlxkybqpdGJ5aoW+kRO+bpJLEzuXtu6Xshw5fOBZw= root@localhost ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHI8u/wAvZLJqIpAd5YSpu9VEaRQOxy0FKzyryeb3kjahkryKPhSX65miZ9Lx7oz5nORFsdeS2xR56ZQj+8HpqM= root@localhost ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDXY+MW1SikusLdkhPrni76LlaZB042p/DVItVeHRCCa root@localhost 524288 524288 1 /machine hvm z13.2-base destroy restart destroy /usr/bin/qemu-system-s390x