id = 1183 title = "KVM crash due to qcow2 out of space condition during virsh-snapshot creation" state = "opened" created_at = "2022-08-31T14:02:11.427Z" closed_at = "n/a" labels = ["qemu-img"] url = "https://gitlab.com/qemu-project/qemu/-/issues/1183" host-os = "n/a" host-arch = "n/a" qemu-version = "qemu-img version 6.2.0 (qemu-kvm-6.2.0-11.module+el8.6.0+15668+464a1f31.2)" guest-os = "n/a" guest-arch = "n/a" description = """virsh snapshot failed due to out of space condition (into the qcow2 image ?) libvirt log: ``` 2022-08-27T06:41:41.164368Z qemu-kvm-one: terminating on signal 15 from pid 1782 (/usr/sbin/libvirtd) 2022-08-27T06:41:41.172667Z qemu-kvm-one: Failed to flush the L2 table cache: Input/output error 2022-08-27T06:41:41.172692Z qemu-kvm-one: Failed to flush the refcount block cache: Input/output error ```""" reproduce = """1. not possible for that moment - i did resize/increase the qcow2 image - now its running again.""" additional = """as i saw - there was a very old qemu-snapshot, which was not properly deleted. After removing this snapshot, i did reszie the image. I do suppose, this could be one reason the image (qcow2) got full ? Because all is THIN i was not aware of it (fs level ok, storage layer ok). Is there any tool, how free space in a thin qcow2 file can be monitored ? ``` PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \\ HOME=/var/lib/libvirt/qemu/domain-13-one-89 \\ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-13-one-89/.local/share \\ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-13-one-89/.cache \\ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-13-one-89/.config \\ QEMU_AUDIO_DRV=none \\ /usr/bin/qemu-kvm-one \\ -name guest=one-89,debug-threads=on \\ -S \\ -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-one-89/master-key.aes \\ -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \\ -cpu qemu64 \\ -m 8192 \\ -overcommit mem-lock=off \\ -smp 4,sockets=4,cores=1,threads=1 \\ -uuid 8c920c7f-f687-4c47-bfc7-671425c7436b \\ -no-user-config \\ -nodefaults \\ -chardev socket,id=charmonitor,fd=40,server,nowait \\ -mon chardev=charmonitor,id=monitor,mode=control \\ -rtc base=utc \\ -no-shutdown \\ -boot strict=on \\ -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \\ -device virtio-scsi-pci,id=scsi0,num_queues=1,bus=pci.0,addr=0x4 \\ -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \\ -blockdev '{"driver":"file","filename":"/var/lib/one//xxxx/disk.0","aio":"threads","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \\ -blockdev '{"node-name":"libvirt-3-format","read-only":false,"discard":"unmap","cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \\ -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-3-format,id=scsi0-0-0-0,bootindex=1,write-cache=off \\ -blockdev '{"driver":"file","filename":"/var/lib/one//xxxx/disk.1","aio":"threads","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \\ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \\ -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,device_id=drive-scsi0-0-1-0,drive=libvirt-2-format,id=scsi0-0-1-0,write-cache=off \\ -blockdev '{"driver":"file","filename":"/var/lib/one//xxxx/disk.2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \\ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \\ -device ide-cd,bus=ide.0,unit=0,drive=libvirt-1-format,id=ide0-0-0 \\ -netdev tap,fd=42,id=hostnet0 \\ -device e1000,netdev=hostnet0,id=net0,mac=02:00:c0:a8:02:17,bus=pci.0,addr=0x3 \\ -chardev socket,id=charchannel0,fd=43,server,nowait \\ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \\ -vnc 0.0.0.0:89 \\ -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \\ -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \\ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \\ -msg timestamp=on ``` as the time of the crash the qcow2 status was: (so i'm not sure the issue is about a space problem or a bug in qemu): ``` qemu-img info xxx/0/xxx image: xxx/0/xxx file format: qcow2 virtual size: 1.46 TiB (1610612736000 bytes) disk size: 988 GiB cluster_size: 65536 Snapshot list: ID TAG VM SIZE DATE VM CLOCK ICOUNT 112 snap-111 0 B 2022-03-11 01:59:15 49:07:53.846 282 snap-281 0 B 2022-08-20 01:59:17538:16:30.416 283 snap-282 0 B 2022-08-21 01:59:16562:10:40.759 284 snap-283 0 B 2022-08-22 01:59:16585:59:16.170 285 snap-284 0 B 2022-08-23 01:59:16609:51:44.825 286 snap-285 0 B 2022-08-24 01:59:16633:45:32.243 287 snap-286 0 B 2022-08-25 01:59:16657:36:44.718 288 snap-287 0 B 2022-08-26 01:59:16681:29:00.793 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false extended l2: false root@proxpve1:~# qemu-img check xxxx/0/xxx No errors were found on the image. 15252433/24576000 = 62.06% allocated, 6.32% fragmented, 0.00% compressed clusters Image end offset: 1062936117248 1rst (OS) Disk on the VM: ------------------------------------------ file format: qcow2 virtual size: 100 GiB (107374182400 bytes) disk size: 190 GiB cluster_size: 65536 Snapshot list: ID TAG VM SIZE DATE VM CLOCK ICOUNT 282 snap-281 7.66 GiB 2022-08-20 01:59:17538:16:30.416 283 snap-282 7.6 GiB 2022-08-21 01:59:16562:10:40.759 284 snap-283 7.62 GiB 2022-08-22 01:59:16585:59:16.170 285 snap-284 7.65 GiB 2022-08-23 01:59:16609:51:44.825 286 snap-285 7.62 GiB 2022-08-24 01:59:16633:45:32.243 287 snap-286 7.63 GiB 2022-08-25 01:59:16657:36:44.718 288 snap-287 7.65 GiB 2022-08-26 01:59:16681:29:00.793 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false extended l2: false No errors were found on the image. 782257/1638400 = 47.75% allocated, 22.16% fragmented, 0.00% compressed clusters Image end offset: 315680292864 ```"""