qemu-img convert a overlay qcow2 image into a entire image I have a base image file "base.qcow2" and a delta qcow2 image file "delta.qcow2" whose backing file is "base.qcow2". Now I use qemu-img to convert "delta.qcow2" and will get a new image file "new.qcow2" which is complete and equivalent to combination of "base.qcow2" and "delta.qcow2". In fact, i just want to convert the delta qcow2 image file and get a new delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I have to admit that this is not bug. Could you please take this as a new feature and enable qemu-img to convert images in-place? On 02/05/2017 09:19 PM, wayen wrote: > > I have a base image file "base.qcow2" and a delta qcow2 image file > "delta.qcow2" whose backing file is "base.qcow2". > > Now I use qemu-img to convert "delta.qcow2" and will get a new image > + file "new.qcow2" which is entire and equivalent to combination of > "base.qcow2" and "delta.qcow2". > > In fact, i just want to convert the delta qcow2 image file and get a new > delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I > have to admit that this is not bug. Could you please take this as a new > feature and enable qemu-img to convert images in-place? This feature already exists. Use: qemu-img rebase -F $backing_fmt -b '' -f qcow2 delta.qcow2 and now delta.qcow2 will be a complete image. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org @Eric Blake, Thanks very much for your help. In your way, I have verified that this feature already exists. @Eric Blake. Sorry, I didn't make it clear. In fact, I don't want to get a complete image. I just want to convert qcow2 overlay and get a new qcow2 overlay. Maybe you think my intention is meaningless， but this is what I want. Can't you simply do a "cp delta.qcow2 new.qcow2" ? @Thomas Huth, when we use qemu-img to convert a image "A" and will get a new image "B" which is equivalent to "A" . But "B" may be a lot different from "A",such as file size. The file size of "B" is usually less than "A" and this is very valuable to me. So I can't simply do a "cp delta.qcow2 new.qcow2" ? I meant to copy B, not A ... but I likely just haven't really understood what you're really trying to do here, so never mind. @Thomas Huth, I just want to reduce the file size of qcow2 overlay image by this conversion. On 02/08/2017 02:16 AM, wayen wrote: > @Thomas Huth, I just want to reduce the file size of qcow2 overlay image > by this conversion. Are you merely trying to sparsify the file (punch holes on the portion of the file that is not mapped to disk), so that the apparent file size is unchanged, but the actual disk usage is minimized? That's probably already possible (virt-sparsify from libguestfs seems to be able to do it; look at what steps it uses). Or are you asking for a way to defragment the file, so that the apparent size shrinks because all occupied clusters are now contiguous, so that there are no longer any need for holes? The end result is the same amount of disk usage, but right now, the only way to defragment is to convert from one image to a copy; we don't support in-place defragmentation yet. So far, no one has come up with a compelling reason why it is needed, or a patch to implement it, but there's no technical reason why it can't be done. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org @Eric Blake, I used virt-sparsify to sparsify the qcow2 overlay image. As you said, the actual disk usage is minimized and the apparent file size is unchanged.It is very valuable for me, because it means that my disk can hold more files. But we need to be careful when transfer the sparse qcow2 overlays. Because some tools do not support sparse file and will convert it into a common file,for example, scp. And I doubt what will happen when transfer sparse files between different file systems. So I want to use qemu-img to convert the sparse qcow2 overlay into common file. If this was feasible, the holes in the sparse overlay will be removed and the above problems will disappear. On Thu, Feb 09, 2017 at 02:05:54AM -0000, wayen wrote: > @Eric Blake, I used virt-sparsify to sparsify the qcow2 overlay image. > As you said, the actual disk usage is minimized and the apparent file > size is unchanged.It is very valuable for me, because it means that my > disk can hold more files. > > But we need to be careful when transfer the sparse qcow2 overlays. > Because some tools do not support sparse file and will convert it into a > common file,for example, scp. And I doubt what will happen when transfer > sparse files between different file systems. > > So I want to use qemu-img to convert the sparse qcow2 overlay into > common file. If this was feasible, the holes in the sparse overlay will > be removed and the above problems will disappear. You can use the drive_mirror HMP command or drive-mirror QMP command on a running QEMU instance to copy the data to a new file and switch to the new file. I'm curious what exactly is being optimized by copying out a fresh qcow2 image. Please post the output of: $ stat delta.qcow2 $ qemu-img map delta.qcow2 $ stat new.qcow2 $ qemu-img map new.qcow2 This will allow us to see the size and data layout differences. Maybe a new command should be added to QEMU to do the optimization in-place. This would avoid the disk space overhead associated with drive-mirror, cp, qemu-img convert, etc. Stefan On Tue, Feb 14, 2017 at 02:17:16AM -0000, wayen wrote: > ** Attachment added: "qemu-img map new.qcow2 output" > https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818564/+files/qemu_img_map_new_qcow2.txt Thanks for posting the attachments. I ran a script to find unallocated clusters in the delta.qcow2 host file. Most are actually qcow2 metadata (L1/L2 tables, refcount blocks). This output shows that any image file size reduction you are hoping to achieve can only come from zero clusters. There are no holes in the files that would result in significant image file size reduction if a new image were written out. Just wanted to share this info in case anyone else is thinking about how to optimize qcow2 files. I still think rewriting images sequentially can be useful - if internal snapshots were used and deleted then COW can result in holes. Hole at 0 size 5.0 clusters Hole at 393216 size 1.0 clusters Hole at 589824 size 1.0 clusters Hole at 1114112 size 1.0 clusters Hole at 1310720 size 1.0 clusters Hole at 1507328 size 1.0 clusters Hole at 1703936 size 1.0 clusters Hole at 2293760 size 1.0 clusters Hole at 2621440 size 1.0 clusters Hole at 3080192 size 1.0 clusters Hole at 5111808 size 1.0 clusters Hole at 6291456 size 1.0 clusters Hole at 30408704 size 1.0 clusters Hole at 47906816 size 1.0 clusters Hole at 142671872 size 1.0 clusters Hole at 219545600 size 1.0 clusters Hole at 667090944 size 1.0 clusters Hole at 853868544 size 1.0 clusters Hole at 1562640384 size 1.0 clusters Hole at 2147483648 size 1.0 clusters Hole at 2617180160 size 1.0 clusters Hole at 3411148800 size 1.0 clusters Hole at 4107075584 size 1.0 clusters Hole at 4294967296 size 1.0 clusters Hole at 4452646912 size 1.0 clusters Hole at 4792057856 size 1.0 clusters Hole at 5494865920 size 1.0 clusters Hole at 5645271040 size 1.0 clusters Hole at 5702483968 size 1.0 clusters Hole at 6187188224 size 1.0 clusters Hole at 6442450944 size 1.0 clusters Hole at 6862995456 size 1.0 clusters Hole at 6987317248 size 1.0 clusters Hole at 7567245312 size 1.0 clusters Hole at 8135245824 size 1.0 clusters Hole at 8590589952 size 1.0 clusters Hole at 8613462016 size 1.0 clusters Hole at 9055436800 size 1.0 clusters Hole at 9703522304 size 1.0 clusters Hole at 10279321600 size 1.0 clusters Hole at 10737418240 size 3.0 clusters Hole at 10844372992 size 1.0 clusters Hole at 11167858688 size 1.0 clusters Hole at 11209605120 size 1.0 clusters Hole at 11209801728 size 1.0 clusters Hole at 11730944000 size 1.0 clusters Hole at 12183207936 size 1.0 clusters Hole at 12705464320 size 1.0 clusters Hole at 12884901888 size 1.0 clusters Hole at 13444120576 size 1.0 clusters Hole at 13910016000 size 1.0 clusters Hole at 14182711296 size 1.0 clusters Hole at 15025635328 size 1.0 clusters Hole at 15032385536 size 1.0 clusters The following script draws the allocated clusters and holes in the image file. I took your qemu-img map output, filtered out any lines with base.qcow2, and sorted using sort -k3 -g to sort on the "Mapped to" field. Then I ran ./qcow2-map-svg.py output.svg. #!/usr/bin/python3 import sys import io COLOR_ALLOCATED = '#ffaaaa' COLOR_HOLE = '#999999' def svg_percentage(value, total): return '{0}%'.format(100.0 * value / total) def svg_rect(x, width, color): print(''.format(x, width, color), file=out) def svg_text(x, y, text): print('{2}'.format(x, y, text), file=out) out = io.StringIO() print(''' ''', file=out) file_map = [] header = True for line in sys.stdin: if header: header = False continue offset, length, mapped, filename = line.split() offset = int(offset, 16) length = int(length, 16) mapped = int(mapped, 16) file_map.append((offset, length, mapped, filename)) file_size = file_map[-1][2] + file_map[-1][1] last_mapped = 0 for _, length, mapped, _ in file_map: if mapped > last_mapped: # if mapped - last_mapped: # print('Hole at {0} size {1} clusters'.format(last_mapped, (mapped - last_mapped) / 65536)) svg_rect(svg_percentage(last_mapped, file_size), svg_percentage(mapped - last_mapped, file_size), COLOR_HOLE) svg_rect(svg_percentage(mapped, file_size), svg_percentage(length, file_size), COLOR_ALLOCATED) last_mapped = mapped + length if last_mapped < file_size: svg_rect(svg_percentage(last_mapped, file_size), svg_percentage(file_size - last_mapped, file_size), COLOR_HOLE) for i in range(10): svg_text(svg_percentage(i, 10), 60, svg_percentage(i, 10)) print('', file=out) print(out.getvalue()) Is there any way to remove holes from qcow2 overlay images? It's very important to me. I am looking forward to your reply. Is there any way to remove holes from sparse qcow2 overlay? It's very important to me. I am looking forward to your reply. Hi wayen: Which version are you used? I also find this problem on old version qemu, and i write a patch for it. It works. I'm not sure the mainline version have solve this problem. what command are you used? On Mon, Apr 10, 2017 at 10:14 AM, wayen