other: 0.887
permissions: 0.886
debug: 0.861
socket: 0.832
semantic: 0.826
network: 0.817
device: 0.812
performance: 0.806
PID: 0.802
graphic: 0.793
KVM: 0.782
files: 0.777
vnc: 0.748
boot: 0.747

qemu-kvm with snapshot option always fails with Permission denied Could not open disk image

I'm trying to use the option: -snapshot       write to temporary files instead of disk image files

How to reproduce? See following log:
2013-06-20 02:13:18.532+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -S -M pc-1.0 -no-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name instance-0000002b -uuid 2d600758-ae56-48b8-bd4d-999744a038e4 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000002b.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -kernel /opt/stack/data/nova/instances/instance-0000002b/kernel -initrd /opt/stack/data/nova/instances/instance-0000002b/ramdisk -append root=/dev/vda console=ttyS0 -drive file=/opt/stack/data/nova/instances/instance-0000002b/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=19,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=fa:16:3e:03:ab:18,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/opt/stack/data/nova/instances/instance-0000002b/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:26868 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -snapshot
Domain id=1 is tainted: custom-argv
char device redirected to /dev/pts/18
qemu-system-x86_64: -drive file=/opt/stack/data/nova/instances/instance-0000002b/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: could not open disk image /opt/stack/data/nova/instances/instance-0000002b/disk: Permission denied
2013-06-20 02:13:18.683+0000: shutting down

Version: QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard

Related info:
The disk is a qcow2 image with a backing file. Both the backing file and the disk are cmodded with 777.

This is a log from dmesg related to apparmor:
[  236.531287] type=1400 audit(1371694399.156:17): apparmor="STATUS" operation="profile_remove" name="libvirt-2d600758-ae56-48b8-bd4d-999744a038e4" pid=4201 comm="apparmor_parser"


libvirt.xml that I'm using:
<domain type="qemu" xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <uuid>2d600758-ae56-48b8-bd4d-999744a038e4</uuid>
  <name>instance-0000002b</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <os>
    <type>hvm</type>
    <kernel>/opt/stack/data/nova/instances/instance-0000002b/kernel</kernel>
    <initrd>/opt/stack/data/nova/instances/instance-0000002b/ramdisk</initrd>
    <cmdline>root=/dev/vda console=ttyS0</cmdline>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc"/>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/opt/stack/data/nova/instances/instance-0000002b/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <disk type="block" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target tray="open" dev="hdc"/>
      <readonly/>
    </disk>
    <interface type="bridge">
      <mac address="fa:16:3e:03:ab:18"/>
      <source bridge="br100"/>
      <filterref filter="nova-instance-instance-0000002b-fa163e03ab18">
        <parameter name="IP" value="10.0.0.3"/>
        <parameter name="DHCPSERVER" value="10.0.0.1"/>
        <parameter name="PROJNET" value="10.0.0.0"/>
        <parameter name="PROJMASK" value="255.255.255.0"/>
      </filterref>
    </interface>
    <serial type="file">
      <source path="/opt/stack/data/nova/instances/instance-0000002b/console.log"/>
    </serial>
    <serial type="pty"/>
    <input type="tablet" bus="usb"/>
    <graphics type="vnc" autoport="no" port="32768" keymap="en-us" listen="127.0.0.1"/>
  </devices>
<qemu:commandline>
   <qemu:arg value='-snapshot'/>
</qemu:commandline>
</domain>

On 06/19/2013 08:42 PM, Sam Stoelinga wrote:
> Public bug reported:
> 
> I'm trying to use the option: -snapshot       write to temporary files
> instead of disk image files
> 

> 
> libvirt.xml that I'm using:
> <domain type="qemu" xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

> <qemu:commandline>
>    <qemu:arg value='-snapshot'/>
> </qemu:commandline>
> </domain>

This is unsupported usage of libvirt, and not a qemu bug.  You'd need to
take this up with the libvirt list to get libvirt to properly support
temporary disk images without needing <qemu:commandline>, and so that
libvirt is properly setting SELinux permissions on the temporary file.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org



Hi, quick question, 
    I thought that using the <transient/> xml tag of <disk> element is the right way to do in libvirt ? 
Why is <qemu:commandline> method being used ?

Also with -snapshot, iiuc the temp. file is created by QEMU internally, so which temp. file and its selinux perms is being referenced above ?


On 07/25/2013 10:09 AM, Deepak C Shetty wrote:
> Hi, quick question, 
>     I thought that using the <transient/> xml tag of <disk> element is the right way to do in libvirt ? 

Yes, that is the designed way.  Unfortunately, it has not been
implemented yet (no one has been clamoring for the feature enough to
write the patch themselves, or for someone else to take interest and
write a patch on their behalf).

> Why is <qemu:commandline> method being used ?

To try and work around the unimplemented nature of the libvirt design.

> 
> Also with -snapshot, iiuc the temp. file is created by QEMU internally,
> so which temp. file and its selinux perms is being referenced above ?
> 

Qemu creating a file itself when libvirt has set SELinux rules on the
qemu instance is very likely to fail, since qemu doesn't know what label
to give the temp file, but the temp file must be labeled to be used.
Hence, this really needs to be implemented properly in libvirt, and is
not a qemu bug.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org



On 07/25/2013 10:32 AM, Eric Blake wrote:
> On 07/25/2013 10:09 AM, Deepak C Shetty wrote:
>> Hi, quick question, 
>>     I thought that using the <transient/> xml tag of <disk> element is the right way to do in libvirt ? 
> 
> Yes, that is the designed way.  Unfortunately, it has not been
> implemented yet (no one has been clamoring for the feature enough to
> write the patch themselves, or for someone else to take interest and
> write a patch on their behalf).

In particular, see this libvirt bug, which is stagnating due to
higher-priority bugs that I am working on first:

https://bugzilla.redhat.com/show_bug.cgi?id=832194

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org



Eric,
    Thanks for the quick reply. I saw the <transient/> desc. in formatdomain.html and thought its supported. So does that mean its supported for other hypervisors but not QEMU/KVM ? If not supported at all, why does it show up in the doc... its misleading.

I had a recent need to start exploiting this feature and landed up here. I am willing to work on supporting <transient/> with your guidance :) since I don't have much knowledge of SELinux.

thanx,
deepak

According to Eric's comments, this was a libvirt bug, not a QEMU bug, so closing this ticket now. If you still encounter this problem, please report it to the libvirt project instead.