Initialization of device cfi.pflash01 failed: Block node is read-only

Hi,

I have several vms defined in libvirt using ovmf for uefi, since a later
update of my server I'm unable to start any of the domains defined. This is
an example of the output given:

# virsh start os-1
error: Failed to start domain os-1
error: internal error: qemu unexpectedly closed the monitor: 2019-03-02T21:23:51.726446Z qemu-system-x86_64: Initialization of device cfi.pflash01 failed: Block node is read-only

an example of domain is like this:

<domain type='kvm'>                                                                                                                                                                                                                                                            
  <name>os-1</name>                                                                                                                                                                                                                                                            
  <uuid>34c41008-ab91-483b-959c-81a7a12ae9be</uuid>                                                                                                                                                                                                                            
  <memory unit='KiB'>8388608</memory>                                                                                                                                                                                                                                          
  <currentMemory unit='KiB'>8388608</currentMemory>                                                                                                                                                                                                                            
  <memoryBacking>                                                                                                                                                                                                                                                              
    <hugepages/>                                                                                                                                                                                                                                                               
  </memoryBacking>                                                                                                                                                                                                                                                             
  <vcpu placement='static' cpuset='10-11,34-35'>4</vcpu>                                                                                                                                                                                                                       
  <os>                                                                                                                                                                                                                                                                         
    <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type>                                                                                                                                                                                                                    
    <loader type='pflash'>/var/lib/libvirt/qemu/nvram/os-1-ovmf.fd</loader>                                                                                                                                                                                                    
    <boot dev='network'/>                                                                                                                                                                                                                                                      
    <boot dev='hd'/>                                                                                                                                                                                                                                                           
  </os>                                                                                                                                                                                                                                                                        
  <features>                                                                                                                                                                                                                                                                   
    <acpi/>                                                                                                                                                                                                                                                                    
    <apic/>                                                                                                                                                                                                                                                                    
  </features>                                                                                                                                                                                                                                                                  
  <cpu mode='host-passthrough' check='partial'/>                                                                                                                                                                                                                               
  <clock offset='utc'>                                                                                                                                                                                                                                                         
    <timer name='rtc' tickpolicy='catchup'/>                                                                                                                                                                                                                                   
    <timer name='pit' tickpolicy='delay'/>                                                                                                                                                                                                                                     
    <timer name='hpet' present='no'/>                                                                                                                                                                                                                                          
  </clock>                                                                                                                                                                                                                                                                     
  <on_poweroff>destroy</on_poweroff>                                                                                                                                                                                                                                           
  <on_reboot>restart</on_reboot>                                                                                                                                                                                                                                               
  <on_crash>destroy</on_crash>                                                                                                                                                                                                                                                 
  <pm>                                                                                                                                                                                                                                                                         
    <suspend-to-mem enabled='no'/>                                                                                                                                                                                                                                             
    <suspend-to-disk enabled='no'/>                                                                                                                                                                                                                                            
  </pm>                                                                                                                                                                                                                                                                        
  <devices>                                                                                                                                                                                                                                                                    
    <emulator>/usr/bin/qemu-system-x86_64</emulator>                                                                                                                                                                                                                           
    <disk type='block' device='disk'>                                                                                                                                                                                                                                          
      <driver name='qemu' type='raw' cache='none' io='native'/>                                                                                                                                                                                                                
      <source dev='/dev/vg0/os-1-vda'/>                                                                                                                                                                                                                                   
      <target dev='vda' bus='virtio'/>                                                                                                                                                                                                                                         
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>                                                                                                                                                                                              
    </disk>                                                                                                                                                                                                                                                                    
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/vg0/os-1-vdb'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:78:cb:97'/>
      <source bridge='virbr0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:1c:4f:22'/>
      <source bridge='virbr1'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

where /var/lib/libvirt/qemu/nvram/os-1-ovmf.fd is a copy of
/usr/share/edk2-ovmf/OVMF_VARS.fd. An the extract from my
/etc/libvirt/qemu.conf to define ovmf:

...
# Location of master nvram file                                                                                                                                                                                                                                                
#                                                                                                                                                                                                                                                                              
# When a domain is configured to use UEFI instead of standard                                                                                                                                                                                                                  
# BIOS it may use a separate storage for UEFI variables. If                                                                                                                                                                                                                    
# that's the case libvirt creates the variable store per domain                                                                                                                                                                                                                
# using this master file as image. Each UEFI firmware can,                                                                                                                                                                                                                     
# however, have different variables store. Therefore the nvram is                                                                                                                                                                                                              
# a list of strings when a single item is in form of:                                                                                                                                                                                                                          
#   ${PATH_TO_UEFI_FW}:${PATH_TO_UEFI_VARS}.                                                                                                                                                                                                                                   
# Later, when libvirt creates per domain variable store, this list is                                                                                                                                                                                                          
# searched for the master image. The UEFI firmware can be called                                                                                                                                                                                                               
# differently for different guest architectures. For instance, it's OVMF                                                                                                                                                                                                       
# for x86_64 and i686, but it's AAVMF for aarch64. The libvirt default                                                                                                                                                                                                         
# follows this scheme.                                                                                                                                                                                                                                                         
nvram = [                                                                                                                                                                                                                                                                      
   "/usr/share/edk2-ovmf/OVMF_CODE.fd:/usr/share/edk2-ovmf/OVMF_VARS.fd",                                                                                                                                                                                                      
]
...

This setup used to work one month ago, now it's setup with:

QEMU emulator version 3.1.0
libvirt-5.0.0
linux-4.19.20

Any help appreciated.

Best regards.

José

Extract from the libvirt log of todays launch of vms.


The uncommented piece of my /etc/libvirt/qemu.conf.

vnc_listen = "192.168.0.61"

user = "root"

group = "root"

dynamic_ownership = 0

nvram = [
   "/usr/share/edk2-ovmf/OVMF_CODE.fd:/usr/share/edk2-ovmf/OVMF_VARS.fd",
]

For some reason, emails directly from mailing list are not hitting my inbox
so I managed to miss this:

https://<email address hidden>/msg600959.html

Please ensure it's sent to my email when answering.

Thanks!

José.

seems to be piece of true on Phil's comments, as extracting the command on the log and executing it manually the result is the following:

2019-04-18T19:37:49.685841Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-os-1/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-6-os-1/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-6-os-1/master-key.aes”: No such file or directory

unless those folders with master keys are dinamically genreated and trashed after execution it would be worthy to involve here libvirt project as well.

José.

Hi José,

your domain XML is bogus with regard to the firmware configuration. You
have:

     <loader type='pflash'>/var/lib/libvirt/qemu/nvram/os-1-ovmf.fd</loader>

and no <nvram> element, and you write that "os-1-ovmf.fd" is a copy of
"OVMF_VARS.fd".

The <loader> element, with @type='pflash', no other attributes, and then
no <nvram> sibling element either, happens to be valid, but it is an
extremely niche use case. It is used when you have a unified, writeable,
OVMF.fd file that contains both the firmware executable and the live
variable store.

Most distros don't provide such unified "OVMF.fd" files, and even if
yours does, you are not passing a full such "OVMF.fd" to the guest, but
just a variable store -- there is no firmware binary at all that the
guest could *execute*.

So please do the following:

- power down the domain

- delete "/var/lib/libvirt/qemu/nvram/os-1-ovmf.fd" for starters

- modify the domain XML as follows, with "virsh edit":

    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>

When you exit your editor, libvirtd will:

- create the <nvram> element, and auto-generate the pathname for the
  <nvram> element

- create the varstore file, pointed-to by the pathname above, from the
  template at "/usr/share/edk2-ovmf/OVMF_VARS.fd".


(more precisely, the nvram element will be auto-generated when you exit "virsh edit", and the nvram file will be created (copied) from the varstore template when you launch the domain)

thanks for the thorough pointers Laszlo, I did and run in different kind
of issues, and given that I was just having the vars around I really wonder
how it ever worked, perhaps I copied the unified and I forgot, as gentoo
thas provide it:

# du -h /usr/share/edk2-ovmf/*
1.9M    /usr/share/edk2-ovmf/OVMF_CODE.fd
2.0M    /usr/share/edk2-ovmf/OVMF.fd
128K    /usr/share/edk2-ovmf/OVMF_VARS.fd

Thanks!

José

oh well, it turns out to be actually related, it turns out
virt-aa-helper is skipping the code file of ovmf:

2019-04-19 10:06:43.576+0000: 9272: error : AppArmorGenSecurityLabel:469 : internal error: cannot load AppArmor profile 'libvirt-34c41008-ab91-483b-959c-81a7a12ae9be'
2019-04-19 10:09:25.751+0000: 9270: error : virCommandWait:2636 : internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/virt-aa-helper -c -u libvirt-34c41008-ab91-483b-959c-81a7a12ae9be) unexpected exit status 1: 2019-04-19 10:09:25.750+0000: 13797: info : libvirt version: 5.2.0
2019-04-19 10:09:25.750+0000: 13797: info : hostname: mole
2019-04-19 10:09:25.750+0000: 13797: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vda': Permission denied
2019-04-19 10:09:25.750+0000: 13797: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vdb': Permission denied
virt-aa-helper: error: /usr/share/edk2-ovmf/OVMF_CODE.fd
virt-aa-helper: error: skipped restricted file
virt-aa-helper: error: invalid VM definition

The domain now looks like:

<domain type='kvm'>                                                                                                                                                                                                                                                    [40/100]
  <name>os-1</name>
  <uuid>34c41008-ab91-483b-959c-81a7a12ae9be</uuid>    
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>                              
  <memoryBacking>
    <hugepages/>             
  </memoryBacking>                      
  <vcpu placement='static' cpuset='10-11,34-35'>4</vcpu>
  <os>                       
    <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type>                      
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/os-1_VARS.fd</nvram>
    <boot dev='network'/>               
    <boot dev='hd'/>           
  </os>                      
  <features>                                                                     
    <acpi/>     
    <apic/>            
  </features>                            
  <cpu mode='host-passthrough' check='partial'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/> 
  </clock>    
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>      
  <on_crash>destroy</on_crash>                    
  <pm>                        
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>                                                          
  <devices>                                                                      
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>                  
      <source dev='/dev/mole-vg0/os-1-vda'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mole-vg0/os-1-vdb'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:78:cb:97'/>
      <source bridge='virbr0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:1c:4f:22'/>
      <source bridge='virbr1'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

this is apparmor profile related in the end, rebooting with apparmor disabled allows
the domain to boot, so I'll deal with the gentoo community, and this can be closed.

Thanks!

José.