1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
|
Initialization of device cfi.pflash01 failed: Block node is read-only
Hi,
I have several vms defined in libvirt using ovmf for uefi, since a later
update of my server I'm unable to start any of the domains defined. This is
an example of the output given:
# virsh start os-1
error: Failed to start domain os-1
error: internal error: qemu unexpectedly closed the monitor: 2019-03-02T21:23:51.726446Z qemu-system-x86_64: Initialization of device cfi.pflash01 failed: Block node is read-only
an example of domain is like this:
<domain type='kvm'>
<name>os-1</name>
<uuid>34c41008-ab91-483b-959c-81a7a12ae9be</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement='static' cpuset='10-11,34-35'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type>
<loader type='pflash'>/var/lib/libvirt/qemu/nvram/os-1-ovmf.fd</loader>
<boot dev='network'/>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-passthrough' check='partial'/>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/vg0/os-1-vda'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/vg0/os-1-vdb'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:78:cb:97'/>
<source bridge='virbr0'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:1c:4f:22'/>
<source bridge='virbr1'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'>
<listen type='address'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</memballoon>
</devices>
</domain>
where /var/lib/libvirt/qemu/nvram/os-1-ovmf.fd is a copy of
/usr/share/edk2-ovmf/OVMF_VARS.fd. An the extract from my
/etc/libvirt/qemu.conf to define ovmf:
...
# Location of master nvram file
#
# When a domain is configured to use UEFI instead of standard
# BIOS it may use a separate storage for UEFI variables. If
# that's the case libvirt creates the variable store per domain
# using this master file as image. Each UEFI firmware can,
# however, have different variables store. Therefore the nvram is
# a list of strings when a single item is in form of:
# ${PATH_TO_UEFI_FW}:${PATH_TO_UEFI_VARS}.
# Later, when libvirt creates per domain variable store, this list is
# searched for the master image. The UEFI firmware can be called
# differently for different guest architectures. For instance, it's OVMF
# for x86_64 and i686, but it's AAVMF for aarch64. The libvirt default
# follows this scheme.
nvram = [
"/usr/share/edk2-ovmf/OVMF_CODE.fd:/usr/share/edk2-ovmf/OVMF_VARS.fd",
]
...
This setup used to work one month ago, now it's setup with:
QEMU emulator version 3.1.0
libvirt-5.0.0
linux-4.19.20
Any help appreciated.
Best regards.
José
Extract from the libvirt log of todays launch of vms.
The uncommented piece of my /etc/libvirt/qemu.conf.
vnc_listen = "192.168.0.61"
user = "root"
group = "root"
dynamic_ownership = 0
nvram = [
"/usr/share/edk2-ovmf/OVMF_CODE.fd:/usr/share/edk2-ovmf/OVMF_VARS.fd",
]
For some reason, emails directly from mailing list are not hitting my inbox
so I managed to miss this:
https://<email address hidden>/msg600959.html
Please ensure it's sent to my email when answering.
Thanks!
José.
seems to be piece of true on Phil's comments, as extracting the command on the log and executing it manually the result is the following:
2019-04-18T19:37:49.685841Z qemu-system-x86_64: -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-os-1/master-key.aes: Unable to read /var/lib/libvirt/qemu/domain-6-os-1/master-key.aes: Failed to open file “/var/lib/libvirt/qemu/domain-6-os-1/master-key.aes”: No such file or directory
unless those folders with master keys are dinamically genreated and trashed after execution it would be worthy to involve here libvirt project as well.
José.
Hi José,
your domain XML is bogus with regard to the firmware configuration. You
have:
<loader type='pflash'>/var/lib/libvirt/qemu/nvram/os-1-ovmf.fd</loader>
and no <nvram> element, and you write that "os-1-ovmf.fd" is a copy of
"OVMF_VARS.fd".
The <loader> element, with @type='pflash', no other attributes, and then
no <nvram> sibling element either, happens to be valid, but it is an
extremely niche use case. It is used when you have a unified, writeable,
OVMF.fd file that contains both the firmware executable and the live
variable store.
Most distros don't provide such unified "OVMF.fd" files, and even if
yours does, you are not passing a full such "OVMF.fd" to the guest, but
just a variable store -- there is no firmware binary at all that the
guest could *execute*.
So please do the following:
- power down the domain
- delete "/var/lib/libvirt/qemu/nvram/os-1-ovmf.fd" for starters
- modify the domain XML as follows, with "virsh edit":
<loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>
When you exit your editor, libvirtd will:
- create the <nvram> element, and auto-generate the pathname for the
<nvram> element
- create the varstore file, pointed-to by the pathname above, from the
template at "/usr/share/edk2-ovmf/OVMF_VARS.fd".
(more precisely, the nvram element will be auto-generated when you exit "virsh edit", and the nvram file will be created (copied) from the varstore template when you launch the domain)
thanks for the thorough pointers Laszlo, I did and run in different kind
of issues, and given that I was just having the vars around I really wonder
how it ever worked, perhaps I copied the unified and I forgot, as gentoo
thas provide it:
# du -h /usr/share/edk2-ovmf/*
1.9M /usr/share/edk2-ovmf/OVMF_CODE.fd
2.0M /usr/share/edk2-ovmf/OVMF.fd
128K /usr/share/edk2-ovmf/OVMF_VARS.fd
Thanks!
José
oh well, it turns out to be actually related, it turns out
virt-aa-helper is skipping the code file of ovmf:
2019-04-19 10:06:43.576+0000: 9272: error : AppArmorGenSecurityLabel:469 : internal error: cannot load AppArmor profile 'libvirt-34c41008-ab91-483b-959c-81a7a12ae9be'
2019-04-19 10:09:25.751+0000: 9270: error : virCommandWait:2636 : internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/virt-aa-helper -c -u libvirt-34c41008-ab91-483b-959c-81a7a12ae9be) unexpected exit status 1: 2019-04-19 10:09:25.750+0000: 13797: info : libvirt version: 5.2.0
2019-04-19 10:09:25.750+0000: 13797: info : hostname: mole
2019-04-19 10:09:25.750+0000: 13797: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vda': Permission denied
2019-04-19 10:09:25.750+0000: 13797: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vdb': Permission denied
virt-aa-helper: error: /usr/share/edk2-ovmf/OVMF_CODE.fd
virt-aa-helper: error: skipped restricted file
virt-aa-helper: error: invalid VM definition
The domain now looks like:
<domain type='kvm'> [40/100]
<name>os-1</name>
<uuid>34c41008-ab91-483b-959c-81a7a12ae9be</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memoryBacking>
<hugepages/>
</memoryBacking>
<vcpu placement='static' cpuset='10-11,34-35'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/os-1_VARS.fd</nvram>
<boot dev='network'/>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-passthrough' check='partial'/>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/mole-vg0/os-1-vda'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/mole-vg0/os-1-vdb'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:78:cb:97'/>
<source bridge='virbr0'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:1c:4f:22'/>
<source bridge='virbr1'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'>
<listen type='address'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</memballoon>
</devices>
</domain>
this is apparmor profile related in the end, rebooting with apparmor disabled allows
the domain to boot, so I'll deal with the gentoo community, and this can be closed.
Thanks!
José.
|