1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
|
peripherals: 0.923
virtual: 0.894
hypervisor: 0.879
ppc: 0.878
KVM: 0.877
register: 0.875
architecture: 0.874
x86: 0.873
files: 0.870
graphic: 0.864
device: 0.860
assembly: 0.859
TCG: 0.859
vnc: 0.858
PID: 0.858
arm: 0.858
boot: 0.856
kernel: 0.855
performance: 0.854
semantic: 0.848
user-level: 0.847
network: 0.841
permissions: 0.841
socket: 0.840
debug: 0.838
risc-v: 0.834
i386: 0.833
VMM: 0.818
mistranslation: 0.804
Domain fails to start when 'readonly' device not writable
This issue is introduced in QEMU 4.2.0 (4.1.0 is working fine)
My root disk is a LVM2 volume thin snapshot that is marked as read-only
But when I try to start the domain (using virt-manager) I get the following error:
Error starting domain: internal error: process exited while connecting to monitor: 2020-04-26T06:55:06.342700Z qemu-system-x86_64: -blockdev {"driver":"host_device","filename":"/dev/vg/vmroot-20200425","aio":"native","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} The device is not writable: Permission denied
Changing the lvm snapshot to writeable allows me to start the domain.
(Making it changes possible during domain is running)
I don't think QEMU should fail when it can't open a (block) device when the read-only option is set.
(why is write access needed?)
Reproduce steps:
* Create LVM read-only volume (I don't think any data is needed)
* Create domain with read-only volume as block device
* Try to start the domain
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
callback(*args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
ret = fn(self, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
self._backend.create()
File "/usr/lib/python3.7/site-packages/libvirt.py", line 1152, in create
if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2020-04-26T07:29:47.463835Z qemu-system-x86_64: -blockdev {"driver":"host_device","filename":"/var/lib/libvirt/vmportage/rootdisk","aio":"native","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: The device is not writable: Permission denied
Can you provide the full guest XML for this - "/etc/libvirt/qemu/$GUESTNAME.xml", and also the full QEMU command line - "/var/log/libvirt/qemu/$GUESTNAME.xml". We need to see whether the disk is considered read-only from libvirt's POV.
> This issue is introduced in QEMU 4.2.0 (4.1.0 is working fine)
That's not neccessarily the case - with QEMU 4.2.0, libvirt switched over to using the new -blockdev command line syntax. When you were testing with 4.1.0, it would have been using the legacy -drive syntax. So the change in behaviour is more likely related to the usage of -blockdev, than any bug introduced in QEMU.
The domain.xml:
<domain type='kvm'>
<name>rotest</name>
<uuid>b4aa0288-8886-42df-abfd-4c8f729e1330</uuid>
<memory unit='KiB'>2048000</memory>
<currentMemory unit='KiB'>2048000</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
<kernel>/var/lib/libvirt/pink/kernel</kernel>
<cmdline>root=/dev/sda ro panic=300 systemd.show_status=1 systemd.unit=graphical.target quiet</cmdline>
<boot dev='hd'/>
</os>
<features>
<acpi/>
</features>
<cpu mode='custom' match='exact' check='none'>
<model fallback='forbid'>qemu64</model>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw'/>
<source dev='/dev/nvmvg/rotest'/>
<target dev='sda' bus='scsi'/>
<readonly/>
<shareable/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/nvmvg/rotest-var'/>
<target dev='sdb' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='pci' index='0' model='pci-root'/>
<controller type='scsi' index='0' model='virtio-scsi'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='usb' index='0' model='piix3-uhci'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
<gl enable='no' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
</graphics>
<video>
<model type='virtio' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>
--------------------------------------------------------------------------------
The qemu command:
2020-04-27 11:57:11.720+0000: starting up libvirt version: 6.0.0, qemu version: 4.2.0, kernel: 5.4.28-gentoo, hostname: gentoo
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-10-rotest \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-10-rotest/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-10-rotest/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-10-rotest/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=rotest,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10-rotest/master-key.aes \
-machine pc-i440fx-2.7,accel=kvm,usb=off,dump-guest-core=off \
-cpu qemu64 \
-m 2000 \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid b4aa0288-8886-42df-abfd-4c8f729e1330 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-kernel /var/lib/libvirt/pink/kernel \
-append 'root=/dev/sda ro panic=300 systemd.show_status=1 systemd.unit=graphical.target quiet' \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 \
-blockdev '{"driver":"host_device","filename":"/dev/nvmvg/rotest","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,share-rw=on,drive=libvirt-2-format,id=scsi0-0-0-0,bootindex=1 \
-blockdev '{"driver":"host_device","filename":"/dev/nvmvg/rotest-var","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=drive-scsi0-0-0-1,drive=libvirt-1-format,id=scsi0-0-0-1,write-cache=on \
-spice port=5900,addr=192.168.1.9,disable-ticketing,seamless-migration=on \
-device virtio-vga,id=video0,max_outputs=1,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-04-27T11:57:11.804028Z qemu-system-x86_64: -blockdev {"driver":"host_device","filename":"/dev/nvmvg/rotest","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}: The device is not writable: Permission denied
2020-04-27 11:57:11.805+0000: shutting down, reason=failed
This was indeed caused by libvirt starting to use -blockdev. The issue is that qemu's 'auto-read-only' property which is used by libvirt for the backing files doesn't properly work if the 'host_device' backend encounters a read-only LV.
The above situation also happens if you have an read-only LV in the backing chain.
In the current upstream situation of qemu's APIs there currently isn't anything that libvirt can do in this case, because any solution would either not fix the problem completely or would require sacrificing other features, the auto-read-only property needs to be fixed in qemu.
I've also filed https://bugzilla.redhat.com/show_bug.cgi?id=1828252 to track this issue.
I saw that that the related issue was implemented in 5.1.0.
So after I updated my QEMU to version 5.1.0. My VM(s) with a LVM read-only volume started again.
Thanks for getting this issue solved.
If I've got that right, this issue got solved, so I'm closing it now. Please file a new ticket in our new tracker at gitlab.com if there is still a problem.
|