1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
|
debug: 0.972
permissions: 0.969
register: 0.966
mistranslation: 0.950
semantic: 0.949
assembly: 0.947
graphic: 0.944
PID: 0.944
architecture: 0.940
user-level: 0.938
peripherals: 0.938
device: 0.937
boot: 0.934
performance: 0.934
kernel: 0.931
socket: 0.929
risc-v: 0.929
arm: 0.921
ppc: 0.918
virtual: 0.913
KVM: 0.908
hypervisor: 0.906
VMM: 0.899
vnc: 0.899
files: 0.898
network: 0.897
TCG: 0.894
i386: 0.839
x86: 0.816
qcow2 image logical corruption after host crash
Description of problem:
In case of power failure disk images that were active and created in qcow2 format can become logically corrupt so that they actually appear as unused (full of zeroes).
Data seems to be there, but at this moment i cannot find any reliable method to recover it. Should it be a raw image, a recovery path would be available, but a qcow2 image only presents zeroes once it gets corrupted. My understanding is that the blockmap of the image gets reset and the image is then assumed to be unused.
My detailed setup :
Kernel 2.6.32-358.18.1.el6.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64
Used via libvirt libvirt-0.10.2-18.el6_4.14.x86_64
The image was used from a NFS share (the nfs server did NOT crash and remained permanently active).
qemu-img check finds no corruption;
qemu-img convert will fully convert the image to raw at a raw image full of zeroes. However, there is data in the file, and the storage backend was not restarted, inactivated during the incident.
I encountered this issue on two different machines, in both cases i was not able to recover the data.
Image was qcow2, thin provisioned, created like this :
qemu-img create -f qcow2 -o cluster_size=2M imagename.img
While addressing the root cause in order to not have this issue repeat would be the ideal scenario, a temporary workaround to run on the affected qcow2 image to "patch" it and recover the data (eventually after a full fsck/recovery inside the guest) would also be good. Otherwise we are basically losing data on a large scale when using qcow2.
Version-Release number of selected component (if applicable):
Kernel 2.6.32-358.18.1.el6.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64
Used via libvirt libvirt-0.10.2-18.el6_4.14.x86_64
How reproducible:
I am not able (and don't have at the moment enough resources to try to manually reproduce it), but the probability of the issue seems quite high as this is the second case of such corruption in weeks.
Additional info:
I can privately provide an image displaying the corruption.
The reported problem has actually two aspects : first is the cause that eventually produces this issue.
The second is the fact that once the logical corruption has occured, qemu-img check finds nothing wrong with the image - this is obviously wrong.
On Tue, Nov 12, 2013 at 09:17:34AM -0000, Blue wrote:
Please post the qemu command-line (ps aux | grep qemu) for the affected
VM.
What kind of workload is accessing the disk? Guest OS and version?
Please also confirm that nothing else is accessing the image file while
the VM is running. It is not safe to use tools like qemu-img on the
file while the VM has it open.
> Image was qcow2, thin provisioned, created like this :
> qemu-img create -f qcow2 -o cluster_size=2M imagename.img
Interesting note, you are setting a custom cluster size. It should work
but it's not the default configuration that is well-tested.
> I can privately provide an image displaying the corruption.
Please send a link to <email address hidden> and <email address hidden>. We can
help you inspect the image to determine the nature of the corruption and
what data can be restored.
Stefan
See also https://bugzilla.redhat.com/show_bug.cgi?id=1029344 for some more information
Indeed, it's the same issue, i opened the report @ centos, redhat and here. Thank you Kevin for posting the link before I got to do it :)
Can we link the reports like we can in rhel-centos bugzilla ?
Did this issue happen ever again with a recent version of QEMU? ... if not, should we close this bug nowadays?
Since then, I switched all my vm images to (sparse) raw and never
experienced corruption problems again.
I could not say if this can still be reproduced today, even then it was
probably a corner case.
I would suggest the closing of the issue as we cannot gather newer and more
relevant data.
On Tue, Jan 31, 2017 at 10:49 PM, Thomas Huth <email address hidden>
wrote:
> Did this issue happen ever again with a recent version of QEMU? ... if
> not, should we close this bug nowadays?
>
> ** Changed in: qemu
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1250360
>
> Title:
> qcow2 image logical corruption after host crash
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1250360/+subscriptions
>
Ok, so let's close this now. Thanks for your reply!
|