1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
|
permissions: 0.977
performance: 0.972
graphic: 0.971
architecture: 0.971
assembly: 0.969
arm: 0.968
semantic: 0.968
debug: 0.966
boot: 0.964
virtual: 0.961
PID: 0.960
register: 0.959
kernel: 0.958
files: 0.955
socket: 0.955
risc-v: 0.955
network: 0.954
device: 0.954
user-level: 0.952
hypervisor: 0.950
mistranslation: 0.944
vnc: 0.937
x86: 0.935
TCG: 0.933
i386: 0.931
ppc: 0.925
KVM: 0.925
peripherals: 0.925
VMM: 0.903
hang in qemu_gluster_init
In qemu_gluster_init, if the call to either glfs_set_volfile_server or glfs_set_logging fails into the "out" case, glfs_fini is called without having first calling glfs_init. This causes glfs_lock to spin forever on this bit:
while (!fs->init)
pthread_cond_wait (&fs->cond, &fs->mutex);
And here's the bottom part of the backtrace when hung:
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1 0x00007feceebf58c3 in glfs_lock (fs=0x7fecf15660b0) at glfs-internal.h:156
#2 glfs_active_subvol (fs=0x7fecf15660b0) at glfs-resolve.c:799
#3 0x00007feceebeb5b4 in glfs_fini (fs=0x7fecf15660b0) at glfs.c:652
#4 0x00007fecf0043c73 in qemu_gluster_init (gconf=<value optimized out>, filename=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block/gluster.c:229
I believe this can be fixed by simply moving the call to glfs_init after the call to glfs_new but before the calls to glfs_set_volfile_server or glfs_set_logging.
Am 16.04.2014 um 15:25 hat John Eckersberg geschrieben:
> Public bug reported:
>
> In qemu_gluster_init, if the call to either glfs_set_volfile_server or
> glfs_set_logging fails into the "out" case, glfs_fini is called without
> having first calling glfs_init. This causes glfs_lock to spin forever
> on this bit:
>
> while (!fs->init)
> pthread_cond_wait (&fs->cond, &fs->mutex);
>
> And here's the bottom part of the backtrace when hung:
>
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
> #1 0x00007feceebf58c3 in glfs_lock (fs=0x7fecf15660b0) at glfs-internal.h:156
> #2 glfs_active_subvol (fs=0x7fecf15660b0) at glfs-resolve.c:799
> #3 0x00007feceebeb5b4 in glfs_fini (fs=0x7fecf15660b0) at glfs.c:652
> #4 0x00007fecf0043c73 in qemu_gluster_init (gconf=<value optimized out>, filename=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block/gluster.c:229
>
> I believe this can be fixed by simply moving the call to glfs_init after
> the call to glfs_new but before the calls to glfs_set_volfile_server or
> glfs_set_logging.
Bharata, can you check whether this is the correct solution, and if so,
send out a patch?
Kevin
On Thu, Apr 17, 2014 at 11:27:52AM +0200, Kevin Wolf wrote:
> Am 16.04.2014 um 15:25 hat John Eckersberg geschrieben:
> > Public bug reported:
> >
> > In qemu_gluster_init, if the call to either glfs_set_volfile_server or
> > glfs_set_logging fails into the "out" case, glfs_fini is called without
> > having first calling glfs_init. This causes glfs_lock to spin forever
> > on this bit:
> >
> > while (!fs->init)
> > pthread_cond_wait (&fs->cond, &fs->mutex);
> >
> > And here's the bottom part of the backtrace when hung:
> >
> > #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
> > #1 0x00007feceebf58c3 in glfs_lock (fs=0x7fecf15660b0) at glfs-internal.h:156
> > #2 glfs_active_subvol (fs=0x7fecf15660b0) at glfs-resolve.c:799
> > #3 0x00007feceebeb5b4 in glfs_fini (fs=0x7fecf15660b0) at glfs.c:652
> > #4 0x00007fecf0043c73 in qemu_gluster_init (gconf=<value optimized out>, filename=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block/gluster.c:229
> >
> > I believe this can be fixed by simply moving the call to glfs_init after
> > the call to glfs_new but before the calls to glfs_set_volfile_server or
> > glfs_set_logging.
>
> Bharata, can you check whether this is the correct solution, and if so,
> send out a patch?
I think glfs_set_volfile_server should have been called before glfs_init.
Anyway will check gluster folks and send out an appropriate patch for this.
BTW we haven't hit this problem till now because glfs_fini was nop when
we wrote this driver but has been functionally implemented now.
Regards,
Bharata.
" glfs_init" cannot be called before since it checks for cmds_args->volfile_server which is allocated only in "glfs_set_volfile_server". We should either modify "glfs_fini" or define a new function to do the cleanup based on if init is done or not.
On Fri, Apr 18, 2014 at 02:38:57PM -0000, Soumya Koduri wrote:
> " glfs_init" cannot be called before since it checks for
> cmds_args->volfile_server which is allocated only in
> "glfs_set_volfile_server". We should either modify "glfs_fini" or
Soumya - Thanks for offering to fix this in gluster.
(Ref: http://lists.gnu.org/archive/html/gluster-devel/2014-04/msg00192.html)
Kevin - I will wait for this fix to be available in libgfapi before
changing QEMU.
Regards,
Bharata.
Also I should add a note to this related bug I filed against gluster - https://bugzilla.redhat.com/show_bug.cgi?id=1088589
This bug is what causes the glfs_set_logging call to fail, and fall into the hang case.
A complete fix has been included in the glusterfs master-branch. It has not (yet) been requested or marked for backporting to a stable (3.5.x) branch.
* https://bugzilla.redhat.com/1091335 with http://review.gluster.org/7857
The issue with glfs_set_logging is fixed in the almost released glusterfs-3.5.1 (https://bugzilla.redhat.com/1103413).
Verified that this fixes the hang and no change is required in gluster
driver of QEMU after this fix in glusterfs code.
On Mon, Jun 23, 2014 at 11:59 PM, nixpanic <email address hidden> wrote:
> A complete fix has been included in the glusterfs master-branch. It has
> not (yet) been requested or marked for backporting to a stable (3.5.x)
> branch.
>
> * https://bugzilla.redhat.com/1091335 with
> http://review.gluster.org/7857
>
> The issue with glfs_set_logging is fixed in the almost released
> glusterfs-3.5.1 (https://bugzilla.redhat.com/1103413).
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1308542
>
> Title:
> hang in qemu_gluster_init
>
> Status in QEMU:
> New
>
> Bug description:
> In qemu_gluster_init, if the call to either glfs_set_volfile_server or
> glfs_set_logging fails into the "out" case, glfs_fini is called
> without having first calling glfs_init. This causes glfs_lock to spin
> forever on this bit:
>
> while (!fs->init)
> pthread_cond_wait (&fs->cond, &fs->mutex);
>
> And here's the bottom part of the backtrace when hung:
>
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
> #1 0x00007feceebf58c3 in glfs_lock (fs=0x7fecf15660b0) at
> glfs-internal.h:156
> #2 glfs_active_subvol (fs=0x7fecf15660b0) at glfs-resolve.c:799
> #3 0x00007feceebeb5b4 in glfs_fini (fs=0x7fecf15660b0) at glfs.c:652
> #4 0x00007fecf0043c73 in qemu_gluster_init (gconf=<value optimized
> out>, filename=<value optimized out>) at
> /usr/src/debug/qemu-kvm-0.12.1.2/block/gluster.c:229
>
> I believe this can be fixed by simply moving the call to glfs_init
> after the call to glfs_new but before the calls to
> glfs_set_volfile_server or glfs_set_logging.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1308542/+subscriptions
>
>
--
http://raobharata.wordpress.com/
Closing according to comment #7.
|