crash at  qemu_iohandler_poll (iohandler.c:124) on macos 10.8.2

I'm seeing consistent hangs / crashes on MacOS 10.8.2 with 1.3.0.  I've tried both gcc-4.2 and clang.  I've tried a half a dozen different images/kernels.

I configured qemu like this:

./configure --disable-sdl --disable-kvm --enable-cocoa --cc=gcc-4.2 --host-cc=gcc-4.2 --enable-debug   --extra-cflags=-g   --extra-ldflags=-g

And ran it like this:

qemu-system-arm -nographic -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1 console=ttyAMA0"

With images, kernel, and initrd described here:

http://psellos.com/2012/08/2012.08.qemu-arm-osx.html

And I get:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x000000010142f2d0
0x000000010142f2d0 in ?? ()

(gdb) bt
#0  0x000000010142f2d0 in ?? ()
#1  0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
#2  0x0000000100172acf in main_loop_wait (nonblocking=0) at main-loop.c:418
#3  0x0000000100207bbf in main_loop () at vl.c:1765
#4  0x000000010020e7b0 in qemu_main (argc=12, argv=0x7fff5fbff360, envp=0x7fff5fbff3c8) at vl.c:3992
#5  0x00000001001d6013 in main (argc=12, argv=0x7fff5fbff360) at ui/cocoa.m:884
(gdb) frame 1
#1  0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
124	                ioh->fd_read(ioh->opaque);
Current language:  auto; currently c
(gdb) p ioh
$1 = (IOHandlerRecord *) 0x10142f110
(gdb) p *ioh
$2 = {
  fd_read_poll = 0, 
  fd_read = 0x10017212b <sigfd_handler>, 
  fd_write = 0, 
  opaque = 0x3, 
  next = {
    le_next = 0x0, 
    le_prev = 0x105d00bc0
  }, 
  fd = 3, 
  deleted = false
}

On Mon, Dec 31, 2012 at 08:46:45PM -0000, Christopher Mason wrote:
> Public bug reported:
> 
> I'm seeing consistent hangs / crashes on MacOS 10.8.2 with 1.3.0.  I've
> tried both gcc-4.2 and clang.  I've tried a half a dozen different
> images/kernels.

Which QEMU version are you building?  Have you tried qemu.git/master?

> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x000000010142f2d0
> 0x000000010142f2d0 in ?? ()
> 
> (gdb) bt
> #0  0x000000010142f2d0 in ?? ()
> #1  0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
> #2  0x0000000100172acf in main_loop_wait (nonblocking=0) at main-loop.c:418
> #3  0x0000000100207bbf in main_loop () at vl.c:1765
> #4  0x000000010020e7b0 in qemu_main (argc=12, argv=0x7fff5fbff360, envp=0x7fff5fbff3c8) at vl.c:3992
> #5  0x00000001001d6013 in main (argc=12, argv=0x7fff5fbff360) at ui/cocoa.m:884
> (gdb) frame 1
> #1  0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
> 124	                ioh->fd_read(ioh->opaque);
> Current language:  auto; currently c
> (gdb) p ioh
> $1 = (IOHandlerRecord *) 0x10142f110
> (gdb) p *ioh
> $2 = {
>   fd_read_poll = 0, 
>   fd_read = 0x10017212b <sigfd_handler>, 

The fd_read() function pointer should be called here.  But somehow we
end up with 0x000000010142f2d0, which is awefully close to the
IOHandlerRecord (0x10142f110).

Perhaps printing out the entire io_handlers list would be interesting
too.

Does this happen at an unspecified point or is it always when the
fd_read sigfd_handler() callback is invoked?  (You could put a
breakpoint on sigfd_handler() and continue the first time it is hit to
check this.)

Stefan


Using qemu master rev dbd99ae..25bbf61 configured with:

./configure --disable-sdl --disable-kvm --enable-cocoa  --enable-debug --extra-cflags=-g --extra-ldflags=-g

(I'm using clang 4.1 now.  Should I be using clang or gcc 4.2? Are these the right config args?)

(gdb) b sigfd_handler
Breakpoint 1 at 0x1001c098d: file main-loop.c, line 41.

(gdb) r -nographic -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1 console=ttyAMA0"
...
Breakpoint 1, sigfd_handler (opaque=0x3) at main-loop.c:41
41	    int fd = (intptr_t)opaque;
(gdb) bt
#0  sigfd_handler (opaque=0x3) at main-loop.c:41
#1  0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
#2  0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
#3  0x000000010027bde4 in main_loop () at vl.c:1765
#4  0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
#5  0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884
Current language:  auto; currently minimal
(gdb) p io_handlers
$1 = {
  lh_first = 0x102102ab0
}
(gdb) p * io_handlers.lh_first
$2 = {
  fd_read_poll = 0x1001fad60 <stdio_read_poll>, 
  fd_read = 0x1001fae20 <stdio_read>, 
  fd_write = 0, 
  opaque = 0x1021029c0, 
  next = {
    le_next = 0x102100000, 
    le_prev = 0x100a09368
  }, 
  fd = 0, 
  deleted = false
}
(gdb) p * io_handlers.lh_first->next.le_prev
$3 = (struct IOHandlerRecord *) 0x102102ab0
(gdb) p * io_handlers.lh_first->next.le_next
$4 = {
  fd_read_poll = 0, 
  fd_read = 0x1001c0970 <sigfd_handler>, 
  fd_write = 0, 
  opaque = 0x3, 
  next = {
    le_next = 0x0, 
    le_prev = 0x102102ad0
  }, 
  fd = 3, 
  deleted = false
}

(gdb) c

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x0000000102100040
0x0000000102100040 in ?? ()
(gdb) bt
#0  0x0000000102100040 in ?? ()
#1  0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
#2  0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
#3  0x000000010027bde4 in main_loop () at vl.c:1765
#4  0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
#5  0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884

(gdb) p io_handlers
$5 = {
  lh_first = 0x102102ab0
}
(gdb) p * io_handlers.lh_first
$6 = {
  fd_read_poll = 0x1001fad60 <stdio_read_poll>, 
  fd_read = 0x1001fae20 <stdio_read>, 
  fd_write = 0, 
  opaque = 0x1021029c0, 
  next = {
    le_next = 0x102100000, 
    le_prev = 0x100a09368
  }, 
  fd = 0, 
  deleted = false
}
(gdb) p * io_handlers.lh_first->next.le_next
$8 = {
  fd_read_poll = 0, 
  fd_read = 0x1001c0970 <sigfd_handler>, 
  fd_write = 0, 
  opaque = 0x3, 
  next = {
    le_next = 0x0, 
    le_prev = 0x102102ad0
  }, 
  fd = 3, 
  deleted = false
}
(gdb) p * io_handlers.lh_first->next.le_prev
$9 = (struct IOHandlerRecord *) 0x102102ab0


On Fri, Jan 04, 2013 at 06:09:30PM -0000, Christopher Mason wrote:
> Using qemu master rev dbd99ae..25bbf61 configured with:
> 
> ./configure --disable-sdl --disable-kvm --enable-cocoa  --enable-debug
> --extra-cflags=-g --extra-ldflags=-g
> 
> (I'm using clang 4.1 now.  Should I be using clang or gcc 4.2? Are these
> the right config args?)

I have never used QEMU on Mac myself, sorry.  Maybe someone else can
help.

> (gdb) b sigfd_handler
> Breakpoint 1 at 0x1001c098d: file main-loop.c, line 41.
> 
> (gdb) r -nographic -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1 console=ttyAMA0"
> ...
> Breakpoint 1, sigfd_handler (opaque=0x3) at main-loop.c:41
> 41	    int fd = (intptr_t)opaque;
> (gdb) bt
> #0  sigfd_handler (opaque=0x3) at main-loop.c:41
> #1  0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
> #2  0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
> #3  0x000000010027bde4 in main_loop () at vl.c:1765
> #4  0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
> #5  0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884
> Current language:  auto; currently minimal
> (gdb) p io_handlers
> $1 = {
>   lh_first = 0x102102ab0
> }
> (gdb) p * io_handlers.lh_first
> $2 = {
>   fd_read_poll = 0x1001fad60 <stdio_read_poll>, 
>   fd_read = 0x1001fae20 <stdio_read>, 
>   fd_write = 0, 
>   opaque = 0x1021029c0, 
>   next = {
>     le_next = 0x102100000, 
>     le_prev = 0x100a09368
>   }, 
>   fd = 0, 
>   deleted = false
> }
> (gdb) p * io_handlers.lh_first->next.le_prev
> $3 = (struct IOHandlerRecord *) 0x102102ab0
> (gdb) p * io_handlers.lh_first->next.le_next
> $4 = {
>   fd_read_poll = 0, 
>   fd_read = 0x1001c0970 <sigfd_handler>, 
>   fd_write = 0, 
>   opaque = 0x3, 
>   next = {
>     le_next = 0x0, 
>     le_prev = 0x102102ad0
>   }, 
>   fd = 3, 
>   deleted = false
> }
> 
> (gdb) c
> 
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x0000000102100040
> 0x0000000102100040 in ?? ()
> (gdb) bt
> #0  0x0000000102100040 in ?? ()
> #1  0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
> #2  0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
> #3  0x000000010027bde4 in main_loop () at vl.c:1765
> #4  0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
> #5  0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884
> 
> (gdb) p io_handlers
> $5 = {
>   lh_first = 0x102102ab0
> }
> (gdb) p * io_handlers.lh_first
> $6 = {
>   fd_read_poll = 0x1001fad60 <stdio_read_poll>, 
>   fd_read = 0x1001fae20 <stdio_read>, 
>   fd_write = 0, 
>   opaque = 0x1021029c0, 
>   next = {
>     le_next = 0x102100000, 
>     le_prev = 0x100a09368
>   }, 
>   fd = 0, 
>   deleted = false
> }
> (gdb) p * io_handlers.lh_first->next.le_next
> $8 = {
>   fd_read_poll = 0, 
>   fd_read = 0x1001c0970 <sigfd_handler>, 
>   fd_write = 0, 
>   opaque = 0x3, 
>   next = {
>     le_next = 0x0, 
>     le_prev = 0x102102ad0
>   }, 
>   fd = 3, 
>   deleted = false
> }
> (gdb) p * io_handlers.lh_first->next.le_prev
> $9 = (struct IOHandlerRecord *) 0x102102ab0

This is interesting.  The iohandlers are intact - there was no
memory corruption there.  The fact that it crashes after executing
sigfd_handler() once is suspicious.

My next suggestion is to break on iohandler.c:124 and find out why
0x0000000102100040 is getting called.  Really it should be
sigfd_handler() that gets called again.  This may require a few tries
and probably familiarity with assembly to debug.

I have pinged other QEMU contributors who have Macs.  Perhaps they can
help better from here.

Stefan


Just a note that IME trying to debug QEMU under gdb on MacOS doesn't work very well. In particular as far as I can tell gdb breaks sigwait() such that the sigwait() in sigwait_compat() can return 0 without setting the int* sig. This causes QEMU to write an uninitialized value into the qemu_signalfd_siginfo struct it sends down the pipe, and then sigfd_handler() calls sigaction() with this bogus data as the signal number. Since sigfd_handler() doesn't check the return value from sigaction() we then proceed to leap off into nowhere. 

sigfd_handler() should probably be checking the return value from sigaction() but the underlying problem is MacOS and/or its gdb breaking sigwait() behaviour somehow.


Can you still reproduce this problem wit the latest release of QEMU (currently version 2.9.0) and macOS, or could we close this bug nowadays?

[Expired for QEMU because there has been no activity for 60 days.]