debug: 0.703
other: 0.690
permissions: 0.676
graphic: 0.672
semantic: 0.647
performance: 0.637
files: 0.628
device: 0.623
PID: 0.620
boot: 0.615
KVM: 0.590
socket: 0.584
vnc: 0.518
network: 0.504

QEMU segfaults during Windows 7 unattended install

During today's automated qemu.git testing, a segmentation fault while installing Windows 7 SP1 happened.

qemu.git top commit: 
10/02 01:30:24 INFO |       git:0150| git commit ID is a684f3cf9b9b9c3cb82be87aafc463de8974610c (tag v1.4.0-4237-ga684f3c)

commit a684f3cf9b9b9c3cb82be87aafc463de8974610c
Merge: 349cd52 1cf9412
Author: Anthony Liguori <email address hidden>
Date:   Mon Sep 30 17:15:27 2013 -0500

    Merge remote-tracking branch 'kraxel/seabios-1.7.3.2' into staging
    
    # By Gerd Hoffmann
    # Via Gerd Hoffmann
    * kraxel/seabios-1.7.3.2:
      update seabios from 1.7.2.2 to 1.7.3.2
    
    Message-id: <email address hidden>

We have the core file saved in our test servers, we can make arrangements to transfer it if there's someone interested in investigating further. The framework saved the 'bt full' of the core file, that was missing some debug info:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/autotest/tests/virt/qemu/qemu -S -name virt-tests-vm1 -M pc -nodefau'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007ffc8fb86cf0 in pixman_image_get_data () from /lib64/libpixman-1.so.0
#0  0x00007ffc8fb86cf0 in pixman_image_get_data () from /lib64/libpixman-1.so.0
No symbol table info available.
#1  0x00007ffc9165b05c in ?? ()
No symbol table info available.
#2  0x00007ffc9382b540 in ?? ()
No symbol table info available.
#3  0x00007ffc8f359a8d in clock_gettime () from /lib64/libc.so.6
No symbol table info available.
#4  0x00007ffc9382b5a8 in ?? ()
No symbol table info available.
#5  0x000000019382b4c0 in ?? ()
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Extra info:

Commits for the submodules:

10/02 01:30:29 DEBUG|base_utils:0134| [stdout] Submodule path 'dtc': checked out 'bc895d6d09695d05ceb8b52486ffe861d6cfbdde'
10/02 01:30:51 DEBUG|base_utils:0134| [stdout] Submodule path 'pixman': checked out '97336fad32acf802003855cd8bd6477fa49a12e3'
10/02 01:30:58 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/SLOF': checked out '8cfdfc43f4c4c8c8dfa4b7cf16f7c19c84eee812'
10/02 01:31:16 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/ipxe': checked out '09c5109b8585178172c7608de8d52e9d9af0b680'
10/02 01:31:20 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/openbios': checked out '0f3d51ef22ec9166beb3ed434d253029ed7cfe84'
10/02 01:31:21 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/qemu-palcode': checked out 'c87a92639b28ac42bc8f6c67443543b405dc479b'
10/02 01:31:27 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/seabios': checked out 'ece025f5980bae88fa677bc9c0d24d2e580e205d'
10/02 01:31:28 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/sgabios': checked out '23d474943dcd55d0550a3d20b3d30e9040a4f15b'
10/02 01:31:31 DEBUG|base_utils:0134| [stdout] Submodule path 'roms/vgabios': checked out '19ea12c230ded95928ecaef0db47a82231c2e485'

Configure options:

10/02 01:31:32 DEBUG|base_utils:0099| Running '/usr/local/autotest/tmp/virt/src/qemu/configure --target-list=x86_64-softmmu --disable-strip --prefix=/usr/local/autotest/tests/virt/qemu/install_root'
10/02 01:31:35 DEBUG|env_proces:0829| (address cache) DHCP lease OK: 00:30:48:c5:d6:e2 --> 10.16.72.38
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Install prefix    /usr/local/autotest/tests/virt/qemu/install_root
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] BIOS directory    /usr/local/autotest/tests/virt/qemu/install_root/share/qemu
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] binary directory  /usr/local/autotest/tests/virt/qemu/install_root/bin
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] library directory /usr/local/autotest/tests/virt/qemu/install_root/lib
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] libexec directory /usr/local/autotest/tests/virt/qemu/install_root/libexec
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] include directory /usr/local/autotest/tests/virt/qemu/install_root/include
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] config directory  /usr/local/autotest/tests/virt/qemu/install_root/etc
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] local state directory   /usr/local/autotest/tests/virt/qemu/install_root/var
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Manual directory  /usr/local/autotest/tests/virt/qemu/install_root/share/man
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] ELF interp prefix /usr/gnemul/qemu-%M
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Source path       /usr/local/autotest/tmp/virt/src/qemu
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] C compiler        cc
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Host C compiler   cc
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] C++ compiler      c++
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Objective-C compiler cc
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] QEMU_CFLAGS       -Werror -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all  -I/usr/include/pixman-1   -I$(SRC_PATH)/dtc/libfdt
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g 
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] make              make
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] install           install
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] python            python -B
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] smbd              /usr/sbin/smbd
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] host CPU          x86_64
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] host big endian   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] target list       x86_64-softmmu
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] tcg debug enabled no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] gprof enabled     no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] sparse enabled    no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] strip binaries    no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] profiler          no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] static build      no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] -Werror enabled   yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] pixman            system
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] SDL support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] GTK support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] curses support    yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] curl support      yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] mingw32 support   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Audio drivers     oss
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Block whitelist (rw) 
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Block whitelist (ro) 
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VirtFS support    yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VNC support       yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VNC TLS support   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VNC SASL support  yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VNC JPEG support  no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VNC PNG support   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] VNC WS support    no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] xen support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] brlapi support    no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] bluez  support    no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Documentation     no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] GUEST_BASE        yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] PIE               yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] vde support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Linux AIO support no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] ATTR/XATTR support yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Install blobs     yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] KVM support       yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] RDMA support      no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] TCG interpreter   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] fdt support       yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] preadv support    yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] fdatasync         yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] madvise           yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] posix_madvise     yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] sigev_thread_id   yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] uuid support      no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] libcap-ng support no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] vhost-net support yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] vhost-scsi support yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Trace backend     nop
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] Trace output file trace-<pid>
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] spice support     no (/)
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] rbd support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] xfsctl support    no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] nss used          no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] libusb            no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] usb net redir     no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] GLX support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] libiscsi support  no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] build guest agent yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] QGA VSS support   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] seccomp support   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] coroutine backend ucontext
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] coroutine pool    yes
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] GlusterFS support no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] virtio-blk-data-plane no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] gcov              gcov
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] gcov enabled      no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] TPM support       no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] libssh2 support   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] TPM passthrough   no
10/02 01:31:40 DEBUG|base_utils:0134| [stdout] QOM debugging     yes
10/02 01:31:40 INFO |build_help:0617| Running parallel make on build dir
10/02 01:31:40 DEBUG|base_utils:0099| Running 'make -j 24'

That's a seabios update.  It is interesting that qemu may crash due to different bios - this smells fishy, and it looks like there's some big security issue waiting to be discovered... ;)

Lucas, I think you want to change --disable-strip into --enable-debug in your configure line, to be able to produce more useful gdb stack traces.

Good point, I've just changed the configure line to include --enable-debug.

About the relation of the crash with the top commit, We can't ensure it was because of this top commit, could be other patches that were applied from one day to another. We only test qemu.git once a day, we don't have enough resources to test commit per commit.

Also, this crash apparently is not 100% reproducible. Today's jobs did not have it, for instance. I guess we don't have enough information about the crash, given that I did not enable debug symbols.

I'm fine with closing this issue, if I see it again, I can reopen it and hopefully this time we'll have a more useful bt full report.

The problem showed up this morning again,  same top commit:

10/07 01:34:42 INFO |       git:0150| git commit ID is a684f3cf9b9b9c3cb82be87aafc463de8974610c (tag v1.4.0-4237-ga684f3c)

This time around, debug symbols were enabled on the configure line:

10/07 01:35:31 DEBUG|build_help:0588| Enabling debug symbols with option: --disable-strip
10/07 01:35:31 INFO |build_help:0607| Running configure on build dir
10/07 01:35:31 DEBUG|base_utils:0099| Running '/usr/local/autotest/tmp/virt/src/qemu/configure --target-list=x86_64-softmmu --enable-debug --disable-strip --prefix=/usr/local/autotest/tests/virt/qemu/install_root'

But no additional info on bt full:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/autotest/tests/virt/qemu/qemu -S -name virt-tests-vm1 -M pc -nodefau'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f11f0f2fcf0 in pixman_image_get_data () from /lib64/libpixman-1.so.0
#0  0x00007f11f0f2fcf0 in pixman_image_get_data () from /lib64/libpixman-1.so.0
No symbol table info available.
#1  0x00007f11f2ac1be0 in ?? ()
No symbol table info available.
#2  0x0000000000000000 in ?? ()
No symbol table info available.

I guess I need the debugging symbols for all involved libraries...

Triaging old bug tickets ... can you somehow still reproduce this problem with the latest version of QEMU (currently v2.9), or could we close this ticket nowadays?

[Expired for QEMU because there has been no activity for 60 days.]