kvm_irqchip_commit_routes: Assertion `ret == 0' failed Several my QEMU instances crashed, and in the qemu log, I can see this assertion failure, qemu-system-x86_64: /build/buildd/qemu-2.0.0+dfsg/kvm-all.c:984: kvm_irqchip_commit_routes: Assertion `ret == 0' failed. The QEMU version is 2.0.0, HV OS is ubuntu 12.04, kernel 3.2.0-38. Guest OS is RHEL 6.3. The problem can be re-produced by the script in the below in link. http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg03739.html i.e. vda_irq_num=25 vdb_irq_num=27 while [ 1 ] do for irq in {1,2,4,8,10,20,40,80} do echo $irq > /proc/irq/$vda_irq_num/smp_affinity echo $irq > /proc/irq/$vdb_irq_num/smp_affinity dd if=/dev/vda of=/dev/zero bs=4K count=100 iflag=direct dd if=/dev/vdb of=/dev/zero bs=4K count=100 iflag=direct done done http://lists.nongnu.org/archive/html/qemu-devel/2014-12/msg03739.html Seems that this patch hasn't been accpeted yet, and also no comments for it. From the debug log, we can see that virq is only 1008, but irq route table has been full, i.e. 1024. In kvm_irqchip_get_virq(), it only calls kvm_flush_dynamic_msi_routes() when all virqs(total gsi_count, 1024 too) have been allocated, but irq route table has two kind of entry type, KVM_IRQ_ROUTING_IRQCHIP and KVM_IRQ_ROUTING_MSI. Seems that 16 KVM_IRQ_ROUTING_IRQCHIP entries has been reserved, if max gsi_count is still 1024, then irq route table is possible to be overflow. The fix could be either set gsi_cout=1008 or increase max irq route count to 1040. kvm_irqchip_send_msi, virq=1008, nr=1024 kvm_irqchip_commit_routes, ret=-22 kvm_irqchip_commit_routes, irq_routes nr=1024 From kvm_pc_setup_irq_routing() function, we can see that 15 routes from PIC and 23 routes from IOAPIC are added into irq route table, but only 23 irq(gsi) are reserved. This leads to irq route table has been full but there are still tens of free gsi. So the "retry" part of kvm_irqchip_get_virq() shall never have chance to be executed. void kvm_pc_setup_irq_routing(bool pci_enabled) { KVMState *s = kvm_state; int i; if (kvm_check_extension(s, KVM_CAP_IRQ_ROUTING)) { for (i = 0; i < 8; ++i) { if (i == 2) { continue; } kvm_irqchip_add_irq_route(s, i, KVM_IRQCHIP_PIC_MASTER, i); } for (i = 8; i < 16; ++i) { kvm_irqchip_add_irq_route(s, i, KVM_IRQCHIP_PIC_SLAVE, i - 8); } if (pci_enabled) { for (i = 0; i < 24; ++i) { if (i == 0) { kvm_irqchip_add_irq_route(s, i, KVM_IRQCHIP_IOAPIC, 2); } else if (i != 2) { kvm_irqchip_add_irq_route(s, i, KVM_IRQCHIP_IOAPIC, i); } } } kvm_irqchip_commit_routes(s); } } static int kvm_irqchip_get_virq(KVMState *s) { uint32_t *word = s->used_gsi_bitmap; int max_words = ALIGN(s->gsi_count, 32) / 32; int i, bit; bool retry = true; again: /* Return the lowest unused GSI in the bitmap */ for (i = 0; i < max_words; i++) { bit = ffs(~word[i]); if (!bit) { continue; } return bit - 1 + i * 32; } if (!s->direct_msi && retry) { retry = false; kvm_flush_dynamic_msi_routes(s); goto again; } return -ENOSPC; } Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command, as it will automatically gather debugging information, in a terminal: apport-collect 1465935 When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs. It appears that the latest version of the patch is here: http://lists.gnu.org/archive/html/qemu-devel/2015-01/msg00822.html However, this hasn't yet be accepted upstream. The most recent discussion requires the submitter to respond to the maintainers questions here: http://lists.gnu.org/archive/html/qemu-devel/2015-01/msg00623.html Have you be able to reproduce this issue on a wily host? What about a different guest? Or is only RHEL6.3 affected? Ryan, Our Hypervisors are running in the internal network which can't access to Launchpad, # apport-collect 1465935 ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out We saw this qemu crash on 18 Hypervisor nodes. So far all our hypervisors are ubuntu 12.04, qemu-2.0.0+dfsg, and guest OS is only RHEL6.3 http://lists.gnu.org/archive/html/qemu-devel/2015-01/msg00822.html Seems that the latest version code has answered maintainers questions. -----Original Message----- From: Paolo Bonzini [mailto: