test_kvm_xen_guest_novector_noapic sometimes fails Description of problem: The test_kvm_xen_guest_novector_noapic test of tests/avocado/kvm_xen_guest.py (soon to be moved to tests/functional/test_x86_64_kvm_xen.py ) is sometimes (maybe 1 time out of 50) failing to boot to the shell prompt. The messages on the serial console are: ``` Linux version 6.3.0-rc3-00031-g1e760fa3596e (alex@zen) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #21 SMP PREEMPT_DYNAMIC Fri Mar 24 15:04:37 GMT 2023 Command line: printk.time=0 root=/dev/xvda console=ttyS0 xen_emul_unplug=ide-disks xen_no_vector_callback noapic x86/fpu: x87 FPU will use FXSAVE signal: max sigframe size: 1440 BIOS-provided physical RAM map: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved BIOS-e820: [mem 0x0000000000100000-0x0000000007fdffff] usable BIOS-e820: [mem 0x0000000007fe0000-0x0000000007ffffff] reserved BIOS-e820: [mem 0x00000000feff8000-0x00000000feffffff] reserved BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved NX (Execute Disable) protection: active SMBIOS 2.8 present. DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 Hypervisor detected: Xen HVM Xen version 4.10. last_pfn = 0x7fe0 max_arch_pfn = 0x400000000 x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT found SMP MP-table at [mem 0x000f5470-0x000f547f] ACPI: Early table checksum verification disabled ACPI: RSDP 0x00000000000F5290 000014 (v00 BOCHS ) ACPI: RSDT 0x0000000007FE237F 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) ACPI: FACP 0x0000000007FE222B 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) ACPI: DSDT 0x0000000007FE0040 0021EB (v01 BOCHS BXPC 00000001 BXPC 00000001) ACPI: FACS 0x0000000007FE0000 000040 ACPI: APIC 0x0000000007FE229F 000080 (v03 BOCHS BXPC 00000001 BXPC 00000001) ACPI: HPET 0x0000000007FE231F 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) ACPI: WAET 0x0000000007FE2357 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) ACPI: Reserving FACP table memory at [mem 0x7fe222b-0x7fe229e] ACPI: Reserving DSDT table memory at [mem 0x7fe0040-0x7fe222a] ACPI: Reserving FACS table memory at [mem 0x7fe0000-0x7fe003f] ACPI: Reserving APIC table memory at [mem 0x7fe229f-0x7fe231e] ACPI: Reserving HPET table memory at [mem 0x7fe231f-0x7fe2356] ACPI: Reserving WAET table memory at [mem 0x7fe2357-0x7fe237e] Zone ranges: DMA [mem 0x0000000000001000-0x0000000000ffffff] DMA32 [mem 0x0000000001000000-0x0000000007fdffff] Normal empty Device empty Movable zone start for each node Early memory node ranges node 0: [mem 0x0000000000001000-0x000000000009efff] node 0: [mem 0x0000000000100000-0x0000000007fdffff] Initmem setup node 0 [mem 0x0000000000001000-0x0000000007fdffff] On node 0, zone DMA: 1 pages in unavailable ranges On node 0, zone DMA: 97 pages in unavailable ranges On node 0, zone DMA32: 32 pages in unavailable ranges ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) ACPI: Skipping IOAPIC probe due to 'noapic' option. ACPI: Using ACPI for processor (LAPIC) configuration information ACPI: HPET id: 0x8086a201 base: 0xfed00000 Intel MultiProcessor Specification v1.4 MPTABLE: OEM ID: BOCHSCPU MPTABLE: Product ID: 0.1 MPTABLE: APIC at: 0xFEE00000 IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 Processors: 2 smpboot: Allowing 2 CPUs, 0 hotplug CPUs [mem 0x08000000-0xfeff7fff] available for PCI devices Booting paravirtualized kernel on Xen HVM clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns setup_percpu: NR_CPUS:64 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1 percpu: Embedded 44 pages/cpu s149304 r0 d30920 u1048576 Built 1 zonelists, mobility grouping on. Total pages: 31968 Kernel command line: printk.time=0 root=/dev/xvda console=ttyS0 xen_emul_unplug=ide-disks xen_no_vector_callback noapic Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear) Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) mem auto-init: stack:off, heap alloc:off, heap free:off Memory: 102364K/130552K available (12288K kernel code, 1699K rwdata, 3004K rodata, 1040K init, 2632K bss, 27928K reserved, 0K cma-reserved) SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1 Dynamic Preempt: full rcu: Preemptible hierarchical RCU implementation. rcu: RCU event tracing is enabled. rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=2. rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 NR_IRQS: 4352, nr_irqs: 440, preallocated irqs: 16 xen:events: Using 2-level ABI rcu: srcu_init: Setting srcu_struct sizes based on contention. Console: colour *CGA 80x25 Cannot get hvm parameter CONSOLE_EVTCHN (18): -22! printk: console [ttyS0] enabled ACPI: Core revision 20221020 ACPI: setting ELCR to 0200 (from 0c00) clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns APIC: Switch to symmetric I/O mode setup Not enabling interrupt remapping due to skipped IO-APIC setup tsc: Unable to calibrate against PIT tsc: using HPET reference calibration tsc: Detected 2496.010 MHz processor clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x23fa80a809f, max_idle_ns: 440795273818 ns Calibrating delay loop (skipped), value calculated using timer frequency.. 4992.02 BogoMIPS (lpj=2496010) pid_max: default: 32768 minimum: 301 LSM: initializing lsm=capability,yama,integrity,selinux Yama: becoming mindful. SELinux: Initializing. Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear) Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear) Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization Spectre V2 : Kernel not compiled with retpoline; no mitigation available! Spectre V2 : Vulnerable Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch Speculative Store Bypass: Vulnerable MDS: Vulnerable: Clear CPU buffers attempted, no microcode MMIO Stale Data: Unknown: No mitigations Freeing SMP alternatives memory: 24K APIC timer disabled due to verification failure smpboot: CPU0: Intel QEMU Virtual CPU version 2.5+ (family: 0xf, model: 0x6b, stepping: 0x1) Performance Events: unsupported Netburst CPU model 107 no PMU driver, software events only. rcu: Hierarchical SRCU implementation. rcu: Max phase no-delay instances is 400. NMI watchdog: Perf NMI watchdog permanently disabled smp: Bringing up secondary CPUs ... x86: Booting SMP configuration: .... node #0, CPUs: #1 smp: Brought up 1 node, 2 CPUs smpboot: Max logical packages: 1 smpboot: Total of 2 processors activated (9984.04 BogoMIPS) devtmpfs: initialized x86/mm: Memory block size: 128MB clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns futex hash table entries: 512 (order: 3, 32768 bytes, linear) PM: RTC time: 12:02:16, date: 2024-12-19 NET: Registered PF_NETLINK/PF_ROUTE protocol family audit: initializing netlink subsys (disabled) audit: type=2000 audit(1734609736.239:1): state=initialized audit_enabled=0 res=1 thermal_sys: Registered thermal governor 'step_wise' thermal_sys: Registered thermal governor 'user_space' cpuidle: using governor ladder cpuidle: using governor menu PCI: Using configuration type 1 for base access HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages HugeTLB: 28 KiB vmemmap can be freed for a 2.00 MiB page cryptd: max_cpu_qlen set to 1000 ACPI: Added _OSI(Module Device) ACPI: Added _OSI(Processor Device) ACPI: Added _OSI(3.0 _SCP Extensions) ACPI: Added _OSI(Processor Aggregator Device) ACPI: 1 ACPI AML tables successfully acquired and loaded ACPI: Interpreter enabled ACPI: PM: (supports S0 S3 S5) ACPI: Using PIC for interrupt routing PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug PCI: Using E820 reservations for host bridge windows ACPI: Enabled 2 GPEs in block 00 to 0F ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] acpi PNP0A03:00: PCIe port services disabled; not requesting _OSC control acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended configuration space under this bridge PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] pci_bus 0000:00: root bus resource [mem 0x08000000-0xfebfffff window] pci_bus 0000:00: root bus resource [mem 0x100000000-0x17fffffff window] pci_bus 0000:00: root bus resource [bus 00-ff] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB pci 0000:00:02.0: [5853:0001] type 00 class 0xff8000 pci 0000:00:02.0: reg 0x10: [io 0xc000-0xc0ff] pci 0000:00:02.0: reg 0x14: [mem 0xfd000000-0xfdffffff pref] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 pci 0000:00:03.0: reg 0x10: [io 0xc100-0xc11f] pci 0000:00:03.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] pci 0000:00:03.0: reg 0x20: [mem 0xfe000000-0xfe003fff 64bit pref] pci 0000:00:03.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 ACPI: PCI: Interrupt link LNKB configured for IRQ 10 ACPI: PCI: Interrupt link LNKC configured for IRQ 11 ACPI: PCI: Interrupt link LNKD configured for IRQ 11 ACPI: PCI: Interrupt link LNKS configured for IRQ 9 xen:balloon: Initialising balloon driver iommu: Default domain type: Translated iommu: DMA domain TLB invalidation policy: lazy mode SCSI subsystem initialized ACPI: bus type USB registered usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb pps_core: LinuxPPS API ver. 1 registered pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti PTP clock support registered Advanced Linux Sound Architecture Driver Initialized. PCI: Using ACPI for IRQ routing hpet: 3 channels of 0 reserved for per-cpu timers clocksource: Switched to clocksource tsc-early FS-Cache: Loaded pnp: PnP ACPI init pnp: PnP ACPI: found 6 devices NET: Registered PF_INET protocol family IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear) tcp_listen_portaddr_hash hash table entries: 128 (order: 0, 4096 bytes, linear) Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) TCP established hash table entries: 1024 (order: 1, 8192 bytes, linear) TCP bind hash table entries: 1024 (order: 4, 65536 bytes, linear) TCP: Hash tables configured (established 1024 bind 1024) UDP hash table entries: 256 (order: 2, 24576 bytes, linear) UDP-Lite hash table entries: 256 (order: 2, 24576 bytes, linear) NET: Registered PF_UNIX/PF_LOCAL protocol family RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] pci_bus 0000:00: resource 7 [mem 0x08000000-0xfebfffff window] pci_bus 0000:00: resource 8 [mem 0x100000000-0x17fffffff window] pci 0000:00:01.0: PIIX3: Enabling Passive Release pci 0000:00:00.0: Limiting direct PCI/PCI transfers PCI: CLS 0 bytes, default 64 kvm_intel: VMX not supported by CPU 0 workingset: timestamp_bits=46 max_order=15 bucket_order=0 squashfs: version 4.0 (2009/01/31) Phillip Lougher fuse: init (API version 7.38) 9p: Installing v9fs 9p2000 file system support Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245) io scheduler mq-deadline registered io scheduler kyber registered ACPI: \_SB_.LNKC: Enabled at IRQ 11 xen:xen_evtchn: Event-channel device installed ACPI: \_SB_.LNKB: Enabled at IRQ 10 xen:grant_table: Grant tables using version 1 layout Grant table initialized Cannot get hvm parameter CONSOLE_EVTCHN (18): -22! Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A Non-volatile memory driver v1.3 ACPI: bus type drm_connector registered loop: module loaded Invalid max_queues (4), will use default max: 2. tun: Universal TUN/TAP device driver, 1.6 e100: Intel(R) PRO/100 Network Driver e100: Copyright(c) 1999-2006 Intel Corporation e1000: Intel(R) PRO/1000 Network Driver e1000: Copyright (c) 1999-2006 Intel Corporation. e1000e: Intel(R) PRO/1000 Network Driver e1000e: Copyright(c) 1999 - 2015 Intel Corporation. igb: Intel(R) Gigabit Ethernet Network Driver igb: Copyright (c) 2007-2014 Intel Corporation. igbvf: Intel(R) Gigabit Virtual Function Network Driver igbvf: Copyright (c) 2009 - 2012 Intel Corporation. VMware vmxnet3 virtual NIC driver - version 1.7.0.0-k-NAPI xen_netfront: Initialising Xen virtual ethernet driver usbcore: registered new interface driver cdc_ether usbcore: registered new interface driver cdc_eem usbcore: registered new interface driver cdc_ncm usbcore: registered new interface driver r8153_ecm usbcore: registered new interface driver cdc_acm cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters usbcore: registered new interface driver usb-storage usbcore: registered new interface driver usbserial_generic usbserial: USB Serial support registered for generic usbcore: registered new interface driver ch341 usbserial: USB Serial support registered for ch341-uart usbcore: registered new interface driver cp210x usbserial: USB Serial support registered for cp210x usbcore: registered new interface driver ftdi_sio usbserial: USB Serial support registered for FTDI USB Serial Device usbcore: registered new interface driver keyspan usbserial: USB Serial support registered for Keyspan - (without firmware) usbserial: USB Serial support registered for Keyspan 1 port adapter usbserial: USB Serial support registered for Keyspan 2 port adapter usbserial: USB Serial support registered for Keyspan 4 port adapter usbcore: registered new interface driver pl2303 usbserial: USB Serial support registered for pl2303 usbcore: registered new interface driver usb_serial_simple usbserial: USB Serial support registered for carelink usbserial: USB Serial support registered for zio usbserial: USB Serial support registered for funsoft usbserial: USB Serial support registered for flashloader usbserial: USB Serial support registered for google usbserial: USB Serial support registered for libtransistor usbserial: USB Serial support registered for vivopay usbserial: USB Serial support registered for moto_modem usbserial: USB Serial support registered for motorola_tetra usbserial: USB Serial support registered for nokia usbserial: USB Serial support registered for novatel_gps usbserial: USB Serial support registered for hp4x usbserial: USB Serial support registered for suunto usbserial: USB Serial support registered for siemens_mpi rtc_cmos 00:05: RTC can wake from S4 rtc_cmos 00:05: registered as rtc0 rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs fail to initialize ptp_kvm intel_pstate: CPU model not supported usbcore: registered new interface driver usbhid usbhid: USB HID core driver GACT probability NOT on xt_time: kernel timezone is -0000 IPVS: Registered protocols (TCP, UDP) IPVS: Connection hash table configured (size=4096, memory=32Kbytes) IPVS: ipvs loaded. IPVS: [rr] scheduler registered. Initializing XFRM netlink socket NET: Registered PF_INET6 protocol family Segment Routing with IPv6 In-situ OAM (IOAM) with IPv6 NET: Registered PF_PACKET protocol family 8021q: 802.1Q VLAN Support v1.8 9pnet: Installing 9P2000 support NET: Registered PF_VSOCK protocol family IPI shorthand broadcast: enabled sched_clock: Marking stable (402156364, -4933103)->(420983909, -23760648) Clockevents: could not switch to one-shot mode: lapic is not functional. Could not switch to high resolution mode on CPU 0 Clockevents: could not switch to one-shot mode: lapic is not functional. Could not switch to high resolution mode on CPU 1 tsc: Refined TSC clocksource calibration: 2495.955 MHz clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x23fa4cd42c8, max_idle_ns: 440795310990 ns clocksource: Switched to clocksource tsc Clockevents: could not switch to one-shot mode: lapic is not functional. Could not switch to high resolution mode on CPU 1 Clockevents: could not switch to one-shot mode: lapic is not functional. Could not switch to high resolution mode on CPU 0 xenbus_probe_frontend: Waiting for devices to initialise: 25s...20s...15s... random: crng init done 10s...5s...0s... ``` Steps to reproduce: Either run the mentioned avocado/functional test, or directly the mentioned QEMU command line >= 50 times Additional information: I think it reproduces more easily if the host machine is under load (not quite sure about it, though). See this discussion on the mailing list for some more details: https://lore.kernel.org/qemu-devel/999a8203f0c800f1305aacdb500dbf6038ebf147.camel@infradead.org/