1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
|
Q35 doesn't support to hot add the 2nd PCIe device to KVM guest
KVM: https://git.kernel.org/pub/scm/virt/kvm/kvm.git branch: next, commit: 4a98623d
Qemu: https://git.qemu.org/git/qemu.git branch: master, commit: 9e2e9fe3
Created a KVM guest with Q35 chipset, and try to hot add 2 PCIe device to guest with qemu internal command device_add, the 1st device can be added successfully, but the 2nd device failed to hot add.
If guest chipset is legacy i440fx, the 2 device can be added successfully.
1. Enable VT-d in BIOS
2. load KVM modules in Linux OS: modprobe kvm; modprobe kvm_intel
3. Bind 2 device to vfio-pci
echo 0000:b1:00.0 > /sys/bus/pci/drivers/i40e/unbind
echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id
echo 0000:b1:00.1 > /sys/bus/pci/drivers/i40e/unbind
echo "8086 1572" > /sys/bus/pci/drivers/vfio-pci/new_id
4. create guest with Q35 chipset:
qemu-system-x86_64 --accel kvm -m 4096 -smp 4 -drive file=/home/rhel8.2.qcow2,if=none,id=virtio-disk0 -device virtio-blk-pci,drive=virtio-disk0 -cpu host -machine q35 -device pcie-root-port,id=root1 -daemonize
5. hot add the 1st device to guest successfully
in guest qemu monitor "device_add vfio-pci,host=b1:00.0,id=nic0,bus=root1"
6. hot add the 2nd device to guest
in guest qemu monitor "device_add vfio-pci,host=b1:00.1,id=nic1,bus=root1"
The 2nd device doesn't be added in guest, and the 1st device is removed from guest.
Guest partial log:
[ 110.452272] pcieport 0000:00:04.0: pciehp: Slot(0): Attention button pressed
[ 110.453314] pcieport 0000:00:04.0: pciehp: Slot(0) Powering on due to button press
[ 110.454156] pcieport 0000:00:04.0: pciehp: Slot(0): Card present
[ 110.454792] pcieport 0000:00:04.0: pciehp: Slot(0): Link Up
[ 110.580927] pci 0000:01:00.0: [8086:1572] type 00 class 0x020000
[ 110.582560] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x007fffff 64bit pref]
[ 110.583453] pci 0000:01:00.0: reg 0x1c: [mem 0x00000000-0x00007fff 64bit pref]
[ 110.584278] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0007ffff pref]
[ 110.585051] pci 0000:01:00.0: Max Payload Size set to 128 (was 512, max 2048)
[ 110.586621] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 110.588140] pci 0000:01:00.0: BAR 0: no space for [mem size 0x00800000 64bit pref]
[ 110.588954] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00800000 64bit pref]
[ 110.589797] pci 0000:01:00.0: BAR 6: assigned [mem 0xfe800000-0xfe87ffff pref]
[ 110.590703] pci 0000:01:00.0: BAR 3: assigned [mem 0xfe000000-0xfe007fff 64bit pref]
[ 110.592085] pcieport 0000:00:04.0: PCI bridge to [bus 01]
[ 110.592755] pcieport 0000:00:04.0: bridge window [io 0x1000-0x1fff]
[ 110.594403] pcieport 0000:00:04.0: bridge window [mem 0xfe800000-0xfe9fffff]
[ 110.595847] pcieport 0000:00:04.0: bridge window [mem 0xfe000000-0xfe1fffff 64bit pref]
[ 110.597867] PCI: No. 2 try to assign unassigned res
[ 110.597870] release child resource [mem 0xfe000000-0xfe007fff 64bit pref]
[ 110.597871] pcieport 0000:00:04.0: resource 15 [mem 0xfe000000-0xfe1fffff 64bit pref] released
[ 110.598881] pcieport 0000:00:04.0: PCI bridge to [bus 01]
[ 110.600789] pcieport 0000:00:04.0: BAR 15: assigned [mem 0x180000000-0x180bfffff 64bit pref]
[ 110.601731] pci 0000:01:00.0: BAR 0: assigned [mem 0x180000000-0x1807fffff 64bit pref]
[ 110.602849] pci 0000:01:00.0: BAR 3: assigned [mem 0x180800000-0x180807fff 64bit pref]
[ 110.604069] pcieport 0000:00:04.0: PCI bridge to [bus 01]
[ 110.604941] pcieport 0000:00:04.0: bridge window [io 0x1000-0x1fff]
[ 110.606237] pcieport 0000:00:04.0: bridge window [mem 0xfe800000-0xfe9fffff]
[ 110.607401] pcieport 0000:00:04.0: bridge window [mem 0x180000000-0x180bfffff 64bit pref]
[ 110.653661] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[ 110.654443] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[ 110.655314] i40e 0000:01:00.0: enabling device (0140 -> 0142)
[ 110.672396] i40e 0000:01:00.0: fw 6.0.48442 api 1.7 nvm 6.01 0x800035b1 1.1747.0 [8086:1572] [8086:0008]
[ 110.750054] i40e 0000:01:00.0: MAC address: 3c:fd:fe:c0:59:98
[ 110.751792] i40e 0000:01:00.0: FW LLDP is enabled
[ 110.764644] i40e 0000:01:00.0 eth1: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[ 110.779390] i40e 0000:01:00.0: PCI-Express: Speed 8.0GT/s Width x8
[ 110.789841] i40e 0000:01:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 4 RSS FD_ATR FD_SB NTUPLE DCB VxLAN Geneve PTP VEPA
[ 111.817553] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[ 205.130288] pcieport 0000:00:04.0: pciehp: Slot(0): Attention button pressed
[ 205.131743] pcieport 0000:00:04.0: pciehp: Slot(0): Powering off due to button press
[ 205.133233] pcieport 0000:00:04.0: pciehp: Slot(0): Card not present
[ 205.135728] i40e 0000:01:00.0: i40e_ptp_stop: removed PHC on eth1
|