1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
|
vfio-pci 0000:00:02.1: VF token required to access device
Description of problem:
I'm trying to use SR-IOV on an i5-12400 trying to create VFs for my UHD Graphics 730.
I had to build this DKMS module for it to work:
https://github.com/strongtz/i915-sriov-dkms
So far I have managed to have 7 VFs created per the dmsg:
```
[root@fedora ~]# dmesg | grep -i vf
[ 0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.1.13-200.fc37.x86_64 root=UUID=a1ec5891-71c6-44ea-9beb-c4f1cde55c0e ro rootflags=subvol=root rhgb quiet intel_iommu=on iommu=pt split_lock_detect=off i915.enable_guc=7 video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init vfio-pci.disable_vga=1 vfio-pci.enable_sriov=1 vfio-pci.ids=8086:4692,8086:7ad0
[ 0.074362] Kernel command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.1.13-200.fc37.x86_64 root=UUID=a1ec5891-71c6-44ea-9beb-c4f1cde55c0e ro rootflags=subvol=root rhgb quiet intel_iommu=on iommu=pt split_lock_detect=off i915.enable_guc=7 video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init v
io-pci.disable_vga=1 vfio-pci.enable_sriov=1 vfio-pci.ids=8086:4692,8086:7ad0
[ 0.288336] pci 0000:00:02.0: VF(n) BAR0 space: [mem 0x60e0000000-0x60e6ffffff 64bit] (contains BAR0 for 7 VFs)
[ 0.288339] pci 0000:00:02.0: VF(n) BAR2 space: [mem 0x6000000000-0x60dfffffff 64bit pref] (contains BAR2 for 7 VFs)
[ 0.293518] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0xa1330000-0xa134ffff 64bit] (contains BAR0 for 8 VFs)
[ 0.336464] VFS: Disk quotas dquot_6.6.0
[ 0.336470] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 1.028560] VFIO - User Level meta-driver version: 0.3
[ 1.039931] vfio-pci 0000:00:02.0: vgaarb: deactivate vga console
[ 1.039933] vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 1.040007] vfio_pci: add [8086:4692[ffffffff:ffffffff]] class 0x000000/00000000
[ 1.040140] vfio_pci: add [8086:7ad0[ffffffff:ffffffff]] class 0x000000/00000000
[ 3.373977] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer
[ 45.696323] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.1 driver_override
[ 45.696356] vfio-pci 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 45.696598] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.2 driver_override
[ 45.696609] vfio-pci 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 45.696724] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.3 driver_override
[ 45.696734] vfio-pci 0000:00:02.3: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 45.696811] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.4 driver_override
[ 45.696825] vfio-pci 0000:00:02.4: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 45.696947] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.5 driver_override
[ 45.696958] vfio-pci 0000:00:02.5: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 45.697050] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.6 driver_override
[ 45.697060] vfio-pci 0000:00:02.6: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[ 45.697127] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.7 driver_override
[ 45.697137] vfio-pci 0000:00:02.7: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
```
I've blacklisted these modules:
```
blacklist igb
blacklist i915
blacklist snd_hda_intel
blacklist snd_sof_pci_intel_tgl
```
And loaded these modules:
```
vfio
vfio-pci
vfio_virqfd
vfio_iommu_type1
```
Kernel args:
```
GRUB_CMDLINE_LINUX="rhgb quiet intel_iommu=on iommu=pt split_lock_detect=off i915.enable_guc=7 video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init vfio-pci.disable_vga=1 vfio-pci.enable_sriov=1 vfio-pci.ids=8086:4692,8086:7ad0"
```
**Error shown:**
```
[root@fedora ~]# ./test.sh
QEMU 7.0.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -device vfio-pci,host=0000:02.2,multifunction=on,bus=pcie.1,addr=0x00,x-vga=on: vfio 0000:00:02.2: error getting device from group 14: Permission denied
Verify all devices in group 14 are bound to vfio-<bus> or pci-stub and not already in use
```
**DMESG shows:**
```
[ 2160.408395] vfio-pci 0000:00:02.2: VF token required to access device
```
This lead me to this conversation / thread:
https://inbox.dpdk.org/dev/CALBAE1MrEoCc8Ch6MNUNTsOcZyJnhr+z+iD0VWjHagQsEdBWCw@mail.gmail.com/#t
Quote: "Something needs to be sorted with the QEMU community."
In fact, something needs to be sorted. **It seems there's no way to specify this VF token anywhere from the CLI args**, so I'm reporting this as a bug (or feature not developed yet?? any ETA?)
**Additional information:** It seems that QEMU might require a patch or a change to allow this VF token to be passed through. It seems that DPDK and other similar projects have already implemented this (it seems Linux has it since Kernel 5.7 - Maybe I'm missing something to pass this token with QEMU considering how old that kernel is? I'd expect this flag to be here in QEMU already)
**Useful code / info:**
* https://patches.dpdk.org/project/dpdk/patch/20200529013710.72302-3-haiyue.wang@intel.com/
* https://github.com/intel/pf-bb-config/blob/master/README.md#usage-example
* https://support.hpe.com/hpesc/public/docDisplay?docId=sd00001790en_us&docLocale=en_US&page=GUID-1D5D76F8-522A-47F5-922B-142BD5177033.html
Thanks,
-Alemar
Steps to reproduce:
1. See description
2. Run QEMU as described
|