vfio-pci 0000:00:02.1: VF token required to access device Description of problem: I'm trying to use SR-IOV on an i5-12400 trying to create VFs for my UHD Graphics 730. I had to build this DKMS module for it to work: https://github.com/strongtz/i915-sriov-dkms So far I have managed to have 7 VFs created per the dmsg: ``` [root@fedora ~]# dmesg | grep -i vf [ 0.000000] Command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.1.13-200.fc37.x86_64 root=UUID=a1ec5891-71c6-44ea-9beb-c4f1cde55c0e ro rootflags=subvol=root rhgb quiet intel_iommu=on iommu=pt split_lock_detect=off i915.enable_guc=7 video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init vfio-pci.disable_vga=1 vfio-pci.enable_sriov=1 vfio-pci.ids=8086:4692,8086:7ad0 [ 0.074362] Kernel command line: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.1.13-200.fc37.x86_64 root=UUID=a1ec5891-71c6-44ea-9beb-c4f1cde55c0e ro rootflags=subvol=root rhgb quiet intel_iommu=on iommu=pt split_lock_detect=off i915.enable_guc=7 video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init v io-pci.disable_vga=1 vfio-pci.enable_sriov=1 vfio-pci.ids=8086:4692,8086:7ad0 [ 0.288336] pci 0000:00:02.0: VF(n) BAR0 space: [mem 0x60e0000000-0x60e6ffffff 64bit] (contains BAR0 for 7 VFs) [ 0.288339] pci 0000:00:02.0: VF(n) BAR2 space: [mem 0x6000000000-0x60dfffffff 64bit pref] (contains BAR2 for 7 VFs) [ 0.293518] pci 0000:01:00.0: VF(n) BAR0 space: [mem 0xa1330000-0xa134ffff 64bit] (contains BAR0 for 8 VFs) [ 0.336464] VFS: Disk quotas dquot_6.6.0 [ 0.336470] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.028560] VFIO - User Level meta-driver version: 0.3 [ 1.039931] vfio-pci 0000:00:02.0: vgaarb: deactivate vga console [ 1.039933] vfio-pci 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem [ 1.040007] vfio_pci: add [8086:4692[ffffffff:ffffffff]] class 0x000000/00000000 [ 1.040140] vfio_pci: add [8086:7ad0[ffffffff:ffffffff]] class 0x000000/00000000 [ 3.373977] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer [ 45.696323] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.1 driver_override [ 45.696356] vfio-pci 0000:00:02.1: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 45.696598] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.2 driver_override [ 45.696609] vfio-pci 0000:00:02.2: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 45.696724] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.3 driver_override [ 45.696734] vfio-pci 0000:00:02.3: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 45.696811] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.4 driver_override [ 45.696825] vfio-pci 0000:00:02.4: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 45.696947] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.5 driver_override [ 45.696958] vfio-pci 0000:00:02.5: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 45.697050] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.6 driver_override [ 45.697060] vfio-pci 0000:00:02.6: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 45.697127] vfio-pci 0000:00:02.0: Captured SR-IOV VF 0000:00:02.7 driver_override [ 45.697137] vfio-pci 0000:00:02.7: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none ``` I've blacklisted these modules: ``` blacklist igb blacklist i915 blacklist snd_hda_intel blacklist snd_sof_pci_intel_tgl ``` And loaded these modules: ``` vfio vfio-pci vfio_virqfd vfio_iommu_type1 ``` Kernel args: ``` GRUB_CMDLINE_LINUX="rhgb quiet intel_iommu=on iommu=pt split_lock_detect=off i915.enable_guc=7 video=vesafb:off video=efifb:off initcall_blacklist=sysfb_init vfio-pci.disable_vga=1 vfio-pci.enable_sriov=1 vfio-pci.ids=8086:4692,8086:7ad0" ``` **Error shown:** ``` [root@fedora ~]# ./test.sh QEMU 7.0.0 monitor - type 'help' for more information (qemu) qemu-system-x86_64: -device vfio-pci,host=0000:02.2,multifunction=on,bus=pcie.1,addr=0x00,x-vga=on: vfio 0000:00:02.2: error getting device from group 14: Permission denied Verify all devices in group 14 are bound to vfio- or pci-stub and not already in use ``` **DMESG shows:** ``` [ 2160.408395] vfio-pci 0000:00:02.2: VF token required to access device ``` This lead me to this conversation / thread: https://inbox.dpdk.org/dev/CALBAE1MrEoCc8Ch6MNUNTsOcZyJnhr+z+iD0VWjHagQsEdBWCw@mail.gmail.com/#t Quote: "Something needs to be sorted with the QEMU community." In fact, something needs to be sorted. **It seems there's no way to specify this VF token anywhere from the CLI args**, so I'm reporting this as a bug (or feature not developed yet?? any ETA?) **Additional information:** It seems that QEMU might require a patch or a change to allow this VF token to be passed through. It seems that DPDK and other similar projects have already implemented this (it seems Linux has it since Kernel 5.7 - Maybe I'm missing something to pass this token with QEMU considering how old that kernel is? I'd expect this flag to be here in QEMU already) **Useful code / info:** * https://patches.dpdk.org/project/dpdk/patch/20200529013710.72302-3-haiyue.wang@intel.com/ * https://github.com/intel/pf-bb-config/blob/master/README.md#usage-example * https://support.hpe.com/hpesc/public/docDisplay?docId=sd00001790en_us&docLocale=en_US&page=GUID-1D5D76F8-522A-47F5-922B-142BD5177033.html Thanks, -Alemar Steps to reproduce: 1. See description 2. Run QEMU as described