graphic: 0.545
semantic: 0.528
instruction: 0.518
device: 0.510
mistranslation: 0.471
other: 0.445
network: 0.396
boot: 0.396
KVM: 0.381
socket: 0.372
assembly: 0.233
vnc: 0.187

qemu-system-x86_64: "VFIO_MAP_DMA : -28" error when we attache 6 VF's to guest machine

We are trying to attach 6 VF's to the guest machine on 4.1.1 qemu emulator.
We are observing "VFIO_MAP_DMA : -28" error.

We are using w-bits=48 bits while lunching VM.

Please provide information how you started QEMU, and some information about your PCI device (e.g. the output of lspci).

qemu-system-x86_64 -name guest=fedora24 -machine q35,accel=kvm,kernel-irqchip=split \
        -enable-kvm \
        -m 4G \
        -smp 8,sockets=1,cores=8,threads=1 \
        -device intel-iommu,intremap=on,caching-mode=on,aw-bits=48  \
        -drive file=<OS_IMAGE_FILE>,format=raw \
        -device ioh3420,id=pcie.1,chassis=1 \
        -device virtio-net-pci,bus=pcie.1,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on,netdev=net0 \
        -netdev user,id=net0,hostfwd=tcp::1111-:22\
        -device vfio-pci,host=3f:02.1  \
		-device vfio-pci,host=3f:02.2  \
		-device vfio-pci,host=3f:02.3 \
		-device vfio-pci,host=3f:02.4 \
        -device vfio-pci,host=3d:02.4 \
		-device vfio-pci,host=3d:02.5 \
		-device vfio-pci,host=3d:02.6 \
        -nographic

Please find the above qemu command to lunch guest machine

Presumably w-bits (aw-bits?) implies using intel-iommu, there's a opportunity for the vfio iommu backend to return -ENOSPC (-28) if we exceed the default number of in-flight DMA mappings per container.  The default limit is 65535.  You can try increasing this by changing the dma_entry_limit module option on the vfio_iommu_type1 module.  Note that in a typical vIOMMU config there's a container per device, so the number of VFs attached is possibly not a factor.  It is however a lot of DMA mappings for a single device if this is the issue and you'd generally want to boot the guest with iommu=pt in order to have reasonable assigned device performance with a vIOMMU, which would also greatly reduce the number of mappings.

After increasing dma_entry_limit limit no issue observed.

But ideal senario device is getting hung and recovery happening only with host hard rebooting.

[Expired for QEMU because there has been no activity for 60 days.]