semantic: 0.926
graphic: 0.883
instruction: 0.851
assembly: 0.836
device: 0.825
mistranslation: 0.805
vnc: 0.803
socket: 0.794
other: 0.750
network: 0.677
KVM: 0.582
boot: 0.506

Cache Layout wrong on many Zen Arch CPUs

AMD CPUs have L3 cache per 2, 3 or 4 cores. Currently, TOPOEXT seems to always map Cache ass if it was an 4-Core per CCX CPU, which is incorrect, and costs upwards 30% performance (more realistically 10%) in L3 Cache Layout aware applications.

Example on a 4-CCX CPU (1950X /w 8 Cores and no SMT): 

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <topology sockets='1' cores='8' threads='1'/>

In windows, coreinfo reports correctly: 

****----  Unified Cache 1, Level 3,    8 MB, Assoc  16, LineSize  64
----****  Unified Cache 6, Level 3,    8 MB, Assoc  16, LineSize  64

On a 3-CCX CPU (3960X /w 6 cores and no SMT):

 <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC-IBPB</model>
    <vendor>AMD</vendor>
    <topology sockets='1' cores='6' threads='1'/>

in windows, coreinfo reports incorrectly: 

****--  Unified Cache  1, Level 3,    8 MB, Assoc  16, LineSize  64
----**  Unified Cache  6, Level 3,    8 MB, Assoc  16, LineSize  64


Validated against 3.0, 3.1, 4.1 and 4.2 versions of qemu-kvm. 

With newer Qemu there is a fix (that does behave correctly) in using the dies parameter: 
 <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/>

The problem is that the dies are exposed differently than how AMD does it natively, they are exposed to Windows as sockets, which means, you can't ever have a machine with more than two CCX (6 cores) as Windows only supports two sockets. (Should this be reported as a separate bug?)

Hi,

I've since confirmed that this bug also exist (as expected) on Linux guests, as well as Zen1 EPYC 7401 CPUs, to make sure this wasn't a problem with the detection of the newer consumer platform. 

Basically it seems (looking at the code with layman eyes) that as long as you have a topology that is dividable by 4 or 8, it will always result in the wrong topology being exposed to the guest, even when the correct option can be built (12, 24 core CPUs, although, it would be great if we could support 9 Core VM CPus as that is a reasonable use case for VMs (3 CCXs of 3 Cores for a total of 9 (or 18 SMT threads)).

Pinging the author and committer of the TopoEXT feature / EPYC cpu model as they should probably know best how to solve this issue.

This is the commit I am referencing: https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8f4202fb1080f86958782b1fca0bf0279f67d136

Damir,
  We normally test Linux guests here. Can you please give me exact qemu command line. Even the SMP parameters(sockets,cores,threads,dies) will also work. I will try to recreate it locally first.
Give me example what works and what does not work.

I have recently sent few more patches to fix another bug. Please check if this makes any difference.
https://patchwork.kernel.org/cover/11272063/
https://lore.kernel<email address hidden>/

This should apply cleanly on git://github.com/ehabkost/qemu.git (branch x86-next)

Note: I will be on vacation until first week of Jan. Responses will be delayed.

Same problem for Ryzen 9 3900X. There should be 4x L3 caches, but there are only 3.

Same results with "host-passthrough" and "EPYC-IBPB". Windows doesn't recognize the correct L3 cache layout.

From coreinfo.exe:

Logical Processor to Cache Map:
**----------------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
********----------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--**--------------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
----**------------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Unified Cache       4, Level 2,  512 KB, Assoc   8, LineSize  64
--------**--------------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------------  Unified Cache       5, Level 2,  512 KB, Assoc   8, LineSize  64
--------********--------  Unified Cache       6, Level 3,   16 MB, Assoc  16, LineSize  64
----------**------------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Unified Cache       7, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----------  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
--------------**--------  Data Cache          7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--------  Instruction Cache   7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--------  Unified Cache       9, Level 2,  512 KB, Assoc   8, LineSize  64
----------------**------  Data Cache          8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Instruction Cache   8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Unified Cache      10, Level 2,  512 KB, Assoc   8, LineSize  64
----------------********  Unified Cache      11, Level 3,   16 MB, Assoc  16, LineSize  64
------------------**----  Data Cache          9, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Instruction Cache   9, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Unified Cache      12, Level 2,  512 KB, Assoc   8, LineSize  64
--------------------**--  Data Cache         10, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------**--  Instruction Cache  10, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------**--  Unified Cache      13, Level 2,  512 KB, Assoc   8, LineSize  64
----------------------**  Data Cache         11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Instruction Cache  11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Unified Cache      14, Level 2,  512 KB, Assoc   8, LineSize  64


AMD does not use dies. For AMD dies is normally set to 1. You probably have to pass dies in some other ways. Did you try the latest qemu v 5.0? Please try it. 

Qemu expects the user to configure the topology based on their requirement.
 
Try replacing <qemu:arg value='cores=3,threads=1,dies=2,sockets=1'/> 
with <qemu:arg value='cores=6,threads=1,dies=1,sockets=1'/>

You can also use the numa configuration. There are multiple ways you can achieve your required configuration.


Damir, Example of how to use numa configuration.
-smp 16,maxcpus=16,cores=16,threads=1,sockets=1 -numa node,nodeid=0,cpus=0-7 -numa node,nodeid=1,cpus=8-15

This will help to put all the cores in correct L3 boundary. I strongly suggest to use the latest qemu release. 

It could be an issue of how the kernel presents the CPU topology. 

Hardware: AMD Ryzen 3900X 12 core 24 threads (SMT)
Host: Kernel 5.6.6, QEMU 4.2

virsh capabilities | grep "cpu id"
            <cpu id='0' socket_id='0' core_id='0' siblings='0,12'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,13'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,14'/>
            <cpu id='3' socket_id='0' core_id='4' siblings='3,15'/>
            <cpu id='4' socket_id='0' core_id='5' siblings='4,16'/>
            <cpu id='5' socket_id='0' core_id='6' siblings='5,17'/>
            <cpu id='6' socket_id='0' core_id='8' siblings='6,18'/>
            <cpu id='7' socket_id='0' core_id='9' siblings='7,19'/>
            <cpu id='8' socket_id='0' core_id='10' siblings='8,20'/>
            <cpu id='9' socket_id='0' core_id='12' siblings='9,21'/>
            <cpu id='10' socket_id='0' core_id='13' siblings='10,22'/>
            <cpu id='11' socket_id='0' core_id='14' siblings='11,23'/>
            <cpu id='12' socket_id='0' core_id='0' siblings='0,12'/>
            <cpu id='13' socket_id='0' core_id='1' siblings='1,13'/>
            <cpu id='14' socket_id='0' core_id='2' siblings='2,14'/>
            <cpu id='15' socket_id='0' core_id='4' siblings='3,15'/>
            <cpu id='16' socket_id='0' core_id='5' siblings='4,16'/>
            <cpu id='17' socket_id='0' core_id='6' siblings='5,17'/>
            <cpu id='18' socket_id='0' core_id='8' siblings='6,18'/>
            <cpu id='19' socket_id='0' core_id='9' siblings='7,19'/>
            <cpu id='20' socket_id='0' core_id='10' siblings='8,20'/>
            <cpu id='21' socket_id='0' core_id='12' siblings='9,21'/>
            <cpu id='22' socket_id='0' core_id='13' siblings='10,22'/>
            <cpu id='23' socket_id='0' core_id='14' siblings='11,23'/>

See how cpu id=3 gets core id=4, and cpu id=6 gets core id=8, etc.

cat /sys/devices/system/cpu/cpu2/topology/core_id
2

cat /sys/devices/system/cpu/cpu3/topology/core_id
4

However, the association of CPU IDs to L3 caches seems to be correct:

echo "Level  CPU list";for file in /sys/devices/system/cpu/cpu*/cache/index3; do echo $(cat $file/id) "    " $(cat $file/shared_cpu_list); done | sort --version-sort
Level  CPU list
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
0      0-2,12-14
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
1      3-5,15-17
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
2      6-8,18-20
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23
3      9-11,21-23

There are 4 L3 caches with the correct CPU lists (6 CPUs/threads each).

Is it possible that this weird CPU ID enumeration is causing the confusion?

Haven't had a chance to check out QEMU 5.0, but hope to do that today.

Finally installed QEMU 5.0.0.154 - still the same. QEMU doesn't recognize the L3 caches and still lists 3 L3 caches instead of 4 with 3 cores/6 threads.

Here the vm.log with the qemu command line (shortened):

2020-05-03 18:23:38.674+0000: starting up libvirt version: 5.10.0, qemu version: 5.0.50v5.0.0-154-g2ef486e76d-dirty, kernel: 5.4.36-1-MANJARO

-machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu host,invtsc=on,hypervisor=on,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-vendor-id=AuthenticAMD,hv-frequencies,hv-crash,kvm=off,host-cache-info=on,l3-cache=off \
-m 49152 \
-mem-prealloc \
-mem-path /dev/hugepages/libvirt/qemu/1-win10 \
-overcommit mem-lock=off \
-smp 24,sockets=1,cores=12,threads=2 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=34,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot menu=off,strict=on \


Hi Seiger,
I am not an expert on libvirt. I mostly use qemu command line for my test. I was able to achieve the 3960X configuration with the following command line. 

# qemu-system-x86_64 -name rhel7  -m 16384 -smp 24,cores=12,threads=2,sockets=1 -hda vdisk.qcow2 -enable-kvm -net nic -net bridge,br=virbr0,helper=/usr/libexec/qemu-bridge-helper -cpu host,+topoext -nographic -numa node,nodeid=0,cpus=0-5 -numa node,nodeid=1,cpus=6-11 -numa node,nodeid=2,cpus=12-17 -numa node,nodeid=3,cpus=18-23

Basically qemu does not have all the information to build the topology for every configuration. It depends on  libvirt for that information. See if this combination works for you.

Hello Babu,

Thanks for the reply and the QEMU command line. I will try to implement it in the XML.

So essentially what you do is to define each group of cpus and associate them with a numa node:

-numa node,nodeid=0,cpus=0-5 -numa node,nodeid=1,cpus=6-11 -numa node,nodeid=2,cpus=12-17 -numa node,nodeid=3,cpus=18-23

Haven't tried it but that might work. Do you need QEMU 5.0 for this to work, or is 4.2 OK?

Yes. Sieger. Please install 5.0 it should work fine. I am not sure about 4.2. 

Hello, 

I took a look today at the layouts when using 1950X (which previously worked, and yes, admittedly, I am using Windows / coreinfo), and any basic config (previously something simple as Sockets=1,Cores=8, Theads=1 (now also Dies=1) worked, but now, the topology presents as if all cores share L3, and that each two cores share L1C/L1D/L2, like if they were smt-siblings. I would call this a serious regression. 

I don't think using Numa Nodes is an ok way to solve this (especially not when at least for 4CCX CPUs, this worked flawlessly before), as that will make numa-aware applications start taking note of numa nodes, and possibly do wierd things (plus, it introduces more configuration where it was not needed before). 

I upgraded to QEMU emulator version 5.0.50
Using q35-5.1 (the latest) and the following libvirt configuration:

  <memory unit="KiB">50331648</memory>
  <currentMemory unit="KiB">50331648</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static">24</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="12"/>
    <vcpupin vcpu="2" cpuset="1"/>
    <vcpupin vcpu="3" cpuset="13"/>
    <vcpupin vcpu="4" cpuset="2"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="6" cpuset="3"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <vcpupin vcpu="8" cpuset="4"/>
    <vcpupin vcpu="9" cpuset="16"/>
    <vcpupin vcpu="10" cpuset="5"/>
    <vcpupin vcpu="11" cpuset="17"/>
    <vcpupin vcpu="12" cpuset="6"/>
    <vcpupin vcpu="13" cpuset="18"/>
    <vcpupin vcpu="14" cpuset="7"/>
    <vcpupin vcpu="15" cpuset="19"/>
    <vcpupin vcpu="16" cpuset="8"/>
    <vcpupin vcpu="17" cpuset="20"/>
    <vcpupin vcpu="18" cpuset="9"/>
    <vcpupin vcpu="19" cpuset="21"/>
    <vcpupin vcpu="20" cpuset="10"/>
    <vcpupin vcpu="21" cpuset="22"/>
    <vcpupin vcpu="22" cpuset="11"/>
    <vcpupin vcpu="23" cpuset="23"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.1">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="no"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <vendor_id state="on" value="AuthenticAMD"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" cores="12" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="require" name="hypervisor"/>
    <feature policy="require" name="topoext"/>
    <numa>
      <cell id="0" cpus="0-2,12-14" memory="12582912" unit="KiB"/>
      <cell id="1" cpus="3-5,15-17" memory="12582912" unit="KiB"/>
      <cell id="2" cpus="6-8,18-20" memory="12582912" unit="KiB"/>
      <cell id="3" cpus="9-11,21-23" memory="12582912" unit="KiB"/>
    </numa>
  </cpu>

...

/var/log/libvirt/qemu/win10.log:

-machine pc-q35-5.1,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu host,invtsc=on,hypervisor=on,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-vendor-id=AuthenticAMD,hv-frequencies,hv-crash,kvm=off,host-cache-info=on,l3-cache=off \
-m 49152 \
-overcommit mem-lock=off \
-smp 24,sockets=1,cores=12,threads=2 \
-mem-prealloc \
-mem-path /dev/hugepages/libvirt/qemu/3-win10 \
-numa node,nodeid=0,cpus=0-2,cpus=12-14,mem=12288 \
-numa node,nodeid=1,cpus=3-5,cpus=15-17,mem=12288 \
-numa node,nodeid=2,cpus=6-8,cpus=18-20,mem=12288 \
-numa node,nodeid=3,cpus=9-11,cpus=21-23,mem=12288 \
...

For some reason I always get l3-cache=off.

CoreInfo.exe in Windows 10 then produces the following report (shortened):

Logical to Physical Processor Map:
**----------------------  Physical Processor 0 (Hyperthreaded)
--*---------------------  Physical Processor 1
---*--------------------  Physical Processor 2
----**------------------  Physical Processor 3 (Hyperthreaded)
------**----------------  Physical Processor 4 (Hyperthreaded)
--------*---------------  Physical Processor 5
---------*--------------  Physical Processor 6
----------**------------  Physical Processor 7 (Hyperthreaded)
------------**----------  Physical Processor 8 (Hyperthreaded)
--------------*---------  Physical Processor 9
---------------*--------  Physical Processor 10
----------------**------  Physical Processor 11 (Hyperthreaded)
------------------**----  Physical Processor 12 (Hyperthreaded)
--------------------*---  Physical Processor 13
---------------------*--  Physical Processor 14
----------------------**  Physical Processor 15 (Hyperthreaded)

Logical Processor to Socket Map:
************************  Socket 0

Logical Processor to NUMA Node Map:
***---------***---------  NUMA Node 0
---***---------***------  NUMA Node 1
------***---------***---  NUMA Node 2
---------***---------***  NUMA Node 3

Approximate Cross-NUMA Node Access Cost (relative to fastest):
     00  01  02  03
00: 1.4 1.2 1.1 1.2
01: 1.1 1.1 1.3 1.1
02: 1.0 1.1 1.0 1.2
03: 1.1 1.2 1.2 1.2

Logical Processor to Cache Map:
**----------------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
***---------------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--*---------------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--*---------------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--*---------------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
---*--------------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
---*--------------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
---*--------------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
---***------------------  Unified Cache       4, Level 3,   16 MB, Assoc  16, LineSize  64
----**------------------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------------  Unified Cache       5, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------------  Unified Cache       6, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------------  Unified Cache       7, Level 3,   16 MB, Assoc  16, LineSize  64
--------*---------------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
--------*---------------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
--------*---------------  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
--------*---------------  Unified Cache       9, Level 3,   16 MB, Assoc  16, LineSize  64
---------*--------------  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
---------*--------------  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
---------*--------------  Unified Cache      10, Level 2,  512 KB, Assoc   8, LineSize  64
---------***------------  Unified Cache      11, Level 3,   16 MB, Assoc  16, LineSize  64
----------**------------  Data Cache          7, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Instruction Cache   7, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------------  Unified Cache      12, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----------  Data Cache          8, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Instruction Cache   8, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----------  Unified Cache      13, Level 2,  512 KB, Assoc   8, LineSize  64
------------***---------  Unified Cache      14, Level 3,   16 MB, Assoc  16, LineSize  64
--------------*---------  Data Cache          9, Level 1,   32 KB, Assoc   8, LineSize  64
--------------*---------  Instruction Cache   9, Level 1,   32 KB, Assoc   8, LineSize  64
--------------*---------  Unified Cache      15, Level 2,  512 KB, Assoc   8, LineSize  64
---------------*--------  Data Cache         10, Level 1,   32 KB, Assoc   8, LineSize  64
---------------*--------  Instruction Cache  10, Level 1,   32 KB, Assoc   8, LineSize  64
---------------*--------  Unified Cache      16, Level 2,  512 KB, Assoc   8, LineSize  64
---------------*--------  Unified Cache      17, Level 3,   16 MB, Assoc  16, LineSize  64
----------------**------  Data Cache         11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Instruction Cache  11, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**------  Unified Cache      18, Level 2,  512 KB, Assoc   8, LineSize  64
----------------**------  Unified Cache      19, Level 3,   16 MB, Assoc  16, LineSize  64
------------------**----  Data Cache         12, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Instruction Cache  12, Level 1,   32 KB, Assoc   8, LineSize  64
------------------**----  Unified Cache      20, Level 2,  512 KB, Assoc   8, LineSize  64
------------------***---  Unified Cache      21, Level 3,   16 MB, Assoc  16, LineSize  64
--------------------*---  Data Cache         13, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------*---  Instruction Cache  13, Level 1,   32 KB, Assoc   8, LineSize  64
--------------------*---  Unified Cache      22, Level 2,  512 KB, Assoc   8, LineSize  64
---------------------*--  Data Cache         14, Level 1,   32 KB, Assoc   8, LineSize  64
---------------------*--  Instruction Cache  14, Level 1,   32 KB, Assoc   8, LineSize  64
---------------------*--  Unified Cache      23, Level 2,  512 KB, Assoc   8, LineSize  64
---------------------***  Unified Cache      24, Level 3,   16 MB, Assoc  16, LineSize  64
----------------------**  Data Cache         15, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Instruction Cache  15, Level 1,   32 KB, Assoc   8, LineSize  64
----------------------**  Unified Cache      25, Level 2,  512 KB, Assoc   8, LineSize  64

Logical Processor to Group Map:
************************  Group 0


The above result is even further away from the actual L3 cache configuration.

So numatune doesn't produce the expected outcome.


Same problem here on 5.0 and 3900x (3 cores per CCX). And as stated before - declaring NUMA nodes is definitely not the right solution if the aim is to emulate the host CPU as close as possible.

The problem is that disabled cores are not taken into account.. ALL Zen2 CPUs have L3 cache group per CCX and every CCX has 4 cores, the problem is that some cores in each CCX (1 for 6 and 12-core CPUs, 2 for 3100) are disabled for some models, but they still use their core ids (as can be seen in virsh capabilities | grep "cpu id" output in above comments). Looking at target/i386/cpu.c:5529, this is not taken into account.

Maybe the cleanest way to fix this is to emulate the host topology by also skipping disabled core ids in the VM? That way, die offset will actually match the real host CPU topology...

A workaround for Linux VMs is to disable CPUs (and setting their number/pinnings accordingly, e.g. every 4th (and 3rd for 3100) core is going to be 'dummy' and disabled system-wide) by e.g. echo 0 > /sys/devices/system/cpu/cpu3/online

No good workaround for Windows VMs exists, as far as I know - the best you can do is setting affinity to specific process(es) and avoid the 'dummy' CPUs, but I am not aware of any possibility to disable specific CPUs (only limiting the overall number).

Hi Jan, 

Problem for me now is why does every config (I can figure out) now result in SMT on/L3 across all cores which is obviously never true on Zen except if you have only less than 4 cores, 8 cores should always result in 2 L3 Caches, and so should 16 Threads /w 8+SMT. This worked in my initial post. 

Latest qemu has removed all the hard coded configurations for AMD. It is leaving everything to customize.  One way is to configure is using numa nodes. This will make sure cpus under one numa node share same L3. Then pin the correct host cpus to guest cpus using vcpupin. I would change this -numa node,nodeid=0,cpus=0-2,cpus=12-14,mem=12288 to -numa node,nodeid=0,cpus=0-2,cpus=3-5,mem=12288. Then have vcpupin map the correct host cpu to guest cpu. Check if this works for you. Can you please post lscpu output from host for everybody's understanding?    


No, creating artificial NUMA nodes is, simply put, never a good solution for CPUs that operate as a single NUMA node - which is the case for all Zen2 CPUs (except maybe EPYCs? not sure about those).

You may workaround the L3 issue that way, but hit many new bugs/problems by introducing multiple NUMA nodes, _especially_ on Windows VMs, because that OS has crappy NUMA handling and multitude of bugs related to it - which was one of the major reasons why even Zen2 Threadrippers are now single NUMA node (e.g. https://www.servethehome.com/wp-content/uploads/2019/11/AMD-Ryzen-Threadripper-3960X-Topology.png ).

The host CPU architecture should be replicated as closely as possible on the VM and for Zen2 CPUs with 4 cores per CCX, _this already works perfectly_ - there are no problems on 3300X/3700(X)/3800X/3950X/3970X/3990X.

There is, unfortunately, no way to customize/specify the "disabled" CPU cores in QEMU, and therefore no way to emulate 1 NUMA node + L3 cache per 2/3 cores - only to passthrough the cache config from host, which is unfortunately not done correctly for CPUs with disabled cores (but again, works perfectly for CPUs with all 4 cores enabled per CCX).

lscpu:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          24
On-line CPU(s) list:             0-23
Thread(s) per core:              2
Core(s) per socket:              12
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      23
Model:                           113
Model name:                      AMD Ryzen 9 3900X 12-Core Processor
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         2972.127
CPU max MHz:                     3800.0000
CPU min MHz:                     2200.0000
BogoMIPS:                        7602.55
Virtualization:                  AMD-V
L1d cache:                       384 KiB
L1i cache:                       384 KiB
L2 cache:                        6 MiB
L3 cache:                        64 MiB
NUMA node0 CPU(s):               0-23
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, IBPB conditional, STIBP conditional, RSB filling
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonsto
                                 p_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a mi
                                 salignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall fsgsbase b
                                 mi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru
                                  wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca


But the important thing has already been posted here in previous comments - notice the skipped core ids belonging to the disabled cores:

virsh capabilities | grep "cpu id":
<cpu id='0' socket_id='0' core_id='0' siblings='0,12'/>
<cpu id='1' socket_id='0' core_id='1' siblings='1,13'/>
<cpu id='2' socket_id='0' core_id='2' siblings='2,14'/>
<cpu id='3' socket_id='0' core_id='4' siblings='3,15'/>
<cpu id='4' socket_id='0' core_id='5' siblings='4,16'/>
<cpu id='5' socket_id='0' core_id='6' siblings='5,17'/>
<cpu id='6' socket_id='0' core_id='8' siblings='6,18'/>
<cpu id='7' socket_id='0' core_id='9' siblings='7,19'/>
<cpu id='8' socket_id='0' core_id='10' siblings='8,20'/>
<cpu id='9' socket_id='0' core_id='12' siblings='9,21'/>
<cpu id='10' socket_id='0' core_id='13' siblings='10,22'/>
<cpu id='11' socket_id='0' core_id='14' siblings='11,23'/>
<cpu id='12' socket_id='0' core_id='0' siblings='0,12'/>
<cpu id='13' socket_id='0' core_id='1' siblings='1,13'/>
<cpu id='14' socket_id='0' core_id='2' siblings='2,14'/>
<cpu id='15' socket_id='0' core_id='4' siblings='3,15'/>
<cpu id='16' socket_id='0' core_id='5' siblings='4,16'/>
<cpu id='17' socket_id='0' core_id='6' siblings='5,17'/>
<cpu id='18' socket_id='0' core_id='8' siblings='6,18'/>
<cpu id='19' socket_id='0' core_id='9' siblings='7,19'/>
<cpu id='20' socket_id='0' core_id='10' siblings='8,20'/>
<cpu id='21' socket_id='0' core_id='12' siblings='9,21'/>
<cpu id='22' socket_id='0' core_id='13' siblings='10,22'/>
<cpu id='23' socket_id='0' core_id='14' siblings='11,23'/>

Damir:
Hm, must be some misconfiguration, then. My config for Linux VMs to utilize 3 out of the 4 CCXs. Important parts of the libvirt domain XML:

  <vcpu placement="static">24</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="3"/>
    <vcpupin vcpu="1" cpuset="15"/>
    <vcpupin vcpu="2" cpuset="4"/>
    <vcpupin vcpu="3" cpuset="16"/>
    <vcpupin vcpu="4" cpuset="5"/>
    <vcpupin vcpu="5" cpuset="17"/>
    <vcpupin vcpu="6" cpuset="0,12"/>
    <vcpupin vcpu="7" cpuset="0,12"/>
    <vcpupin vcpu="8" cpuset="6"/>
    <vcpupin vcpu="9" cpuset="18"/>
    <vcpupin vcpu="10" cpuset="7"/>
    <vcpupin vcpu="11" cpuset="19"/>
    <vcpupin vcpu="12" cpuset="8"/>
    <vcpupin vcpu="13" cpuset="20"/>
    <vcpupin vcpu="14" cpuset="0,12"/>
    <vcpupin vcpu="15" cpuset="0,12"/>
    <vcpupin vcpu="16" cpuset="9"/>
    <vcpupin vcpu="17" cpuset="21"/>
    <vcpupin vcpu="18" cpuset="10"/>
    <vcpupin vcpu="19" cpuset="22"/>
    <vcpupin vcpu="20" cpuset="11"/>
    <vcpupin vcpu="21" cpuset="23"/>
    <vcpupin vcpu="22" cpuset="0,12"/>
    <vcpupin vcpu="23" cpuset="0,12"/>
    <emulatorpin cpuset="1,13"/>
    <iothreadpin iothread="1" cpuset="2,14"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/ccxtest-clone_VARS.fd</nvram>
  </os>
.
.
.
  <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="host,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,host-cache-info=on,-amd-stibp"/>
  </qemu:commandline>

The CPUs with cpuset="0,12" are disabled once booted. The host-cache-info=on is the part that makes sure that the cache config is passed to the VM (but unfortunately does not take disabled cores into account, which results in incorrect config). The qemu:commandline is added because I need to add -amd-stibp, otherwise I wouldn't be able to boot. This overrides most parts in the <cpu> XML part.

"The CPUs with cpuset="0,12" are disabled once booted. The host-cache-info=on is the part that makes sure that the cache config is passed to the VM (but unfortunately does not take disabled cores into account, which results in incorrect config). The qemu:commandline is added because I need to add -amd-stibp, otherwise I wouldn't be able to boot. This overrides most parts in the <cpu> XML part."

Is there a XML equivalent for host-cache-info=on ?

Will that work with model EPYC-IBPB as well?

Sieger, I am not an expert on XML. So, I dont know. Qemu probably cannot handle disabled cores. I am still trying to learn more about this problem. 

With regard to Jan's comment earlier and the virsh capabilities listing the cores and siblings, also note the following lines from virsh capabilities for a 3900X CPU:

    <cache>
      <bank id='0' level='3' type='both' size='16' unit='MiB' cpus='0-2,12-14'/>
      <bank id='1' level='3' type='both' size='16' unit='MiB' cpus='3-5,15-17'/>
      <bank id='2' level='3' type='both' size='16' unit='MiB' cpus='6-8,18-20'/>
      <bank id='3' level='3' type='both' size='16' unit='MiB' cpus='9-11,21-23'/>
    </cache>

virsh capabilities is perfectly able to identify the L3 cache structure and associate the right cpus. It would be ideal to just use the above output inside the libvirt domain configuration to "manually" define the L3 cache, or something to that effect on the qemu command line.

Users could then decide to pin only part of the cpus, usually a multiple of 6 (in the case of the 3900X) to align with the CCX.

I'm now on kernel 5.6.11 and QEMU v5.0.0.r533.gdebe78ce14-1 (from Arch Linux AUR qemu-git), running q35-5.1. I will try the host-passthrough with host-cache-info=on option Jan posted. Question - is host-cache-info=on the same as <cache mode="passthrough"/> under <cpu mode=host-passthrough...?

<cache mode="passthrough"/>

adds "host-cache-info=on,l3-cache=off"

to the qemu -cpu args

I believe l3-cache=off is useless with host-cache-info=on

So <cache mode="passthrough"/> should do what you want.

Thanks Jan. I had some new hardware/software issues combined with the QEMU 5.0.. issues that had my Windows VM crash after some minutes.

I totally overlooked the following:
    <vcpupin vcpu="6" cpuset="0,12"/>
    <vcpupin vcpu="7" cpuset="0,12"/>

So I guess you posted to answer to this: https://www.reddit.com/r/VFIO/comments/erwzrg/think_i_found_a_workaround_to_get_l3_cache_shared/

As it's late, I'll try tomorrow. Sorry for all the confusion but I had a real tough time with this Ryzen build.

Jan, I tried your suggestion but it didn't make a difference. Here is my current setup:

h/w: AMD Ryzen 9 3900X
kernel: 5.4
QEMU: 5.0.0-6
Chipset selection: Q35-5.0

Configuration: host-passthrough, cache enabled

Use CoreInfo.exe inside Windows. The problem is this:

Logical Processor to Cache Map:
**---------------------- Data Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64
**---------------------- Instruction Cache 0, Level 1, 32 KB, Assoc 8, LineSize 64
**---------------------- Unified Cache 0, Level 2, 512 KB, Assoc 8, LineSize 64
********---------------- Unified Cache 1, Level 3, 16 MB, Assoc 16, LineSize 64

The last line above should be as follows:

******------------------ Unified Cache 0, Level 3, 16 MB, Assoc 16, LineSize 64

The cache is supposed to be associated with 3 cores a 2 threads in group 0. Yet it shows 8 (2x4) vcpus inside a cache that is associated with the next group.

In total, I always get 3 L3 caches instead of 4 L4 caches for my 12 cores / 24 threads. Also see my next post.


This is the CPU cache layout as shown by lscpu -a -e

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ
  0    0      0    0 0:0:0:0          yes 3800.0000 2200.0000
  1    0      0    1 1:1:1:0          yes 3800.0000 2200.0000
  2    0      0    2 2:2:2:0          yes 3800.0000 2200.0000
  3    0      0    3 3:3:3:1          yes 3800.0000 2200.0000
  4    0      0    4 4:4:4:1          yes 3800.0000 2200.0000
  5    0      0    5 5:5:5:1          yes 3800.0000 2200.0000
  6    0      0    6 6:6:6:2          yes 3800.0000 2200.0000
  7    0      0    7 7:7:7:2          yes 3800.0000 2200.0000
  8    0      0    8 8:8:8:2          yes 3800.0000 2200.0000
  9    0      0    9 9:9:9:3          yes 3800.0000 2200.0000
 10    0      0   10 10:10:10:3       yes 3800.0000 2200.0000
 11    0      0   11 11:11:11:3       yes 3800.0000 2200.0000
 12    0      0    0 0:0:0:0          yes 3800.0000 2200.0000
 13    0      0    1 1:1:1:0          yes 3800.0000 2200.0000
 14    0      0    2 2:2:2:0          yes 3800.0000 2200.0000
 15    0      0    3 3:3:3:1          yes 3800.0000 2200.0000
 16    0      0    4 4:4:4:1          yes 3800.0000 2200.0000
 17    0      0    5 5:5:5:1          yes 3800.0000 2200.0000
 18    0      0    6 6:6:6:2          yes 3800.0000 2200.0000
 19    0      0    7 7:7:7:2          yes 3800.0000 2200.0000
 20    0      0    8 8:8:8:2          yes 3800.0000 2200.0000
 21    0      0    9 9:9:9:3          yes 3800.0000 2200.0000
 22    0      0   10 10:10:10:3       yes 3800.0000 2200.0000
 23    0      0   11 11:11:11:3       yes 3800.0000 2200.0000

I was trying to allocate cache using the cachetune feature in libvirt, but it turns out to be either misleading or much too complicated to be usable. Here is what I tried:

  <vcpu placement="static">24</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="12"/>
    <vcpupin vcpu="2" cpuset="1"/>
    <vcpupin vcpu="3" cpuset="13"/>
    <vcpupin vcpu="4" cpuset="2"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="6" cpuset="3"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <vcpupin vcpu="8" cpuset="4"/>
    <vcpupin vcpu="9" cpuset="16"/>
    <vcpupin vcpu="10" cpuset="5"/>
    <vcpupin vcpu="11" cpuset="17"/>
    <vcpupin vcpu="12" cpuset="6"/>
    <vcpupin vcpu="13" cpuset="18"/>
    <vcpupin vcpu="14" cpuset="7"/>
    <vcpupin vcpu="15" cpuset="19"/>
    <vcpupin vcpu="16" cpuset="8"/>
    <vcpupin vcpu="17" cpuset="20"/>
    <vcpupin vcpu="18" cpuset="9"/>
    <vcpupin vcpu="19" cpuset="21"/>
    <vcpupin vcpu="20" cpuset="10"/>
    <vcpupin vcpu="21" cpuset="22"/>
    <vcpupin vcpu="22" cpuset="11"/>
    <vcpupin vcpu="23" cpuset="23"/>
    <cachetune vcpus="0-2,12-14">
      <cache id="0" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="0-2,12-14"/>
    </cachetune>
    <cachetune vcpus="3-5,15-17">
      <cache id="1" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="3-5,15-17"/>
    </cachetune>
    <cachetune vcpus="6-8,18-20">
      <cache id="2" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="6-8,18-20"/>
    </cachetune>
    <cachetune vcpus="9-11,21-23">
      <cache id="3" level="3" type="both" size="16" unit="MiB"/>
      <monitor level="3" vcpus="9-11,21-23"/>
    </cachetune>
  </cputune>

Unfortunately it gives the following error when I try to start the VM:

Error starting domain: internal error: Missing or inconsistent resctrl info for memory bandwidth allocation

I have resctrl mounted like this:

mount -t resctrl resctrl /sys/fs/resctrl

This error leads to the following description on how to allocate memory bandwith: https://software.intel.com/content/www/us/en/develop/articles/use-intel-resource-director-technology-to-allocate-memory-bandwidth.html

I think this is over the top and perhaps I'm trying the wrong approach. All I can say is that every suggestion I've seen and tried so far has led me to one conclusion: QEMU does NOT support the L3 cache layout of the new ZEN 2 arch CPUs such as the Ryzen 9 3900X.

h-sieger,
that is a misunderstanding, read my comment carefully again:
"A workaround for Linux VMs is to disable CPUs (and setting their number/pinnings accordingly, e.g. every 4th (and 3rd for 3100) core is going to be 'dummy' and disabled system-wide) by e.g. echo 0 > /sys/devices/system/cpu/cpu3/online

No good workaround for Windows VMs exists, as far as I know - the best you can do is setting affinity to specific process(es) and avoid the 'dummy' CPUs, but I am not aware of any possibility to disable specific CPUs (only limiting the overall number)."

I do NOT have a fix - only a very ugly workaround for Linux guests only - I cannot fix the cache layout, but on Linux, I can get around that by adding dummy CPUs that I then disable in the guest during startup, so they are not used - effectively making sure that only the correct 6 vCPUs / 3 cores are used. On Windows, you cannot do that, AFAIK.

Thanks for clarifying, Jan.

In the meantime I tried a number of so-called solutions published on Reddit and other places, none of which seems to work.

So if I understand it correctly, there is currently no solution to the incorrect l3 cache layout for Zen architecture CPUs. At best a workaround for Linux guests.

I hope somebody is looking into that.

Thanks for clarifying, Jan.

In the meantime I tried a number of so-called solutions published on Reddit and other places, none of which seems to work.

So if I understand it correctly, there is currently no solution to the incorrect l3 cache layout for Zen architecture CPUs. At best a workaround for Linux guests.

I hope somebody is looking into that.

The problem is caused by the fact that with Ryzen CPUs with disabled cores, the APIC IDs are not sequential on host - in order for cache topology to be configured properly, there is a 'hole' in APIC ID and core ID numbering (I have added full output of cpuid for my 3900X). Unfortunately, adding holes to the numbering is the only way to achieve what is needed for 3 cores per CCX as CPUID Fn8000_001D_EAX NumSharingCache parameter rounds to  powers of two (for Ryzen 3100 with 2 cores per CCX, lowering NumSharingCache should also work, correctly setting the L3 cache cores with their IDs still being sequential).

A small hack in x86_apicid_from_topo_ids() in include/hw/i386/topology.h can introduce a correct numbering (at least if you do not have epyc set as your cpu, then _epyc variant of the functions are used). But to fix this properly will probably require some thought - maybe introduce the ability to assign APIC IDs directly somehow? Or the ability to specify the 'holes' somehow in the -smt param, or maybe -cpu host,topoext=on should do this automatically? I don't know...

e.g. For 3 core per CCX CPUs, to fix this, at include/hw/i386/topology.h:220 change:

(topo_ids->core_id << apicid_core_offset(topo_info)) |

to

((topo_ids->core_id + (topo_ids->core_id / 3)) << apicid_core_offset(topo_info)) |


The cache topology is now correct (-cpu host,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,host-cache-info=on -smp 18,sockets=1,dies=1,cores=9,threads=2), even in Windows:

Logical Processor to Cache Map:
**----------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
******------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--**--------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
----**------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Unified Cache       4, Level 2,  512 KB, Assoc   8, LineSize  64
------******------  Unified Cache       5, Level 3,   16 MB, Assoc  16, LineSize  64
--------**--------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Unified Cache       6, Level 2,  512 KB, Assoc   8, LineSize  64
----------**------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Unified Cache       7, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
------------******  Unified Cache       9, Level 3,   16 MB, Assoc  16, LineSize  64



@Jan: this coreinfo output looks good.

I finally managed to get the core /cache alignment right, I believe:

  <vcpu placement="static" current="24">32</vcpu>
  <vcpus>
    <vcpu id="0" enabled="yes" hotpluggable="no"/>
    <vcpu id="1" enabled="yes" hotpluggable="yes"/>
    <vcpu id="2" enabled="yes" hotpluggable="yes"/>
    <vcpu id="3" enabled="yes" hotpluggable="yes"/>
    <vcpu id="4" enabled="yes" hotpluggable="yes"/>
    <vcpu id="5" enabled="yes" hotpluggable="yes"/>
    <vcpu id="6" enabled="no" hotpluggable="yes"/>
    <vcpu id="7" enabled="no" hotpluggable="yes"/>
    <vcpu id="8" enabled="yes" hotpluggable="yes"/>
    <vcpu id="9" enabled="yes" hotpluggable="yes"/>
    <vcpu id="10" enabled="yes" hotpluggable="yes"/>
    <vcpu id="11" enabled="yes" hotpluggable="yes"/>
    <vcpu id="12" enabled="yes" hotpluggable="yes"/>
    <vcpu id="13" enabled="yes" hotpluggable="yes"/>
    <vcpu id="14" enabled="no" hotpluggable="yes"/>
    <vcpu id="15" enabled="no" hotpluggable="yes"/>
    <vcpu id="16" enabled="yes" hotpluggable="yes"/>
    <vcpu id="17" enabled="yes" hotpluggable="yes"/>
    <vcpu id="18" enabled="yes" hotpluggable="yes"/>
    <vcpu id="19" enabled="yes" hotpluggable="yes"/>
    <vcpu id="20" enabled="yes" hotpluggable="yes"/>
    <vcpu id="21" enabled="yes" hotpluggable="yes"/>
    <vcpu id="22" enabled="no" hotpluggable="yes"/>
    <vcpu id="23" enabled="no" hotpluggable="yes"/>
    <vcpu id="24" enabled="yes" hotpluggable="yes"/>
    <vcpu id="25" enabled="yes" hotpluggable="yes"/>
    <vcpu id="26" enabled="yes" hotpluggable="yes"/>
    <vcpu id="27" enabled="yes" hotpluggable="yes"/>
    <vcpu id="28" enabled="yes" hotpluggable="yes"/>
    <vcpu id="29" enabled="yes" hotpluggable="yes"/>
    <vcpu id="30" enabled="no" hotpluggable="yes"/>
    <vcpu id="31" enabled="no" hotpluggable="yes"/>
  </vcpus>
  <cputune>
    <vcpupin vcpu="0" cpuset="0"/>
    <vcpupin vcpu="1" cpuset="12"/>
    <vcpupin vcpu="2" cpuset="1"/>
    <vcpupin vcpu="3" cpuset="13"/>
    <vcpupin vcpu="4" cpuset="2"/>
    <vcpupin vcpu="5" cpuset="14"/>
    <vcpupin vcpu="8" cpuset="3"/>
    <vcpupin vcpu="9" cpuset="15"/>
    <vcpupin vcpu="10" cpuset="4"/>
    <vcpupin vcpu="11" cpuset="16"/>
    <vcpupin vcpu="12" cpuset="5"/>
    <vcpupin vcpu="13" cpuset="17"/>
    <vcpupin vcpu="16" cpuset="6"/>
    <vcpupin vcpu="17" cpuset="18"/>
    <vcpupin vcpu="18" cpuset="7"/>
    <vcpupin vcpu="19" cpuset="19"/>
    <vcpupin vcpu="20" cpuset="8"/>
    <vcpupin vcpu="21" cpuset="20"/>
    <vcpupin vcpu="24" cpuset="9"/>
    <vcpupin vcpu="25" cpuset="21"/>
    <vcpupin vcpu="26" cpuset="10"/>
    <vcpupin vcpu="27" cpuset="22"/>
    <vcpupin vcpu="28" cpuset="11"/>
    <vcpupin vcpu="29" cpuset="23"/>
  </cputune>

...
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" dies="1" cores="16" threads="2"/>
    <cache mode="passthrough"/>


The Windows Coreinfo output is this:

Logical to Physical Processor Map:
**----------------  Physical Processor 0 (Hyperthreaded)
--**--------------  Physical Processor 1 (Hyperthreaded)
----**------------  Physical Processor 2 (Hyperthreaded)
------**----------  Physical Processor 3 (Hyperthreaded)
--------**--------  Physical Processor 4 (Hyperthreaded)
----------**------  Physical Processor 5 (Hyperthreaded)
------------**----  Physical Processor 6 (Hyperthreaded)
--------------**--  Physical Processor 7 (Hyperthreaded)
----------------**  Physical Processor 8 (Hyperthreaded)

Logical Processor to Socket Map:
******************  Socket 0

Logical Processor to NUMA Node Map:
******************  NUMA Node 0

No NUMA nodes.

Logical Processor to Cache Map:
**----------------  Data Cache          0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Instruction Cache   0, Level 1,   32 KB, Assoc   8, LineSize  64
**----------------  Unified Cache       0, Level 2,  512 KB, Assoc   8, LineSize  64
******------------  Unified Cache       1, Level 3,   16 MB, Assoc  16, LineSize  64
--**--------------  Data Cache          1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Instruction Cache   1, Level 1,   32 KB, Assoc   8, LineSize  64
--**--------------  Unified Cache       2, Level 2,  512 KB, Assoc   8, LineSize  64
----**------------  Data Cache          2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Instruction Cache   2, Level 1,   32 KB, Assoc   8, LineSize  64
----**------------  Unified Cache       3, Level 2,  512 KB, Assoc   8, LineSize  64
------**----------  Data Cache          3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Instruction Cache   3, Level 1,   32 KB, Assoc   8, LineSize  64
------**----------  Unified Cache       4, Level 2,  512 KB, Assoc   8, LineSize  64
------******------  Unified Cache       5, Level 3,   16 MB, Assoc  16, LineSize  64
--------**--------  Data Cache          4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Instruction Cache   4, Level 1,   32 KB, Assoc   8, LineSize  64
--------**--------  Unified Cache       6, Level 2,  512 KB, Assoc   8, LineSize  64
----------**------  Data Cache          5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Instruction Cache   5, Level 1,   32 KB, Assoc   8, LineSize  64
----------**------  Unified Cache       7, Level 2,  512 KB, Assoc   8, LineSize  64
------------**----  Data Cache          6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Instruction Cache   6, Level 1,   32 KB, Assoc   8, LineSize  64
------------**----  Unified Cache       8, Level 2,  512 KB, Assoc   8, LineSize  64
------------******  Unified Cache       9, Level 3,   16 MB, Assoc  16, LineSize  64
--------------**--  Data Cache          7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--  Instruction Cache   7, Level 1,   32 KB, Assoc   8, LineSize  64
--------------**--  Unified Cache      10, Level 2,  512 KB, Assoc   8, LineSize  64
----------------**  Data Cache          8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**  Instruction Cache   8, Level 1,   32 KB, Assoc   8, LineSize  64
----------------**  Unified Cache      11, Level 2,  512 KB, Assoc   8, LineSize  64

Logical Processor to Group Map:
******************  Group 0


Haven't been able to test if it performs as expected. Need to do that.

Of course it would be great if QEMU was patched to recognize correct CCX alignment as I'm not sure if and what will be the penalty of this weird setup.

Yep, I read the Reddit thread, had no idea this was possible.

Still, both solutions are ugly workarounds and it would be nice to fix this properly. But at least I don't have to patch and compile QEMU on my own anymore.

h-sieger,
Your XML gave me very significant performance gains.
Is there any way to do this with more than 24 assigned cores?


@sanjaybmd

I'm glad to read that it worked for you. In fact, since I posted the XML I didn't have the time to do benchmarking, now my motherboard is dead and I have to wait for repair/replacement.

Do you have any data to quantify the performance gain?

As to the number of cores, you will notice that my 3900X has only 12 physical cores, that is 24 threads. Yet I assigned 32 vcpus in total. 8 of them are disabled. This is to align the vcpus to the actual CCX topology of 3 cores per CCX.

QEMU thinks the cores per CCX should be a multiple of 2, e.g. 2, 4, etc. cores. So I assign 4 cores = 8 vcpus, and disable 2 vcpus to simulate the actual topology.

If your CPU has more cores, you could scale it up. Be aware that the 3950X should not have this issue as it has 4 cores per CCX, if I remember correctly.

Note: I took this idea from a Reddit post (see link somewhere above).

h-sieger, 
I did some testing with geekbench 5:

baseline multicore score = 12733
https://browser.geekbench.com/v5/cpu/3069626

score with <cache="passthrough"> option = 12775
https://browser.geekbench.com/v5/cpu/3069415

best score with your xml above = 16960
https://browser.geekbench.com/v5/cpu/3066003

I'm running a 3960x and it is 3 cores per CCX so your xml above works well. I'm just now learning about all this so I'm still trying to figure out how to modify your xml to assign more cores. Anyway, I'm getting better performance out of my Windows 10 VM now assigning 24 vcpu as opposed to the 32 that I was assigning before!
By the way, I tried to email you directly because I'm not sure this is appropriate discussion for this bug report but I could not create an account on your website (captcha was malfunctioning). Hope you you can fix that soon. 


Sanjay,

You can just increase the number of vcpus, such as:

<vcpu placement="static" current="48">64</vcpu>

then continue to define the vcpus:

    <vcpu id="32" enabled="yes" hotpluggable="yes"/>
    <vcpu id="33" enabled="yes" hotpluggable="yes"/>
    <vcpu id="34" enabled="yes" hotpluggable="yes"/>
    <vcpu id="35" enabled="yes" hotpluggable="yes"/>
    <vcpu id="36" enabled="yes" hotpluggable="yes"/>
    <vcpu id="37" enabled="yes" hotpluggable="yes"/>
    <vcpu id="38" enabled="no" hotpluggable="yes"/>
    <vcpu id="39" enabled="no" hotpluggable="yes"/>
    <vcpu id="40" enabled="yes" hotpluggable="yes"/>
    <vcpu id="41" enabled="yes" hotpluggable="yes"/>
    <vcpu id="42" enabled="yes" hotpluggable="yes"/>
    <vcpu id="43" enabled="yes" hotpluggable="yes"/>
    <vcpu id="44" enabled="yes" hotpluggable="yes"/>
    <vcpu id="45" enabled="yes" hotpluggable="yes"/>
    <vcpu id="46" enabled="no" hotpluggable="yes"/>
    <vcpu id="47" enabled="no" hotpluggable="yes"/>
    <vcpu id="48" enabled="yes" hotpluggable="yes"/>
    <vcpu id="49" enabled="yes" hotpluggable="yes"/>
    <vcpu id="50" enabled="yes" hotpluggable="yes"/>
    <vcpu id="51" enabled="yes" hotpluggable="yes"/>
    <vcpu id="52" enabled="yes" hotpluggable="yes"/>
    <vcpu id="53" enabled="yes" hotpluggable="yes"/>
    <vcpu id="54" enabled="no" hotpluggable="yes"/>
    <vcpu id="55" enabled="no" hotpluggable="yes"/>
    <vcpu id="56" enabled="yes" hotpluggable="yes"/>
    <vcpu id="57" enabled="yes" hotpluggable="yes"/>
    <vcpu id="58" enabled="yes" hotpluggable="yes"/>
    <vcpu id="59" enabled="yes" hotpluggable="yes"/>
    <vcpu id="60" enabled="yes" hotpluggable="yes"/>
    <vcpu id="61" enabled="yes" hotpluggable="yes"/>
    <vcpu id="62" enabled="no" hotpluggable="yes"/>
    <vcpu id="63" enabled="no" hotpluggable="yes"/>

(6x enabled=yes, then 2x enabled=no.)

You will get more vcpu ids than you have threads, but since you disable 16 out of 64, you will have 48 active.

vcpupin should continue as follows:

    <vcpupin vcpu="32" cpuset="24"/>
    <vcpupin vcpu="33" cpuset="36"/>
    <vcpupin vcpu="34" cpuset="25"/>
    <vcpupin vcpu="35" cpuset="37"/>
    <vcpupin vcpu="36" cpuset="26"/>
    <vcpupin vcpu="37" cpuset="38"/>
    <vcpupin vcpu="40" cpuset="27"/>
    <vcpupin vcpu="41" cpuset="39"/>
    <vcpupin vcpu="42" cpuset="28"/>
    <vcpupin vcpu="43" cpuset="40"/>
    <vcpupin vcpu="44" cpuset="29"/>
    <vcpupin vcpu="45" cpuset="41"/>
    <vcpupin vcpu="48" cpuset="30"/>
    <vcpupin vcpu="49" cpuset="42"/>
    <vcpupin vcpu="50" cpuset="31"/>
    <vcpupin vcpu="51" cpuset="43"/>
    <vcpupin vcpu="52" cpuset="32"/>
    <vcpupin vcpu="53" cpuset="44"/>
    <vcpupin vcpu="56" cpuset="33"/>
    <vcpupin vcpu="57" cpuset="45"/>
    <vcpupin vcpu="58" cpuset="34"/>
    <vcpupin vcpu="59" cpuset="46"/>
    <vcpupin vcpu="60" cpuset="35"/>
    <vcpupin vcpu="61" cpuset="47"/>

This is if you pin all vcpus to the VM, which may not be the best thing to do. The maximum number of vcpus you can pin on a Threadripper 3960X are 48.

The QEMU project is currently considering to move its bug tracking to
another system. For this we need to know which bugs are still valid
and which could be closed already. Thus we are setting older bugs to
"Incomplete" now.

If you still think this bug report here is valid, then please switch
the state back to "New" within the next 60 days, otherwise this report
will be marked as "Expired". Or please mark it as "Fix Released" if
the problem has been solved with a newer version of QEMU already.

Thank you and sorry for the inconvenience.


[Expired for QEMU because there has been no activity for 60 days.]