diff options
Diffstat (limited to '')
| -rw-r--r-- | results/classifier/118/all/1871842 | 1360 |
1 files changed, 1360 insertions, 0 deletions
diff --git a/results/classifier/118/all/1871842 b/results/classifier/118/all/1871842 new file mode 100644 index 00000000..cdb09b08 --- /dev/null +++ b/results/classifier/118/all/1871842 @@ -0,0 +1,1360 @@ +register: 0.973 +permissions: 0.971 +debug: 0.968 +semantic: 0.967 +assembly: 0.963 +peripherals: 0.962 +device: 0.961 +graphic: 0.961 +performance: 0.961 +boot: 0.959 +network: 0.958 +architecture: 0.956 +kernel: 0.951 +PID: 0.950 +hypervisor: 0.942 +socket: 0.938 +arm: 0.936 +mistranslation: 0.935 +user-level: 0.935 +risc-v: 0.927 +vnc: 0.923 +VMM: 0.917 +ppc: 0.915 +files: 0.914 +TCG: 0.881 +x86: 0.880 +KVM: 0.874 +virtual: 0.852 +i386: 0.772 + +AMD CPUID leaf 0x8000'0008 reported number of cores inconsistent with ACPI.MADT + +Setup: +CPU: AMD EPYC-v2 or host's EPYC cpu +Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +qemu version: self build +git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' + +Cmdline: +qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 + +Issue: +We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] + +The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +The lower four bits of ECX are the [NC] field and all set. + +When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +Note, ECX is zero. Indicating that this is no SMP capabale CPU. + +I'm debugging it using my local machine and the QEMU provided EPYC-v2 CPU model and it is reproducible there as well and reports: EAX: 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 + +I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. + +The following table shows my findings with the -smp option: +Option | Qemu guest observed ECX value +-smp 4 | 0x0 +-smp 4,cores=4 | 0x3 +-smp 4,cores=2,thread=2 | 0x3 +-smp 4,cores=4,threads=2 | QEMU boot error: topology false. + +Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? + +Querying 0x8000'0008 on the physical processor results in different reports than quering QEMU's model as does it with -enable-kvm -cpu host. + +Furthermore, the ACPI.MADT shows 4 local APICs to be present while the CPU leave reports a single core processor. + +This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the wrong number. + + +Please let me know, if you need more information from my side. + + +[0] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d71328e2b879cbbd19/src/kern/ia32/kernel_thread-ia32.cpp#L109 +[1] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d71328e2b879cbbd19/src/kern/ia32/cpu-ia32.cpp#L1120 +[2] https://github.com/qemu/qemu/blob/f2a8261110c32c4dccd84e774d8dd7a0524e00fb/target/i386/cpu.c#L5835 +[3] https://www.amd.com/system/files/TechDocs/24594.pdf + +On Thu, 09 Apr 2020 12:58:11 -0000 +Philipp Eppelt <email address hidden> wrote: + +> Public bug reported: +> +> Setup: +> CPU: AMD EPYC-v2 or host's EPYC cpu +> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +> qemu version: self build +> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +> +> Cmdline: +> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +> +> Issue: +> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +> +> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +> The lower four bits of ECX are the [NC] field and all set. +> +> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +> +> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +> CPU model and it is reproducible there as well and reports: EAX: +> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +> +> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +> +> The following table shows my findings with the -smp option: +> Option | Qemu guest observed ECX value +> -smp 4 | 0x0 +> -smp 4,cores=4 | 0x3 +> -smp 4,cores=2,thread=2 | 0x3 +> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +> +> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +I'd say X corresponds to number of logical CPUs. +Depending on presence of other options these are distributed among them in magical manner +(see pc_smp_parse() for starters) + +> Querying 0x8000'0008 on the physical processor results in different +> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +> +> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +> CPU leave reports a single core processor. +it matches -smp X as it should be. + +> +> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +> wrong number. +CCed author of recent epyc patches who might know how AMD should work better than me. + +> +> Please let me know, if you need more information from my side. +> +> +> [0] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d71328e2b879cbbd19/src/kern/ia32/kernel_thread-ia32.cpp#L109 +> [1] https://github.com/kernkonzept/fiasco/blob/522ccc5f29ab120213cf02d71328e2b879cbbd19/src/kern/ia32/cpu-ia32.cpp#L1120 +> [2] https://github.com/qemu/qemu/blob/f2a8261110c32c4dccd84e774d8dd7a0524e00fb/target/i386/cpu.c#L5835 +> [3] https://www.amd.com/system/files/TechDocs/24594.pdf +> +> ** Affects: qemu +> Importance: Undecided +> Status: New +> + + + + + +On 4/9/20 9:00 AM, Igor Mammedov wrote: +> On Thu, 09 Apr 2020 12:58:11 -0000 +> Philipp Eppelt <email address hidden> wrote: +> +>> Public bug reported: +>> +>> Setup: +>> CPU: AMD EPYC-v2 or host's EPYC cpu +>> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +>> qemu version: self build +>> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +>> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +>> +>> Cmdline: +>> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +>> +>> Issue: +>> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +>> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +>> +>> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +>> The lower four bits of ECX are the [NC] field and all set. +>> +>> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +>> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +>> +>> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +>> CPU model and it is reproducible there as well and reports: EAX: +>> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +>> +>> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +>> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +>> +>> The following table shows my findings with the -smp option: +>> Option | Qemu guest observed ECX value +>> -smp 4 | 0x0 +>> -smp 4,cores=4 | 0x3 +>> -smp 4,cores=2,thread=2 | 0x3 +>> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +>> +>> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +>> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +> I'd say X corresponds to number of logical CPUs. +> Depending on presence of other options these are distributed among them in magical manner +> (see pc_smp_parse() for starters) +> +>> Querying 0x8000'0008 on the physical processor results in different +>> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +>> +>> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +>> CPU leave reports a single core processor. +> it matches -smp X as it should be. +> +>> +>> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +>> wrong number. +> CCed author of recent epyc patches who might know how AMD should work better than me. + +Hmm.. Interesting.. Not sure why this did not come up during my testing. +Probably this cpuid information is not widely used. + +Yes. I am looking at it right now. I see that EPYC model is reporting +wrong. Not sure why -cpu host is reporting wrong. I thought -cpu host gets +the information directly from the host. Will investigate. + + +Philipp, + Can you please check if this patch works for you. + +diff --git a/target/i386/cpu.c b/target/i386/cpu.c +index 90ffc5f..e467fee 100644 +--- a/target/i386/cpu.c ++++ b/target/i386/cpu.c +@@ -5831,10 +5831,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t +index, uint32_t count, + } + *ebx = env->features[FEAT_8000_0008_EBX]; + *ecx = 0; +- *edx = 0; + if (cs->nr_cores * cs->nr_threads > 1) { +- *ecx |= (cs->nr_cores * cs->nr_threads) - 1; ++ unsigned long max_apicids, bits_required; ++ ++ max_apicids = (cs->nr_cores * cs->nr_threads) - 1; ++ if (max_apicids) { ++ /* Find out the number of bits to represent all the +apicids */ ++ bits_required = find_last_bit(&max_apicids, +BITS_PER_BYTE) + 1; ++ *ecx |= bits_required << 12 | max_apicids; ++ } + } ++ *edx = 0; + break; + case 0x8000000A: + if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { + + + +On 4/9/20 9:00 AM, Igor Mammedov wrote: +> On Thu, 09 Apr 2020 12:58:11 -0000 +> Philipp Eppelt <email address hidden> wrote: +> +>> Public bug reported: +>> +>> Setup: +>> CPU: AMD EPYC-v2 or host's EPYC cpu +>> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +>> qemu version: self build +>> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +>> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +>> +>> Cmdline: +>> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +>> +>> Issue: +>> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +>> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +>> +>> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +>> The lower four bits of ECX are the [NC] field and all set. +>> +>> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +>> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +>> +>> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +>> CPU model and it is reproducible there as well and reports: EAX: +>> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +>> +>> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +>> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +>> +>> The following table shows my findings with the -smp option: +>> Option | Qemu guest observed ECX value +>> -smp 4 | 0x0 +>> -smp 4,cores=4 | 0x3 +>> -smp 4,cores=2,thread=2 | 0x3 +>> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +>> +>> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +>> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +> I'd say X corresponds to number of logical CPUs. +> Depending on presence of other options these are distributed among them in magical manner +> (see pc_smp_parse() for starters) +> +>> Querying 0x8000'0008 on the physical processor results in different +>> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +>> +>> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +>> CPU leave reports a single core processor. +> it matches -smp X as it should be. +> +>> +>> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +>> wrong number. +> CCed author of recent epyc patches who might know how AMD should work better than me. +> +>> +>> Please let me know, if you need more information from my side. +>> +>> +>> [0] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fkernel_thread-ia32.cpp%23L109&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=hcFJzLAVQoIh5IN9CP%2F9cUQNOZoBnpRA6FliJur1wzQ%3D&reserved=0 +>> [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fcpu-ia32.cpp%23L1120&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=ANJIbYKbwfq2bDelH%2FRLKnDPIUZc1BwxHspmgxLU7gs%3D&reserved=0 +>> [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fblob%2Ff2a8261110c32c4dccd84e774d8dd7a0524e00fb%2Ftarget%2Fi386%2Fcpu.c%23L5835&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=oj3mv9e5YOzUsfUjXK44gC8LybyWgMKo8JBIrRR%2BmDA%3D&reserved=0 +>> [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%2F24594.pdf&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=7Yr3J9ihlqSqXCXKN5JJNTByO3NGI%2BGMz2EqBF2Y4hw%3D&reserved=0 +>> +>> ** Affects: qemu +>> Importance: Undecided +>> Status: New +>> +> + + +Hi, + +thanks for looking into this so quickly. + +With this patch applied ontop of git commit +f3bac27cc1e303e1860cc55b9b6889ba39dee587 I still have the issue and it +reports the same numbers. I like the new usage of the ApicIdSize field. + + +I looked into the mentioned pc_smp_parse() and had it print the topology +for -smp 4: + +qemu-system-x86_64: warning: cpu topology: sockets (4) , dies (1) , +cores (1) , threads (1) , maxcpus (4), cpus (4) + +and with -smp 4,cores=4: + +qemu-system-x86_64: warning: cpu topology: sockets (1) , dies (1) , +cores (4) , threads (1) , maxcpus (4), cpus (4) + +As far as I understand it, these are the numbers the cpuid:8000'0008 +code relies on: +`cs->nr_cores`, `cs->nr_threads` with `cs` being of type CPUState. + +So I think the issue is rooted with the preferring sockets over cores +when the -smp cmdline option is parsed, as stated in hw/i386/pc.c:729. + +I guess this is the same code for Intel and AMD CPUs alike and this +issue just didn't surface for us on Intel CPUs, as they don't have this +CPUID leaf and we don't look at the topology. + +This seems to boil down to a more careful use of the -smp option on my end. + +Thanks again for looking into this. + +Cheers, +Philipp + + + +On 4/10/20 2:12 AM, Babu Moger wrote: +> Philipp, +> Can you please check if this patch works for you. +> +> diff --git a/target/i386/cpu.c b/target/i386/cpu.c +> index 90ffc5f..e467fee 100644 +> --- a/target/i386/cpu.c +> +++ b/target/i386/cpu.c +> @@ -5831,10 +5831,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t +> index, uint32_t count, +> } +> *ebx = env->features[FEAT_8000_0008_EBX]; +> *ecx = 0; +> - *edx = 0; +> if (cs->nr_cores * cs->nr_threads > 1) { +> - *ecx |= (cs->nr_cores * cs->nr_threads) - 1; +> + unsigned long max_apicids, bits_required; +> + +> + max_apicids = (cs->nr_cores * cs->nr_threads) - 1; +> + if (max_apicids) { +> + /* Find out the number of bits to represent all the +> apicids */ +> + bits_required = find_last_bit(&max_apicids, +> BITS_PER_BYTE) + 1; +> + *ecx |= bits_required << 12 | max_apicids; +> + } +> } +> + *edx = 0; +> break; +> case 0x8000000A: +> if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { +> +> +> On 4/9/20 9:00 AM, Igor Mammedov wrote: +>> On Thu, 09 Apr 2020 12:58:11 -0000 +>> Philipp Eppelt <email address hidden> wrote: +>> +>>> Public bug reported: +>>> +>>> Setup: +>>> CPU: AMD EPYC-v2 or host's EPYC cpu +>>> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +>>> qemu version: self build +>>> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +>>> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +>>> +>>> Cmdline: +>>> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +>>> +>>> Issue: +>>> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +>>> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +>>> +>>> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +>>> The lower four bits of ECX are the [NC] field and all set. +>>> +>>> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +>>> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +>>> +>>> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +>>> CPU model and it is reproducible there as well and reports: EAX: +>>> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +>>> +>>> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +>>> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +>>> +>>> The following table shows my findings with the -smp option: +>>> Option | Qemu guest observed ECX value +>>> -smp 4 | 0x0 +>>> -smp 4,cores=4 | 0x3 +>>> -smp 4,cores=2,thread=2 | 0x3 +>>> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +>>> +>>> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +>>> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +>> I'd say X corresponds to number of logical CPUs. +>> Depending on presence of other options these are distributed among them in magical manner +>> (see pc_smp_parse() for starters) +>> +>>> Querying 0x8000'0008 on the physical processor results in different +>>> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +>>> +>>> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +>>> CPU leave reports a single core processor. +>> it matches -smp X as it should be. +>> +>>> +>>> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +>>> wrong number. +>> CCed author of recent epyc patches who might know how AMD should work better than me. +>> +>>> +>>> Please let me know, if you need more information from my side. +>>> +>>> +>>> [0] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fkernel_thread-ia32.cpp%23L109&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=hcFJzLAVQoIh5IN9CP%2F9cUQNOZoBnpRA6FliJur1wzQ%3D&reserved=0 +>>> [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fcpu-ia32.cpp%23L1120&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=ANJIbYKbwfq2bDelH%2FRLKnDPIUZc1BwxHspmgxLU7gs%3D&reserved=0 +>>> [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fblob%2Ff2a8261110c32c4dccd84e774d8dd7a0524e00fb%2Ftarget%2Fi386%2Fcpu.c%23L5835&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=oj3mv9e5YOzUsfUjXK44gC8LybyWgMKo8JBIrRR%2BmDA%3D&reserved=0 +>>> [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%2F24594.pdf&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=7Yr3J9ihlqSqXCXKN5JJNTByO3NGI%2BGMz2EqBF2Y4hw%3D&reserved=0 +>>> +>>> ** Affects: qemu +>>> Importance: Undecided +>>> Status: New +>>> +>> +> + +-- +<email address hidden> - Tel. 0351-41 883 221 +http://www.kernkonzept.com + +Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. +Geschäftsführer: Dr.-Ing. Michael Hohmuth + + +Hi, + +I have to clarify some things mentioned in my last post: + +I only tested the change with an emulated EPYC-v2 CPU, I cannot test on +a physical EPYC CPU at the moment. However, I doubt that the results +will be different regarding the 0x8000_0008.ECX result. + +The topology information printed is from the EPYC-v2 CPU model. I try to +get access to the machine and have a look if -cpu host affects this +topology. + +So there is still the open question for the -enable-kvm -cpu host -smp 4 +case. Shouldn't in this case the topology of the host CPU be reported? + + +In all emulated-CPU cases it's on the user to define the topology or to +live with the generated one (although I think preferring multi-socket +systems is outdated, but it's likely just the case in my 'world'). + + +Cheers, +Philipp + + +On 4/14/20 10:24 AM, Philipp Eppelt wrote: +> Hi, +> +> thanks for looking into this so quickly. +> +> With this patch applied ontop of git commit +> f3bac27cc1e303e1860cc55b9b6889ba39dee587 I still have the issue and it +> reports the same numbers. I like the new usage of the ApicIdSize field. +> +> +> I looked into the mentioned pc_smp_parse() and had it print the topology +> for -smp 4: +> +> qemu-system-x86_64: warning: cpu topology: sockets (4) , dies (1) , +> cores (1) , threads (1) , maxcpus (4), cpus (4) +> +> and with -smp 4,cores=4: +> +> qemu-system-x86_64: warning: cpu topology: sockets (1) , dies (1) , +> cores (4) , threads (1) , maxcpus (4), cpus (4) +> +> As far as I understand it, these are the numbers the cpuid:8000'0008 +> code relies on: +> `cs->nr_cores`, `cs->nr_threads` with `cs` being of type CPUState. +> +> So I think the issue is rooted with the preferring sockets over cores +> when the -smp cmdline option is parsed, as stated in hw/i386/pc.c:729. +> +> I guess this is the same code for Intel and AMD CPUs alike and this +> issue just didn't surface for us on Intel CPUs, as they don't have this +> CPUID leaf and we don't look at the topology. +> +> This seems to boil down to a more careful use of the -smp option on my end. +> +> Thanks again for looking into this. +> +> Cheers, +> Philipp +> +> +> +> On 4/10/20 2:12 AM, Babu Moger wrote: +>> Philipp, +>> Can you please check if this patch works for you. +>> +>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c +>> index 90ffc5f..e467fee 100644 +>> --- a/target/i386/cpu.c +>> +++ b/target/i386/cpu.c +>> @@ -5831,10 +5831,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t +>> index, uint32_t count, +>> } +>> *ebx = env->features[FEAT_8000_0008_EBX]; +>> *ecx = 0; +>> - *edx = 0; +>> if (cs->nr_cores * cs->nr_threads > 1) { +>> - *ecx |= (cs->nr_cores * cs->nr_threads) - 1; +>> + unsigned long max_apicids, bits_required; +>> + +>> + max_apicids = (cs->nr_cores * cs->nr_threads) - 1; +>> + if (max_apicids) { +>> + /* Find out the number of bits to represent all the +>> apicids */ +>> + bits_required = find_last_bit(&max_apicids, +>> BITS_PER_BYTE) + 1; +>> + *ecx |= bits_required << 12 | max_apicids; +>> + } +>> } +>> + *edx = 0; +>> break; +>> case 0x8000000A: +>> if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { +>> +>> +>> On 4/9/20 9:00 AM, Igor Mammedov wrote: +>>> On Thu, 09 Apr 2020 12:58:11 -0000 +>>> Philipp Eppelt <email address hidden> wrote: +>>> +>>>> Public bug reported: +>>>> +>>>> Setup: +>>>> CPU: AMD EPYC-v2 or host's EPYC cpu +>>>> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +>>>> qemu version: self build +>>>> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +>>>> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +>>>> +>>>> Cmdline: +>>>> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +>>>> +>>>> Issue: +>>>> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +>>>> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +>>>> +>>>> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +>>>> The lower four bits of ECX are the [NC] field and all set. +>>>> +>>>> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +>>>> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +>>>> +>>>> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +>>>> CPU model and it is reproducible there as well and reports: EAX: +>>>> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +>>>> +>>>> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +>>>> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +>>>> +>>>> The following table shows my findings with the -smp option: +>>>> Option | Qemu guest observed ECX value +>>>> -smp 4 | 0x0 +>>>> -smp 4,cores=4 | 0x3 +>>>> -smp 4,cores=2,thread=2 | 0x3 +>>>> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +>>>> +>>>> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +>>>> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +>>> I'd say X corresponds to number of logical CPUs. +>>> Depending on presence of other options these are distributed among them in magical manner +>>> (see pc_smp_parse() for starters) +>>> +>>>> Querying 0x8000'0008 on the physical processor results in different +>>>> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +>>>> +>>>> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +>>>> CPU leave reports a single core processor. +>>> it matches -smp X as it should be. +>>> +>>>> +>>>> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +>>>> wrong number. +>>> CCed author of recent epyc patches who might know how AMD should work better than me. +>>> +>>>> +>>>> Please let me know, if you need more information from my side. +>>>> +>>>> +>>>> [0] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fkernel_thread-ia32.cpp%23L109&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=hcFJzLAVQoIh5IN9CP%2F9cUQNOZoBnpRA6FliJur1wzQ%3D&reserved=0 +>>>> [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fcpu-ia32.cpp%23L1120&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=ANJIbYKbwfq2bDelH%2FRLKnDPIUZc1BwxHspmgxLU7gs%3D&reserved=0 +>>>> [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fblob%2Ff2a8261110c32c4dccd84e774d8dd7a0524e00fb%2Ftarget%2Fi386%2Fcpu.c%23L5835&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=oj3mv9e5YOzUsfUjXK44gC8LybyWgMKo8JBIrRR%2BmDA%3D&reserved=0 +>>>> [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%2F24594.pdf&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=7Yr3J9ihlqSqXCXKN5JJNTByO3NGI%2BGMz2EqBF2Y4hw%3D&reserved=0 +>>>> +>>>> ** Affects: qemu +>>>> Importance: Undecided +>>>> Status: New +>>>> +>>> +>> +> + +-- +<email address hidden> - Tel. 0351-41 883 221 +http://www.kernkonzept.com + +Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. +Geschäftsführer: Dr.-Ing. Michael Hohmuth + + +On Tue, 14 Apr 2020 13:27:34 -0000 +Philipp Eppelt <email address hidden> wrote: + +> Hi, +> +> I have to clarify some things mentioned in my last post: +> +> I only tested the change with an emulated EPYC-v2 CPU, I cannot test on +> a physical EPYC CPU at the moment. However, I doubt that the results +> will be different regarding the 0x8000_0008.ECX result. +> +> The topology information printed is from the EPYC-v2 CPU model. I try to +> get access to the machine and have a look if -cpu host affects this +> topology. +> +> So there is still the open question for the -enable-kvm -cpu host -smp 4 +> case. Shouldn't in this case the topology of the host CPU be reported? +topology was never affected by the choice of -cpu, it's up to users to +define it using -smp the way they prefer. + + +> In all emulated-CPU cases it's on the user to define the topology or to +> live with the generated one (although I think preferring multi-socket +> systems is outdated, but it's likely just the case in my 'world'). +> +> +> Cheers, +> Philipp +> +> +> On 4/14/20 10:24 AM, Philipp Eppelt wrote: +> > Hi, +> > +> > thanks for looking into this so quickly. +> > +> > With this patch applied ontop of git commit +> > f3bac27cc1e303e1860cc55b9b6889ba39dee587 I still have the issue and it +> > reports the same numbers. I like the new usage of the ApicIdSize field. +> > +> > +> > I looked into the mentioned pc_smp_parse() and had it print the topology +> > for -smp 4: +> > +> > qemu-system-x86_64: warning: cpu topology: sockets (4) , dies (1) , +> > cores (1) , threads (1) , maxcpus (4), cpus (4) +> > +> > and with -smp 4,cores=4: +> > +> > qemu-system-x86_64: warning: cpu topology: sockets (1) , dies (1) , +> > cores (4) , threads (1) , maxcpus (4), cpus (4) +> > +> > As far as I understand it, these are the numbers the cpuid:8000'0008 +> > code relies on: +> > `cs->nr_cores`, `cs->nr_threads` with `cs` being of type CPUState. +> > +> > So I think the issue is rooted with the preferring sockets over cores +> > when the -smp cmdline option is parsed, as stated in hw/i386/pc.c:729. +> > +> > I guess this is the same code for Intel and AMD CPUs alike and this +> > issue just didn't surface for us on Intel CPUs, as they don't have this +> > CPUID leaf and we don't look at the topology. +> > +> > This seems to boil down to a more careful use of the -smp option on my end. +> > +> > Thanks again for looking into this. +> > +> > Cheers, +> > Philipp +> > +> > +> > +> > On 4/10/20 2:12 AM, Babu Moger wrote: +> >> Philipp, +> >> Can you please check if this patch works for you. +> >> +> >> diff --git a/target/i386/cpu.c b/target/i386/cpu.c +> >> index 90ffc5f..e467fee 100644 +> >> --- a/target/i386/cpu.c +> >> +++ b/target/i386/cpu.c +> >> @@ -5831,10 +5831,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t +> >> index, uint32_t count, +> >> } +> >> *ebx = env->features[FEAT_8000_0008_EBX]; +> >> *ecx = 0; +> >> - *edx = 0; +> >> if (cs->nr_cores * cs->nr_threads > 1) { +> >> - *ecx |= (cs->nr_cores * cs->nr_threads) - 1; +> >> + unsigned long max_apicids, bits_required; +> >> + +> >> + max_apicids = (cs->nr_cores * cs->nr_threads) - 1; +> >> + if (max_apicids) { +> >> + /* Find out the number of bits to represent all the +> >> apicids */ +> >> + bits_required = find_last_bit(&max_apicids, +> >> BITS_PER_BYTE) + 1; +> >> + *ecx |= bits_required << 12 | max_apicids; +> >> + } +> >> } +> >> + *edx = 0; +> >> break; +> >> case 0x8000000A: +> >> if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { +> >> +> >> +> >> On 4/9/20 9:00 AM, Igor Mammedov wrote: +> >>> On Thu, 09 Apr 2020 12:58:11 -0000 +> >>> Philipp Eppelt <email address hidden> wrote: +> >>> +> >>>> Public bug reported: +> >>>> +> >>>> Setup: +> >>>> CPU: AMD EPYC-v2 or host's EPYC cpu +> >>>> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +> >>>> qemu version: self build +> >>>> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +> >>>> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +> >>>> +> >>>> Cmdline: +> >>>> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +> >>>> +> >>>> Issue: +> >>>> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +> >>>> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +> >>>> +> >>>> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +> >>>> The lower four bits of ECX are the [NC] field and all set. +> >>>> +> >>>> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +> >>>> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +> >>>> +> >>>> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +> >>>> CPU model and it is reproducible there as well and reports: EAX: +> >>>> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +> >>>> +> >>>> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +> >>>> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +> >>>> +> >>>> The following table shows my findings with the -smp option: +> >>>> Option | Qemu guest observed ECX value +> >>>> -smp 4 | 0x0 +> >>>> -smp 4,cores=4 | 0x3 +> >>>> -smp 4,cores=2,thread=2 | 0x3 +> >>>> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +> >>>> +> >>>> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +> >>>> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +> >>> I'd say X corresponds to number of logical CPUs. +> >>> Depending on presence of other options these are distributed among them in magical manner +> >>> (see pc_smp_parse() for starters) +> >>> +> >>>> Querying 0x8000'0008 on the physical processor results in different +> >>>> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +> >>>> +> >>>> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +> >>>> CPU leave reports a single core processor. +> >>> it matches -smp X as it should be. +> >>> +> >>>> +> >>>> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +> >>>> wrong number. +> >>> CCed author of recent epyc patches who might know how AMD should work better than me. +> >>> +> >>>> +> >>>> Please let me know, if you need more information from my side. +> >>>> +> >>>> +> >>>> [0] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fkernel_thread-ia32.cpp%23L109&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=hcFJzLAVQoIh5IN9CP%2F9cUQNOZoBnpRA6FliJur1wzQ%3D&reserved=0 +> >>>> [1] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkernkonzept%2Ffiasco%2Fblob%2F522ccc5f29ab120213cf02d71328e2b879cbbd19%2Fsrc%2Fkern%2Fia32%2Fcpu-ia32.cpp%23L1120&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=ANJIbYKbwfq2bDelH%2FRLKnDPIUZc1BwxHspmgxLU7gs%3D&reserved=0 +> >>>> [2] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fqemu%2Fqemu%2Fblob%2Ff2a8261110c32c4dccd84e774d8dd7a0524e00fb%2Ftarget%2Fi386%2Fcpu.c%23L5835&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=oj3mv9e5YOzUsfUjXK44gC8LybyWgMKo8JBIrRR%2BmDA%3D&reserved=0 +> >>>> [3] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%2F24594.pdf&data=02%7C01%7Cbabu.moger%40amd.com%7C57569f7959744399655b08d7dc8e6e24%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637220379083511672&sdata=7Yr3J9ihlqSqXCXKN5JJNTByO3NGI%2BGMz2EqBF2Y4hw%3D&reserved=0 +> >>>> +> >>>> ** Affects: qemu +> >>>> Importance: Undecided +> >>>> Status: New +> >>>> +> >>> +> >> +> > +> + + + +Thanks. I saw the update in the thread. +https://<email address hidden>/ +Looks like you have found a way to take care of your problem. +But We need to fix the CPUID leaf 0x8000'0008 anyways. +Will send the patch to review later this week. Thanks + + +On 4/9/20 12:48 PM, Babu Moger wrote: +> +> +> On 4/9/20 9:00 AM, Igor Mammedov wrote: +>> On Thu, 09 Apr 2020 12:58:11 -0000 +>> Philipp Eppelt <email address hidden> wrote: +>> +>>> Public bug reported: +>>> +>>> Setup: +>>> CPU: AMD EPYC-v2 or host's EPYC cpu +>>> Linux 64-bit fedora host; Kernel version 5.5.15-200.fc31 +>>> qemu version: self build +>>> git-head: f3bac27cc1e303e1860cc55b9b6889ba39dee587 +>>> config: Configured with: '../configure' '--target-list=x86_64-softmmu,mips64el-softmmu,mips64-softmmu,mipsel-softmmu,mips-softmmu,i386-softmmu,aarch64-softmmu,arm-softmmu' '--prefix=/opt/qemu-master' +>>> +>>> Cmdline: +>>> qemu-system-x86_64 -kernel /home/peppelt/code/l4/internal/.build-x86_64/bin/amd64_gen/bootstrap -append "" -initrd "./fiasco/.build-x86_64/fiasco , ... " -serial stdio -nographic -monitor none -nographic -monitor none -cpu EPYC-v2 -m 4G -smp 4 +>>> +>>> Issue: +>>> We are developing an microkernel operating system called L4Re. We recently got an AMD EPYC server for testing and we couldn't execute SMP tests of our system when running Linux + qemu + VM w/ L4Re. +>>> In fact, the kernel did not recognize any APs at all. On AMD CPUs the kernel checks for the number of cores reported in CPUID leaf 0x8000_0008.ECX[NC] or [ApicIdSize]. [0][1] +>>> +>>> The physical machine reports for leaf 0x8000_0008: EAX: 0x3030 EBX: 0x18cf757 ECX: 0x703f EDX: 0x1000 +>>> The lower four bits of ECX are the [NC] field and all set. +>>> +>>> When querying inside qemu with -enable-kvm -cpu host -smp 4 (basically as replacement and addition to the above cmdline) the CPUID leaf shows: EAX: 0x3024, EBX: 0x1001000, ECX: 0x0, EDX: 0x0 +>>> Note, ECX is zero. Indicating that this is no SMP capabale CPU. +>>> +>>> I'm debugging it using my local machine and the QEMU provided EPYC-v2 +>>> CPU model and it is reproducible there as well and reports: EAX: +>>> 0x3028, EBX: 0x0, ECX: 0x0, EDX: 0x0 +>>> +>>> I checked other AMD based CPU models (phenom, opteron_g3/g5) and they behave the same. [2] shows the CPUID 0x8000'0008 handling in the QEMU source. +>>> I believe that behavior here is wrong as ECX[NC] should report the number of cores per processor, as stated in the AMD manual [2] p.584. In my understanding -smp 4 should then lead to ECX[NC] = 0x3. +>>> +>>> The following table shows my findings with the -smp option: +>>> Option | Qemu guest observed ECX value +>>> -smp 4 | 0x0 +>>> -smp 4,cores=4 | 0x3 +>>> -smp 4,cores=2,thread=2 | 0x3 +>>> -smp 4,cores=4,threads=2 | QEMU boot error: topology false. +>>> +>>> Now, I'm asking myself how the terminology of the AMD manual maps to QEMU's -smp option. +>>> Obviously, nr_cores and nr_threads correspond to the cores and threads options on the cmdline and cores * threads <= 4 (in this example), but what corresponds the X in -smp X to? +>> I'd say X corresponds to number of logical CPUs. +>> Depending on presence of other options these are distributed among them in magical manner +>> (see pc_smp_parse() for starters) +>> +>>> Querying 0x8000'0008 on the physical processor results in different +>>> reports than quering QEMU's model as does it with -enable-kvm -cpu host. +>>> +>>> Furthermore, the ACPI.MADT shows 4 local APICs to be present while the +>>> CPU leave reports a single core processor. +>> it matches -smp X as it should be. +>> +>>> +>>> This leads me to the conclusion that CPUID 0x8000'0008.ECX reports the +>>> wrong number. +>> CCed author of recent epyc patches who might know how AMD should work better than me. +> +> Hmm.. Interesting.. Not sure why this did not come up during my testing. +> Probably this cpuid information is not widely used. +> +> Yes. I am looking at it right now. I see that EPYC model is reporting +> wrong. Not sure why -cpu host is reporting wrong. I thought -cpu host gets +> the information directly from the host. Will investigate. +> +> + + +CPUID leaf CPUID_Fn80000008_ECX provides information about the +number of threads supported by the processor. It was found that +the field ApicIdSize(bits 15-12) was not set correctly. + +ApicIdSize is defined as the number of bits required to represent +all the ApicId values within a package. + +Valid Values: Value Description +3h-0h Reserved. +4h up to 16 threads. +5h up to 32 threads. +6h up to 64 threads. +7h up to 128 threads. +Fh-8h Reserved. + +Fix the bit appropriately. + +This came up during following thread. +https://lore.kernel.<email address hidden>/#t + +Refer the Processor Programming Reference (PPR) for AMD Family 17h +Model 01h, Revision B1 Processors. The documentation is available +from the bugzilla Link below. +Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 + +Reported-by: Philipp Eppelt <email address hidden> +Signed-off-by: Babu Moger <email address hidden> +--- + target/i386/cpu.c | 12 +++++++++--- + 1 file changed, 9 insertions(+), 3 deletions(-) + +diff --git a/target/i386/cpu.c b/target/i386/cpu.c +index 90ffc5f..68210f6 100644 +--- a/target/i386/cpu.c ++++ b/target/i386/cpu.c +@@ -5830,11 +5830,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, + *eax = cpu->phys_bits; + } + *ebx = env->features[FEAT_8000_0008_EBX]; +- *ecx = 0; +- *edx = 0; + if (cs->nr_cores * cs->nr_threads > 1) { +- *ecx |= (cs->nr_cores * cs->nr_threads) - 1; ++ unsigned int max_apicids, bits_required; ++ ++ max_apicids = (cs->nr_cores * cs->nr_threads) - 1; ++ /* Find out the number of bits to represent all the apicids */ ++ bits_required = 32 - clz32(max_apicids); ++ *ecx = bits_required << 12 | max_apicids; ++ } else { ++ *ecx = 0; + } ++ *edx = 0; + break; + case 0x8000000A: + if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { + + + +Good catch, thanks for the patch. Comments below: + +On Fri, Apr 17, 2020 at 10:14:32AM -0500, Babu Moger wrote: +> CPUID leaf CPUID_Fn80000008_ECX provides information about the +> number of threads supported by the processor. It was found that +> the field ApicIdSize(bits 15-12) was not set correctly. +> +> ApicIdSize is defined as the number of bits required to represent +> all the ApicId values within a package. +> +> Valid Values: Value Description +> 3h-0h Reserved. +> 4h up to 16 threads. +> 5h up to 32 threads. +> 6h up to 64 threads. +> 7h up to 128 threads. +> Fh-8h Reserved. +> +> Fix the bit appropriately. +> +> This came up during following thread. +> https://lore.kernel.<email address hidden>/#t +> +> Refer the Processor Programming Reference (PPR) for AMD Family 17h +> Model 01h, Revision B1 Processors. The documentation is available +> from the bugzilla Link below. +> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 +> +> Reported-by: Philipp Eppelt <email address hidden> +> Signed-off-by: Babu Moger <email address hidden> +> --- +> target/i386/cpu.c | 12 +++++++++--- +> 1 file changed, 9 insertions(+), 3 deletions(-) +> +> diff --git a/target/i386/cpu.c b/target/i386/cpu.c +> index 90ffc5f..68210f6 100644 +> --- a/target/i386/cpu.c +> +++ b/target/i386/cpu.c +> @@ -5830,11 +5830,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, +> *eax = cpu->phys_bits; +> } +> *ebx = env->features[FEAT_8000_0008_EBX]; +> - *ecx = 0; +> - *edx = 0; +> if (cs->nr_cores * cs->nr_threads > 1) { +> - *ecx |= (cs->nr_cores * cs->nr_threads) - 1; + +I'm not sure we want a compatibility flag to keep ABI on older +machine types, here. Strictly speaking, CPUID must never change +on older machine types, but sometimes trying hard to emulate bugs +of old QEMU versions is a pointless exercise. + + +> + unsigned int max_apicids, bits_required; +> + +> + max_apicids = (cs->nr_cores * cs->nr_threads) - 1; +> + /* Find out the number of bits to represent all the apicids */ +> + bits_required = 32 - clz32(max_apicids); + +This won't work if nr_cores > 1 and nr_threads is not a power of +2, will it? + +For reference, the field is documented[1] as: + +"The number of bits in the initial Core::X86::Apic::ApicId[ApicId] +value that indicate thread ID within a package" + +This sounds like the value already stored at +CPUX86State::pkg_offset. + + +> + *ecx = bits_required << 12 | max_apicids; + +Bits 7:0 are documented as "The number of threads in the package +is NC+1", with no reference to APIC IDs at all. + +Using ((nr_cores * nr_threads) - 1) for bits 7:0 sounds correct, +but the variable name seems misleading. + + +> + } else { +> + *ecx = 0; +> } +> + *edx = 0; +> break; +> case 0x8000000A: +> if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { +> +> + +References: + +[1] Processor Programming Reference (PPR) for + AMD Family 17h Model 18h, Revision B1 Processors + 55570-B1 Rev 3.14 - Sep 26, 2019 + https://bugzilla.kernel.org/attachment.cgi?id=287395&action=edit + + +-- +Eduardo + + + + + +On 4/17/20 2:15 PM, Eduardo Habkost wrote: +> Good catch, thanks for the patch. Comments below: +> +> On Fri, Apr 17, 2020 at 10:14:32AM -0500, Babu Moger wrote: +>> CPUID leaf CPUID_Fn80000008_ECX provides information about the +>> number of threads supported by the processor. It was found that +>> the field ApicIdSize(bits 15-12) was not set correctly. +>> +>> ApicIdSize is defined as the number of bits required to represent +>> all the ApicId values within a package. +>> +>> Valid Values: Value Description +>> 3h-0h Reserved. +>> 4h up to 16 threads. +>> 5h up to 32 threads. +>> 6h up to 64 threads. +>> 7h up to 128 threads. +>> Fh-8h Reserved. +>> +>> Fix the bit appropriately. +>> +>> This came up during following thread. +>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kernel.org%2Fqemu-devel%2F158643709116.17430.15995069125716778943.malonedeb%40wampee.canonical.com%2F%23t&data=02%7C01%7Cbabu.moger%40amd.com%7C1b8d59370cdb403dd54308d7e303adb7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637227477274521298&sdata=NZHLwOkQrbjkGeqYSI0wgRNUd3QHRCf7lBtdqoR5XfI%3D&reserved=0 +>> +>> Refer the Processor Programming Reference (PPR) for AMD Family 17h +>> Model 01h, Revision B1 Processors. The documentation is available +>> from the bugzilla Link below. +>> Link: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fshow_bug.cgi%3Fid%3D206537&data=02%7C01%7Cbabu.moger%40amd.com%7C1b8d59370cdb403dd54308d7e303adb7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637227477274521298&sdata=oNLqu0J49eTrJ8pQ6GKg64ZUDfV3egZN2VVkU0DwMaU%3D&reserved=0 +>> +>> Reported-by: Philipp Eppelt <email address hidden> +>> Signed-off-by: Babu Moger <email address hidden> +>> --- +>> target/i386/cpu.c | 12 +++++++++--- +>> 1 file changed, 9 insertions(+), 3 deletions(-) +>> +>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c +>> index 90ffc5f..68210f6 100644 +>> --- a/target/i386/cpu.c +>> +++ b/target/i386/cpu.c +>> @@ -5830,11 +5830,17 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, +>> *eax = cpu->phys_bits; +>> } +>> *ebx = env->features[FEAT_8000_0008_EBX]; +>> - *ecx = 0; +>> - *edx = 0; +>> if (cs->nr_cores * cs->nr_threads > 1) { +>> - *ecx |= (cs->nr_cores * cs->nr_threads) - 1; +> +> I'm not sure we want a compatibility flag to keep ABI on older +> machine types, here. Strictly speaking, CPUID must never change +> on older machine types, but sometimes trying hard to emulate bugs +> of old QEMU versions is a pointless exercise. + +Not sure about this. But it seemed like nobody cared about this field before. +> +> +>> + unsigned int max_apicids, bits_required; +>> + +>> + max_apicids = (cs->nr_cores * cs->nr_threads) - 1; +>> + /* Find out the number of bits to represent all the apicids */ +>> + bits_required = 32 - clz32(max_apicids); +> +> This won't work if nr_cores > 1 and nr_threads is not a power of +> 2, will it? + +It seem to work. Tested with threads=5,cores=3. + +> +> For reference, the field is documented[1] as: +> +> "The number of bits in the initial Core::X86::Apic::ApicId[ApicId] +> value that indicate thread ID within a package" +> +> This sounds like the value already stored at +> CPUX86State::pkg_offset. + +Yes, it is already in pkg_offset. We can use it. + +> +> +>> + *ecx = bits_required << 12 | max_apicids; +> +> Bits 7:0 are documented as "The number of threads in the package +> is NC+1", with no reference to APIC IDs at all. +> +> Using ((nr_cores * nr_threads) - 1) for bits 7:0 sounds correct, +> but the variable name seems misleading. + +I can change the variable name to num_threads. +> +> +>> + } else { +>> + *ecx = 0; +>> } +>> + *edx = 0; +>> break; +>> case 0x8000000A: +>> if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { +>> +>> +> +> References: +> +> [1] Processor Programming Reference (PPR) for +> AMD Family 17h Model 18h, Revision B1 Processors +> 55570-B1 Rev 3.14 - Sep 26, 2019 +> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.kernel.org%2Fattachment.cgi%3Fid%3D287395%26action%3Dedit&data=02%7C01%7Cbabu.moger%40amd.com%7C1b8d59370cdb403dd54308d7e303adb7%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637227477274521298&sdata=UsM3h4vp3dTgigqOvt7GrGiIUHvH8Kn1g%2BO%2FfGMav%2Bc%3D&reserved=0 +> +> + + +CPUID leaf CPUID_Fn80000008_ECX provides information about the +number of threads supported by the processor. It was found that +the field ApicIdSize(bits 15-12) was not set correctly. + +ApicIdSize is defined as the number of bits required to represent +all the ApicId values within a package. + +Valid Values: Value Description +3h-0h Reserved. +4h up to 16 threads. +5h up to 32 threads. +6h up to 64 threads. +7h up to 128 threads. +Fh-8h Reserved. + +Fix the bit appropriately. + +This came up during following thread. +https://lore.kernel.<email address hidden>/#t + +Refer the Processor Programming Reference (PPR) for AMD Family 17h +Model 01h, Revision B1 Processors. The documentation is available +from the bugzilla Link below. +Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 + +Reported-by: Philipp Eppelt <email address hidden> +Signed-off-by: Babu Moger <email address hidden> +--- +v2: + Used env->pkg_offset for bits 15:12 which is already available. + + target/i386/cpu.c | 15 ++++++++++++--- + 1 file changed, 12 insertions(+), 3 deletions(-) + +diff --git a/target/i386/cpu.c b/target/i386/cpu.c +index 90ffc5f..5e5a605 100644 +--- a/target/i386/cpu.c ++++ b/target/i386/cpu.c +@@ -5830,11 +5830,20 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, + *eax = cpu->phys_bits; + } + *ebx = env->features[FEAT_8000_0008_EBX]; +- *ecx = 0; +- *edx = 0; + if (cs->nr_cores * cs->nr_threads > 1) { +- *ecx |= (cs->nr_cores * cs->nr_threads) - 1; ++ /* ++ * Bits 15:12 is "The number of bits in the initial ++ * Core::X86::Apic::ApicId[ApicId] value that indicate ++ * thread ID within a package". This is already stored at ++ * CPUX86State::pkg_offset. ++ * Bits 7:0 is "The number of threads in the package is NC+1" ++ */ ++ *ecx = (env->pkg_offset << 12) | ++ ((cs->nr_cores * cs->nr_threads) - 1); ++ } else { ++ *ecx = 0; + } ++ *edx = 0; + break; + case 0x8000000A: + if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { + + + +Hi, + +thanks for the patch, I tested it in my setup and I'm seeing numbers +that make sense. + +However, I can create a setup which places the value 02h* into the +ApicIdSize field, which is reserved. However, I deem this a +configuration issue as well. + +* -cpu EPYC-v2 -smp 4,cores=4 --> 0x8000_0008[ECX] = 0x2003 + +Cheers, +Philipp + +On 4/17/20 11:55 PM, Babu Moger wrote: +> CPUID leaf CPUID_Fn80000008_ECX provides information about the +> number of threads supported by the processor. It was found that +> the field ApicIdSize(bits 15-12) was not set correctly. +> +> ApicIdSize is defined as the number of bits required to represent +> all the ApicId values within a package. +> +> Valid Values: Value Description +> 3h-0h Reserved. +> 4h up to 16 threads. +> 5h up to 32 threads. +> 6h up to 64 threads. +> 7h up to 128 threads. +> Fh-8h Reserved. +> +> Fix the bit appropriately. +> +> This came up during following thread. +> https://lore.kernel.<email address hidden>/#t +> +> Refer the Processor Programming Reference (PPR) for AMD Family 17h +> Model 01h, Revision B1 Processors. The documentation is available +> from the bugzilla Link below. +> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537 +> +> Reported-by: Philipp Eppelt <email address hidden> +> Signed-off-by: Babu Moger <email address hidden> +> --- +> v2: +> Used env->pkg_offset for bits 15:12 which is already available. +> +> target/i386/cpu.c | 15 ++++++++++++--- +> 1 file changed, 12 insertions(+), 3 deletions(-) +> +> diff --git a/target/i386/cpu.c b/target/i386/cpu.c +> index 90ffc5f..5e5a605 100644 +> --- a/target/i386/cpu.c +> +++ b/target/i386/cpu.c +> @@ -5830,11 +5830,20 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count, +> *eax = cpu->phys_bits; +> } +> *ebx = env->features[FEAT_8000_0008_EBX]; +> - *ecx = 0; +> - *edx = 0; +> if (cs->nr_cores * cs->nr_threads > 1) { +> - *ecx |= (cs->nr_cores * cs->nr_threads) - 1; +> + /* +> + * Bits 15:12 is "The number of bits in the initial +> + * Core::X86::Apic::ApicId[ApicId] value that indicate +> + * thread ID within a package". This is already stored at +> + * CPUX86State::pkg_offset. +> + * Bits 7:0 is "The number of threads in the package is NC+1" +> + */ +> + *ecx = (env->pkg_offset << 12) | +> + ((cs->nr_cores * cs->nr_threads) - 1); +> + } else { +> + *ecx = 0; +> } +> + *edx = 0; +> break; +> case 0x8000000A: +> if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) { +> + +-- +<email address hidden> - Tel. 0351-41 883 221 +http://www.kernkonzept.com + +Kernkonzept GmbH. Sitz: Dresden. Amtsgericht Dresden, HRB 31129. +Geschäftsführer: Dr.-Ing. Michael Hohmuth + + +If I got that right, there were some patches proposed for this bug ... has this been fixed already? Or is there still anything left to do? + +The patch mentioned earlier has been included here: +https://gitlab.com/qemu-project/qemu/-/commit/cac9edfc4dad2a7d2ad7e +So I assume this has been fixed. If you still have problems, please open a new ticket in the new bug tracker at gitlab. + |