From dee4dcba78baf712cab403d47d9db319ab7f95d6 Mon Sep 17 00:00:00 2001 From: Christian Krinitsin Date: Thu, 3 Jul 2025 19:39:53 +0200 Subject: restructure results --- results/classifier/016/015-016 | 86 - results/classifier/016/KVM/25842545 | 229 - results/classifier/016/KVM/26095107 | 185 - results/classifier/016/KVM/55961334 | 66 - results/classifier/016/TCG-i386/13442371 | 396 -- results/classifier/016/TCG-x86/14488057 | 738 -- results/classifier/016/TCG-x86/66743673 | 391 - results/classifier/016/TCG-x86/80615920 | 375 - results/classifier/016/TCG/95154278 | 182 - results/classifier/016/arm/33802194 | 4966 ------------- results/classifier/016/arm/57195159 | 342 - results/classifier/016/assembly-arm/1809546 | 109 - results/classifier/016/assembly-x86/1156313 | 147 - .../016/assembly-x86/gitlab_semantic_addsubps | 55 - .../016/assembly-x86/gitlab_semantic_adox | 68 - .../016/assembly-x86/gitlab_semantic_bextr | 57 - .../016/assembly-x86/gitlab_semantic_blsi | 52 - .../016/assembly-x86/gitlab_semantic_bzhi | 70 - results/classifier/016/boot/16056596 | 125 - results/classifier/016/boot/43643137 | 565 -- results/classifier/016/boot/51610399 | 335 - results/classifier/016/categories.csv | 21 - results/classifier/016/debug/02572177 | 448 -- results/classifier/016/debug/05479587 | 110 - results/classifier/016/debug/11357571 | 74 - results/classifier/016/debug/11933524 | 1152 --- results/classifier/016/debug/12360755 | 323 - results/classifier/016/debug/16228234 | 1871 ----- results/classifier/016/debug/21247035 | 1348 ---- results/classifier/016/debug/22219210 | 70 - results/classifier/016/debug/23270873 | 719 -- results/classifier/016/debug/30680944 | 622 -- results/classifier/016/debug/32484936 | 250 - results/classifier/016/debug/42226390 | 214 - results/classifier/016/debug/55247116 | 1337 ---- results/classifier/016/debug/56937788 | 371 - results/classifier/016/debug/63565653 | 76 - results/classifier/016/debug/64322995 | 81 - results/classifier/016/debug/68897003 | 743 -- results/classifier/016/debug/71456293 | 1513 ---- results/classifier/016/debug/73660729 | 58 - results/classifier/016/debug/74545755 | 371 - results/classifier/016/debug/80570214 | 427 -- results/classifier/016/debug/85542195 | 147 - results/classifier/016/debug/88225572 | 2927 -------- results/classifier/016/debug/88281850 | 308 - results/classifier/016/debug/96782458 | 1026 --- results/classifier/016/hypervisor/31349848 | 181 - results/classifier/016/hypervisor/64571620 | 812 --- results/classifier/016/hypervisor/81775929 | 262 - results/classifier/016/i386/26430026 | 192 - results/classifier/016/i386/74715356 | 153 - results/classifier/016/kernel/12869209 | 115 - results/classifier/016/kernel/59540920 | 403 -- results/classifier/016/kernel/60339453 | 88 - results/classifier/016/network/62179944 | 58 - results/classifier/016/none/23300761 | 340 - results/classifier/016/none/42613410 | 176 - results/classifier/016/none/42974450 | 456 -- results/classifier/016/none/48245039 | 557 -- results/classifier/016/none/50773216 | 137 - results/classifier/016/none/55753058 | 320 - results/classifier/016/none/56309929 | 207 - results/classifier/016/none/65781993 | 2820 -------- results/classifier/016/none/70294255 | 1088 --- results/classifier/016/none/70868267 | 67 - results/classifier/016/none/80604314 | 1507 ---- results/classifier/016/operating system/14887122 | 285 - .../016/semantic-x86/gitlab_semantic_blsmsk | 59 - results/classifier/016/user-level-x86/21221931 | 355 - results/classifier/016/user-level/02364653 | 390 - results/classifier/016/user-level/23448582 | 292 - results/classifier/016/virtual/04472277 | 603 -- results/classifier/016/virtual/16201167 | 127 - results/classifier/016/virtual/24190340 | 2083 ------ results/classifier/016/virtual/24930826 | 60 - results/classifier/016/virtual/25892827 | 1104 --- results/classifier/016/virtual/35170175 | 548 -- results/classifier/016/virtual/36568044 | 4608 ------------ results/classifier/016/virtual/46572227 | 433 -- results/classifier/016/virtual/53568181 | 105 - results/classifier/016/virtual/57231878 | 269 - results/classifier/016/virtual/67821138 | 226 - results/classifier/016/virtual/70021271 | 7475 -------------------- results/classifier/016/virtual/70416488 | 1206 ---- results/classifier/016/virtual/74466963 | 1905 ----- results/classifier/016/virtual/79834768 | 436 -- results/classifier/016/x86/17743720 | 798 --- results/classifier/016/x86/28596630 | 140 - results/classifier/016/x86/55367348 | 559 -- results/classifier/016/x86/57756589 | 1448 ---- results/classifier/016/x86/92957605 | 445 -- results/classifier/016/x86/99674399 | 175 - 93 files changed, 62219 deletions(-) delete mode 100644 results/classifier/016/015-016 delete mode 100644 results/classifier/016/KVM/25842545 delete mode 100644 results/classifier/016/KVM/26095107 delete mode 100644 results/classifier/016/KVM/55961334 delete mode 100644 results/classifier/016/TCG-i386/13442371 delete mode 100644 results/classifier/016/TCG-x86/14488057 delete mode 100644 results/classifier/016/TCG-x86/66743673 delete mode 100644 results/classifier/016/TCG-x86/80615920 delete mode 100644 results/classifier/016/TCG/95154278 delete mode 100644 results/classifier/016/arm/33802194 delete mode 100644 results/classifier/016/arm/57195159 delete mode 100644 results/classifier/016/assembly-arm/1809546 delete mode 100644 results/classifier/016/assembly-x86/1156313 delete mode 100644 results/classifier/016/assembly-x86/gitlab_semantic_addsubps delete mode 100644 results/classifier/016/assembly-x86/gitlab_semantic_adox delete mode 100644 results/classifier/016/assembly-x86/gitlab_semantic_bextr delete mode 100644 results/classifier/016/assembly-x86/gitlab_semantic_blsi delete mode 100644 results/classifier/016/assembly-x86/gitlab_semantic_bzhi delete mode 100644 results/classifier/016/boot/16056596 delete mode 100644 results/classifier/016/boot/43643137 delete mode 100644 results/classifier/016/boot/51610399 delete mode 100644 results/classifier/016/categories.csv delete mode 100644 results/classifier/016/debug/02572177 delete mode 100644 results/classifier/016/debug/05479587 delete mode 100644 results/classifier/016/debug/11357571 delete mode 100644 results/classifier/016/debug/11933524 delete mode 100644 results/classifier/016/debug/12360755 delete mode 100644 results/classifier/016/debug/16228234 delete mode 100644 results/classifier/016/debug/21247035 delete mode 100644 results/classifier/016/debug/22219210 delete mode 100644 results/classifier/016/debug/23270873 delete mode 100644 results/classifier/016/debug/30680944 delete mode 100644 results/classifier/016/debug/32484936 delete mode 100644 results/classifier/016/debug/42226390 delete mode 100644 results/classifier/016/debug/55247116 delete mode 100644 results/classifier/016/debug/56937788 delete mode 100644 results/classifier/016/debug/63565653 delete mode 100644 results/classifier/016/debug/64322995 delete mode 100644 results/classifier/016/debug/68897003 delete mode 100644 results/classifier/016/debug/71456293 delete mode 100644 results/classifier/016/debug/73660729 delete mode 100644 results/classifier/016/debug/74545755 delete mode 100644 results/classifier/016/debug/80570214 delete mode 100644 results/classifier/016/debug/85542195 delete mode 100644 results/classifier/016/debug/88225572 delete mode 100644 results/classifier/016/debug/88281850 delete mode 100644 results/classifier/016/debug/96782458 delete mode 100644 results/classifier/016/hypervisor/31349848 delete mode 100644 results/classifier/016/hypervisor/64571620 delete mode 100644 results/classifier/016/hypervisor/81775929 delete mode 100644 results/classifier/016/i386/26430026 delete mode 100644 results/classifier/016/i386/74715356 delete mode 100644 results/classifier/016/kernel/12869209 delete mode 100644 results/classifier/016/kernel/59540920 delete mode 100644 results/classifier/016/kernel/60339453 delete mode 100644 results/classifier/016/network/62179944 delete mode 100644 results/classifier/016/none/23300761 delete mode 100644 results/classifier/016/none/42613410 delete mode 100644 results/classifier/016/none/42974450 delete mode 100644 results/classifier/016/none/48245039 delete mode 100644 results/classifier/016/none/50773216 delete mode 100644 results/classifier/016/none/55753058 delete mode 100644 results/classifier/016/none/56309929 delete mode 100644 results/classifier/016/none/65781993 delete mode 100644 results/classifier/016/none/70294255 delete mode 100644 results/classifier/016/none/70868267 delete mode 100644 results/classifier/016/none/80604314 delete mode 100644 results/classifier/016/operating system/14887122 delete mode 100644 results/classifier/016/semantic-x86/gitlab_semantic_blsmsk delete mode 100644 results/classifier/016/user-level-x86/21221931 delete mode 100644 results/classifier/016/user-level/02364653 delete mode 100644 results/classifier/016/user-level/23448582 delete mode 100644 results/classifier/016/virtual/04472277 delete mode 100644 results/classifier/016/virtual/16201167 delete mode 100644 results/classifier/016/virtual/24190340 delete mode 100644 results/classifier/016/virtual/24930826 delete mode 100644 results/classifier/016/virtual/25892827 delete mode 100644 results/classifier/016/virtual/35170175 delete mode 100644 results/classifier/016/virtual/36568044 delete mode 100644 results/classifier/016/virtual/46572227 delete mode 100644 results/classifier/016/virtual/53568181 delete mode 100644 results/classifier/016/virtual/57231878 delete mode 100644 results/classifier/016/virtual/67821138 delete mode 100644 results/classifier/016/virtual/70021271 delete mode 100644 results/classifier/016/virtual/70416488 delete mode 100644 results/classifier/016/virtual/74466963 delete mode 100644 results/classifier/016/virtual/79834768 delete mode 100644 results/classifier/016/x86/17743720 delete mode 100644 results/classifier/016/x86/28596630 delete mode 100644 results/classifier/016/x86/55367348 delete mode 100644 results/classifier/016/x86/57756589 delete mode 100644 results/classifier/016/x86/92957605 delete mode 100644 results/classifier/016/x86/99674399 (limited to 'results/classifier/016') diff --git a/results/classifier/016/015-016 b/results/classifier/016/015-016 deleted file mode 100644 index baa779f5..00000000 --- a/results/classifier/016/015-016 +++ /dev/null @@ -1,86 +0,0 @@ -85 changes: -11357571: vnc -> debug -55367348: none -> x86 -43643137: none -> boot -68897003: none -> debug -33802194: none -> arm -11933524: none -> debug -16228234: none -> debug -21247035: none -> debug -71456293: none -> debug -73660729: architecture-arm -> debug -96782458: all -> debug -95154278: all -> TCG -70021271: all -> virtual -74715356: all -> i386 -16056596: all -> boot -92957605: all -> x86 -02364653: all -> user-level -23300761: all -> none -79834768: all -> virtual -26095107: all -> KVM -80570214: all -> debug -64571620: all -> hypervisor -16201167: all -> virtual -53568181: all -> virtual -70416488: all -> virtual -55247116: all -> debug -14488057: all -> TCG-x86 -80604314: all -> none -17743720: all -> x86 -42974450: all -> none -59540920: all -> kernel -74466963: all -> virtual -36568044: all -> virtual -23448582: all -> user-level -51610399: all -> boot -21221931: all -> user-level-x86 -88281850: all -> debug -14887122: all -> operating system -88225572: all -> debug -26430026: all -> i386 -48245039: all -> none -42226390: all -> debug -12869209: all -> kernel -46572227: all -> virtual -66743673: all -> TCG-x86 -85542195: all -> debug -24190340: debug -> virtual -67821138: permissions -> virtual -99674399: permissions -> x86 -57231878: permissions -> virtual -35170175: permissions -> virtual -50773216: permissions -> none -12360755: permissions -> debug -81775929: virtual -> hypervisor -22219210: virtual -> debug -74545755: risc-v -> debug -05479587: network -> debug -1156313: semantic -> assembly-x86 -62179944: operating system -> network -24930826: operating system -> virtual -63565653: operating system -> debug -23270873: unknown -> debug -80615920: unknown -> TCG-x86 -13442371: unknown -> TCG-i386 -25842545: unknown -> KVM -32484936: unknown -> debug -57756589: unknown -> x86 -04472277: unknown -> virtual -02572177: unknown -> debug -31349848: unknown -> hypervisor -70294255: unknown -> none -25892827: unknown -> virtual -56937788: peripherals -> debug -60339453: peripherals -> kernel -28596630: user-level -> x86 -30680944: graphic -> debug -55961334: graphic -> KVM -1809546: semantic-arm -> assembly-arm -57195159: register-arm -> arm -gitlab_semantic_bextr: semantic-x86 -> assembly-x86 -gitlab_semantic_addsubps: semantic-x86 -> assembly-x86 -gitlab_semantic_bzhi: semantic-x86 -> assembly-x86 -gitlab_semantic_adox: semantic-x86 -> assembly-x86 -gitlab_semantic_blsi: semantic-x86 -> assembly-x86 -64322995: files -> debug diff --git a/results/classifier/016/KVM/25842545 b/results/classifier/016/KVM/25842545 deleted file mode 100644 index 6228f8d3..00000000 --- a/results/classifier/016/KVM/25842545 +++ /dev/null @@ -1,229 +0,0 @@ -KVM: 0.983 -debug: 0.931 -hypervisor: 0.924 -virtual: 0.913 -socket: 0.682 -x86: 0.566 -register: 0.248 -TCG: 0.136 -operating system: 0.125 -kernel: 0.073 -PID: 0.068 -device: 0.047 -files: 0.047 -performance: 0.045 -assembly: 0.018 -i386: 0.012 -network: 0.012 -semantic: 0.011 -VMM: 0.009 -peripherals: 0.008 -user-level: 0.008 -boot: 0.008 -architecture: 0.007 -risc-v: 0.006 -ppc: 0.006 -permissions: 0.004 -alpha: 0.004 -graphic: 0.003 -vnc: 0.003 -arm: 0.001 -mistranslation: 0.001 - -[Qemu-devel] [Bug?] Guest pause because VMPTRLD failed in KVM - -Hello, - - We encountered a problem that a guest paused because the KMOD report VMPTRLD -failed. - -The related information is as follows: - -1) Qemu command: - /usr/bin/qemu-kvm -name omu1 -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu -host -m 15625 -realtime mlock=off -smp 8,sockets=1,cores=8,threads=1 -uuid -a2aacfff-6583-48b4-b6a4-e6830e519931 -no-user-config -nodefaults -chardev -socket,id=charmonitor,path=/var/lib/libvirt/qemu/omu1.monitor,server,nowait --mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown --boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device -virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive -file=/home/env/guest1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,aio=native - -device -virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0 - -drive -file=/home/env/guest_300G.img,if=none,id=drive-virtio-disk1,format=raw,cache=none,aio=native - -device -virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 - -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device -virtio-net-pci,netdev=hostnet0,id=net0,mac=00:00:80:05:00:00,bus=pci.0,addr=0x3 --netdev tap,fd=27,id=hostnet1,vhost=on,vhostfd=28 -device -virtio-net-pci,netdev=hostnet1,id=net1,mac=00:00:80:05:00:01,bus=pci.0,addr=0x4 --chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 --device usb-tablet,id=input0 -vnc 0.0.0.0:0 -device -cirrus-vga,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -device -virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on - - 2) Qemu log: - KVM: entry failed, hardware error 0x4 - RAX=00000000ffffffed RBX=ffff8803fa2d7fd8 RCX=0100000000000000 -RDX=0000000000000000 - RSI=0000000000000000 RDI=0000000000000046 RBP=ffff8803fa2d7e90 -RSP=ffff8803fa2efe90 - R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 -R11=000000000000b69a - R12=0000000000000001 R13=ffffffff81a25b40 R14=0000000000000000 -R15=ffff8803fa2d7fd8 - RIP=ffffffff81053e16 RFL=00000286 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 - ES =0000 0000000000000000 ffffffff 00c00000 - CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA] - SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS [-WA] - DS =0000 0000000000000000 ffffffff 00c00000 - FS =0000 0000000000000000 ffffffff 00c00000 - GS =0000 ffff88040f540000 ffffffff 00c00000 - LDT=0000 0000000000000000 ffffffff 00c00000 - TR =0040 ffff88040f550a40 00002087 00008b00 DPL=0 TSS64-busy - GDT= ffff88040f549000 0000007f - IDT= ffffffffff529000 00000fff - CR0=80050033 CR2=00007f81ca0c5000 CR3=00000003f5081000 CR4=000407e0 - DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 -DR3=0000000000000000 - DR6=00000000ffff0ff0 DR7=0000000000000400 - EFER=0000000000000d01 - Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? -?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? - - 3) Demsg - [347315.028339] kvm: vmptrld ffff8817ec5f0000/17ec5f0000 failed - klogd 1.4.1, ---------- state change ---------- - [347315.039506] kvm: vmptrld ffff8817ec5f0000/17ec5f0000 failed - [347315.051728] kvm: vmptrld ffff8817ec5f0000/17ec5f0000 failed - [347315.057472] vmwrite error: reg 6c0a value ffff88307e66e480 (err -2120672384) - [347315.064567] Pid: 69523, comm: qemu-kvm Tainted: GF X -3.0.93-0.8-default #1 - [347315.064569] Call Trace: - [347315.064587] [] dump_trace+0x75/0x300 - [347315.064595] [] dump_stack+0x69/0x6f - [347315.064617] [] vmx_vcpu_load+0x11e/0x1d0 [kvm_intel] - [347315.064647] [] kvm_arch_vcpu_load+0x44/0x1d0 [kvm] - [347315.064669] [] finish_task_switch+0x81/0xe0 - [347315.064676] [] thread_return+0x3b/0x2a7 - [347315.064687] [] kvm_vcpu_block+0x65/0xa0 [kvm] - [347315.064703] [] __vcpu_run+0xd1/0x260 [kvm] - [347315.064732] [] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 -[kvm] - [347315.064759] [] kvm_vcpu_ioctl+0x38e/0x580 [kvm] - [347315.064771] [] do_vfs_ioctl+0x8b/0x3b0 - [347315.064776] [] sys_ioctl+0xa1/0xb0 - [347315.064783] [] system_call_fastpath+0x16/0x1b - [347315.064797] [<00007fee51969ce7>] 0x7fee51969ce6 - [347315.064799] vmwrite error: reg 6c0c value ffff88307e664000 (err -2120630272) - [347315.064802] Pid: 69523, comm: qemu-kvm Tainted: GF X -3.0.93-0.8-default #1 - [347315.064803] Call Trace: - [347315.064807] [] dump_trace+0x75/0x300 - [347315.064811] [] dump_stack+0x69/0x6f - [347315.064817] [] vmx_vcpu_load+0x12c/0x1d0 [kvm_intel] - [347315.064832] [] kvm_arch_vcpu_load+0x44/0x1d0 [kvm] - [347315.064851] [] finish_task_switch+0x81/0xe0 - [347315.064855] [] thread_return+0x3b/0x2a7 - [347315.064865] [] kvm_vcpu_block+0x65/0xa0 [kvm] - [347315.064880] [] __vcpu_run+0xd1/0x260 [kvm] - [347315.064907] [] kvm_arch_vcpu_ioctl_run+0x68/0x1a0 -[kvm] - [347315.064933] [] kvm_vcpu_ioctl+0x38e/0x580 [kvm] - [347315.064943] [] do_vfs_ioctl+0x8b/0x3b0 - [347315.064947] [] sys_ioctl+0xa1/0xb0 - [347315.064951] [] system_call_fastpath+0x16/0x1b - [347315.064957] [<00007fee51969ce7>] 0x7fee51969ce6 - [347315.064959] vmwrite error: reg 6c10 value 0 (err 0) - - 4) The isssue can't be reporduced. I search the Intel VMX sepc about reaseons -of vmptrld failure: - The instruction fails if its operand is not properly aligned, sets -unsupported physical-address bits, or is equal to the VMXON - pointer. In addition, the instruction fails if the 32 bits in memory -referenced by the operand do not match the VMCS - revision identifier supported by this processor. - - But I can't find any cues from the KVM source code. It seems each - error conditions is impossible in theory. :( - -Any suggestions will be appreciated! Paolo? - --- -Regards, --Gonglei - -On 10/11/2016 15:10, gong lei wrote: -> -4) The isssue can't be reporduced. I search the Intel VMX sepc about -> -reaseons -> -of vmptrld failure: -> -The instruction fails if its operand is not properly aligned, sets -> -unsupported physical-address bits, or is equal to the VMXON -> -pointer. In addition, the instruction fails if the 32 bits in memory -> -referenced by the operand do not match the VMCS -> -revision identifier supported by this processor. -> -> -But I can't find any cues from the KVM source code. It seems each -> -error conditions is impossible in theory. :( -Yes, it should not happen. :( - -If it's not reproducible, it's really hard to say what it was, except a -random memory corruption elsewhere or even a bit flip (!). - -Paolo - -On 2016/11/17 20:39, Paolo Bonzini wrote: -> -> -On 10/11/2016 15:10, gong lei wrote: -> -> 4) The isssue can't be reporduced. I search the Intel VMX sepc about -> -> reaseons -> -> of vmptrld failure: -> -> The instruction fails if its operand is not properly aligned, sets -> -> unsupported physical-address bits, or is equal to the VMXON -> -> pointer. In addition, the instruction fails if the 32 bits in memory -> -> referenced by the operand do not match the VMCS -> -> revision identifier supported by this processor. -> -> -> -> But I can't find any cues from the KVM source code. It seems each -> -> error conditions is impossible in theory. :( -> -Yes, it should not happen. :( -> -> -If it's not reproducible, it's really hard to say what it was, except a -> -random memory corruption elsewhere or even a bit flip (!). -> -> -Paolo -Thanks for your reply, Paolo :) - --- -Regards, --Gonglei - diff --git a/results/classifier/016/KVM/26095107 b/results/classifier/016/KVM/26095107 deleted file mode 100644 index 90f47e7b..00000000 --- a/results/classifier/016/KVM/26095107 +++ /dev/null @@ -1,185 +0,0 @@ -KVM: 0.987 -debug: 0.905 -x86: 0.855 -hypervisor: 0.806 -virtual: 0.770 -register: 0.560 -assembly: 0.451 -user-level: 0.387 -operating system: 0.082 -VMM: 0.081 -PID: 0.046 -kernel: 0.039 -i386: 0.032 -files: 0.026 -device: 0.022 -socket: 0.014 -TCG: 0.013 -semantic: 0.012 -performance: 0.012 -network: 0.010 -architecture: 0.008 -peripherals: 0.003 -risc-v: 0.003 -alpha: 0.003 -graphic: 0.003 -boot: 0.003 -vnc: 0.002 -ppc: 0.002 -permissions: 0.002 -arm: 0.001 -mistranslation: 0.001 - -[Qemu-devel] [Bug Report] vm paused after succeeding to migrate - -Hi, all -I encounterd a bug when I try to migrate a windows vm. - -Enviroment information: -host A: cpu E5620(model WestmereEP without flag xsave) -host B: cpu E5-2643(model SandyBridgeEP with xsave) - -The reproduce steps is : -1. Start a windows 2008 vm with -cpu host(which means host-passthrough). -2. Migrate the vm to host B when cr4.OSXSAVE=0 (successfully). -3. Vm runs on host B for a while so that cr4.OSXSAVE changes to 1. -4. Then migrate the vm to host A (successfully), but vm was paused, and qemu -printed log as followed: - -KVM: entry failed, hardware error 0x80000021 - -If you're running a guest on an Intel machine without unrestricted mode -support, the failure can be most likely due to the guest entering an invalid -state for Intel VT. For example, the guest maybe running in big real mode -which is not supported on less recent Intel processors. - -EAX=019b3bb0 EBX=01a3ae80 ECX=01a61ce8 EDX=00000000 -ESI=01a62000 EDI=00000000 EBP=00000000 ESP=01718b20 -EIP=0185d982 EFL=00000286 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 -ES =0000 00000000 0000ffff 00009300 -CS =f000 ffff0000 0000ffff 00009b00 -SS =0000 00000000 0000ffff 00009300 -DS =0000 00000000 0000ffff 00009300 -FS =0000 00000000 0000ffff 00009300 -GS =0000 00000000 0000ffff 00009300 -LDT=0000 00000000 0000ffff 00008200 -TR =0000 00000000 0000ffff 00008b00 -GDT= 00000000 0000ffff -IDT= 00000000 0000ffff -CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 -DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 -DR3=0000000000000000 -DR6=00000000ffff0ff0 DR7=0000000000000400 -EFER=0000000000000000 -Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 -00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - -I have found that problem happened when kvm_put_sregs returns err -22(called by -kvm_arch_put_registers(qemu)). -Because kvm_arch_vcpu_ioctl_set_sregs(kvm-mod) checked that guest_cpuid_has no -X86_FEATURE_XSAVE but cr4.OSXSAVE=1. -So should we cancel migration when kvm_arch_put_registers returns error? - -* linzhecheng (address@hidden) wrote: -> -Hi, all -> -I encounterd a bug when I try to migrate a windows vm. -> -> -Enviroment information: -> -host A: cpu E5620(model WestmereEP without flag xsave) -> -host B: cpu E5-2643(model SandyBridgeEP with xsave) -> -> -The reproduce steps is : -> -1. Start a windows 2008 vm with -cpu host(which means host-passthrough). -> -2. Migrate the vm to host B when cr4.OSXSAVE=0 (successfully). -> -3. Vm runs on host B for a while so that cr4.OSXSAVE changes to 1. -> -4. Then migrate the vm to host A (successfully), but vm was paused, and qemu -> -printed log as followed: -Remember that migrating using -cpu host across different CPU models is NOT -expected to work. - -> -KVM: entry failed, hardware error 0x80000021 -> -> -If you're running a guest on an Intel machine without unrestricted mode -> -support, the failure can be most likely due to the guest entering an invalid -> -state for Intel VT. For example, the guest maybe running in big real mode -> -which is not supported on less recent Intel processors. -> -> -EAX=019b3bb0 EBX=01a3ae80 ECX=01a61ce8 EDX=00000000 -> -ESI=01a62000 EDI=00000000 EBP=00000000 ESP=01718b20 -> -EIP=0185d982 EFL=00000286 [--S--P-] CPL=0 II=0 A20=1 SMM=0 HLT=0 -> -ES =0000 00000000 0000ffff 00009300 -> -CS =f000 ffff0000 0000ffff 00009b00 -> -SS =0000 00000000 0000ffff 00009300 -> -DS =0000 00000000 0000ffff 00009300 -> -FS =0000 00000000 0000ffff 00009300 -> -GS =0000 00000000 0000ffff 00009300 -> -LDT=0000 00000000 0000ffff 00008200 -> -TR =0000 00000000 0000ffff 00008b00 -> -GDT= 00000000 0000ffff -> -IDT= 00000000 0000ffff -> -CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 -> -DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 -> -DR3=0000000000000000 -> -DR6=00000000ffff0ff0 DR7=0000000000000400 -> -EFER=0000000000000000 -> -Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 -> -00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 -> -00 -> -> -I have found that problem happened when kvm_put_sregs returns err -22(called -> -by kvm_arch_put_registers(qemu)). -> -Because kvm_arch_vcpu_ioctl_set_sregs(kvm-mod) checked that guest_cpuid_has -> -no X86_FEATURE_XSAVE but cr4.OSXSAVE=1. -> -So should we cancel migration when kvm_arch_put_registers returns error? -It would seem good if we can make the migration fail there rather than -hitting that KVM error. -It looks like we need to do a bit of plumbing to convert the places that -call it to return a bool rather than void. - -Dave - --- -Dr. David Alan Gilbert / address@hidden / Manchester, UK - diff --git a/results/classifier/016/KVM/55961334 b/results/classifier/016/KVM/55961334 deleted file mode 100644 index ede09a18..00000000 --- a/results/classifier/016/KVM/55961334 +++ /dev/null @@ -1,66 +0,0 @@ -KVM: 0.924 -x86: 0.885 -debug: 0.797 -virtual: 0.715 -operating system: 0.649 -hypervisor: 0.394 -TCG: 0.049 -register: 0.039 -network: 0.028 -kernel: 0.021 -PID: 0.016 -performance: 0.016 -user-level: 0.014 -files: 0.011 -architecture: 0.009 -socket: 0.007 -ppc: 0.004 -assembly: 0.004 -semantic: 0.003 -alpha: 0.003 -device: 0.003 -VMM: 0.003 -risc-v: 0.002 -vnc: 0.002 -boot: 0.001 -permissions: 0.001 -peripherals: 0.001 -graphic: 0.000 -arm: 0.000 -mistranslation: 0.000 -i386: 0.000 - -[Bug] "-ht" flag ignored under KVM - guest still reports HT - -Hi Community, -We have observed that the 'ht' feature bit cannot be disabled when QEMU runs -with KVM acceleration. -qemu-system-x86_64 \ - --enable-kvm \ - -machine q35 \ - -cpu host,-ht \ - -smp 4 \ - -m 4G \ - -drive file=rootfs.img,format=raw \ - -nographic \ - -append 'console=ttyS0 root=/dev/sda rw' -Because '-ht' is specified, the guest should expose no HT capability -(cpuid.1.edx[28] = 0), and /proc/cpuinfo shouldn't show HT feature, but we still -saw ht in linux guest when run 'cat /proc/cpuinfo'. -XiaoYao mentioned that: - -It has been the behavior of QEMU since - - commit 400281af34e5ee6aa9f5496b53d8f82c6fef9319 - Author: Andre Przywara - Date: Wed Aug 19 15:42:42 2009 +0200 - - set CPUID bits to present cores and threads topology - -that we cannot remove HT CPUID bit from guest via "-cpu xxx,-ht" if the -VM has >= 2 vcpus. -I'd like to know whether there's a plan to address this issue, or if the current -behaviour is considered acceptable. -Best regards, -Ewan. - diff --git a/results/classifier/016/TCG-i386/13442371 b/results/classifier/016/TCG-i386/13442371 deleted file mode 100644 index 1db702bd..00000000 --- a/results/classifier/016/TCG-i386/13442371 +++ /dev/null @@ -1,396 +0,0 @@ -i386: 0.997 -TCG: 0.898 -debug: 0.821 -x86: 0.754 -operating system: 0.376 -virtual: 0.277 -kernel: 0.203 -risc-v: 0.180 -files: 0.109 -PID: 0.057 -register: 0.055 -hypervisor: 0.039 -user-level: 0.033 -boot: 0.032 -socket: 0.029 -device: 0.027 -vnc: 0.025 -semantic: 0.017 -network: 0.015 -assembly: 0.010 -architecture: 0.009 -VMM: 0.008 -performance: 0.005 -alpha: 0.005 -graphic: 0.003 -permissions: 0.003 -peripherals: 0.002 -mistranslation: 0.001 -KVM: 0.001 -ppc: 0.000 -arm: 0.000 - -[Qemu-devel] [BUG] nanoMIPS support problem related to extract2 support for i386 TCG target - -Hello, Richard, Peter, and others. - -As a part of activities before 4.1 release, I tested nanoMIPS support -in QEMU (which was officially fully integrated in 4.0, is currently -limited to system mode only, and was tested in a similar fashion right -prior to 4.0). - -This support appears to be broken now. Following command line works in -4.0, but results in kernel panic for the current tip of the tree: - -~/Build/qemu-test-revert-c6fb8c0cf704/mipsel-softmmu/qemu-system-mipsel --cpu I7200 -kernel generic_nano32r6el_page4k -M malta -serial stdio -m -1G -hda nanomips32r6_le_sf_2017.05-03-59-gf5595d6.ext4 -append -"mem=256m@0x0 rw console=ttyS0 vga=cirrus vesa=0x111 root=/dev/sda" - -(kernel and rootfs image files used in this commend line can be -downloaded from the locations mentioned in our user guide) - -The quick bisect points to the commit: - -commit c6fb8c0cf704c4a1a48c3e99e995ad4c58150dab -Author: Richard Henderson -Date: Mon Feb 25 11:42:35 2019 -0800 - - tcg/i386: Support INDEX_op_extract2_{i32,i64} - - Signed-off-by: Richard Henderson - -Please advise on further actions. - -Yours, -Aleksandar - -On Fri, Jul 12, 2019 at 8:09 PM Aleksandar Markovic - wrote: -> -> -Hello, Richard, Peter, and others. -> -> -As a part of activities before 4.1 release, I tested nanoMIPS support -> -in QEMU (which was officially fully integrated in 4.0, is currently -> -limited to system mode only, and was tested in a similar fashion right -> -prior to 4.0). -> -> -This support appears to be broken now. Following command line works in -> -4.0, but results in kernel panic for the current tip of the tree: -> -> -~/Build/qemu-test-revert-c6fb8c0cf704/mipsel-softmmu/qemu-system-mipsel -> --cpu I7200 -kernel generic_nano32r6el_page4k -M malta -serial stdio -m -> -1G -hda nanomips32r6_le_sf_2017.05-03-59-gf5595d6.ext4 -append -> -"mem=256m@0x0 rw console=ttyS0 vga=cirrus vesa=0x111 root=/dev/sda" -> -> -(kernel and rootfs image files used in this commend line can be -> -downloaded from the locations mentioned in our user guide) -> -> -The quick bisect points to the commit: -> -> -commit c6fb8c0cf704c4a1a48c3e99e995ad4c58150dab -> -Author: Richard Henderson -> -Date: Mon Feb 25 11:42:35 2019 -0800 -> -> -tcg/i386: Support INDEX_op_extract2_{i32,i64} -> -> -Signed-off-by: Richard Henderson -> -> -Please advise on further actions. -> -Just to add a data point: - -If the following change is applied: - -diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h -index 928e8b8..b6a4cf2 100644 ---- a/tcg/i386/tcg-target.h -+++ b/tcg/i386/tcg-target.h -@@ -124,7 +124,7 @@ extern bool have_avx2; - #define TCG_TARGET_HAS_deposit_i32 1 - #define TCG_TARGET_HAS_extract_i32 1 - #define TCG_TARGET_HAS_sextract_i32 1 --#define TCG_TARGET_HAS_extract2_i32 1 -+#define TCG_TARGET_HAS_extract2_i32 0 - #define TCG_TARGET_HAS_movcond_i32 1 - #define TCG_TARGET_HAS_add2_i32 1 - #define TCG_TARGET_HAS_sub2_i32 1 -@@ -163,7 +163,7 @@ extern bool have_avx2; - #define TCG_TARGET_HAS_deposit_i64 1 - #define TCG_TARGET_HAS_extract_i64 1 - #define TCG_TARGET_HAS_sextract_i64 0 --#define TCG_TARGET_HAS_extract2_i64 1 -+#define TCG_TARGET_HAS_extract2_i64 0 - #define TCG_TARGET_HAS_movcond_i64 1 - #define TCG_TARGET_HAS_add2_i64 1 - #define TCG_TARGET_HAS_sub2_i64 1 - -... the problem disappears. - - -> -Yours, -> -Aleksandar - -On Fri, Jul 12, 2019 at 8:19 PM Aleksandar Markovic - wrote: -> -> -On Fri, Jul 12, 2019 at 8:09 PM Aleksandar Markovic -> - wrote: -> -> -> -> Hello, Richard, Peter, and others. -> -> -> -> As a part of activities before 4.1 release, I tested nanoMIPS support -> -> in QEMU (which was officially fully integrated in 4.0, is currently -> -> limited to system mode only, and was tested in a similar fashion right -> -> prior to 4.0). -> -> -> -> This support appears to be broken now. Following command line works in -> -> 4.0, but results in kernel panic for the current tip of the tree: -> -> -> -> ~/Build/qemu-test-revert-c6fb8c0cf704/mipsel-softmmu/qemu-system-mipsel -> -> -cpu I7200 -kernel generic_nano32r6el_page4k -M malta -serial stdio -m -> -> 1G -hda nanomips32r6_le_sf_2017.05-03-59-gf5595d6.ext4 -append -> -> "mem=256m@0x0 rw console=ttyS0 vga=cirrus vesa=0x111 root=/dev/sda" -> -> -> -> (kernel and rootfs image files used in this commend line can be -> -> downloaded from the locations mentioned in our user guide) -> -> -> -> The quick bisect points to the commit: -> -> -> -> commit c6fb8c0cf704c4a1a48c3e99e995ad4c58150dab -> -> Author: Richard Henderson -> -> Date: Mon Feb 25 11:42:35 2019 -0800 -> -> -> -> tcg/i386: Support INDEX_op_extract2_{i32,i64} -> -> -> -> Signed-off-by: Richard Henderson -> -> -> -> Please advise on further actions. -> -> -> -> -Just to add a data point: -> -> -If the following change is applied: -> -> -diff --git a/tcg/i386/tcg-target.h b/tcg/i386/tcg-target.h -> -index 928e8b8..b6a4cf2 100644 -> ---- a/tcg/i386/tcg-target.h -> -+++ b/tcg/i386/tcg-target.h -> -@@ -124,7 +124,7 @@ extern bool have_avx2; -> -#define TCG_TARGET_HAS_deposit_i32 1 -> -#define TCG_TARGET_HAS_extract_i32 1 -> -#define TCG_TARGET_HAS_sextract_i32 1 -> --#define TCG_TARGET_HAS_extract2_i32 1 -> -+#define TCG_TARGET_HAS_extract2_i32 0 -> -#define TCG_TARGET_HAS_movcond_i32 1 -> -#define TCG_TARGET_HAS_add2_i32 1 -> -#define TCG_TARGET_HAS_sub2_i32 1 -> -@@ -163,7 +163,7 @@ extern bool have_avx2; -> -#define TCG_TARGET_HAS_deposit_i64 1 -> -#define TCG_TARGET_HAS_extract_i64 1 -> -#define TCG_TARGET_HAS_sextract_i64 0 -> --#define TCG_TARGET_HAS_extract2_i64 1 -> -+#define TCG_TARGET_HAS_extract2_i64 0 -> -#define TCG_TARGET_HAS_movcond_i64 1 -> -#define TCG_TARGET_HAS_add2_i64 1 -> -#define TCG_TARGET_HAS_sub2_i64 1 -> -> -... the problem disappears. -> -It looks the problem is in this code segment in of tcg_gen_deposit_i32(): - - if (ofs == 0) { - tcg_gen_extract2_i32(ret, arg1, arg2, len); - tcg_gen_rotli_i32(ret, ret, len); - goto done; - } - -) - -If that code segment is deleted altogether (which effectively forces -usage of "fallback" part of tcg_gen_deposit_i32()), the problem also -vanishes (without changes from my previous mail). - -> -> -> Yours, -> -> Aleksandar - -Aleksandar Markovic writes: - -> -Hello, Richard, Peter, and others. -> -> -As a part of activities before 4.1 release, I tested nanoMIPS support -> -in QEMU (which was officially fully integrated in 4.0, is currently -> -limited to system mode only, and was tested in a similar fashion right -> -prior to 4.0). -> -> -This support appears to be broken now. Following command line works in -> -4.0, but results in kernel panic for the current tip of the tree: -> -> -~/Build/qemu-test-revert-c6fb8c0cf704/mipsel-softmmu/qemu-system-mipsel -> --cpu I7200 -kernel generic_nano32r6el_page4k -M malta -serial stdio -m -> -1G -hda nanomips32r6_le_sf_2017.05-03-59-gf5595d6.ext4 -append -> -"mem=256m@0x0 rw console=ttyS0 vga=cirrus vesa=0x111 root=/dev/sda" -> -> -(kernel and rootfs image files used in this commend line can be -> -downloaded from the locations mentioned in our user guide) -> -> -The quick bisect points to the commit: -> -> -commit c6fb8c0cf704c4a1a48c3e99e995ad4c58150dab -> -Author: Richard Henderson -> -Date: Mon Feb 25 11:42:35 2019 -0800 -> -> -tcg/i386: Support INDEX_op_extract2_{i32,i64} -> -> -Signed-off-by: Richard Henderson -> -> -Please advise on further actions. -Please see the fix: - - Subject: [PATCH for-4.1] tcg: Fix constant folding of INDEX_op_extract2_i32 - Date: Tue, 9 Jul 2019 14:19:00 +0200 - Message-Id: - -> -> -Yours, -> -Aleksandar --- -Alex Bennée - -On Sat, Jul 13, 2019 at 9:21 AM Alex Bennée wrote: -> -> -Please see the fix: -> -> -Subject: [PATCH for-4.1] tcg: Fix constant folding of INDEX_op_extract2_i32 -> -Date: Tue, 9 Jul 2019 14:19:00 +0200 -> -Message-Id: -> -Thanks, this fixed the behavior. - -Sincerely, -Aleksandar - -> -> -> -> -> Yours, -> -> Aleksandar -> -> -> --- -> -Alex Bennée -> - diff --git a/results/classifier/016/TCG-x86/14488057 b/results/classifier/016/TCG-x86/14488057 deleted file mode 100644 index 32717019..00000000 --- a/results/classifier/016/TCG-x86/14488057 +++ /dev/null @@ -1,738 +0,0 @@ -x86: 0.998 -TCG: 0.890 -virtual: 0.788 -operating system: 0.750 -permissions: 0.093 -i386: 0.064 -files: 0.048 -VMM: 0.039 -assembly: 0.034 -debug: 0.029 -hypervisor: 0.028 -PID: 0.016 -mistranslation: 0.013 -kernel: 0.010 -semantic: 0.006 -register: 0.006 -user-level: 0.005 -network: 0.005 -architecture: 0.004 -alpha: 0.004 -device: 0.003 -performance: 0.003 -graphic: 0.002 -boot: 0.002 -risc-v: 0.002 -socket: 0.002 -peripherals: 0.001 -ppc: 0.001 -vnc: 0.001 -KVM: 0.000 -arm: 0.000 - -[Qemu-devel] [BUG] user-to-root privesc inside VM via bad translation caching - -This is an issue in QEMU's system emulation for X86 in TCG mode. -The issue permits an attacker who can execute code in guest ring 3 -with normal user privileges to inject code into other processes that -are running in guest ring 3, in particular root-owned processes. - -== reproduction steps == - - - Create an x86-64 VM and install Debian Jessie in it. The following - steps should all be executed inside the VM. - - Verify that procmail is installed and the correct version: - address@hidden:~# apt-cache show procmail | egrep 'Version|SHA' - Version: 3.22-24 - SHA1: 54ed2d51db0e76f027f06068ab5371048c13434c - SHA256: 4488cf6975af9134a9b5238d5d70e8be277f70caa45a840dfbefd2dc444bfe7f - - Install build-essential and nasm ("apt install build-essential nasm"). - - Unpack the exploit, compile it and run it: - address@hidden:~$ tar xvf procmail_cache_attack.tar - procmail_cache_attack/ - procmail_cache_attack/shellcode.asm - procmail_cache_attack/xp.c - procmail_cache_attack/compile.sh - procmail_cache_attack/attack.c - address@hidden:~$ cd procmail_cache_attack - address@hidden:~/procmail_cache_attack$ ./compile.sh - address@hidden:~/procmail_cache_attack$ ./attack - memory mappings set up - child is dead, codegen should be complete - executing code as root! :) - address@hidden:~/procmail_cache_attack# id - uid=0(root) gid=0(root) groups=0(root),[...] - -Note: While the exploit depends on the precise version of procmail, -the actual vulnerability is in QEMU, not in procmail. procmail merely -serves as a seldomly-executed setuid root binary into which code can -be injected. - - -== detailed issue description == -QEMU caches translated basic blocks. To look up a translated basic -block, the function tb_find() is used, which uses tb_htable_lookup() -in its slowpath, which in turn compares translated basic blocks -(TranslationBlock) to the lookup information (struct tb_desc) using -tb_cmp(). - -tb_cmp() attempts to ensure (among other things) that both the virtual -start address of the basic block and the physical addresses that the -basic block covers match. When checking the physical addresses, it -assumes that a basic block can span at most two pages. - -gen_intermediate_code() attempts to enforce this by stopping the -translation of a basic block if nearly one page of instructions has -been translated already: - - /* if too long translation, stop generation too */ - if (tcg_op_buf_full() || - (pc_ptr - pc_start) >= (TARGET_PAGE_SIZE - 32) || - num_insns >= max_insns) { - gen_jmp_im(pc_ptr - dc->cs_base); - gen_eob(dc); - break; - } - -However, while real X86 processors have a maximum instruction length -of 15 bytes, QEMU's instruction decoder for X86 does not place any -limit on the instruction length or the number of instruction prefixes. -Therefore, it is possible to create an arbitrarily long instruction -by e.g. prepending an arbitrary number of LOCK prefixes to a normal -instruction. This permits creating a basic block that spans three -pages by simply appending an approximately page-sized instruction to -the end of a normal basic block that starts close to the end of a -page. - -Such an overlong basic block causes the basic block caching to fail as -follows: If code is generated and cached for a basic block that spans -the physical pages (A,E,B), this basic block will be returned by -lookups in a process in which the physical pages (A,B,C) are mapped -in the same virtual address range (assuming that all other lookup -parameters match). - -This behavior can be abused by an attacker e.g. as follows: If a -non-relocatable world-readable setuid executable legitimately contains -the pages (A,B,C), an attacker can map (A,E,B) into his own process, -at the normal load address of A, where E is an attacker-controlled -page. If a legitimate basic block spans the pages A and B, an attacker -can write arbitrary non-branch instructions at the start of E, then -append an overlong instruction -that ends behind the start of C, yielding a modified basic block that -spans all three pages. If the attacker then executes the modified -basic block in his process, the modified basic block is cached. -Next, the attacker can execute the setuid binary, which will reuse the -cached modified basic block, executing attacker-controlled -instructions in the context of the privileged process. - -I am sending this to qemu-devel because a QEMU security contact -told me that QEMU does not consider privilege escalation inside a -TCG VM to be a security concern. -procmail_cache_attack.tar -Description: -Unix tar archive - -On 20 March 2017 at 14:36, Jann Horn wrote: -> -This is an issue in QEMU's system emulation for X86 in TCG mode. -> -The issue permits an attacker who can execute code in guest ring 3 -> -with normal user privileges to inject code into other processes that -> -are running in guest ring 3, in particular root-owned processes. -> -I am sending this to qemu-devel because a QEMU security contact -> -told me that QEMU does not consider privilege escalation inside a -> -TCG VM to be a security concern. -Correct; it's just a bug. Don't trust TCG QEMU as a security boundary. - -We should really fix the crossing-a-page-boundary code for x86. -I believe we do get it correct for ARM Thumb instructions. - -thanks --- PMM - -On Mon, Mar 20, 2017 at 10:46 AM, Peter Maydell wrote: -> -On 20 March 2017 at 14:36, Jann Horn wrote: -> -> This is an issue in QEMU's system emulation for X86 in TCG mode. -> -> The issue permits an attacker who can execute code in guest ring 3 -> -> with normal user privileges to inject code into other processes that -> -> are running in guest ring 3, in particular root-owned processes. -> -> -> I am sending this to qemu-devel because a QEMU security contact -> -> told me that QEMU does not consider privilege escalation inside a -> -> TCG VM to be a security concern. -> -> -Correct; it's just a bug. Don't trust TCG QEMU as a security boundary. -> -> -We should really fix the crossing-a-page-boundary code for x86. -> -I believe we do get it correct for ARM Thumb instructions. -How about doing the instruction size check as follows? - -diff --git a/target/i386/translate.c b/target/i386/translate.c -index 72c1b03a2a..94cf3da719 100644 ---- a/target/i386/translate.c -+++ b/target/i386/translate.c -@@ -8235,6 +8235,10 @@ static target_ulong disas_insn(CPUX86State -*env, DisasContext *s, - default: - goto unknown_op; - } -+ if (s->pc - pc_start > 15) { -+ s->pc = pc_start; -+ goto illegal_op; -+ } - return s->pc; - illegal_op: - gen_illegal_opcode(s); - -Thanks, --- -Pranith - -On 22 March 2017 at 14:55, Pranith Kumar wrote: -> -On Mon, Mar 20, 2017 at 10:46 AM, Peter Maydell wrote: -> -> On 20 March 2017 at 14:36, Jann Horn wrote: -> ->> This is an issue in QEMU's system emulation for X86 in TCG mode. -> ->> The issue permits an attacker who can execute code in guest ring 3 -> ->> with normal user privileges to inject code into other processes that -> ->> are running in guest ring 3, in particular root-owned processes. -> -> -> ->> I am sending this to qemu-devel because a QEMU security contact -> ->> told me that QEMU does not consider privilege escalation inside a -> ->> TCG VM to be a security concern. -> -> -> -> Correct; it's just a bug. Don't trust TCG QEMU as a security boundary. -> -> -> -> We should really fix the crossing-a-page-boundary code for x86. -> -> I believe we do get it correct for ARM Thumb instructions. -> -> -How about doing the instruction size check as follows? -> -> -diff --git a/target/i386/translate.c b/target/i386/translate.c -> -index 72c1b03a2a..94cf3da719 100644 -> ---- a/target/i386/translate.c -> -+++ b/target/i386/translate.c -> -@@ -8235,6 +8235,10 @@ static target_ulong disas_insn(CPUX86State -> -*env, DisasContext *s, -> -default: -> -goto unknown_op; -> -} -> -+ if (s->pc - pc_start > 15) { -> -+ s->pc = pc_start; -> -+ goto illegal_op; -> -+ } -> -return s->pc; -> -illegal_op: -> -gen_illegal_opcode(s); -This doesn't look right because it means we'll check -only after we've emitted all the code to do the -instruction operation, so the effect will be -"execute instruction, then take illegal-opcode -exception". - -We should check what the x86 architecture spec actually -says and implement that. - -thanks --- PMM - -On Wed, Mar 22, 2017 at 11:04 AM, Peter Maydell - wrote: -> -> -> -> How about doing the instruction size check as follows? -> -> -> -> diff --git a/target/i386/translate.c b/target/i386/translate.c -> -> index 72c1b03a2a..94cf3da719 100644 -> -> --- a/target/i386/translate.c -> -> +++ b/target/i386/translate.c -> -> @@ -8235,6 +8235,10 @@ static target_ulong disas_insn(CPUX86State -> -> *env, DisasContext *s, -> -> default: -> -> goto unknown_op; -> -> } -> -> + if (s->pc - pc_start > 15) { -> -> + s->pc = pc_start; -> -> + goto illegal_op; -> -> + } -> -> return s->pc; -> -> illegal_op: -> -> gen_illegal_opcode(s); -> -> -This doesn't look right because it means we'll check -> -only after we've emitted all the code to do the -> -instruction operation, so the effect will be -> -"execute instruction, then take illegal-opcode -> -exception". -> -The pc is restored to original address (s->pc = pc_start), so the -exception will overwrite the generated illegal instruction and will be -executed first. - -But yes, it's better to follow the architecture manual. - -Thanks, --- -Pranith - -On 22 March 2017 at 15:14, Pranith Kumar wrote: -> -On Wed, Mar 22, 2017 at 11:04 AM, Peter Maydell -> - wrote: -> -> This doesn't look right because it means we'll check -> -> only after we've emitted all the code to do the -> -> instruction operation, so the effect will be -> -> "execute instruction, then take illegal-opcode -> -> exception". -> -The pc is restored to original address (s->pc = pc_start), so the -> -exception will overwrite the generated illegal instruction and will be -> -executed first. -s->pc is the guest PC -- moving that backwards will -not do anything about the generated TCG IR that's -already been written. You'd need to rewind the -write pointer in the IR stream, which there is -no support for doing AFAIK. - -thanks --- PMM - -On Wed, Mar 22, 2017 at 11:21 AM, Peter Maydell - wrote: -> -On 22 March 2017 at 15:14, Pranith Kumar wrote: -> -> On Wed, Mar 22, 2017 at 11:04 AM, Peter Maydell -> -> wrote: -> ->> This doesn't look right because it means we'll check -> ->> only after we've emitted all the code to do the -> ->> instruction operation, so the effect will be -> ->> "execute instruction, then take illegal-opcode -> ->> exception". -> -> -> The pc is restored to original address (s->pc = pc_start), so the -> -> exception will overwrite the generated illegal instruction and will be -> -> executed first. -> -> -s->pc is the guest PC -- moving that backwards will -> -not do anything about the generated TCG IR that's -> -already been written. You'd need to rewind the -> -write pointer in the IR stream, which there is -> -no support for doing AFAIK. -Ah, OK. Thanks for the explanation. May be we should check the size of -the instruction while decoding the prefixes and error out once we -exceed the limit. We would not generate any IR code. - --- -Pranith - -On 03/23/2017 02:29 AM, Pranith Kumar wrote: -On Wed, Mar 22, 2017 at 11:21 AM, Peter Maydell - wrote: -On 22 March 2017 at 15:14, Pranith Kumar wrote: -On Wed, Mar 22, 2017 at 11:04 AM, Peter Maydell - wrote: -This doesn't look right because it means we'll check -only after we've emitted all the code to do the -instruction operation, so the effect will be -"execute instruction, then take illegal-opcode -exception". -The pc is restored to original address (s->pc = pc_start), so the -exception will overwrite the generated illegal instruction and will be -executed first. -s->pc is the guest PC -- moving that backwards will -not do anything about the generated TCG IR that's -already been written. You'd need to rewind the -write pointer in the IR stream, which there is -no support for doing AFAIK. -Ah, OK. Thanks for the explanation. May be we should check the size of -the instruction while decoding the prefixes and error out once we -exceed the limit. We would not generate any IR code. -Yes. -It would not enforce a true limit of 15 bytes, since you can't know that until -you've done the rest of the decode. But you'd be able to say that no more than -14 prefix + 1 opc + 6 modrm+sib+ofs + 4 immediate = 25 bytes is used. -Which does fix the bug. - - -r~ - -On 22/03/2017 21:01, Richard Henderson wrote: -> -> -> -> Ah, OK. Thanks for the explanation. May be we should check the size of -> -> the instruction while decoding the prefixes and error out once we -> -> exceed the limit. We would not generate any IR code. -> -> -Yes. -> -> -It would not enforce a true limit of 15 bytes, since you can't know that -> -until you've done the rest of the decode. But you'd be able to say that -> -no more than 14 prefix + 1 opc + 6 modrm+sib+ofs + 4 immediate = 25 -> -bytes is used. -> -> -Which does fix the bug. -Yeah, that would work for 2.9 if somebody wants to put together a patch. - Ensuring that all instruction fetching happens before translation side -effects is a little harder, but perhaps it's also the opportunity to get -rid of s->rip_offset which is a little ugly. - -Paolo - -On Thu, Mar 23, 2017 at 6:27 AM, Paolo Bonzini wrote: -> -> -> -On 22/03/2017 21:01, Richard Henderson wrote: -> ->> -> ->> Ah, OK. Thanks for the explanation. May be we should check the size of -> ->> the instruction while decoding the prefixes and error out once we -> ->> exceed the limit. We would not generate any IR code. -> -> -> -> Yes. -> -> -> -> It would not enforce a true limit of 15 bytes, since you can't know that -> -> until you've done the rest of the decode. But you'd be able to say that -> -> no more than 14 prefix + 1 opc + 6 modrm+sib+ofs + 4 immediate = 25 -> -> bytes is used. -> -> -> -> Which does fix the bug. -> -> -Yeah, that would work for 2.9 if somebody wants to put together a patch. -> -Ensuring that all instruction fetching happens before translation side -> -effects is a little harder, but perhaps it's also the opportunity to get -> -rid of s->rip_offset which is a little ugly. -How about the following? - -diff --git a/target/i386/translate.c b/target/i386/translate.c -index 72c1b03a2a..67c58b8900 100644 ---- a/target/i386/translate.c -+++ b/target/i386/translate.c -@@ -4418,6 +4418,11 @@ static target_ulong disas_insn(CPUX86State -*env, DisasContext *s, - s->vex_l = 0; - s->vex_v = 0; - next_byte: -+ /* The prefixes can atmost be 14 bytes since x86 has an upper -+ limit of 15 bytes for the instruction */ -+ if (s->pc - pc_start > 14) { -+ goto illegal_op; -+ } - b = cpu_ldub_code(env, s->pc); - s->pc++; - /* Collect prefixes. */ - --- -Pranith - -On 23/03/2017 17:50, Pranith Kumar wrote: -> -On Thu, Mar 23, 2017 at 6:27 AM, Paolo Bonzini wrote: -> -> -> -> -> -> On 22/03/2017 21:01, Richard Henderson wrote: -> ->>> -> ->>> Ah, OK. Thanks for the explanation. May be we should check the size of -> ->>> the instruction while decoding the prefixes and error out once we -> ->>> exceed the limit. We would not generate any IR code. -> ->> -> ->> Yes. -> ->> -> ->> It would not enforce a true limit of 15 bytes, since you can't know that -> ->> until you've done the rest of the decode. But you'd be able to say that -> ->> no more than 14 prefix + 1 opc + 6 modrm+sib+ofs + 4 immediate = 25 -> ->> bytes is used. -> ->> -> ->> Which does fix the bug. -> -> -> -> Yeah, that would work for 2.9 if somebody wants to put together a patch. -> -> Ensuring that all instruction fetching happens before translation side -> -> effects is a little harder, but perhaps it's also the opportunity to get -> -> rid of s->rip_offset which is a little ugly. -> -> -How about the following? -> -> -diff --git a/target/i386/translate.c b/target/i386/translate.c -> -index 72c1b03a2a..67c58b8900 100644 -> ---- a/target/i386/translate.c -> -+++ b/target/i386/translate.c -> -@@ -4418,6 +4418,11 @@ static target_ulong disas_insn(CPUX86State -> -*env, DisasContext *s, -> -s->vex_l = 0; -> -s->vex_v = 0; -> -next_byte: -> -+ /* The prefixes can atmost be 14 bytes since x86 has an upper -> -+ limit of 15 bytes for the instruction */ -> -+ if (s->pc - pc_start > 14) { -> -+ goto illegal_op; -> -+ } -> -b = cpu_ldub_code(env, s->pc); -> -s->pc++; -> -/* Collect prefixes. */ -Please make the comment more verbose, based on Richard's remark. We -should apply it to 2.9. - -Also, QEMU usually formats comments with stars on every line. - -Paolo - -On Thu, Mar 23, 2017 at 1:37 PM, Paolo Bonzini wrote: -> -> -> -On 23/03/2017 17:50, Pranith Kumar wrote: -> -> On Thu, Mar 23, 2017 at 6:27 AM, Paolo Bonzini wrote: -> ->> -> ->> -> ->> On 22/03/2017 21:01, Richard Henderson wrote: -> ->>>> -> ->>>> Ah, OK. Thanks for the explanation. May be we should check the size of -> ->>>> the instruction while decoding the prefixes and error out once we -> ->>>> exceed the limit. We would not generate any IR code. -> ->>> -> ->>> Yes. -> ->>> -> ->>> It would not enforce a true limit of 15 bytes, since you can't know that -> ->>> until you've done the rest of the decode. But you'd be able to say that -> ->>> no more than 14 prefix + 1 opc + 6 modrm+sib+ofs + 4 immediate = 25 -> ->>> bytes is used. -> ->>> -> ->>> Which does fix the bug. -> ->> -> ->> Yeah, that would work for 2.9 if somebody wants to put together a patch. -> ->> Ensuring that all instruction fetching happens before translation side -> ->> effects is a little harder, but perhaps it's also the opportunity to get -> ->> rid of s->rip_offset which is a little ugly. -> -> -> -> How about the following? -> -> -> -> diff --git a/target/i386/translate.c b/target/i386/translate.c -> -> index 72c1b03a2a..67c58b8900 100644 -> -> --- a/target/i386/translate.c -> -> +++ b/target/i386/translate.c -> -> @@ -4418,6 +4418,11 @@ static target_ulong disas_insn(CPUX86State -> -> *env, DisasContext *s, -> -> s->vex_l = 0; -> -> s->vex_v = 0; -> -> next_byte: -> -> + /* The prefixes can atmost be 14 bytes since x86 has an upper -> -> + limit of 15 bytes for the instruction */ -> -> + if (s->pc - pc_start > 14) { -> -> + goto illegal_op; -> -> + } -> -> b = cpu_ldub_code(env, s->pc); -> -> s->pc++; -> -> /* Collect prefixes. */ -> -> -Please make the comment more verbose, based on Richard's remark. We -> -should apply it to 2.9. -> -> -Also, QEMU usually formats comments with stars on every line. -OK. I'll send a proper patch with updated comment. - -Thanks, --- -Pranith - diff --git a/results/classifier/016/TCG-x86/66743673 b/results/classifier/016/TCG-x86/66743673 deleted file mode 100644 index 6109dbb8..00000000 --- a/results/classifier/016/TCG-x86/66743673 +++ /dev/null @@ -1,391 +0,0 @@ -x86: 0.976 -TCG: 0.891 -kernel: 0.468 -debug: 0.421 -virtual: 0.306 -hypervisor: 0.247 -operating system: 0.237 -files: 0.057 -PID: 0.046 -register: 0.044 -architecture: 0.041 -socket: 0.027 -boot: 0.022 -i386: 0.021 -semantic: 0.016 -device: 0.012 -vnc: 0.012 -risc-v: 0.009 -user-level: 0.008 -VMM: 0.008 -KVM: 0.008 -assembly: 0.008 -performance: 0.005 -network: 0.004 -permissions: 0.002 -graphic: 0.002 -alpha: 0.002 -peripherals: 0.001 -ppc: 0.001 -mistranslation: 0.001 -arm: 0.000 - -[Bug] QEMU TCG warnings after commit c6bd2dd63420 - HTT / CMP_LEG bits - -Hi Community, - -This email contains 3 bugs appear to share the same root cause. - -[1] We ran into the following warnings when running QEMU v10.0.0 in TCG mode: - -qemu-system-x86_64 \ - -machine q35 \ - -m 4G -smp 4 \ - -kernel ./arch/x86/boot/bzImage \ - -bios /usr/share/ovmf/OVMF.fd \ - -drive file=~/kernel/rootfs.ext4,index=0,format=raw,media=disk \ - -drive file=~/kernel/swap.img,index=1,format=raw,media=disk \ - -nographic \ - -append 'root=/dev/sda rw resume=/dev/sdb console=ttyS0 nokaslr' -qemu-system-x86_64: warning: TCG doesn't support requested feature: -CPUID.01H:EDX.ht [bit 28] -qemu-system-x86_64: warning: TCG doesn't support requested feature: -CPUID.80000001H:ECX.cmp-legacy [bit 1] -(repeats 4 times, once per vCPU) -Tracing the history shows that commit c6bd2dd63420 "i386/cpu: Set up CPUID_HT in -x86_cpu_expand_features() instead of cpu_x86_cpuid()" is what introduced the -warnings. -Since that commit, TCG unconditionally advertises HTT (CPUID 1 EDX[28]) and -CMP_LEG (CPUID 8000_0001 ECX[1]). Because TCG itself has no SMT support, these -bits trigger the warnings above. -[2] Also, Zhao pointed me to a similar report on GitLab: -https://gitlab.com/qemu-project/qemu/-/issues/2894 -The symptoms there look identical to what we're seeing. -By convention we file one issue per email, but these two appear to share the -same root cause, so I'm describing them together here. -[3] My colleague Alan noticed what appears to be a related problem: if we launch -a guest with '-cpu ,-ht --enable-kvm', which means explicitly removing -the ht flag, but the guest still reports HT(cat /proc/cpuinfo in linux guest) -enabled. In other words, under KVM the ht bit seems to be forced on even when -the user tries to disable it. -Best regards, -Ewan - -On 4/29/25 11:02 AM, Ewan Hai wrote: -Hi Community, - -This email contains 3 bugs appear to share the same root cause. - -[1] We ran into the following warnings when running QEMU v10.0.0 in TCG mode: - -qemu-system-x86_64 \ -   -machine q35 \ -   -m 4G -smp 4 \ -   -kernel ./arch/x86/boot/bzImage \ -   -bios /usr/share/ovmf/OVMF.fd \ -   -drive file=~/kernel/rootfs.ext4,index=0,format=raw,media=disk \ -   -drive file=~/kernel/swap.img,index=1,format=raw,media=disk \ -   -nographic \ -   -append 'root=/dev/sda rw resume=/dev/sdb console=ttyS0 nokaslr' -qemu-system-x86_64: warning: TCG doesn't support requested feature: -CPUID.01H:EDX.ht [bit 28] -qemu-system-x86_64: warning: TCG doesn't support requested feature: -CPUID.80000001H:ECX.cmp-legacy [bit 1] -(repeats 4 times, once per vCPU) -Tracing the history shows that commit c6bd2dd63420 "i386/cpu: Set up CPUID_HT in -x86_cpu_expand_features() instead of cpu_x86_cpuid()" is what introduced the -warnings. -Since that commit, TCG unconditionally advertises HTT (CPUID 1 EDX[28]) and -CMP_LEG (CPUID 8000_0001 ECX[1]). Because TCG itself has no SMT support, these -bits trigger the warnings above. -[2] Also, Zhao pointed me to a similar report on GitLab: -https://gitlab.com/qemu-project/qemu/-/issues/2894 -The symptoms there look identical to what we're seeing. -By convention we file one issue per email, but these two appear to share the -same root cause, so I'm describing them together here. -[3] My colleague Alan noticed what appears to be a related problem: if we launch -a guest with '-cpu ,-ht --enable-kvm', which means explicitly removing -the ht flag, but the guest still reports HT(cat /proc/cpuinfo in linux guest) -enabled. In other words, under KVM the ht bit seems to be forced on even when -the user tries to disable it. -XiaoYao reminded me that issue [3] stems from a different patch. Please ignore -it for now—I'll start a separate thread to discuss that one independently. -Best regards, -Ewan - -On 4/29/2025 11:02 AM, Ewan Hai wrote: -Hi Community, - -This email contains 3 bugs appear to share the same root cause. -[1] We ran into the following warnings when running QEMU v10.0.0 in TCG -mode: -qemu-system-x86_64 \ -   -machine q35 \ -   -m 4G -smp 4 \ -   -kernel ./arch/x86/boot/bzImage \ -   -bios /usr/share/ovmf/OVMF.fd \ -   -drive file=~/kernel/rootfs.ext4,index=0,format=raw,media=disk \ -   -drive file=~/kernel/swap.img,index=1,format=raw,media=disk \ -   -nographic \ -   -append 'root=/dev/sda rw resume=/dev/sdb console=ttyS0 nokaslr' -qemu-system-x86_64: warning: TCG doesn't support requested feature: -CPUID.01H:EDX.ht [bit 28] -qemu-system-x86_64: warning: TCG doesn't support requested feature: -CPUID.80000001H:ECX.cmp-legacy [bit 1] -(repeats 4 times, once per vCPU) -Tracing the history shows that commit c6bd2dd63420 "i386/cpu: Set up -CPUID_HT in x86_cpu_expand_features() instead of cpu_x86_cpuid()" is -what introduced the warnings. -Since that commit, TCG unconditionally advertises HTT (CPUID 1 EDX[28]) -and CMP_LEG (CPUID 8000_0001 ECX[1]). Because TCG itself has no SMT -support, these bits trigger the warnings above. -[2] Also, Zhao pointed me to a similar report on GitLab: -https://gitlab.com/qemu-project/qemu/-/issues/2894 -The symptoms there look identical to what we're seeing. -By convention we file one issue per email, but these two appear to share -the same root cause, so I'm describing them together here. -It was caused by my two patches. I think the fix can be as follow. -If no objection from the community, I can submit the formal patch. - -diff --git a/target/i386/cpu.c b/target/i386/cpu.c -index 1f970aa4daa6..fb95aadd6161 100644 ---- a/target/i386/cpu.c -+++ b/target/i386/cpu.c -@@ -776,11 +776,12 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t -vendor1, -CPUID_PAE | CPUID_MCE | CPUID_CX8 | CPUID_APIC | CPUID_SEP | \ - CPUID_MTRR | CPUID_PGE | CPUID_MCA | CPUID_CMOV | CPUID_PAT | \ - CPUID_PSE36 | CPUID_CLFLUSH | CPUID_ACPI | CPUID_MMX | \ -- CPUID_FXSR | CPUID_SSE | CPUID_SSE2 | CPUID_SS | CPUID_DE) -+ CPUID_FXSR | CPUID_SSE | CPUID_SSE2 | CPUID_SS | CPUID_DE | \ -+ CPUID_HT) - /* partly implemented: - CPUID_MTRR, CPUID_MCA, CPUID_CLFLUSH (needed for Win64) */ - /* missing: -- CPUID_VME, CPUID_DTS, CPUID_SS, CPUID_HT, CPUID_TM, CPUID_PBE */ -+ CPUID_VME, CPUID_DTS, CPUID_SS, CPUID_TM, CPUID_PBE */ - - /* - * Kernel-only features that can be shown to usermode programs even if -@@ -848,7 +849,8 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t -vendor1, -#define TCG_EXT3_FEATURES (CPUID_EXT3_LAHF_LM | CPUID_EXT3_SVM | \ - CPUID_EXT3_CR8LEG | CPUID_EXT3_ABM | CPUID_EXT3_SSE4A | \ -- CPUID_EXT3_3DNOWPREFETCH | CPUID_EXT3_KERNEL_FEATURES) -+ CPUID_EXT3_3DNOWPREFETCH | CPUID_EXT3_KERNEL_FEATURES | \ -+ CPUID_EXT3_CMP_LEG) - - #define TCG_EXT4_FEATURES 0 -[3] My colleague Alan noticed what appears to be a related problem: if -we launch a guest with '-cpu ,-ht --enable-kvm', which means -explicitly removing the ht flag, but the guest still reports HT(cat / -proc/cpuinfo in linux guest) enabled. In other words, under KVM the ht -bit seems to be forced on even when the user tries to disable it. -This has been the behavior of QEMU for many years, not some regression -introduced by my patches. We can discuss how to address it separately. -Best regards, -Ewan - -On Tue, Apr 29, 2025 at 01:55:59PM +0800, Xiaoyao Li wrote: -> -Date: Tue, 29 Apr 2025 13:55:59 +0800 -> -From: Xiaoyao Li -> -Subject: Re: [Bug] QEMU TCG warnings after commit c6bd2dd63420 - HTT / -> -CMP_LEG bits -> -> -On 4/29/2025 11:02 AM, Ewan Hai wrote: -> -> Hi Community, -> -> -> -> This email contains 3 bugs appear to share the same root cause. -> -> -> -> [1] We ran into the following warnings when running QEMU v10.0.0 in TCG -> -> mode: -> -> -> -> qemu-system-x86_64 \ -> ->   -machine q35 \ -> ->   -m 4G -smp 4 \ -> ->   -kernel ./arch/x86/boot/bzImage \ -> ->   -bios /usr/share/ovmf/OVMF.fd \ -> ->   -drive file=~/kernel/rootfs.ext4,index=0,format=raw,media=disk \ -> ->   -drive file=~/kernel/swap.img,index=1,format=raw,media=disk \ -> ->   -nographic \ -> ->   -append 'root=/dev/sda rw resume=/dev/sdb console=ttyS0 nokaslr' -> -> -> -> qemu-system-x86_64: warning: TCG doesn't support requested feature: -> -> CPUID.01H:EDX.ht [bit 28] -> -> qemu-system-x86_64: warning: TCG doesn't support requested feature: -> -> CPUID.80000001H:ECX.cmp-legacy [bit 1] -> -> (repeats 4 times, once per vCPU) -> -> -> -> Tracing the history shows that commit c6bd2dd63420 "i386/cpu: Set up -> -> CPUID_HT in x86_cpu_expand_features() instead of cpu_x86_cpuid()" is -> -> what introduced the warnings. -> -> -> -> Since that commit, TCG unconditionally advertises HTT (CPUID 1 EDX[28]) -> -> and CMP_LEG (CPUID 8000_0001 ECX[1]). Because TCG itself has no SMT -> -> support, these bits trigger the warnings above. -> -> -> -> [2] Also, Zhao pointed me to a similar report on GitLab: -> -> -https://gitlab.com/qemu-project/qemu/-/issues/2894 -> -> The symptoms there look identical to what we're seeing. -> -> -> -> By convention we file one issue per email, but these two appear to share -> -> the same root cause, so I'm describing them together here. -> -> -It was caused by my two patches. I think the fix can be as follow. -> -If no objection from the community, I can submit the formal patch. -> -> -diff --git a/target/i386/cpu.c b/target/i386/cpu.c -> -index 1f970aa4daa6..fb95aadd6161 100644 -> ---- a/target/i386/cpu.c -> -+++ b/target/i386/cpu.c -> -@@ -776,11 +776,12 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t -> -vendor1, -> -CPUID_PAE | CPUID_MCE | CPUID_CX8 | CPUID_APIC | CPUID_SEP | \ -> -CPUID_MTRR | CPUID_PGE | CPUID_MCA | CPUID_CMOV | CPUID_PAT | \ -> -CPUID_PSE36 | CPUID_CLFLUSH | CPUID_ACPI | CPUID_MMX | \ -> -- CPUID_FXSR | CPUID_SSE | CPUID_SSE2 | CPUID_SS | CPUID_DE) -> -+ CPUID_FXSR | CPUID_SSE | CPUID_SSE2 | CPUID_SS | CPUID_DE | \ -> -+ CPUID_HT) -> -/* partly implemented: -> -CPUID_MTRR, CPUID_MCA, CPUID_CLFLUSH (needed for Win64) */ -> -/* missing: -> -- CPUID_VME, CPUID_DTS, CPUID_SS, CPUID_HT, CPUID_TM, CPUID_PBE */ -> -+ CPUID_VME, CPUID_DTS, CPUID_SS, CPUID_TM, CPUID_PBE */ -> -> -/* -> -* Kernel-only features that can be shown to usermode programs even if -> -@@ -848,7 +849,8 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t -> -vendor1, -> -> -#define TCG_EXT3_FEATURES (CPUID_EXT3_LAHF_LM | CPUID_EXT3_SVM | \ -> -CPUID_EXT3_CR8LEG | CPUID_EXT3_ABM | CPUID_EXT3_SSE4A | \ -> -- CPUID_EXT3_3DNOWPREFETCH | CPUID_EXT3_KERNEL_FEATURES) -> -+ CPUID_EXT3_3DNOWPREFETCH | CPUID_EXT3_KERNEL_FEATURES | \ -> -+ CPUID_EXT3_CMP_LEG) -> -> -#define TCG_EXT4_FEATURES 0 -This fix is fine for me...at least from SDM, HTT depends on topology and -it should exist when user sets "-smp 4". - -> -> [3] My colleague Alan noticed what appears to be a related problem: if -> -> we launch a guest with '-cpu ,-ht --enable-kvm', which means -> -> explicitly removing the ht flag, but the guest still reports HT(cat -> -> /proc/cpuinfo in linux guest) enabled. In other words, under KVM the ht -> -> bit seems to be forced on even when the user tries to disable it. -> -> -XiaoYao reminded me that issue [3] stems from a different patch. Please -> -ignore it for now—I'll start a separate thread to discuss that one -> -independently. -I haven't found any other thread :-). - -By the way, just curious, in what cases do you need to disbale the HT -flag? "-smp 4" means 4 cores with 1 thread per core, and is it not -enough? - -As for the “-ht” behavior, I'm also unsure whether this should be fixed -or not - one possible consideration is whether “-ht” would be useful. - -On 5/8/25 5:04 PM, Zhao Liu wrote: -[3] My colleague Alan noticed what appears to be a related problem: if -we launch a guest with '-cpu ,-ht --enable-kvm', which means -explicitly removing the ht flag, but the guest still reports HT(cat -/proc/cpuinfo in linux guest) enabled. In other words, under KVM the ht -bit seems to be forced on even when the user tries to disable it. -XiaoYao reminded me that issue [3] stems from a different patch. Please -ignore it for now—I'll start a separate thread to discuss that one -independently. -I haven't found any other thread :-). -Please refer to -https://lore.kernel.org/all/db6ae3bb-f4e5-4719-9beb-623fcff56af2@zhaoxin.com/ -. -By the way, just curious, in what cases do you need to disbale the HT -flag? "-smp 4" means 4 cores with 1 thread per core, and is it not -enough? - -As for the “-ht” behavior, I'm also unsure whether this should be fixed -or not - one possible consideration is whether “-ht” would be useful. -I wasn't trying to target any specific use case, using "-ht" was simply a way to -check how the ht feature behaves under both KVM and TCG. There's no special -workload behind it; I just wanted to confirm that the flag is respected (or not) -in each mode. - diff --git a/results/classifier/016/TCG-x86/80615920 b/results/classifier/016/TCG-x86/80615920 deleted file mode 100644 index 6ccc1e19..00000000 --- a/results/classifier/016/TCG-x86/80615920 +++ /dev/null @@ -1,375 +0,0 @@ -TCG: 0.928 -operating system: 0.873 -x86: 0.854 -kernel: 0.848 -debug: 0.296 -i386: 0.187 -performance: 0.112 -virtual: 0.069 -files: 0.046 -risc-v: 0.036 -PID: 0.035 -user-level: 0.023 -VMM: 0.020 -register: 0.012 -architecture: 0.009 -boot: 0.007 -assembly: 0.005 -KVM: 0.005 -device: 0.005 -semantic: 0.005 -hypervisor: 0.004 -alpha: 0.004 -graphic: 0.002 -ppc: 0.002 -network: 0.001 -peripherals: 0.001 -socket: 0.001 -vnc: 0.001 -permissions: 0.001 -arm: 0.001 -mistranslation: 0.001 - -[BUG] accel/tcg: cpu_exec_longjmp_cleanup: assertion failed: (cpu == current_cpu) - -It seems there is a bug in SIGALRM handling when 486 system emulates x86_64 -code. - -This code: - -#include -#include -#include -#include -#include - -pthread_t thread1, thread2; - -// Signal handler for SIGALRM -void alarm_handler(int sig) { - // Do nothing, just wake up the other thread -} - -// Thread 1 function -void* thread1_func(void* arg) { - // Set up the signal handler for SIGALRM - signal(SIGALRM, alarm_handler); - - // Wait for 5 seconds - sleep(1); - - // Send SIGALRM signal to thread 2 - pthread_kill(thread2, SIGALRM); - - return NULL; -} - -// Thread 2 function -void* thread2_func(void* arg) { - // Wait for the SIGALRM signal - pause(); - - printf("Thread 2 woke up!\n"); - - return NULL; -} - -int main() { - // Create thread 1 - if (pthread_create(&thread1, NULL, thread1_func, NULL) != 0) { - fprintf(stderr, "Failed to create thread 1\n"); - return 1; - } - - // Create thread 2 - if (pthread_create(&thread2, NULL, thread2_func, NULL) != 0) { - fprintf(stderr, "Failed to create thread 2\n"); - return 1; - } - - // Wait for both threads to finish - pthread_join(thread1, NULL); - pthread_join(thread2, NULL); - - return 0; -} - - -Fails with this -strace log (there are also unsupported syscalls 334 and 435, -but it seems it doesn't affect the code much): - -... -736 rt_sigaction(SIGALRM,0x000000001123ec20,0x000000001123ecc0) = 0 -736 clock_nanosleep(CLOCK_REALTIME,0,{tv_sec = 1,tv_nsec = 0},{tv_sec = -1,tv_nsec = 0}) -736 rt_sigprocmask(SIG_BLOCK,0x00000000109fad20,0x0000000010800b38,8) = 0 -736 Unknown syscall 435 -736 -clone(CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID| - ... -736 rt_sigprocmask(SIG_SETMASK,0x0000000010800b38,NULL,8) -736 set_robust_list(0x11a419a0,0) = -1 errno=38 (Function not implemented) -736 rt_sigprocmask(SIG_SETMASK,0x0000000011a41fb0,NULL,8) = 0 - = 0 -736 pause(0,0,2,277186368,0,295966400) -736 -futex(0x000000001123f990,FUTEX_CLOCK_REALTIME|FUTEX_WAIT_BITSET,738,NULL,NULL,0) - = 0 -736 rt_sigprocmask(SIG_BLOCK,0x00000000109fad20,0x000000001123ee88,8) = 0 -736 getpid() = 736 -736 tgkill(736,739,SIGALRM) = 0 - = -1 errno=4 (Interrupted system call) ---- SIGALRM {si_signo=SIGALRM, si_code=SI_TKILL, si_pid=736, si_uid=0} --- -0x48874a != 0x3c69e10 -736 rt_sigprocmask(SIG_SETMASK,0x000000001123ee88,NULL,8) = 0 -** -ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion failed: -(cpu == current_cpu) -Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion -failed: (cpu == current_cpu) -0x48874a != 0x3c69e10 -** -ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion failed: -(cpu == current_cpu) -Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion -failed: (cpu == current_cpu) -# - -The code fails either with or without -singlestep, the command line: - -/usr/bin/qemu-x86_64 -L /opt/x86_64 -strace -singlestep /opt/x86_64/alarm.bin - -Source code of QEMU 8.1.1 was modified with patch "[PATCH] qemu/timer: Don't -use RDTSC on i486" [1], -with added few ioctls (not relevant) and cpu_exec_longjmp_cleanup() now prints -current pointers of -cpu and current_cpu (line "0x48874a != 0x3c69e10"). - -config.log (built as a part of buildroot, basically the minimal possible -configuration for running x86_64 on 486): - -# Configured with: -'/mnt/hd_8tb_p1/p1/home/crossgen/buildroot_486_2/output/build/qemu-8.1.1/configure' - '--prefix=/usr' -'--cross-prefix=/mnt/hd_8tb_p1/p1/home/crossgen/buildroot_486_2/output/host/bin/i486-buildroot-linux-gnu-' - '--audio-drv-list=' -'--python=/mnt/hd_8tb_p1/p1/home/crossgen/buildroot_486_2/output/host/bin/python3' - -'--ninja=/mnt/hd_8tb_p1/p1/home/crossgen/buildroot_486_2/output/host/bin/ninja' -'--disable-alsa' '--disable-bpf' '--disable-brlapi' '--disable-bsd-user' -'--disable-cap-ng' '--disable-capstone' '--disable-containers' -'--disable-coreaudio' '--disable-curl' '--disable-curses' -'--disable-dbus-display' '--disable-docs' '--disable-dsound' '--disable-hvf' -'--disable-jack' '--disable-libiscsi' '--disable-linux-aio' -'--disable-linux-io-uring' '--disable-malloc-trim' '--disable-membarrier' -'--disable-mpath' '--disable-netmap' '--disable-opengl' '--disable-oss' -'--disable-pa' '--disable-rbd' '--disable-sanitizers' '--disable-selinux' -'--disable-sparse' '--disable-strip' '--disable-vde' '--disable-vhost-crypto' -'--disable-vhost-user-blk-server' '--disable-virtfs' '--disable-whpx' -'--disable-xen' '--disable-attr' '--disable-kvm' '--disable-vhost-net' -'--disable-download' '--disable-hexagon-idef-parser' '--disable-system' -'--enable-linux-user' '--target-list=x86_64-linux-user' '--disable-vhost-user' -'--disable-slirp' '--disable-sdl' '--disable-fdt' '--enable-trace-backends=nop' -'--disable-tools' '--disable-guest-agent' '--disable-fuse' -'--disable-fuse-lseek' '--disable-seccomp' '--disable-libssh' -'--disable-libusb' '--disable-vnc' '--disable-nettle' '--disable-numa' -'--disable-pipewire' '--disable-spice' '--disable-usb-redir' -'--disable-install-blobs' - -Emulation of the same x86_64 code with qemu 6.2.0 installed on another x86_64 -native machine works fine. - -[1] -https://lists.nongnu.org/archive/html/qemu-devel/2023-11/msg05387.html -Best regards, -Petr - -On Sat, 25 Nov 2023 at 13:09, Petr Cvek wrote: -> -> -It seems there is a bug in SIGALRM handling when 486 system emulates x86_64 -> -code. -486 host is pretty well out of support currently. Can you reproduce -this on a less ancient host CPU type ? - -> -ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion failed: -> -(cpu == current_cpu) -> -Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: -> -assertion failed: (cpu == current_cpu) -> -0x48874a != 0x3c69e10 -> -** -> -ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion failed: -> -(cpu == current_cpu) -> -Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: -> -assertion failed: (cpu == current_cpu) -What compiler version do you build QEMU with? That -assert is there because we have seen some buggy compilers -in the past which don't correctly preserve the variable -value as the setjmp/longjmp spec requires them to. - -thanks --- PMM - -Dne 27. 11. 23 v 10:37 Peter Maydell napsal(a): -> -On Sat, 25 Nov 2023 at 13:09, Petr Cvek wrote: -> -> -> -> It seems there is a bug in SIGALRM handling when 486 system emulates x86_64 -> -> code. -> -> -486 host is pretty well out of support currently. Can you reproduce -> -this on a less ancient host CPU type ? -> -It seems it only fails when the code is compiled for i486. QEMU built with the -same compiler with -march=i586 and above runs on the same physical hardware -without a problem. All -march= variants were executed on ryzen 3600. - -> -> ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion -> -> failed: (cpu == current_cpu) -> -> Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: -> -> assertion failed: (cpu == current_cpu) -> -> 0x48874a != 0x3c69e10 -> -> ** -> -> ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion -> -> failed: (cpu == current_cpu) -> -> Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: -> -> assertion failed: (cpu == current_cpu) -> -> -What compiler version do you build QEMU with? That -> -assert is there because we have seen some buggy compilers -> -in the past which don't correctly preserve the variable -> -value as the setjmp/longjmp spec requires them to. -> -i486 and i586+ code variants were compiled with GCC 13.2.0 (more exactly, -slackware64 current multilib distribution). - -i486 binary which runs on the real 486 is also GCC 13.2.0 and installed as a -part of the buildroot crosscompiler (about two week old git snapshot). - -> -thanks -> --- PMM -best regards, -Petr - -On 11/25/23 07:08, Petr Cvek wrote: -ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion failed: -(cpu == current_cpu) -Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion -failed: (cpu == current_cpu) -# - -The code fails either with or without -singlestep, the command line: - -/usr/bin/qemu-x86_64 -L /opt/x86_64 -strace -singlestep /opt/x86_64/alarm.bin - -Source code of QEMU 8.1.1 was modified with patch "[PATCH] qemu/timer: Don't use -RDTSC on i486" [1], -with added few ioctls (not relevant) and cpu_exec_longjmp_cleanup() now prints -current pointers of -cpu and current_cpu (line "0x48874a != 0x3c69e10"). -If you try this again with 8.2-rc2, you should not see an assertion failure. -You should see instead - -QEMU internal SIGILL {code=ILLOPC, addr=0x12345678} -which I think more accurately summarizes the situation of attempting RDTSC on hardware -that does not support it. -r~ - -Dne 29. 11. 23 v 15:25 Richard Henderson napsal(a): -> -On 11/25/23 07:08, Petr Cvek wrote: -> -> ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: assertion -> -> failed: (cpu == current_cpu) -> -> Bail out! ERROR:../accel/tcg/cpu-exec.c:546:cpu_exec_longjmp_cleanup: -> -> assertion failed: (cpu == current_cpu) -> -> # -> -> -> -> The code fails either with or without -singlestep, the command line: -> -> -> -> /usr/bin/qemu-x86_64 -L /opt/x86_64 -strace -singlestep -> -> /opt/x86_64/alarm.bin -> -> -> -> Source code of QEMU 8.1.1 was modified with patch "[PATCH] qemu/timer: Don't -> -> use RDTSC on i486" [1], -> -> with added few ioctls (not relevant) and cpu_exec_longjmp_cleanup() now -> -> prints current pointers of -> -> cpu and current_cpu (line "0x48874a != 0x3c69e10"). -> -> -> -If you try this again with 8.2-rc2, you should not see an assertion failure. -> -You should see instead -> -> -QEMU internal SIGILL {code=ILLOPC, addr=0x12345678} -> -> -which I think more accurately summarizes the situation of attempting RDTSC on -> -hardware that does not support it. -> -> -Compilation of vanilla qemu v8.2.0-rc2 with -march=i486 by GCC 13.2.0 and -running the resulting binary on ryzen still leads to: - -** -ERROR:../accel/tcg/cpu-exec.c:533:cpu_exec_longjmp_cleanup: assertion failed: -(cpu == current_cpu) -Bail out! ERROR:../accel/tcg/cpu-exec.c:533:cpu_exec_longjmp_cleanup: assertion -failed: (cpu == current_cpu) -Aborted - -> -> -r~ -Petr - diff --git a/results/classifier/016/TCG/95154278 b/results/classifier/016/TCG/95154278 deleted file mode 100644 index 8c1a96f7..00000000 --- a/results/classifier/016/TCG/95154278 +++ /dev/null @@ -1,182 +0,0 @@ -TCG: 0.893 -boot: 0.799 -files: 0.766 -register: 0.658 -device: 0.619 -PID: 0.516 -VMM: 0.497 -risc-v: 0.481 -vnc: 0.477 -debug: 0.235 -peripherals: 0.180 -operating system: 0.170 -hypervisor: 0.162 -kernel: 0.124 -alpha: 0.121 -virtual: 0.104 -architecture: 0.073 -assembly: 0.048 -semantic: 0.039 -user-level: 0.019 -socket: 0.019 -KVM: 0.014 -performance: 0.012 -permissions: 0.011 -network: 0.008 -graphic: 0.004 -arm: 0.002 -x86: 0.002 -mistranslation: 0.002 -ppc: 0.002 -i386: 0.002 - -[Qemu-devel] [BUG] checkpatch.pl hangs on target/mips/msa_helper.c - -If checkpatch.pl is applied (using switch "-f") on file -target/mips/msa_helper.c, it will hang. - -There is a workaround for this particular file: - -These lines in msa_helper.c: - - uint## BITS ##_t S = _S, T = _T; \ - uint## BITS ##_t as, at, xs, xt, xd; \ - -should be replaced with: - - uint## BITS ## _t S = _S, T = _T; \ - uint## BITS ## _t as, at, xs, xt, xd; \ - -(a space is added after the second "##" in each line) - -The workaround is found by partial deleting and undeleting of the code in -msa_helper.c in binary search fashion. - -This workaround will soon be submitted by me as a patch within a series on misc -MIPS issues. - -I took a look at checkpatch.pl code, and it looks it is fairly complicated to -fix the issue, since it happens in the code segment involving intricate logic -conditions. - -Regards, -Aleksandar - -On Wed, Jul 04, 2018 at 03:35:18PM +0000, Aleksandar Markovic wrote: -> -If checkpatch.pl is applied (using switch "-f") on file -> -target/mips/msa_helper.c, it will hang. -> -> -There is a workaround for this particular file: -> -> -These lines in msa_helper.c: -> -> -uint## BITS ##_t S = _S, T = _T; \ -> -uint## BITS ##_t as, at, xs, xt, xd; \ -> -> -should be replaced with: -> -> -uint## BITS ## _t S = _S, T = _T; \ -> -uint## BITS ## _t as, at, xs, xt, xd; \ -> -> -(a space is added after the second "##" in each line) -> -> -The workaround is found by partial deleting and undeleting of the code in -> -msa_helper.c in binary search fashion. -> -> -This workaround will soon be submitted by me as a patch within a series on -> -misc MIPS issues. -> -> -I took a look at checkpatch.pl code, and it looks it is fairly complicated to -> -fix the issue, since it happens in the code segment involving intricate logic -> -conditions. -Thanks for figuring this out, Aleksandar. Not sure if anyone else has -the apetite to fix checkpatch.pl. - -Stefan -signature.asc -Description: -PGP signature - -On 07/11/2018 09:36 AM, Stefan Hajnoczi wrote: -> -On Wed, Jul 04, 2018 at 03:35:18PM +0000, Aleksandar Markovic wrote: -> -> If checkpatch.pl is applied (using switch "-f") on file -> -> target/mips/msa_helper.c, it will hang. -> -> -> -> There is a workaround for this particular file: -> -> -> -> These lines in msa_helper.c: -> -> -> -> uint## BITS ##_t S = _S, T = _T; \ -> -> uint## BITS ##_t as, at, xs, xt, xd; \ -> -> -> -> should be replaced with: -> -> -> -> uint## BITS ## _t S = _S, T = _T; \ -> -> uint## BITS ## _t as, at, xs, xt, xd; \ -> -> -> -> (a space is added after the second "##" in each line) -> -> -> -> The workaround is found by partial deleting and undeleting of the code in -> -> msa_helper.c in binary search fashion. -> -> -> -> This workaround will soon be submitted by me as a patch within a series on -> -> misc MIPS issues. -> -> -> -> I took a look at checkpatch.pl code, and it looks it is fairly complicated -> -> to fix the issue, since it happens in the code segment involving intricate -> -> logic conditions. -> -> -Thanks for figuring this out, Aleksandar. Not sure if anyone else has -> -the apetite to fix checkpatch.pl. -Anyone else but Paolo ;P -http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg01250.html -signature.asc -Description: -OpenPGP digital signature - diff --git a/results/classifier/016/arm/33802194 b/results/classifier/016/arm/33802194 deleted file mode 100644 index a1a7b8bf..00000000 --- a/results/classifier/016/arm/33802194 +++ /dev/null @@ -1,4966 +0,0 @@ -arm: 0.990 -virtual: 0.605 -debug: 0.560 -operating system: 0.515 -kernel: 0.338 -register: 0.156 -hypervisor: 0.149 -TCG: 0.149 -user-level: 0.060 -PID: 0.041 -device: 0.031 -architecture: 0.014 -boot: 0.014 -files: 0.013 -VMM: 0.013 -performance: 0.011 -risc-v: 0.011 -socket: 0.010 -semantic: 0.008 -assembly: 0.007 -network: 0.007 -KVM: 0.005 -peripherals: 0.004 -graphic: 0.003 -permissions: 0.003 -vnc: 0.001 -mistranslation: 0.001 -alpha: 0.001 -x86: 0.000 -ppc: 0.000 -i386: 0.000 - -[BUG] cxl can not create region - -Hi list - -I want to test cxl functions in arm64, and found some problems I can't -figure out. - -My test environment: - -1. build latest bios from -https://github.com/tianocore/edk2.git -master -branch(cc2db6ebfb6d9d85ba4c7b35fba1fa37fffc0bc2) -2. build latest qemu-system-aarch64 from git://git.qemu.org/qemu.git -master branch(846dcf0ba4eff824c295f06550b8673ff3f31314). With cxl arm -support patch: -https://patchwork.kernel.org/project/cxl/cover/20220616141950.23374-1-Jonathan.Cameron@huawei.com/ -3. build Linux kernel from -https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git -preview -branch(65fc1c3d26b96002a5aa1f4012fae4dc98fd5683) -4. build latest ndctl tools from -https://github.com/pmem/ndctl -create_region branch(8558b394e449779e3a4f3ae90fae77ede0bca159) - -And my qemu test commands: -sudo $QEMU_BIN -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 \ - -cpu max -smp 8 -nographic -no-reboot \ - -kernel $KERNEL -bios $BIOS_BIN \ - -drive if=none,file=$ROOTFS,format=qcow2,id=hd \ - -device virtio-blk-pci,drive=hd -append 'root=/dev/vda1 -nokaslr dyndbg="module cxl* +p"' \ - -object memory-backend-ram,size=4G,id=mem0 \ - -numa node,nodeid=0,cpus=0-7,memdev=mem0 \ - -net nic -net user,hostfwd=tcp::2222-:22 -enable-kvm \ - -object -memory-backend-file,id=cxl-mem0,share=on,mem-path=/tmp/cxltest.raw,size=256M -\ - -object -memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M -\ - -object -memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M -\ - -object -memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M -\ - -object -memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa0.raw,size=256M -\ - -object -memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M -\ - -object -memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M -\ - -object -memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M -\ - -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ - -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \ - -device cxl-upstream,bus=root_port0,id=us0 \ - -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ - -device -cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0 \ - -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ - -device -cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1 \ - -device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ - -device -cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2 \ - -device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ - -device -cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3 \ - -M -cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k - -And I have got two problems. -1. When I want to create x1 region with command: "cxl create-region -d -decoder0.0 -w 1 -g 4096 mem0", kernel crashed with null pointer -reference. Crash log: - -[ 534.697324] cxl_region region0: config state: 0 -[ 534.697346] cxl_region region0: probe: -6 -[ 534.697368] cxl_acpi ACPI0017:00: decoder0.0: created region0 -[ 534.699115] cxl region0: mem0:endpoint3 decoder3.0 add: -mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1 -[ 534.699149] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1 -[ 534.699167] cxl region0: ACPI0016:00:port1 decoder1.0 add: -mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1 -[ 534.699176] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256 -[ 534.699182] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0 -for mem0:decoder3.0 @ 0 -[ 534.699189] cxl region0: 0000:0d:00.0:port2 iw: 1 ig: 256 -[ 534.699193] cxl region0: 0000:0d:00.0:port2 target[0] = -0000:0e:00.0 for mem0:decoder3.0 @ 0 -[ 534.699405] Unable to handle kernel NULL pointer dereference at -virtual address 0000000000000000 -[ 534.701474] Mem abort info: -[ 534.701994] ESR = 0x0000000086000004 -[ 534.702653] EC = 0x21: IABT (current EL), IL = 32 bits -[ 534.703616] SET = 0, FnV = 0 -[ 534.704174] EA = 0, S1PTW = 0 -[ 534.704803] FSC = 0x04: level 0 translation fault -[ 534.705694] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010144a000 -[ 534.706875] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000 -[ 534.709855] Internal error: Oops: 86000004 [#1] PREEMPT SMP -[ 534.710301] Modules linked in: -[ 534.710546] CPU: 7 PID: 331 Comm: cxl Not tainted -5.19.0-rc3-00064-g65fc1c3d26b9-dirty #11 -[ 534.715393] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 -[ 534.717179] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) -[ 534.719190] pc : 0x0 -[ 534.719928] lr : commit_store+0x118/0x2cc -[ 534.721007] sp : ffff80000aec3c30 -[ 534.721793] x29: ffff80000aec3c30 x28: ffff0000da62e740 x27: ffff0000c0c06b30 -[ 534.723875] x26: 0000000000000000 x25: ffff0000c0a2a400 x24: ffff0000c0a29400 -[ 534.725440] x23: 0000000000000003 x22: 0000000000000000 x21: ffff0000c0c06800 -[ 534.727312] x20: 0000000000000000 x19: ffff0000c1559800 x18: 0000000000000000 -[ 534.729138] x17: 0000000000000000 x16: 0000000000000000 x15: 0000ffffd41fe838 -[ 534.731046] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 -[ 534.732402] x11: 0000000000000000 x10: 0000000000000000 x9 : 0000000000000000 -[ 534.734432] x8 : 0000000000000000 x7 : 0000000000000000 x6 : ffff0000c0906e80 -[ 534.735921] x5 : 0000000000000000 x4 : 0000000000000000 x3 : ffff80000aec3bf0 -[ 534.737437] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000c155a000 -[ 534.738878] Call trace: -[ 534.739368] 0x0 -[ 534.739713] dev_attr_store+0x1c/0x30 -[ 534.740186] sysfs_kf_write+0x48/0x58 -[ 534.740961] kernfs_fop_write_iter+0x128/0x184 -[ 534.741872] new_sync_write+0xdc/0x158 -[ 534.742706] vfs_write+0x1ac/0x2a8 -[ 534.743440] ksys_write+0x68/0xf0 -[ 534.744328] __arm64_sys_write+0x1c/0x28 -[ 534.745180] invoke_syscall+0x44/0xf0 -[ 534.745989] el0_svc_common+0x4c/0xfc -[ 534.746661] do_el0_svc+0x60/0xa8 -[ 534.747378] el0_svc+0x2c/0x78 -[ 534.748066] el0t_64_sync_handler+0xb8/0x12c -[ 534.748919] el0t_64_sync+0x18c/0x190 -[ 534.749629] Code: bad PC value -[ 534.750169] ---[ end trace 0000000000000000 ]--- - -2. When I want to create x4 region with command: "cxl create-region -d -decoder0.0 -w 4 -g 4096 -m mem0 mem1 mem2 mem3". I got below errors: - -cxl region: create_region: region0: failed to set target3 to mem3 -cxl region: cmd_create_region: created 0 regions - -And kernel log as below: -[ 60.536663] cxl_region region0: config state: 0 -[ 60.536675] cxl_region region0: probe: -6 -[ 60.536696] cxl_acpi ACPI0017:00: decoder0.0: created region0 -[ 60.538251] cxl region0: mem0:endpoint3 decoder3.0 add: -mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1 -[ 60.538278] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1 -[ 60.538295] cxl region0: ACPI0016:00:port1 decoder1.0 add: -mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1 -[ 60.538647] cxl region0: mem1:endpoint4 decoder4.0 add: -mem1:decoder4.0 @ 1 next: none nr_eps: 1 nr_targets: 1 -[ 60.538663] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -mem1:decoder4.0 @ 1 next: mem1 nr_eps: 2 nr_targets: 2 -[ 60.538675] cxl region0: ACPI0016:00:port1 decoder1.0 add: -mem1:decoder4.0 @ 1 next: 0000:0d:00.0 nr_eps: 2 nr_targets: 1 -[ 60.539311] cxl region0: mem2:endpoint5 decoder5.0 add: -mem2:decoder5.0 @ 2 next: none nr_eps: 1 nr_targets: 1 -[ 60.539332] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -mem2:decoder5.0 @ 2 next: mem2 nr_eps: 3 nr_targets: 3 -[ 60.539343] cxl region0: ACPI0016:00:port1 decoder1.0 add: -mem2:decoder5.0 @ 2 next: 0000:0d:00.0 nr_eps: 3 nr_targets: 1 -[ 60.539711] cxl region0: mem3:endpoint6 decoder6.0 add: -mem3:decoder6.0 @ 3 next: none nr_eps: 1 nr_targets: 1 -[ 60.539723] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -mem3:decoder6.0 @ 3 next: mem3 nr_eps: 4 nr_targets: 4 -[ 60.539735] cxl region0: ACPI0016:00:port1 decoder1.0 add: -mem3:decoder6.0 @ 3 next: 0000:0d:00.0 nr_eps: 4 nr_targets: 1 -[ 60.539742] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256 -[ 60.539747] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0 -for mem0:decoder3.0 @ 0 -[ 60.539754] cxl region0: 0000:0d:00.0:port2 iw: 4 ig: 512 -[ 60.539758] cxl region0: 0000:0d:00.0:port2 target[0] = -0000:0e:00.0 for mem0:decoder3.0 @ 0 -[ 60.539764] cxl region0: ACPI0016:00:port1: cannot host mem1:decoder4.0 at 1 - -I have tried to write sysfs node manually, got same errors. - -Hope I can get some helps here. - -Bob - -On Fri, 5 Aug 2022 10:20:23 +0800 -Bobo WL wrote: - -> -Hi list -> -> -I want to test cxl functions in arm64, and found some problems I can't -> -figure out. -Hi Bob, - -Glad to see people testing this code. - -> -> -My test environment: -> -> -1. build latest bios from -https://github.com/tianocore/edk2.git -master -> -branch(cc2db6ebfb6d9d85ba4c7b35fba1fa37fffc0bc2) -> -2. build latest qemu-system-aarch64 from git://git.qemu.org/qemu.git -> -master branch(846dcf0ba4eff824c295f06550b8673ff3f31314). With cxl arm -> -support patch: -> -https://patchwork.kernel.org/project/cxl/cover/20220616141950.23374-1-Jonathan.Cameron@huawei.com/ -> -3. build Linux kernel from -> -https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git -preview -> -branch(65fc1c3d26b96002a5aa1f4012fae4dc98fd5683) -> -4. build latest ndctl tools from -https://github.com/pmem/ndctl -> -create_region branch(8558b394e449779e3a4f3ae90fae77ede0bca159) -> -> -And my qemu test commands: -> -sudo $QEMU_BIN -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 \ -> --cpu max -smp 8 -nographic -no-reboot \ -> --kernel $KERNEL -bios $BIOS_BIN \ -> --drive if=none,file=$ROOTFS,format=qcow2,id=hd \ -> --device virtio-blk-pci,drive=hd -append 'root=/dev/vda1 -> -nokaslr dyndbg="module cxl* +p"' \ -> --object memory-backend-ram,size=4G,id=mem0 \ -> --numa node,nodeid=0,cpus=0-7,memdev=mem0 \ -> --net nic -net user,hostfwd=tcp::2222-:22 -enable-kvm \ -> --object -> -memory-backend-file,id=cxl-mem0,share=on,mem-path=/tmp/cxltest.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa0.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M -> -\ -> --device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -> --device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \ -Probably not related to your problem, but there is a disconnect in QEMU / -kernel assumptionsaround the presence of an HDM decoder when a HB only -has a single root port. Spec allows it to be provided or not as an -implementation choice. -Kernel assumes it isn't provide. Qemu assumes it is. - -The temporary solution is to throw in a second root port on the HB and not -connect anything to it. Longer term I may special case this so that the -particular -decoder defaults to pass through settings in QEMU if there is only one root -port. - -> --device cxl-upstream,bus=root_port0,id=us0 \ -> --device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ -> --device -> -cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0 \ -> --device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ -> --device -> -cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1 \ -> --device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ -> --device -> -cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2 \ -> --device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ -> --device -> -cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3 \ -> --M -> -cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k -> -> -And I have got two problems. -> -1. When I want to create x1 region with command: "cxl create-region -d -> -decoder0.0 -w 1 -g 4096 mem0", kernel crashed with null pointer -> -reference. Crash log: -> -> -[ 534.697324] cxl_region region0: config state: 0 -> -[ 534.697346] cxl_region region0: probe: -6 -Seems odd this is up here. But maybe fine. - -> -[ 534.697368] cxl_acpi ACPI0017:00: decoder0.0: created region0 -> -[ 534.699115] cxl region0: mem0:endpoint3 decoder3.0 add: -> -mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1 -> -[ 534.699149] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1 -> -[ 534.699167] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1 -> -[ 534.699176] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256 -> -[ 534.699182] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0 -> -for mem0:decoder3.0 @ 0 -> -[ 534.699189] cxl region0: 0000:0d:00.0:port2 iw: 1 ig: 256 -> -[ 534.699193] cxl region0: 0000:0d:00.0:port2 target[0] = -> -0000:0e:00.0 for mem0:decoder3.0 @ 0 -> -[ 534.699405] Unable to handle kernel NULL pointer dereference at -> -virtual address 0000000000000000 -> -[ 534.701474] Mem abort info: -> -[ 534.701994] ESR = 0x0000000086000004 -> -[ 534.702653] EC = 0x21: IABT (current EL), IL = 32 bits -> -[ 534.703616] SET = 0, FnV = 0 -> -[ 534.704174] EA = 0, S1PTW = 0 -> -[ 534.704803] FSC = 0x04: level 0 translation fault -> -[ 534.705694] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010144a000 -> -[ 534.706875] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000 -> -[ 534.709855] Internal error: Oops: 86000004 [#1] PREEMPT SMP -> -[ 534.710301] Modules linked in: -> -[ 534.710546] CPU: 7 PID: 331 Comm: cxl Not tainted -> -5.19.0-rc3-00064-g65fc1c3d26b9-dirty #11 -> -[ 534.715393] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 -> -[ 534.717179] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) -> -[ 534.719190] pc : 0x0 -> -[ 534.719928] lr : commit_store+0x118/0x2cc -> -[ 534.721007] sp : ffff80000aec3c30 -> -[ 534.721793] x29: ffff80000aec3c30 x28: ffff0000da62e740 x27: -> -ffff0000c0c06b30 -> -[ 534.723875] x26: 0000000000000000 x25: ffff0000c0a2a400 x24: -> -ffff0000c0a29400 -> -[ 534.725440] x23: 0000000000000003 x22: 0000000000000000 x21: -> -ffff0000c0c06800 -> -[ 534.727312] x20: 0000000000000000 x19: ffff0000c1559800 x18: -> -0000000000000000 -> -[ 534.729138] x17: 0000000000000000 x16: 0000000000000000 x15: -> -0000ffffd41fe838 -> -[ 534.731046] x14: 0000000000000000 x13: 0000000000000000 x12: -> -0000000000000000 -> -[ 534.732402] x11: 0000000000000000 x10: 0000000000000000 x9 : -> -0000000000000000 -> -[ 534.734432] x8 : 0000000000000000 x7 : 0000000000000000 x6 : -> -ffff0000c0906e80 -> -[ 534.735921] x5 : 0000000000000000 x4 : 0000000000000000 x3 : -> -ffff80000aec3bf0 -> -[ 534.737437] x2 : 0000000000000000 x1 : 0000000000000000 x0 : -> -ffff0000c155a000 -> -[ 534.738878] Call trace: -> -[ 534.739368] 0x0 -> -[ 534.739713] dev_attr_store+0x1c/0x30 -> -[ 534.740186] sysfs_kf_write+0x48/0x58 -> -[ 534.740961] kernfs_fop_write_iter+0x128/0x184 -> -[ 534.741872] new_sync_write+0xdc/0x158 -> -[ 534.742706] vfs_write+0x1ac/0x2a8 -> -[ 534.743440] ksys_write+0x68/0xf0 -> -[ 534.744328] __arm64_sys_write+0x1c/0x28 -> -[ 534.745180] invoke_syscall+0x44/0xf0 -> -[ 534.745989] el0_svc_common+0x4c/0xfc -> -[ 534.746661] do_el0_svc+0x60/0xa8 -> -[ 534.747378] el0_svc+0x2c/0x78 -> -[ 534.748066] el0t_64_sync_handler+0xb8/0x12c -> -[ 534.748919] el0t_64_sync+0x18c/0x190 -> -[ 534.749629] Code: bad PC value -> -[ 534.750169] ---[ end trace 0000000000000000 ]--- -> -> -2. When I want to create x4 region with command: "cxl create-region -d -> -decoder0.0 -w 4 -g 4096 -m mem0 mem1 mem2 mem3". I got below errors: -> -> -cxl region: create_region: region0: failed to set target3 to mem3 -> -cxl region: cmd_create_region: created 0 regions -> -> -And kernel log as below: -> -[ 60.536663] cxl_region region0: config state: 0 -> -[ 60.536675] cxl_region region0: probe: -6 -> -[ 60.536696] cxl_acpi ACPI0017:00: decoder0.0: created region0 -> -[ 60.538251] cxl region0: mem0:endpoint3 decoder3.0 add: -> -mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.538278] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1 -> -[ 60.538295] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1 -> -[ 60.538647] cxl region0: mem1:endpoint4 decoder4.0 add: -> -mem1:decoder4.0 @ 1 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.538663] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem1:decoder4.0 @ 1 next: mem1 nr_eps: 2 nr_targets: 2 -> -[ 60.538675] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem1:decoder4.0 @ 1 next: 0000:0d:00.0 nr_eps: 2 nr_targets: 1 -> -[ 60.539311] cxl region0: mem2:endpoint5 decoder5.0 add: -> -mem2:decoder5.0 @ 2 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.539332] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem2:decoder5.0 @ 2 next: mem2 nr_eps: 3 nr_targets: 3 -> -[ 60.539343] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem2:decoder5.0 @ 2 next: 0000:0d:00.0 nr_eps: 3 nr_targets: 1 -> -[ 60.539711] cxl region0: mem3:endpoint6 decoder6.0 add: -> -mem3:decoder6.0 @ 3 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.539723] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem3:decoder6.0 @ 3 next: mem3 nr_eps: 4 nr_targets: 4 -> -[ 60.539735] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem3:decoder6.0 @ 3 next: 0000:0d:00.0 nr_eps: 4 nr_targets: 1 -> -[ 60.539742] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256 -> -[ 60.539747] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0 -> -for mem0:decoder3.0 @ 0 -> -[ 60.539754] cxl region0: 0000:0d:00.0:port2 iw: 4 ig: 512 -This looks like off by 1 that should be fixed in the below mentioned -cxl/pending branch. That ig should be 256. Note the fix was -for a test case with a fat HB and no switch, but certainly looks -like this is the same issue. - -> -[ 60.539758] cxl region0: 0000:0d:00.0:port2 target[0] = -> -0000:0e:00.0 for mem0:decoder3.0 @ 0 -> -[ 60.539764] cxl region0: ACPI0016:00:port1: cannot host mem1:decoder4.0 at -> -1 -> -> -I have tried to write sysfs node manually, got same errors. -When stepping through by hand, which sysfs write triggers the crash above? - -Not sure it's related, but I've just sent out a fix to the -target register handling in QEMU. -20220808122051.14822-1-Jonathan.Cameron@huawei.com -/T/#m47ff985412ce44559e6b04d677c302f8cd371330">https://lore.kernel.org/linux-cxl/ -20220808122051.14822-1-Jonathan.Cameron@huawei.com -/T/#m47ff985412ce44559e6b04d677c302f8cd371330 -I did have one instance last week of triggering what looked to be a race -condition but -the stack trace doesn't looks related to what you've hit. - -It will probably be a few days before I have time to take a look at replicating -what you have seen. - -If you have time, try using the kernel.org cxl/pending branch as there are -a few additional fixes on there since you sent this email. Optimistic to hope -this is covered by one of those, but at least it will mean we are trying to -replicate -on same branch. - -Jonathan - - -> -> -Hope I can get some helps here. -> -> -Bob - -Hi Jonathan - -Thanks for your reply! - -On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron - wrote: -> -> -Probably not related to your problem, but there is a disconnect in QEMU / -> -kernel assumptionsaround the presence of an HDM decoder when a HB only -> -has a single root port. Spec allows it to be provided or not as an -> -implementation choice. -> -Kernel assumes it isn't provide. Qemu assumes it is. -> -> -The temporary solution is to throw in a second root port on the HB and not -> -connect anything to it. Longer term I may special case this so that the -> -particular -> -decoder defaults to pass through settings in QEMU if there is only one root -> -port. -> -You are right! After adding an extra HB in qemu, I can create a x1 -region successfully. -But have some errors in Nvdimm: - -[ 74.925838] Unknown online node for memory at 0x10000000000, assuming node 0 -[ 74.925846] Unknown target node for memory at 0x10000000000, assuming node 0 -[ 74.927470] nd_region region0: nmem0: is disabled, failing probe - -And x4 region still failed with same errors, using latest cxl/preview -branch don't work. -I have picked "Two CXL emulation fixes" patches in qemu, still not working. - -Bob - -On Tue, 9 Aug 2022 21:07:06 +0800 -Bobo WL wrote: - -> -Hi Jonathan -> -> -Thanks for your reply! -> -> -On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> - wrote: -> -> -> -> Probably not related to your problem, but there is a disconnect in QEMU / -> -> kernel assumptionsaround the presence of an HDM decoder when a HB only -> -> has a single root port. Spec allows it to be provided or not as an -> -> implementation choice. -> -> Kernel assumes it isn't provide. Qemu assumes it is. -> -> -> -> The temporary solution is to throw in a second root port on the HB and not -> -> connect anything to it. Longer term I may special case this so that the -> -> particular -> -> decoder defaults to pass through settings in QEMU if there is only one root -> -> port. -> -> -> -> -You are right! After adding an extra HB in qemu, I can create a x1 -> -region successfully. -> -But have some errors in Nvdimm: -> -> -[ 74.925838] Unknown online node for memory at 0x10000000000, assuming node > 0 -> -[ 74.925846] Unknown target node for memory at 0x10000000000, assuming node > 0 -> -[ 74.927470] nd_region region0: nmem0: is disabled, failing probe -Ah. I've seen this one, but not chased it down yet. Was on my todo list to -chase -down. Once I reach this state I can verify the HDM Decode is correct which is -what -I've been using to test (Which wasn't true until earlier this week). -I'm currently testing via devmem, more for historical reasons than because it -makes -that much sense anymore. - -> -> -And x4 region still failed with same errors, using latest cxl/preview -> -branch don't work. -> -I have picked "Two CXL emulation fixes" patches in qemu, still not working. -> -> -Bob - -On Tue, 9 Aug 2022 17:08:25 +0100 -Jonathan Cameron wrote: - -> -On Tue, 9 Aug 2022 21:07:06 +0800 -> -Bobo WL wrote: -> -> -> Hi Jonathan -> -> -> -> Thanks for your reply! -> -> -> -> On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> wrote: -> -> > -> -> > Probably not related to your problem, but there is a disconnect in QEMU / -> -> > kernel assumptionsaround the presence of an HDM decoder when a HB only -> -> > has a single root port. Spec allows it to be provided or not as an -> -> > implementation choice. -> -> > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > -> -> > The temporary solution is to throw in a second root port on the HB and not -> -> > connect anything to it. Longer term I may special case this so that the -> -> > particular -> -> > decoder defaults to pass through settings in QEMU if there is only one -> -> > root port. -> -> > -> -> -> -> You are right! After adding an extra HB in qemu, I can create a x1 -> -> region successfully. -> -> But have some errors in Nvdimm: -> -> -> -> [ 74.925838] Unknown online node for memory at 0x10000000000, assuming -> -> node 0 -> -> [ 74.925846] Unknown target node for memory at 0x10000000000, assuming -> -> node 0 -> -> [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> -Ah. I've seen this one, but not chased it down yet. Was on my todo list to -> -chase -> -down. Once I reach this state I can verify the HDM Decode is correct which is -> -what -> -I've been using to test (Which wasn't true until earlier this week). -> -I'm currently testing via devmem, more for historical reasons than because it -> -makes -> -that much sense anymore. -*embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -I'd forgotten that was still on the todo list. I don't think it will -be particularly hard to do and will take a look in next few days. - -Very very indirectly this error is causing a driver probe fail that means that -we hit a code path that has a rather odd looking check on NDD_LABELING. -Should not have gotten near that path though - hence the problem is actually -when we call cxl_pmem_get_config_data() and it returns an error because -we haven't fully connected up the command in QEMU. - -Jonathan - - -> -> -> -> -> And x4 region still failed with same errors, using latest cxl/preview -> -> branch don't work. -> -> I have picked "Two CXL emulation fixes" patches in qemu, still not working. -> -> -> -> Bob - -On Thu, 11 Aug 2022 18:08:57 +0100 -Jonathan Cameron via wrote: - -> -On Tue, 9 Aug 2022 17:08:25 +0100 -> -Jonathan Cameron wrote: -> -> -> On Tue, 9 Aug 2022 21:07:06 +0800 -> -> Bobo WL wrote: -> -> -> -> > Hi Jonathan -> -> > -> -> > Thanks for your reply! -> -> > -> -> > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > wrote: -> -> > > -> -> > > Probably not related to your problem, but there is a disconnect in QEMU -> -> > > / -> -> > > kernel assumptionsaround the presence of an HDM decoder when a HB only -> -> > > has a single root port. Spec allows it to be provided or not as an -> -> > > implementation choice. -> -> > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > -> -> > > The temporary solution is to throw in a second root port on the HB and -> -> > > not -> -> > > connect anything to it. Longer term I may special case this so that -> -> > > the particular -> -> > > decoder defaults to pass through settings in QEMU if there is only one -> -> > > root port. -> -> > > -> -> > -> -> > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > region successfully. -> -> > But have some errors in Nvdimm: -> -> > -> -> > [ 74.925838] Unknown online node for memory at 0x10000000000, assuming -> -> > node 0 -> -> > [ 74.925846] Unknown target node for memory at 0x10000000000, assuming -> -> > node 0 -> -> > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> -> -> Ah. I've seen this one, but not chased it down yet. Was on my todo list to -> -> chase -> -> down. Once I reach this state I can verify the HDM Decode is correct which -> -> is what -> -> I've been using to test (Which wasn't true until earlier this week). -> -> I'm currently testing via devmem, more for historical reasons than because -> -> it makes -> -> that much sense anymore. -> -> -*embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -I'd forgotten that was still on the todo list. I don't think it will -> -be particularly hard to do and will take a look in next few days. -> -> -Very very indirectly this error is causing a driver probe fail that means that -> -we hit a code path that has a rather odd looking check on NDD_LABELING. -> -Should not have gotten near that path though - hence the problem is actually -> -when we call cxl_pmem_get_config_data() and it returns an error because -> -we haven't fully connected up the command in QEMU. -So a least one bug in QEMU. We were not supporting variable length payloads on -mailbox -inputs (but were on outputs). That hasn't mattered until we get to LSA writes. -We just need to relax condition on the supplied length. - -diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -index c352a935c4..fdda9529fe 100644 ---- a/hw/cxl/cxl-mailbox-utils.c -+++ b/hw/cxl/cxl-mailbox-utils.c -@@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) - cxl_cmd = &cxl_cmd_set[set][cmd]; - h = cxl_cmd->handler; - if (h) { -- if (len == cxl_cmd->in) { -+ if (len == cxl_cmd->in || !cxl_cmd->in) { - cxl_cmd->payload = cxl_dstate->mbox_reg_state + - A_CXL_DEV_CMD_PAYLOAD; - ret = (*h)(cxl_cmd, cxl_dstate, &len); - - -This lets the nvdimm/region probe fine, but I'm getting some issues with -namespace capacity so I'll look at what is causing that next. -Unfortunately I'm not that familiar with the driver/nvdimm side of things -so it's take a while to figure out what kicks off what! - -Jonathan - -> -> -Jonathan -> -> -> -> -> -> > -> -> > And x4 region still failed with same errors, using latest cxl/preview -> -> > branch don't work. -> -> > I have picked "Two CXL emulation fixes" patches in qemu, still not -> -> > working. -> -> > -> -> > Bob -> -> - -Jonathan Cameron wrote: -> -On Thu, 11 Aug 2022 18:08:57 +0100 -> -Jonathan Cameron via wrote: -> -> -> On Tue, 9 Aug 2022 17:08:25 +0100 -> -> Jonathan Cameron wrote: -> -> -> -> > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > Bobo WL wrote: -> -> > -> -> > > Hi Jonathan -> -> > > -> -> > > Thanks for your reply! -> -> > > -> -> > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > wrote: -> -> > > > -> -> > > > Probably not related to your problem, but there is a disconnect in -> -> > > > QEMU / -> -> > > > kernel assumptionsaround the presence of an HDM decoder when a HB only -> -> > > > has a single root port. Spec allows it to be provided or not as an -> -> > > > implementation choice. -> -> > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > -> -> > > > The temporary solution is to throw in a second root port on the HB -> -> > > > and not -> -> > > > connect anything to it. Longer term I may special case this so that -> -> > > > the particular -> -> > > > decoder defaults to pass through settings in QEMU if there is only -> -> > > > one root port. -> -> > > > -> -> > > -> -> > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > region successfully. -> -> > > But have some errors in Nvdimm: -> -> > > -> -> > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > assuming node 0 -> -> > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > assuming node 0 -> -> > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > -> -> > Ah. I've seen this one, but not chased it down yet. Was on my todo list -> -> > to chase -> -> > down. Once I reach this state I can verify the HDM Decode is correct -> -> > which is what -> -> > I've been using to test (Which wasn't true until earlier this week). -> -> > I'm currently testing via devmem, more for historical reasons than -> -> > because it makes -> -> > that much sense anymore. -> -> -> -> *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> I'd forgotten that was still on the todo list. I don't think it will -> -> be particularly hard to do and will take a look in next few days. -> -> -> -> Very very indirectly this error is causing a driver probe fail that means -> -> that -> -> we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> Should not have gotten near that path though - hence the problem is actually -> -> when we call cxl_pmem_get_config_data() and it returns an error because -> -> we haven't fully connected up the command in QEMU. -> -> -So a least one bug in QEMU. We were not supporting variable length payloads -> -on mailbox -> -inputs (but were on outputs). That hasn't mattered until we get to LSA -> -writes. -> -We just need to relax condition on the supplied length. -> -> -diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -index c352a935c4..fdda9529fe 100644 -> ---- a/hw/cxl/cxl-mailbox-utils.c -> -+++ b/hw/cxl/cxl-mailbox-utils.c -> -@@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -cxl_cmd = &cxl_cmd_set[set][cmd]; -> -h = cxl_cmd->handler; -> -if (h) { -> -- if (len == cxl_cmd->in) { -> -+ if (len == cxl_cmd->in || !cxl_cmd->in) { -> -cxl_cmd->payload = cxl_dstate->mbox_reg_state + -> -A_CXL_DEV_CMD_PAYLOAD; -> -ret = (*h)(cxl_cmd, cxl_dstate, &len); -> -> -> -This lets the nvdimm/region probe fine, but I'm getting some issues with -> -namespace capacity so I'll look at what is causing that next. -> -Unfortunately I'm not that familiar with the driver/nvdimm side of things -> -so it's take a while to figure out what kicks off what! -The whirlwind tour is that 'struct nd_region' instances that represent a -persitent memory address range are composed of one more mappings of -'struct nvdimm' objects. The nvdimm object is driven by the dimm driver -in drivers/nvdimm/dimm.c. That driver is mainly charged with unlocking -the dimm (if locked) and interrogating the label area to look for -namespace labels. - -The label command calls are routed to the '->ndctl()' callback that was -registered when the CXL nvdimm_bus_descriptor was created. That callback -handles both 'bus' scope calls, currently none for CXL, and per nvdimm -calls. cxl_pmem_nvdimm_ctl() translates those generic LIBNVDIMM commands -to CXL commands. - -The 'struct nvdimm' objects that the CXL side registers have the -NDD_LABELING flag set which means that namespaces need to be explicitly -created / provisioned from region capacity. Otherwise, if -drivers/nvdimm/dimm.c does not find a namespace-label-index block then -the region reverts to label-less mode and a default namespace equal to -the size of the region is instantiated. - -If you are seeing small mismatches in namespace capacity then it may -just be the fact that by default 'ndctl create-namespace' results in an -'fsdax' mode namespace which just means that it is a block device where -1.5% of the capacity is reserved for 'struct page' metadata. You should -be able to see namespace capacity == region capacity by doing "ndctl -create-namespace -m raw", and disable DAX operation. - -Hope that helps. - -On Fri, 12 Aug 2022 09:03:02 -0700 -Dan Williams wrote: - -> -Jonathan Cameron wrote: -> -> On Thu, 11 Aug 2022 18:08:57 +0100 -> -> Jonathan Cameron via wrote: -> -> -> -> > On Tue, 9 Aug 2022 17:08:25 +0100 -> -> > Jonathan Cameron wrote: -> -> > -> -> > > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > > Bobo WL wrote: -> -> > > -> -> > > > Hi Jonathan -> -> > > > -> -> > > > Thanks for your reply! -> -> > > > -> -> > > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > > wrote: -> -> > > > > -> -> > > > > Probably not related to your problem, but there is a disconnect in -> -> > > > > QEMU / -> -> > > > > kernel assumptionsaround the presence of an HDM decoder when a HB -> -> > > > > only -> -> > > > > has a single root port. Spec allows it to be provided or not as an -> -> > > > > implementation choice. -> -> > > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > > -> -> > > > > The temporary solution is to throw in a second root port on the HB -> -> > > > > and not -> -> > > > > connect anything to it. Longer term I may special case this so -> -> > > > > that the particular -> -> > > > > decoder defaults to pass through settings in QEMU if there is only -> -> > > > > one root port. -> -> > > > > -> -> > > > -> -> > > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > > region successfully. -> -> > > > But have some errors in Nvdimm: -> -> > > > -> -> > > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > > assuming node 0 -> -> > > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > > assuming node 0 -> -> > > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > > > -> -> > > -> -> > > Ah. I've seen this one, but not chased it down yet. Was on my todo -> -> > > list to chase -> -> > > down. Once I reach this state I can verify the HDM Decode is correct -> -> > > which is what -> -> > > I've been using to test (Which wasn't true until earlier this week). -> -> > > I'm currently testing via devmem, more for historical reasons than -> -> > > because it makes -> -> > > that much sense anymore. -> -> > -> -> > *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> > I'd forgotten that was still on the todo list. I don't think it will -> -> > be particularly hard to do and will take a look in next few days. -> -> > -> -> > Very very indirectly this error is causing a driver probe fail that means -> -> > that -> -> > we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> > Should not have gotten near that path though - hence the problem is -> -> > actually -> -> > when we call cxl_pmem_get_config_data() and it returns an error because -> -> > we haven't fully connected up the command in QEMU. -> -> -> -> So a least one bug in QEMU. We were not supporting variable length payloads -> -> on mailbox -> -> inputs (but were on outputs). That hasn't mattered until we get to LSA -> -> writes. -> -> We just need to relax condition on the supplied length. -> -> -> -> diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -> index c352a935c4..fdda9529fe 100644 -> -> --- a/hw/cxl/cxl-mailbox-utils.c -> -> +++ b/hw/cxl/cxl-mailbox-utils.c -> -> @@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -> cxl_cmd = &cxl_cmd_set[set][cmd]; -> -> h = cxl_cmd->handler; -> -> if (h) { -> -> - if (len == cxl_cmd->in) { -> -> + if (len == cxl_cmd->in || !cxl_cmd->in) { -> -> cxl_cmd->payload = cxl_dstate->mbox_reg_state + -> -> A_CXL_DEV_CMD_PAYLOAD; -> -> ret = (*h)(cxl_cmd, cxl_dstate, &len); -> -> -> -> -> -> This lets the nvdimm/region probe fine, but I'm getting some issues with -> -> namespace capacity so I'll look at what is causing that next. -> -> Unfortunately I'm not that familiar with the driver/nvdimm side of things -> -> so it's take a while to figure out what kicks off what! -> -> -The whirlwind tour is that 'struct nd_region' instances that represent a -> -persitent memory address range are composed of one more mappings of -> -'struct nvdimm' objects. The nvdimm object is driven by the dimm driver -> -in drivers/nvdimm/dimm.c. That driver is mainly charged with unlocking -> -the dimm (if locked) and interrogating the label area to look for -> -namespace labels. -> -> -The label command calls are routed to the '->ndctl()' callback that was -> -registered when the CXL nvdimm_bus_descriptor was created. That callback -> -handles both 'bus' scope calls, currently none for CXL, and per nvdimm -> -calls. cxl_pmem_nvdimm_ctl() translates those generic LIBNVDIMM commands -> -to CXL commands. -> -> -The 'struct nvdimm' objects that the CXL side registers have the -> -NDD_LABELING flag set which means that namespaces need to be explicitly -> -created / provisioned from region capacity. Otherwise, if -> -drivers/nvdimm/dimm.c does not find a namespace-label-index block then -> -the region reverts to label-less mode and a default namespace equal to -> -the size of the region is instantiated. -> -> -If you are seeing small mismatches in namespace capacity then it may -> -just be the fact that by default 'ndctl create-namespace' results in an -> -'fsdax' mode namespace which just means that it is a block device where -> -1.5% of the capacity is reserved for 'struct page' metadata. You should -> -be able to see namespace capacity == region capacity by doing "ndctl -> -create-namespace -m raw", and disable DAX operation. -Currently ndctl create-namespace crashes qemu ;) -Which isn't ideal! - -> -> -Hope that helps. -Got me looking at the right code. Thanks! - -Jonathan - -On Fri, 12 Aug 2022 17:15:09 +0100 -Jonathan Cameron wrote: - -> -On Fri, 12 Aug 2022 09:03:02 -0700 -> -Dan Williams wrote: -> -> -> Jonathan Cameron wrote: -> -> > On Thu, 11 Aug 2022 18:08:57 +0100 -> -> > Jonathan Cameron via wrote: -> -> > -> -> > > On Tue, 9 Aug 2022 17:08:25 +0100 -> -> > > Jonathan Cameron wrote: -> -> > > -> -> > > > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > > > Bobo WL wrote: -> -> > > > -> -> > > > > Hi Jonathan -> -> > > > > -> -> > > > > Thanks for your reply! -> -> > > > > -> -> > > > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > > > wrote: -> -> > > > > > -> -> > > > > > Probably not related to your problem, but there is a disconnect -> -> > > > > > in QEMU / -> -> > > > > > kernel assumptionsaround the presence of an HDM decoder when a HB -> -> > > > > > only -> -> > > > > > has a single root port. Spec allows it to be provided or not as -> -> > > > > > an implementation choice. -> -> > > > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > > > -> -> > > > > > The temporary solution is to throw in a second root port on the -> -> > > > > > HB and not -> -> > > > > > connect anything to it. Longer term I may special case this so -> -> > > > > > that the particular -> -> > > > > > decoder defaults to pass through settings in QEMU if there is -> -> > > > > > only one root port. -> -> > > > > > -> -> > > > > -> -> > > > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > > > region successfully. -> -> > > > > But have some errors in Nvdimm: -> -> > > > > -> -> > > > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > > > assuming node 0 -> -> > > > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > > > assuming node 0 -> -> > > > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > > > > -> -> > > > -> -> > > > Ah. I've seen this one, but not chased it down yet. Was on my todo -> -> > > > list to chase -> -> > > > down. Once I reach this state I can verify the HDM Decode is correct -> -> > > > which is what -> -> > > > I've been using to test (Which wasn't true until earlier this week). -> -> > > > I'm currently testing via devmem, more for historical reasons than -> -> > > > because it makes -> -> > > > that much sense anymore. -> -> > > -> -> > > *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> > > I'd forgotten that was still on the todo list. I don't think it will -> -> > > be particularly hard to do and will take a look in next few days. -> -> > > -> -> > > Very very indirectly this error is causing a driver probe fail that -> -> > > means that -> -> > > we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> > > Should not have gotten near that path though - hence the problem is -> -> > > actually -> -> > > when we call cxl_pmem_get_config_data() and it returns an error because -> -> > > we haven't fully connected up the command in QEMU. -> -> > -> -> > So a least one bug in QEMU. We were not supporting variable length -> -> > payloads on mailbox -> -> > inputs (but were on outputs). That hasn't mattered until we get to LSA -> -> > writes. -> -> > We just need to relax condition on the supplied length. -> -> > -> -> > diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -> > index c352a935c4..fdda9529fe 100644 -> -> > --- a/hw/cxl/cxl-mailbox-utils.c -> -> > +++ b/hw/cxl/cxl-mailbox-utils.c -> -> > @@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -> > cxl_cmd = &cxl_cmd_set[set][cmd]; -> -> > h = cxl_cmd->handler; -> -> > if (h) { -> -> > - if (len == cxl_cmd->in) { -> -> > + if (len == cxl_cmd->in || !cxl_cmd->in) { -> -> > cxl_cmd->payload = cxl_dstate->mbox_reg_state + -> -> > A_CXL_DEV_CMD_PAYLOAD; -> -> > ret = (*h)(cxl_cmd, cxl_dstate, &len); -> -> > -> -> > -> -> > This lets the nvdimm/region probe fine, but I'm getting some issues with -> -> > namespace capacity so I'll look at what is causing that next. -> -> > Unfortunately I'm not that familiar with the driver/nvdimm side of things -> -> > so it's take a while to figure out what kicks off what! -> -> -> -> The whirlwind tour is that 'struct nd_region' instances that represent a -> -> persitent memory address range are composed of one more mappings of -> -> 'struct nvdimm' objects. The nvdimm object is driven by the dimm driver -> -> in drivers/nvdimm/dimm.c. That driver is mainly charged with unlocking -> -> the dimm (if locked) and interrogating the label area to look for -> -> namespace labels. -> -> -> -> The label command calls are routed to the '->ndctl()' callback that was -> -> registered when the CXL nvdimm_bus_descriptor was created. That callback -> -> handles both 'bus' scope calls, currently none for CXL, and per nvdimm -> -> calls. cxl_pmem_nvdimm_ctl() translates those generic LIBNVDIMM commands -> -> to CXL commands. -> -> -> -> The 'struct nvdimm' objects that the CXL side registers have the -> -> NDD_LABELING flag set which means that namespaces need to be explicitly -> -> created / provisioned from region capacity. Otherwise, if -> -> drivers/nvdimm/dimm.c does not find a namespace-label-index block then -> -> the region reverts to label-less mode and a default namespace equal to -> -> the size of the region is instantiated. -> -> -> -> If you are seeing small mismatches in namespace capacity then it may -> -> just be the fact that by default 'ndctl create-namespace' results in an -> -> 'fsdax' mode namespace which just means that it is a block device where -> -> 1.5% of the capacity is reserved for 'struct page' metadata. You should -> -> be able to see namespace capacity == region capacity by doing "ndctl -> -> create-namespace -m raw", and disable DAX operation. -> -> -Currently ndctl create-namespace crashes qemu ;) -> -Which isn't ideal! -> -Found a cause for this one. Mailbox payload may be as small as 256 bytes. -We have code in kernel sanity checking that output payload fits in the -mailbox, but nothing on the input payload. Symptom is that we write just -off the end whatever size the payload is. Note doing this shouldn't crash -qemu - so I need to fix a range check somewhere. - -I think this is because cxl_pmem_get_config_size() returns the mailbox -payload size as being the available LSA size, forgetting to remove the -size of the headers on the set_lsa side of things. -https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/tree/drivers/cxl/pmem.c?h=next#n110 -I've hacked the max_payload to be -8 - -Now we still don't succeed in creating the namespace, but bonus is it doesn't -crash any more. - - -Jonathan - - - -> -> -> -> Hope that helps. -> -Got me looking at the right code. Thanks! -> -> -Jonathan -> -> - -On Mon, 15 Aug 2022 15:18:09 +0100 -Jonathan Cameron via wrote: - -> -On Fri, 12 Aug 2022 17:15:09 +0100 -> -Jonathan Cameron wrote: -> -> -> On Fri, 12 Aug 2022 09:03:02 -0700 -> -> Dan Williams wrote: -> -> -> -> > Jonathan Cameron wrote: -> -> > > On Thu, 11 Aug 2022 18:08:57 +0100 -> -> > > Jonathan Cameron via wrote: -> -> > > -> -> > > > On Tue, 9 Aug 2022 17:08:25 +0100 -> -> > > > Jonathan Cameron wrote: -> -> > > > -> -> > > > > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > > > > Bobo WL wrote: -> -> > > > > -> -> > > > > > Hi Jonathan -> -> > > > > > -> -> > > > > > Thanks for your reply! -> -> > > > > > -> -> > > > > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > > > > wrote: -> -> > > > > > > -> -> > > > > > > Probably not related to your problem, but there is a disconnect -> -> > > > > > > in QEMU / -> -> > > > > > > kernel assumptionsaround the presence of an HDM decoder when a -> -> > > > > > > HB only -> -> > > > > > > has a single root port. Spec allows it to be provided or not as -> -> > > > > > > an implementation choice. -> -> > > > > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > > > > -> -> > > > > > > The temporary solution is to throw in a second root port on the -> -> > > > > > > HB and not -> -> > > > > > > connect anything to it. Longer term I may special case this so -> -> > > > > > > that the particular -> -> > > > > > > decoder defaults to pass through settings in QEMU if there is -> -> > > > > > > only one root port. -> -> > > > > > > -> -> > > > > > -> -> > > > > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > > > > region successfully. -> -> > > > > > But have some errors in Nvdimm: -> -> > > > > > -> -> > > > > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > > > > assuming node 0 -> -> > > > > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > > > > assuming node 0 -> -> > > > > > [ 74.927470] nd_region region0: nmem0: is disabled, failing -> -> > > > > > probe -> -> > > > > -> -> > > > > Ah. I've seen this one, but not chased it down yet. Was on my todo -> -> > > > > list to chase -> -> > > > > down. Once I reach this state I can verify the HDM Decode is -> -> > > > > correct which is what -> -> > > > > I've been using to test (Which wasn't true until earlier this -> -> > > > > week). -> -> > > > > I'm currently testing via devmem, more for historical reasons than -> -> > > > > because it makes -> -> > > > > that much sense anymore. -> -> > > > -> -> > > > *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> > > > I'd forgotten that was still on the todo list. I don't think it will -> -> > > > be particularly hard to do and will take a look in next few days. -> -> > > > -> -> > > > Very very indirectly this error is causing a driver probe fail that -> -> > > > means that -> -> > > > we hit a code path that has a rather odd looking check on -> -> > > > NDD_LABELING. -> -> > > > Should not have gotten near that path though - hence the problem is -> -> > > > actually -> -> > > > when we call cxl_pmem_get_config_data() and it returns an error -> -> > > > because -> -> > > > we haven't fully connected up the command in QEMU. -> -> > > -> -> > > So a least one bug in QEMU. We were not supporting variable length -> -> > > payloads on mailbox -> -> > > inputs (but were on outputs). That hasn't mattered until we get to LSA -> -> > > writes. -> -> > > We just need to relax condition on the supplied length. -> -> > > -> -> > > diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -> > > index c352a935c4..fdda9529fe 100644 -> -> > > --- a/hw/cxl/cxl-mailbox-utils.c -> -> > > +++ b/hw/cxl/cxl-mailbox-utils.c -> -> > > @@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -> > > cxl_cmd = &cxl_cmd_set[set][cmd]; -> -> > > h = cxl_cmd->handler; -> -> > > if (h) { -> -> > > - if (len == cxl_cmd->in) { -> -> > > + if (len == cxl_cmd->in || !cxl_cmd->in) { -> -> > > cxl_cmd->payload = cxl_dstate->mbox_reg_state + -> -> > > A_CXL_DEV_CMD_PAYLOAD; -> -> > > ret = (*h)(cxl_cmd, cxl_dstate, &len); -> -> > > -> -> > > -> -> > > This lets the nvdimm/region probe fine, but I'm getting some issues with -> -> > > namespace capacity so I'll look at what is causing that next. -> -> > > Unfortunately I'm not that familiar with the driver/nvdimm side of -> -> > > things -> -> > > so it's take a while to figure out what kicks off what! -> -> > -> -> > The whirlwind tour is that 'struct nd_region' instances that represent a -> -> > persitent memory address range are composed of one more mappings of -> -> > 'struct nvdimm' objects. The nvdimm object is driven by the dimm driver -> -> > in drivers/nvdimm/dimm.c. That driver is mainly charged with unlocking -> -> > the dimm (if locked) and interrogating the label area to look for -> -> > namespace labels. -> -> > -> -> > The label command calls are routed to the '->ndctl()' callback that was -> -> > registered when the CXL nvdimm_bus_descriptor was created. That callback -> -> > handles both 'bus' scope calls, currently none for CXL, and per nvdimm -> -> > calls. cxl_pmem_nvdimm_ctl() translates those generic LIBNVDIMM commands -> -> > to CXL commands. -> -> > -> -> > The 'struct nvdimm' objects that the CXL side registers have the -> -> > NDD_LABELING flag set which means that namespaces need to be explicitly -> -> > created / provisioned from region capacity. Otherwise, if -> -> > drivers/nvdimm/dimm.c does not find a namespace-label-index block then -> -> > the region reverts to label-less mode and a default namespace equal to -> -> > the size of the region is instantiated. -> -> > -> -> > If you are seeing small mismatches in namespace capacity then it may -> -> > just be the fact that by default 'ndctl create-namespace' results in an -> -> > 'fsdax' mode namespace which just means that it is a block device where -> -> > 1.5% of the capacity is reserved for 'struct page' metadata. You should -> -> > be able to see namespace capacity == region capacity by doing "ndctl -> -> > create-namespace -m raw", and disable DAX operation. -> -> -> -> Currently ndctl create-namespace crashes qemu ;) -> -> Which isn't ideal! -> -> -> -> -Found a cause for this one. Mailbox payload may be as small as 256 bytes. -> -We have code in kernel sanity checking that output payload fits in the -> -mailbox, but nothing on the input payload. Symptom is that we write just -> -off the end whatever size the payload is. Note doing this shouldn't crash -> -qemu - so I need to fix a range check somewhere. -> -> -I think this is because cxl_pmem_get_config_size() returns the mailbox -> -payload size as being the available LSA size, forgetting to remove the -> -size of the headers on the set_lsa side of things. -> -https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/tree/drivers/cxl/pmem.c?h=next#n110 -> -> -I've hacked the max_payload to be -8 -> -> -Now we still don't succeed in creating the namespace, but bonus is it doesn't -> -crash any more. -In the interests of defensive / correct handling from QEMU I took a -look into why it was crashing. Turns out that providing a NULL write callback -for -the memory device region (that the above overlarge write was spilling into) -isn't -a safe thing to do. Needs a stub. Oops. - -On plus side we might never have noticed this was going wrong without the crash -*silver lining in every cloud* - -Fix to follow... - -Jonathan - - -> -> -> -Jonathan -> -> -> -> -> > -> -> > Hope that helps. -> -> Got me looking at the right code. Thanks! -> -> -> -> Jonathan -> -> -> -> -> -> - -On Mon, 15 Aug 2022 at 15:55, Jonathan Cameron via wrote: -> -In the interests of defensive / correct handling from QEMU I took a -> -look into why it was crashing. Turns out that providing a NULL write -> -callback for -> -the memory device region (that the above overlarge write was spilling into) -> -isn't -> -a safe thing to do. Needs a stub. Oops. -Yeah. We've talked before about adding an assert so that that kind of -"missing function" bug is caught at device creation rather than only -if the guest tries to access the device, but we never quite got around -to it... - --- PMM - -On Fri, 12 Aug 2022 16:44:03 +0100 -Jonathan Cameron wrote: - -> -On Thu, 11 Aug 2022 18:08:57 +0100 -> -Jonathan Cameron via wrote: -> -> -> On Tue, 9 Aug 2022 17:08:25 +0100 -> -> Jonathan Cameron wrote: -> -> -> -> > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > Bobo WL wrote: -> -> > -> -> > > Hi Jonathan -> -> > > -> -> > > Thanks for your reply! -> -> > > -> -> > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > wrote: -> -> > > > -> -> > > > Probably not related to your problem, but there is a disconnect in -> -> > > > QEMU / -> -> > > > kernel assumptionsaround the presence of an HDM decoder when a HB only -> -> > > > has a single root port. Spec allows it to be provided or not as an -> -> > > > implementation choice. -> -> > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > -> -> > > > The temporary solution is to throw in a second root port on the HB -> -> > > > and not -> -> > > > connect anything to it. Longer term I may special case this so that -> -> > > > the particular -> -> > > > decoder defaults to pass through settings in QEMU if there is only -> -> > > > one root port. -> -> > > > -> -> > > -> -> > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > region successfully. -> -> > > But have some errors in Nvdimm: -> -> > > -> -> > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > assuming node 0 -> -> > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > assuming node 0 -> -> > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > > -> -> > -> -> > Ah. I've seen this one, but not chased it down yet. Was on my todo list -> -> > to chase -> -> > down. Once I reach this state I can verify the HDM Decode is correct -> -> > which is what -> -> > I've been using to test (Which wasn't true until earlier this week). -> -> > I'm currently testing via devmem, more for historical reasons than -> -> > because it makes -> -> > that much sense anymore. -> -> -> -> *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> I'd forgotten that was still on the todo list. I don't think it will -> -> be particularly hard to do and will take a look in next few days. -> -> -> -> Very very indirectly this error is causing a driver probe fail that means -> -> that -> -> we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> Should not have gotten near that path though - hence the problem is actually -> -> when we call cxl_pmem_get_config_data() and it returns an error because -> -> we haven't fully connected up the command in QEMU. -> -> -So a least one bug in QEMU. We were not supporting variable length payloads -> -on mailbox -> -inputs (but were on outputs). That hasn't mattered until we get to LSA -> -writes. -> -We just need to relax condition on the supplied length. -> -> -diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -index c352a935c4..fdda9529fe 100644 -> ---- a/hw/cxl/cxl-mailbox-utils.c -> -+++ b/hw/cxl/cxl-mailbox-utils.c -> -@@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -cxl_cmd = &cxl_cmd_set[set][cmd]; -> -h = cxl_cmd->handler; -> -if (h) { -> -- if (len == cxl_cmd->in) { -> -+ if (len == cxl_cmd->in || !cxl_cmd->in) { -Fix is wrong as we use ~0 as the placeholder for variable payload, not 0. - -With that fixed we hit new fun paths - after some errors we get the -worrying - not totally sure but looks like a failure on an error cleanup. -I'll chase down the error source, but even then this is probably triggerable by -hardware problem or similar. Some bonus prints in here from me chasing -error paths, but it's otherwise just cxl/next + the fix I posted earlier today. - -[ 69.919877] nd_bus ndbus0: START: nd_region.probe(region0) -[ 69.920108] nd_region_probe -[ 69.920623] ------------[ cut here ]------------ -[ 69.920675] refcount_t: addition on 0; use-after-free. -[ 69.921314] WARNING: CPU: 3 PID: 710 at lib/refcount.c:25 -refcount_warn_saturate+0xa0/0x144 -[ 69.926949] Modules linked in: cxl_pmem cxl_mem cxl_pci cxl_port cxl_acpi -cxl_core -[ 69.928830] CPU: 3 PID: 710 Comm: kworker/u8:9 Not tainted 5.19.0-rc3+ #399 -[ 69.930596] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 02/06/2015 -[ 69.931482] Workqueue: events_unbound async_run_entry_fn -[ 69.932403] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) -[ 69.934023] pc : refcount_warn_saturate+0xa0/0x144 -[ 69.935161] lr : refcount_warn_saturate+0xa0/0x144 -[ 69.936541] sp : ffff80000890b960 -[ 69.937921] x29: ffff80000890b960 x28: 0000000000000000 x27: 0000000000000000 -[ 69.940917] x26: ffffa54a90d5cb10 x25: ffffa54a90809e98 x24: 0000000000000000 -[ 69.942537] x23: ffffa54a91a3d8d8 x22: ffff0000c5254800 x21: ffff0000c5254800 -[ 69.944013] x20: ffff0000ce924180 x19: ffff0000c5254800 x18: ffffffffffffffff -[ 69.946100] x17: ffff5ab66e5ef000 x16: ffff80000801c000 x15: 0000000000000000 -[ 69.947585] x14: 0000000000000001 x13: 0a2e656572662d72 x12: 657466612d657375 -[ 69.948670] x11: 203b30206e6f206e x10: 6f69746964646120 x9 : ffffa54a8f63d288 -[ 69.950679] x8 : 206e6f206e6f6974 x7 : 69646461203a745f x6 : 00000000fffff31e -[ 69.952113] x5 : ffff0000ff61ba08 x4 : 00000000fffff31e x3 : ffff5ab66e5ef000 -root@debian:/sys/bus/cxl/devices/decoder0.0/region0# [ 69.954752] x2 : -0000000000000000 x1 : 0000000000000000 x0 : ffff0000c512e740 -[ 69.957098] Call trace: -[ 69.957959] refcount_warn_saturate+0xa0/0x144 -[ 69.958773] get_ndd+0x5c/0x80 -[ 69.959294] nd_region_register_namespaces+0xe4/0xe90 -[ 69.960253] nd_region_probe+0x100/0x290 -[ 69.960796] nvdimm_bus_probe+0xf4/0x1c0 -[ 69.962087] really_probe+0x19c/0x3f0 -[ 69.962620] __driver_probe_device+0x11c/0x190 -[ 69.963258] driver_probe_device+0x44/0xf4 -[ 69.963773] __device_attach_driver+0xa4/0x140 -[ 69.964471] bus_for_each_drv+0x84/0xe0 -[ 69.965068] __device_attach+0xb0/0x1f0 -[ 69.966101] device_initial_probe+0x20/0x30 -[ 69.967142] bus_probe_device+0xa4/0xb0 -[ 69.968104] device_add+0x3e8/0x910 -[ 69.969111] nd_async_device_register+0x24/0x74 -[ 69.969928] async_run_entry_fn+0x40/0x150 -[ 69.970725] process_one_work+0x1dc/0x450 -[ 69.971796] worker_thread+0x154/0x450 -[ 69.972700] kthread+0x118/0x120 -[ 69.974141] ret_from_fork+0x10/0x20 -[ 69.975141] ---[ end trace 0000000000000000 ]--- -[ 70.117887] Into nd_namespace_pmem_set_resource() - -> -cxl_cmd->payload = cxl_dstate->mbox_reg_state + -> -A_CXL_DEV_CMD_PAYLOAD; -> -ret = (*h)(cxl_cmd, cxl_dstate, &len); -> -> -> -This lets the nvdimm/region probe fine, but I'm getting some issues with -> -namespace capacity so I'll look at what is causing that next. -> -Unfortunately I'm not that familiar with the driver/nvdimm side of things -> -so it's take a while to figure out what kicks off what! -> -> -Jonathan -> -> -> -> -> Jonathan -> -> -> -> -> -> > -> -> > > -> -> > > And x4 region still failed with same errors, using latest cxl/preview -> -> > > branch don't work. -> -> > > I have picked "Two CXL emulation fixes" patches in qemu, still not -> -> > > working. -> -> > > -> -> > > Bob -> -> -> -> -> - -On Mon, 15 Aug 2022 18:04:44 +0100 -Jonathan Cameron wrote: - -> -On Fri, 12 Aug 2022 16:44:03 +0100 -> -Jonathan Cameron wrote: -> -> -> On Thu, 11 Aug 2022 18:08:57 +0100 -> -> Jonathan Cameron via wrote: -> -> -> -> > On Tue, 9 Aug 2022 17:08:25 +0100 -> -> > Jonathan Cameron wrote: -> -> > -> -> > > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > > Bobo WL wrote: -> -> > > -> -> > > > Hi Jonathan -> -> > > > -> -> > > > Thanks for your reply! -> -> > > > -> -> > > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > > wrote: -> -> > > > > -> -> > > > > Probably not related to your problem, but there is a disconnect in -> -> > > > > QEMU / -> -> > > > > kernel assumptionsaround the presence of an HDM decoder when a HB -> -> > > > > only -> -> > > > > has a single root port. Spec allows it to be provided or not as an -> -> > > > > implementation choice. -> -> > > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > > -> -> > > > > The temporary solution is to throw in a second root port on the HB -> -> > > > > and not -> -> > > > > connect anything to it. Longer term I may special case this so -> -> > > > > that the particular -> -> > > > > decoder defaults to pass through settings in QEMU if there is only -> -> > > > > one root port. -> -> > > > > -> -> > > > -> -> > > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > > region successfully. -> -> > > > But have some errors in Nvdimm: -> -> > > > -> -> > > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > > assuming node 0 -> -> > > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > > assuming node 0 -> -> > > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > > > -> -> > > -> -> > > Ah. I've seen this one, but not chased it down yet. Was on my todo -> -> > > list to chase -> -> > > down. Once I reach this state I can verify the HDM Decode is correct -> -> > > which is what -> -> > > I've been using to test (Which wasn't true until earlier this week). -> -> > > I'm currently testing via devmem, more for historical reasons than -> -> > > because it makes -> -> > > that much sense anymore. -> -> > -> -> > *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> > I'd forgotten that was still on the todo list. I don't think it will -> -> > be particularly hard to do and will take a look in next few days. -> -> > -> -> > Very very indirectly this error is causing a driver probe fail that means -> -> > that -> -> > we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> > Should not have gotten near that path though - hence the problem is -> -> > actually -> -> > when we call cxl_pmem_get_config_data() and it returns an error because -> -> > we haven't fully connected up the command in QEMU. -> -> -> -> So a least one bug in QEMU. We were not supporting variable length payloads -> -> on mailbox -> -> inputs (but were on outputs). That hasn't mattered until we get to LSA -> -> writes. -> -> We just need to relax condition on the supplied length. -> -> -> -> diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -> index c352a935c4..fdda9529fe 100644 -> -> --- a/hw/cxl/cxl-mailbox-utils.c -> -> +++ b/hw/cxl/cxl-mailbox-utils.c -> -> @@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -> cxl_cmd = &cxl_cmd_set[set][cmd]; -> -> h = cxl_cmd->handler; -> -> if (h) { -> -> - if (len == cxl_cmd->in) { -> -> + if (len == cxl_cmd->in || !cxl_cmd->in) { -> -Fix is wrong as we use ~0 as the placeholder for variable payload, not 0. -Cause of the error is a failure in GET_LSA. -Reason, payload length is wrong in QEMU but was hidden previously by my wrong -fix here. Probably still a good idea to inject an error in GET_LSA and chase -down the refcount issue. - - -diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -index fdda9529fe..e8565fbd6e 100644 ---- a/hw/cxl/cxl-mailbox-utils.c -+++ b/hw/cxl/cxl-mailbox-utils.c -@@ -489,7 +489,7 @@ static struct cxl_cmd cxl_cmd_set[256][256] = { - cmd_identify_memory_device, 0, 0 }, - [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO", - cmd_ccls_get_partition_info, 0, 0 }, -- [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 0, 0 }, -+ [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 8, 0 }, - [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa, - ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE }, - [MEDIA_AND_POISON][GET_POISON_LIST] = { "MEDIA_AND_POISON_GET_POISON_LIST", -@@ -510,12 +510,13 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) - cxl_cmd = &cxl_cmd_set[set][cmd]; - h = cxl_cmd->handler; - if (h) { -- if (len == cxl_cmd->in || !cxl_cmd->in) { -+ if (len == cxl_cmd->in || cxl_cmd->in == ~0) { - cxl_cmd->payload = cxl_dstate->mbox_reg_state + - A_CXL_DEV_CMD_PAYLOAD; - -And woot, we get a namespace in the LSA :) - -I'll post QEMU fixes in next day or two. Kernel side now seems more or less -fine be it with suspicious refcount underflow. - -> -> -With that fixed we hit new fun paths - after some errors we get the -> -worrying - not totally sure but looks like a failure on an error cleanup. -> -I'll chase down the error source, but even then this is probably triggerable -> -by -> -hardware problem or similar. Some bonus prints in here from me chasing -> -error paths, but it's otherwise just cxl/next + the fix I posted earlier -> -today. -> -> -[ 69.919877] nd_bus ndbus0: START: nd_region.probe(region0) -> -[ 69.920108] nd_region_probe -> -[ 69.920623] ------------[ cut here ]------------ -> -[ 69.920675] refcount_t: addition on 0; use-after-free. -> -[ 69.921314] WARNING: CPU: 3 PID: 710 at lib/refcount.c:25 -> -refcount_warn_saturate+0xa0/0x144 -> -[ 69.926949] Modules linked in: cxl_pmem cxl_mem cxl_pci cxl_port cxl_acpi -> -cxl_core -> -[ 69.928830] CPU: 3 PID: 710 Comm: kworker/u8:9 Not tainted 5.19.0-rc3+ #399 -> -[ 69.930596] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 02/06/2015 -> -[ 69.931482] Workqueue: events_unbound async_run_entry_fn -> -[ 69.932403] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) -> -[ 69.934023] pc : refcount_warn_saturate+0xa0/0x144 -> -[ 69.935161] lr : refcount_warn_saturate+0xa0/0x144 -> -[ 69.936541] sp : ffff80000890b960 -> -[ 69.937921] x29: ffff80000890b960 x28: 0000000000000000 x27: -> -0000000000000000 -> -[ 69.940917] x26: ffffa54a90d5cb10 x25: ffffa54a90809e98 x24: -> -0000000000000000 -> -[ 69.942537] x23: ffffa54a91a3d8d8 x22: ffff0000c5254800 x21: -> -ffff0000c5254800 -> -[ 69.944013] x20: ffff0000ce924180 x19: ffff0000c5254800 x18: -> -ffffffffffffffff -> -[ 69.946100] x17: ffff5ab66e5ef000 x16: ffff80000801c000 x15: -> -0000000000000000 -> -[ 69.947585] x14: 0000000000000001 x13: 0a2e656572662d72 x12: -> -657466612d657375 -> -[ 69.948670] x11: 203b30206e6f206e x10: 6f69746964646120 x9 : -> -ffffa54a8f63d288 -> -[ 69.950679] x8 : 206e6f206e6f6974 x7 : 69646461203a745f x6 : -> -00000000fffff31e -> -[ 69.952113] x5 : ffff0000ff61ba08 x4 : 00000000fffff31e x3 : -> -ffff5ab66e5ef000 -> -root@debian:/sys/bus/cxl/devices/decoder0.0/region0# [ 69.954752] x2 : -> -0000000000000000 x1 : 0000000000000000 x0 : ffff0000c512e740 -> -[ 69.957098] Call trace: -> -[ 69.957959] refcount_warn_saturate+0xa0/0x144 -> -[ 69.958773] get_ndd+0x5c/0x80 -> -[ 69.959294] nd_region_register_namespaces+0xe4/0xe90 -> -[ 69.960253] nd_region_probe+0x100/0x290 -> -[ 69.960796] nvdimm_bus_probe+0xf4/0x1c0 -> -[ 69.962087] really_probe+0x19c/0x3f0 -> -[ 69.962620] __driver_probe_device+0x11c/0x190 -> -[ 69.963258] driver_probe_device+0x44/0xf4 -> -[ 69.963773] __device_attach_driver+0xa4/0x140 -> -[ 69.964471] bus_for_each_drv+0x84/0xe0 -> -[ 69.965068] __device_attach+0xb0/0x1f0 -> -[ 69.966101] device_initial_probe+0x20/0x30 -> -[ 69.967142] bus_probe_device+0xa4/0xb0 -> -[ 69.968104] device_add+0x3e8/0x910 -> -[ 69.969111] nd_async_device_register+0x24/0x74 -> -[ 69.969928] async_run_entry_fn+0x40/0x150 -> -[ 69.970725] process_one_work+0x1dc/0x450 -> -[ 69.971796] worker_thread+0x154/0x450 -> -[ 69.972700] kthread+0x118/0x120 -> -[ 69.974141] ret_from_fork+0x10/0x20 -> -[ 69.975141] ---[ end trace 0000000000000000 ]--- -> -[ 70.117887] Into nd_namespace_pmem_set_resource() -> -> -> cxl_cmd->payload = cxl_dstate->mbox_reg_state + -> -> A_CXL_DEV_CMD_PAYLOAD; -> -> ret = (*h)(cxl_cmd, cxl_dstate, &len); -> -> -> -> -> -> This lets the nvdimm/region probe fine, but I'm getting some issues with -> -> namespace capacity so I'll look at what is causing that next. -> -> Unfortunately I'm not that familiar with the driver/nvdimm side of things -> -> so it's take a while to figure out what kicks off what! -> -> -> -> Jonathan -> -> -> -> > -> -> > Jonathan -> -> > -> -> > -> -> > > -> -> > > > -> -> > > > And x4 region still failed with same errors, using latest cxl/preview -> -> > > > branch don't work. -> -> > > > I have picked "Two CXL emulation fixes" patches in qemu, still not -> -> > > > working. -> -> > > > -> -> > > > Bob -> -> > -> -> > -> -> -> - -Jonathan Cameron wrote: -> -On Fri, 12 Aug 2022 16:44:03 +0100 -> -Jonathan Cameron wrote: -> -> -> On Thu, 11 Aug 2022 18:08:57 +0100 -> -> Jonathan Cameron via wrote: -> -> -> -> > On Tue, 9 Aug 2022 17:08:25 +0100 -> -> > Jonathan Cameron wrote: -> -> > -> -> > > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > > Bobo WL wrote: -> -> > > -> -> > > > Hi Jonathan -> -> > > > -> -> > > > Thanks for your reply! -> -> > > > -> -> > > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > > wrote: -> -> > > > > -> -> > > > > Probably not related to your problem, but there is a disconnect in -> -> > > > > QEMU / -> -> > > > > kernel assumptionsaround the presence of an HDM decoder when a HB -> -> > > > > only -> -> > > > > has a single root port. Spec allows it to be provided or not as an -> -> > > > > implementation choice. -> -> > > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > > -> -> > > > > The temporary solution is to throw in a second root port on the HB -> -> > > > > and not -> -> > > > > connect anything to it. Longer term I may special case this so -> -> > > > > that the particular -> -> > > > > decoder defaults to pass through settings in QEMU if there is only -> -> > > > > one root port. -> -> > > > > -> -> > > > -> -> > > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > > region successfully. -> -> > > > But have some errors in Nvdimm: -> -> > > > -> -> > > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > > assuming node 0 -> -> > > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > > assuming node 0 -> -> > > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > > > -> -> > > -> -> > > Ah. I've seen this one, but not chased it down yet. Was on my todo -> -> > > list to chase -> -> > > down. Once I reach this state I can verify the HDM Decode is correct -> -> > > which is what -> -> > > I've been using to test (Which wasn't true until earlier this week). -> -> > > I'm currently testing via devmem, more for historical reasons than -> -> > > because it makes -> -> > > that much sense anymore. -> -> > -> -> > *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> > I'd forgotten that was still on the todo list. I don't think it will -> -> > be particularly hard to do and will take a look in next few days. -> -> > -> -> > Very very indirectly this error is causing a driver probe fail that means -> -> > that -> -> > we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> > Should not have gotten near that path though - hence the problem is -> -> > actually -> -> > when we call cxl_pmem_get_config_data() and it returns an error because -> -> > we haven't fully connected up the command in QEMU. -> -> -> -> So a least one bug in QEMU. We were not supporting variable length payloads -> -> on mailbox -> -> inputs (but were on outputs). That hasn't mattered until we get to LSA -> -> writes. -> -> We just need to relax condition on the supplied length. -> -> -> -> diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -> index c352a935c4..fdda9529fe 100644 -> -> --- a/hw/cxl/cxl-mailbox-utils.c -> -> +++ b/hw/cxl/cxl-mailbox-utils.c -> -> @@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -> cxl_cmd = &cxl_cmd_set[set][cmd]; -> -> h = cxl_cmd->handler; -> -> if (h) { -> -> - if (len == cxl_cmd->in) { -> -> + if (len == cxl_cmd->in || !cxl_cmd->in) { -> -Fix is wrong as we use ~0 as the placeholder for variable payload, not 0. -> -> -With that fixed we hit new fun paths - after some errors we get the -> -worrying - not totally sure but looks like a failure on an error cleanup. -> -I'll chase down the error source, but even then this is probably triggerable -> -by -> -hardware problem or similar. Some bonus prints in here from me chasing -> -error paths, but it's otherwise just cxl/next + the fix I posted earlier -> -today. -One of the scenarios that I cannot rule out is nvdimm_probe() racing -nd_region_probe(), but given all the work it takes to create a region I -suspect all the nvdimm_probe() work to have completed... - -It is at least one potentially wrong hypothesis that needs to be chased -down. - -> -> -[ 69.919877] nd_bus ndbus0: START: nd_region.probe(region0) -> -[ 69.920108] nd_region_probe -> -[ 69.920623] ------------[ cut here ]------------ -> -[ 69.920675] refcount_t: addition on 0; use-after-free. -> -[ 69.921314] WARNING: CPU: 3 PID: 710 at lib/refcount.c:25 -> -refcount_warn_saturate+0xa0/0x144 -> -[ 69.926949] Modules linked in: cxl_pmem cxl_mem cxl_pci cxl_port cxl_acpi -> -cxl_core -> -[ 69.928830] CPU: 3 PID: 710 Comm: kworker/u8:9 Not tainted 5.19.0-rc3+ #399 -> -[ 69.930596] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 02/06/2015 -> -[ 69.931482] Workqueue: events_unbound async_run_entry_fn -> -[ 69.932403] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) -> -[ 69.934023] pc : refcount_warn_saturate+0xa0/0x144 -> -[ 69.935161] lr : refcount_warn_saturate+0xa0/0x144 -> -[ 69.936541] sp : ffff80000890b960 -> -[ 69.937921] x29: ffff80000890b960 x28: 0000000000000000 x27: -> -0000000000000000 -> -[ 69.940917] x26: ffffa54a90d5cb10 x25: ffffa54a90809e98 x24: -> -0000000000000000 -> -[ 69.942537] x23: ffffa54a91a3d8d8 x22: ffff0000c5254800 x21: -> -ffff0000c5254800 -> -[ 69.944013] x20: ffff0000ce924180 x19: ffff0000c5254800 x18: -> -ffffffffffffffff -> -[ 69.946100] x17: ffff5ab66e5ef000 x16: ffff80000801c000 x15: -> -0000000000000000 -> -[ 69.947585] x14: 0000000000000001 x13: 0a2e656572662d72 x12: -> -657466612d657375 -> -[ 69.948670] x11: 203b30206e6f206e x10: 6f69746964646120 x9 : -> -ffffa54a8f63d288 -> -[ 69.950679] x8 : 206e6f206e6f6974 x7 : 69646461203a745f x6 : -> -00000000fffff31e -> -[ 69.952113] x5 : ffff0000ff61ba08 x4 : 00000000fffff31e x3 : -> -ffff5ab66e5ef000 -> -root@debian:/sys/bus/cxl/devices/decoder0.0/region0# [ 69.954752] x2 : -> -0000000000000000 x1 : 0000000000000000 x0 : ffff0000c512e740 -> -[ 69.957098] Call trace: -> -[ 69.957959] refcount_warn_saturate+0xa0/0x144 -> -[ 69.958773] get_ndd+0x5c/0x80 -> -[ 69.959294] nd_region_register_namespaces+0xe4/0xe90 -> -[ 69.960253] nd_region_probe+0x100/0x290 -> -[ 69.960796] nvdimm_bus_probe+0xf4/0x1c0 -> -[ 69.962087] really_probe+0x19c/0x3f0 -> -[ 69.962620] __driver_probe_device+0x11c/0x190 -> -[ 69.963258] driver_probe_device+0x44/0xf4 -> -[ 69.963773] __device_attach_driver+0xa4/0x140 -> -[ 69.964471] bus_for_each_drv+0x84/0xe0 -> -[ 69.965068] __device_attach+0xb0/0x1f0 -> -[ 69.966101] device_initial_probe+0x20/0x30 -> -[ 69.967142] bus_probe_device+0xa4/0xb0 -> -[ 69.968104] device_add+0x3e8/0x910 -> -[ 69.969111] nd_async_device_register+0x24/0x74 -> -[ 69.969928] async_run_entry_fn+0x40/0x150 -> -[ 69.970725] process_one_work+0x1dc/0x450 -> -[ 69.971796] worker_thread+0x154/0x450 -> -[ 69.972700] kthread+0x118/0x120 -> -[ 69.974141] ret_from_fork+0x10/0x20 -> -[ 69.975141] ---[ end trace 0000000000000000 ]--- -> -[ 70.117887] Into nd_namespace_pmem_set_resource() - -On Mon, 15 Aug 2022 15:55:15 -0700 -Dan Williams wrote: - -> -Jonathan Cameron wrote: -> -> On Fri, 12 Aug 2022 16:44:03 +0100 -> -> Jonathan Cameron wrote: -> -> -> -> > On Thu, 11 Aug 2022 18:08:57 +0100 -> -> > Jonathan Cameron via wrote: -> -> > -> -> > > On Tue, 9 Aug 2022 17:08:25 +0100 -> -> > > Jonathan Cameron wrote: -> -> > > -> -> > > > On Tue, 9 Aug 2022 21:07:06 +0800 -> -> > > > Bobo WL wrote: -> -> > > > -> -> > > > > Hi Jonathan -> -> > > > > -> -> > > > > Thanks for your reply! -> -> > > > > -> -> > > > > On Mon, Aug 8, 2022 at 8:37 PM Jonathan Cameron -> -> > > > > wrote: -> -> > > > > > -> -> > > > > > Probably not related to your problem, but there is a disconnect -> -> > > > > > in QEMU / -> -> > > > > > kernel assumptionsaround the presence of an HDM decoder when a HB -> -> > > > > > only -> -> > > > > > has a single root port. Spec allows it to be provided or not as -> -> > > > > > an implementation choice. -> -> > > > > > Kernel assumes it isn't provide. Qemu assumes it is. -> -> > > > > > -> -> > > > > > The temporary solution is to throw in a second root port on the -> -> > > > > > HB and not -> -> > > > > > connect anything to it. Longer term I may special case this so -> -> > > > > > that the particular -> -> > > > > > decoder defaults to pass through settings in QEMU if there is -> -> > > > > > only one root port. -> -> > > > > > -> -> > > > > -> -> > > > > You are right! After adding an extra HB in qemu, I can create a x1 -> -> > > > > region successfully. -> -> > > > > But have some errors in Nvdimm: -> -> > > > > -> -> > > > > [ 74.925838] Unknown online node for memory at 0x10000000000, -> -> > > > > assuming node 0 -> -> > > > > [ 74.925846] Unknown target node for memory at 0x10000000000, -> -> > > > > assuming node 0 -> -> > > > > [ 74.927470] nd_region region0: nmem0: is disabled, failing probe -> -> > > > > -> -> > > > -> -> > > > Ah. I've seen this one, but not chased it down yet. Was on my todo -> -> > > > list to chase -> -> > > > down. Once I reach this state I can verify the HDM Decode is correct -> -> > > > which is what -> -> > > > I've been using to test (Which wasn't true until earlier this week). -> -> > > > I'm currently testing via devmem, more for historical reasons than -> -> > > > because it makes -> -> > > > that much sense anymore. -> -> > > -> -> > > *embarassed cough*. We haven't fully hooked the LSA up in qemu yet. -> -> > > I'd forgotten that was still on the todo list. I don't think it will -> -> > > be particularly hard to do and will take a look in next few days. -> -> > > -> -> > > Very very indirectly this error is causing a driver probe fail that -> -> > > means that -> -> > > we hit a code path that has a rather odd looking check on NDD_LABELING. -> -> > > Should not have gotten near that path though - hence the problem is -> -> > > actually -> -> > > when we call cxl_pmem_get_config_data() and it returns an error because -> -> > > we haven't fully connected up the command in QEMU. -> -> > -> -> > So a least one bug in QEMU. We were not supporting variable length -> -> > payloads on mailbox -> -> > inputs (but were on outputs). That hasn't mattered until we get to LSA -> -> > writes. -> -> > We just need to relax condition on the supplied length. -> -> > -> -> > diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c -> -> > index c352a935c4..fdda9529fe 100644 -> -> > --- a/hw/cxl/cxl-mailbox-utils.c -> -> > +++ b/hw/cxl/cxl-mailbox-utils.c -> -> > @@ -510,7 +510,7 @@ void cxl_process_mailbox(CXLDeviceState *cxl_dstate) -> -> > cxl_cmd = &cxl_cmd_set[set][cmd]; -> -> > h = cxl_cmd->handler; -> -> > if (h) { -> -> > - if (len == cxl_cmd->in) { -> -> > + if (len == cxl_cmd->in || !cxl_cmd->in) { -> -> Fix is wrong as we use ~0 as the placeholder for variable payload, not 0. -> -> -> -> With that fixed we hit new fun paths - after some errors we get the -> -> worrying - not totally sure but looks like a failure on an error cleanup. -> -> I'll chase down the error source, but even then this is probably -> -> triggerable by -> -> hardware problem or similar. Some bonus prints in here from me chasing -> -> error paths, but it's otherwise just cxl/next + the fix I posted earlier -> -> today. -> -> -One of the scenarios that I cannot rule out is nvdimm_probe() racing -> -nd_region_probe(), but given all the work it takes to create a region I -> -suspect all the nvdimm_probe() work to have completed... -> -> -It is at least one potentially wrong hypothesis that needs to be chased -> -down. -Maybe there should be a special award for the non-intuitive -ndctl create-namespace command (modifies existing namespace and might create -a different empty one...) I'm sure there is some interesting history behind -that one :) - -Upshot is I just threw a filesystem on fsdax and wrote some text files on it -to allow easy grepping. The right data ends up in the memory and a plausible -namespace description is stored in the LSA. - -So to some degree at least it's 'working' on an 8 way direct connected -set of emulated devices. - -One snag is that serial number support isn't yet upstream in QEMU. -(I have had it in my tree for a while but not posted it yet because of - QEMU feature freeze) -https://gitlab.com/jic23/qemu/-/commit/144c783ea8a5fbe169f46ea1ba92940157f42733 -That's needed for meaningful cookie generation. Otherwise you can build the -namespace once, but it won't work on next probe as the cookie is 0 and you -hit some error paths. - -Maybe sensible to add a sanity check and fail namespace creation if -cookie is 0? (Silly side question, but is there a theoretical risk of -a serial number / other data combination leading to a fletcher64() -checksum that happens to be 0 - that would give a very odd bug report!) - -So to make it work the following is needed: - -1) The kernel fix for mailbox buffer overflow. -2) Qemu fix for size of arguements for get_lsa -3) Qemu fix to allow variable size input arguements (for set_lsa) -4) Serial number patch above + command lines to qemu to set appropriate - serial numbers. - -I'll send out the QEMU fixes shortly and post the Serial number patch, -though that almost certainly won't go in until next QEMU development -cycle starts in a few weeks. - -Next up, run through same tests on some other topologies. - -Jonathan - -> -> -> -> -> [ 69.919877] nd_bus ndbus0: START: nd_region.probe(region0) -> -> [ 69.920108] nd_region_probe -> -> [ 69.920623] ------------[ cut here ]------------ -> -> [ 69.920675] refcount_t: addition on 0; use-after-free. -> -> [ 69.921314] WARNING: CPU: 3 PID: 710 at lib/refcount.c:25 -> -> refcount_warn_saturate+0xa0/0x144 -> -> [ 69.926949] Modules linked in: cxl_pmem cxl_mem cxl_pci cxl_port -> -> cxl_acpi cxl_core -> -> [ 69.928830] CPU: 3 PID: 710 Comm: kworker/u8:9 Not tainted 5.19.0-rc3+ -> -> #399 -> -> [ 69.930596] Hardware name: QEMU QEMU Virtual Machine, BIOS 0.0.0 -> -> 02/06/2015 -> -> [ 69.931482] Workqueue: events_unbound async_run_entry_fn -> -> [ 69.932403] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS -> -> BTYPE=--) -> -> [ 69.934023] pc : refcount_warn_saturate+0xa0/0x144 -> -> [ 69.935161] lr : refcount_warn_saturate+0xa0/0x144 -> -> [ 69.936541] sp : ffff80000890b960 -> -> [ 69.937921] x29: ffff80000890b960 x28: 0000000000000000 x27: -> -> 0000000000000000 -> -> [ 69.940917] x26: ffffa54a90d5cb10 x25: ffffa54a90809e98 x24: -> -> 0000000000000000 -> -> [ 69.942537] x23: ffffa54a91a3d8d8 x22: ffff0000c5254800 x21: -> -> ffff0000c5254800 -> -> [ 69.944013] x20: ffff0000ce924180 x19: ffff0000c5254800 x18: -> -> ffffffffffffffff -> -> [ 69.946100] x17: ffff5ab66e5ef000 x16: ffff80000801c000 x15: -> -> 0000000000000000 -> -> [ 69.947585] x14: 0000000000000001 x13: 0a2e656572662d72 x12: -> -> 657466612d657375 -> -> [ 69.948670] x11: 203b30206e6f206e x10: 6f69746964646120 x9 : -> -> ffffa54a8f63d288 -> -> [ 69.950679] x8 : 206e6f206e6f6974 x7 : 69646461203a745f x6 : -> -> 00000000fffff31e -> -> [ 69.952113] x5 : ffff0000ff61ba08 x4 : 00000000fffff31e x3 : -> -> ffff5ab66e5ef000 -> -> root@debian:/sys/bus/cxl/devices/decoder0.0/region0# [ 69.954752] x2 : -> -> 0000000000000000 x1 : 0000000000000000 x0 : ffff0000c512e740 -> -> [ 69.957098] Call trace: -> -> [ 69.957959] refcount_warn_saturate+0xa0/0x144 -> -> [ 69.958773] get_ndd+0x5c/0x80 -> -> [ 69.959294] nd_region_register_namespaces+0xe4/0xe90 -> -> [ 69.960253] nd_region_probe+0x100/0x290 -> -> [ 69.960796] nvdimm_bus_probe+0xf4/0x1c0 -> -> [ 69.962087] really_probe+0x19c/0x3f0 -> -> [ 69.962620] __driver_probe_device+0x11c/0x190 -> -> [ 69.963258] driver_probe_device+0x44/0xf4 -> -> [ 69.963773] __device_attach_driver+0xa4/0x140 -> -> [ 69.964471] bus_for_each_drv+0x84/0xe0 -> -> [ 69.965068] __device_attach+0xb0/0x1f0 -> -> [ 69.966101] device_initial_probe+0x20/0x30 -> -> [ 69.967142] bus_probe_device+0xa4/0xb0 -> -> [ 69.968104] device_add+0x3e8/0x910 -> -> [ 69.969111] nd_async_device_register+0x24/0x74 -> -> [ 69.969928] async_run_entry_fn+0x40/0x150 -> -> [ 69.970725] process_one_work+0x1dc/0x450 -> -> [ 69.971796] worker_thread+0x154/0x450 -> -> [ 69.972700] kthread+0x118/0x120 -> -> [ 69.974141] ret_from_fork+0x10/0x20 -> -> [ 69.975141] ---[ end trace 0000000000000000 ]--- -> -> [ 70.117887] Into nd_namespace_pmem_set_resource() - -Bobo WL wrote: -> -Hi list -> -> -I want to test cxl functions in arm64, and found some problems I can't -> -figure out. -> -> -My test environment: -> -> -1. build latest bios from -https://github.com/tianocore/edk2.git -master -> -branch(cc2db6ebfb6d9d85ba4c7b35fba1fa37fffc0bc2) -> -2. build latest qemu-system-aarch64 from git://git.qemu.org/qemu.git -> -master branch(846dcf0ba4eff824c295f06550b8673ff3f31314). With cxl arm -> -support patch: -> -https://patchwork.kernel.org/project/cxl/cover/20220616141950.23374-1-Jonathan.Cameron@huawei.com/ -> -3. build Linux kernel from -> -https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git -preview -> -branch(65fc1c3d26b96002a5aa1f4012fae4dc98fd5683) -> -4. build latest ndctl tools from -https://github.com/pmem/ndctl -> -create_region branch(8558b394e449779e3a4f3ae90fae77ede0bca159) -> -> -And my qemu test commands: -> -sudo $QEMU_BIN -M virt,gic-version=3,cxl=on -m 4g,maxmem=8G,slots=8 \ -> --cpu max -smp 8 -nographic -no-reboot \ -> --kernel $KERNEL -bios $BIOS_BIN \ -> --drive if=none,file=$ROOTFS,format=qcow2,id=hd \ -> --device virtio-blk-pci,drive=hd -append 'root=/dev/vda1 -> -nokaslr dyndbg="module cxl* +p"' \ -> --object memory-backend-ram,size=4G,id=mem0 \ -> --numa node,nodeid=0,cpus=0-7,memdev=mem0 \ -> --net nic -net user,hostfwd=tcp::2222-:22 -enable-kvm \ -> --object -> -memory-backend-file,id=cxl-mem0,share=on,mem-path=/tmp/cxltest.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa0.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M -> -\ -> --object -> -memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M -> -\ -> --device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -> --device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \ -> --device cxl-upstream,bus=root_port0,id=us0 \ -> --device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ -> --device -> -cxl-type3,bus=swport0,memdev=cxl-mem0,lsa=cxl-lsa0,id=cxl-pmem0 \ -> --device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ -> --device -> -cxl-type3,bus=swport1,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem1 \ -> --device cxl-downstream,port=2,bus=us0,id=swport2,chassis=0,slot=6 \ -> --device -> -cxl-type3,bus=swport2,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem2 \ -> --device cxl-downstream,port=3,bus=us0,id=swport3,chassis=0,slot=7 \ -> --device -> -cxl-type3,bus=swport3,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem3 \ -> --M -> -cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k -> -> -And I have got two problems. -> -1. When I want to create x1 region with command: "cxl create-region -d -> -decoder0.0 -w 1 -g 4096 mem0", kernel crashed with null pointer -> -reference. Crash log: -> -> -[ 534.697324] cxl_region region0: config state: 0 -> -[ 534.697346] cxl_region region0: probe: -6 -> -[ 534.697368] cxl_acpi ACPI0017:00: decoder0.0: created region0 -> -[ 534.699115] cxl region0: mem0:endpoint3 decoder3.0 add: -> -mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1 -> -[ 534.699149] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1 -> -[ 534.699167] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1 -> -[ 534.699176] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256 -> -[ 534.699182] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0 -> -for mem0:decoder3.0 @ 0 -> -[ 534.699189] cxl region0: 0000:0d:00.0:port2 iw: 1 ig: 256 -> -[ 534.699193] cxl region0: 0000:0d:00.0:port2 target[0] = -> -0000:0e:00.0 for mem0:decoder3.0 @ 0 -> -[ 534.699405] Unable to handle kernel NULL pointer dereference at -> -virtual address 0000000000000000 -> -[ 534.701474] Mem abort info: -> -[ 534.701994] ESR = 0x0000000086000004 -> -[ 534.702653] EC = 0x21: IABT (current EL), IL = 32 bits -> -[ 534.703616] SET = 0, FnV = 0 -> -[ 534.704174] EA = 0, S1PTW = 0 -> -[ 534.704803] FSC = 0x04: level 0 translation fault -> -[ 534.705694] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010144a000 -> -[ 534.706875] [0000000000000000] pgd=0000000000000000, p4d=0000000000000000 -> -[ 534.709855] Internal error: Oops: 86000004 [#1] PREEMPT SMP -> -[ 534.710301] Modules linked in: -> -[ 534.710546] CPU: 7 PID: 331 Comm: cxl Not tainted -> -5.19.0-rc3-00064-g65fc1c3d26b9-dirty #11 -> -[ 534.715393] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 -> -[ 534.717179] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) -> -[ 534.719190] pc : 0x0 -> -[ 534.719928] lr : commit_store+0x118/0x2cc -> -[ 534.721007] sp : ffff80000aec3c30 -> -[ 534.721793] x29: ffff80000aec3c30 x28: ffff0000da62e740 x27: -> -ffff0000c0c06b30 -> -[ 534.723875] x26: 0000000000000000 x25: ffff0000c0a2a400 x24: -> -ffff0000c0a29400 -> -[ 534.725440] x23: 0000000000000003 x22: 0000000000000000 x21: -> -ffff0000c0c06800 -> -[ 534.727312] x20: 0000000000000000 x19: ffff0000c1559800 x18: -> -0000000000000000 -> -[ 534.729138] x17: 0000000000000000 x16: 0000000000000000 x15: -> -0000ffffd41fe838 -> -[ 534.731046] x14: 0000000000000000 x13: 0000000000000000 x12: -> -0000000000000000 -> -[ 534.732402] x11: 0000000000000000 x10: 0000000000000000 x9 : -> -0000000000000000 -> -[ 534.734432] x8 : 0000000000000000 x7 : 0000000000000000 x6 : -> -ffff0000c0906e80 -> -[ 534.735921] x5 : 0000000000000000 x4 : 0000000000000000 x3 : -> -ffff80000aec3bf0 -> -[ 534.737437] x2 : 0000000000000000 x1 : 0000000000000000 x0 : -> -ffff0000c155a000 -> -[ 534.738878] Call trace: -> -[ 534.739368] 0x0 -> -[ 534.739713] dev_attr_store+0x1c/0x30 -> -[ 534.740186] sysfs_kf_write+0x48/0x58 -> -[ 534.740961] kernfs_fop_write_iter+0x128/0x184 -> -[ 534.741872] new_sync_write+0xdc/0x158 -> -[ 534.742706] vfs_write+0x1ac/0x2a8 -> -[ 534.743440] ksys_write+0x68/0xf0 -> -[ 534.744328] __arm64_sys_write+0x1c/0x28 -> -[ 534.745180] invoke_syscall+0x44/0xf0 -> -[ 534.745989] el0_svc_common+0x4c/0xfc -> -[ 534.746661] do_el0_svc+0x60/0xa8 -> -[ 534.747378] el0_svc+0x2c/0x78 -> -[ 534.748066] el0t_64_sync_handler+0xb8/0x12c -> -[ 534.748919] el0t_64_sync+0x18c/0x190 -> -[ 534.749629] Code: bad PC value -> -[ 534.750169] ---[ end trace 0000000000000000 ]--- -What was the top kernel commit when you ran this test? What is the line -number of "commit_store+0x118"? - -> -2. When I want to create x4 region with command: "cxl create-region -d -> -decoder0.0 -w 4 -g 4096 -m mem0 mem1 mem2 mem3". I got below errors: -> -> -cxl region: create_region: region0: failed to set target3 to mem3 -> -cxl region: cmd_create_region: created 0 regions -> -> -And kernel log as below: -> -[ 60.536663] cxl_region region0: config state: 0 -> -[ 60.536675] cxl_region region0: probe: -6 -> -[ 60.536696] cxl_acpi ACPI0017:00: decoder0.0: created region0 -> -[ 60.538251] cxl region0: mem0:endpoint3 decoder3.0 add: -> -mem0:decoder3.0 @ 0 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.538278] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem0:decoder3.0 @ 0 next: mem0 nr_eps: 1 nr_targets: 1 -> -[ 60.538295] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem0:decoder3.0 @ 0 next: 0000:0d:00.0 nr_eps: 1 nr_targets: 1 -> -[ 60.538647] cxl region0: mem1:endpoint4 decoder4.0 add: -> -mem1:decoder4.0 @ 1 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.538663] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem1:decoder4.0 @ 1 next: mem1 nr_eps: 2 nr_targets: 2 -> -[ 60.538675] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem1:decoder4.0 @ 1 next: 0000:0d:00.0 nr_eps: 2 nr_targets: 1 -> -[ 60.539311] cxl region0: mem2:endpoint5 decoder5.0 add: -> -mem2:decoder5.0 @ 2 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.539332] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem2:decoder5.0 @ 2 next: mem2 nr_eps: 3 nr_targets: 3 -> -[ 60.539343] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem2:decoder5.0 @ 2 next: 0000:0d:00.0 nr_eps: 3 nr_targets: 1 -> -[ 60.539711] cxl region0: mem3:endpoint6 decoder6.0 add: -> -mem3:decoder6.0 @ 3 next: none nr_eps: 1 nr_targets: 1 -> -[ 60.539723] cxl region0: 0000:0d:00.0:port2 decoder2.0 add: -> -mem3:decoder6.0 @ 3 next: mem3 nr_eps: 4 nr_targets: 4 -> -[ 60.539735] cxl region0: ACPI0016:00:port1 decoder1.0 add: -> -mem3:decoder6.0 @ 3 next: 0000:0d:00.0 nr_eps: 4 nr_targets: 1 -> -[ 60.539742] cxl region0: ACPI0016:00:port1 iw: 1 ig: 256 -> -[ 60.539747] cxl region0: ACPI0016:00:port1 target[0] = 0000:0c:00.0 -> -for mem0:decoder3.0 @ 0 -> -[ 60.539754] cxl region0: 0000:0d:00.0:port2 iw: 4 ig: 512 -> -[ 60.539758] cxl region0: 0000:0d:00.0:port2 target[0] = -> -0000:0e:00.0 for mem0:decoder3.0 @ 0 -> -[ 60.539764] cxl region0: ACPI0016:00:port1: cannot host mem1:decoder4.0 at -> -1 -> -> -I have tried to write sysfs node manually, got same errors. -> -> -Hope I can get some helps here. -What is the output of: - - cxl list -MDTu -d decoder0.0 - -...? It might be the case that mem1 cannot be mapped by decoder0.0, or -at least not in the specified order, or that validation check is broken. - -Hi Dan, - -Thanks for your reply! - -On Mon, Aug 8, 2022 at 11:58 PM Dan Williams wrote: -> -> -What is the output of: -> -> -cxl list -MDTu -d decoder0.0 -> -> -...? It might be the case that mem1 cannot be mapped by decoder0.0, or -> -at least not in the specified order, or that validation check is broken. -Command "cxl list -MDTu -d decoder0.0" output: - -[ - { - "memdevs":[ - { - "memdev":"mem2", - "pmem_size":"256.00 MiB (268.44 MB)", - "ram_size":0, - "serial":"0", - "host":"0000:11:00.0" - }, - { - "memdev":"mem1", - "pmem_size":"256.00 MiB (268.44 MB)", - "ram_size":0, - "serial":"0", - "host":"0000:10:00.0" - }, - { - "memdev":"mem0", - "pmem_size":"256.00 MiB (268.44 MB)", - "ram_size":0, - "serial":"0", - "host":"0000:0f:00.0" - }, - { - "memdev":"mem3", - "pmem_size":"256.00 MiB (268.44 MB)", - "ram_size":0, - "serial":"0", - "host":"0000:12:00.0" - } - ] - }, - { - "root decoders":[ - { - "decoder":"decoder0.0", - "resource":"0x10000000000", - "size":"4.00 GiB (4.29 GB)", - "pmem_capable":true, - "volatile_capable":true, - "accelmem_capable":true, - "nr_targets":1, - "targets":[ - { - "target":"ACPI0016:01", - "alias":"pci0000:0c", - "position":0, - "id":"0xc" - } - ] - } - ] - } -] - -Bobo WL wrote: -> -Hi Dan, -> -> -Thanks for your reply! -> -> -On Mon, Aug 8, 2022 at 11:58 PM Dan Williams wrote: -> -> -> -> What is the output of: -> -> -> -> cxl list -MDTu -d decoder0.0 -> -> -> -> ...? It might be the case that mem1 cannot be mapped by decoder0.0, or -> -> at least not in the specified order, or that validation check is broken. -> -> -Command "cxl list -MDTu -d decoder0.0" output: -Thanks for this, I think I know the problem, but will try some -experiments with cxl_test first. - -Did the commit_store() crash stop reproducing with latest cxl/preview -branch? - -On Tue, Aug 9, 2022 at 11:17 PM Dan Williams wrote: -> -> -Bobo WL wrote: -> -> Hi Dan, -> -> -> -> Thanks for your reply! -> -> -> -> On Mon, Aug 8, 2022 at 11:58 PM Dan Williams -> -> wrote: -> -> > -> -> > What is the output of: -> -> > -> -> > cxl list -MDTu -d decoder0.0 -> -> > -> -> > ...? It might be the case that mem1 cannot be mapped by decoder0.0, or -> -> > at least not in the specified order, or that validation check is broken. -> -> -> -> Command "cxl list -MDTu -d decoder0.0" output: -> -> -Thanks for this, I think I know the problem, but will try some -> -experiments with cxl_test first. -> -> -Did the commit_store() crash stop reproducing with latest cxl/preview -> -branch? -No, still hitting this bug if don't add extra HB device in qemu - -Dan Williams wrote: -> -Bobo WL wrote: -> -> Hi Dan, -> -> -> -> Thanks for your reply! -> -> -> -> On Mon, Aug 8, 2022 at 11:58 PM Dan Williams -> -> wrote: -> -> > -> -> > What is the output of: -> -> > -> -> > cxl list -MDTu -d decoder0.0 -> -> > -> -> > ...? It might be the case that mem1 cannot be mapped by decoder0.0, or -> -> > at least not in the specified order, or that validation check is broken. -> -> -> -> Command "cxl list -MDTu -d decoder0.0" output: -> -> -Thanks for this, I think I know the problem, but will try some -> -experiments with cxl_test first. -Hmm, so my cxl_test experiment unfortunately passed so I'm not -reproducing the failure mode. This is the result of creating x4 region -with devices directly attached to a single host-bridge: - -# cxl create-region -d decoder3.5 -w 4 -m -g 256 mem{12,10,9,11} -s $((1<<30)) -{ - "region":"region8", - "resource":"0xf1f0000000", - "size":"1024.00 MiB (1073.74 MB)", - "interleave_ways":4, - "interleave_granularity":256, - "decode_state":"commit", - "mappings":[ - { - "position":3, - "memdev":"mem11", - "decoder":"decoder21.0" - }, - { - "position":2, - "memdev":"mem9", - "decoder":"decoder19.0" - }, - { - "position":1, - "memdev":"mem10", - "decoder":"decoder20.0" - }, - { - "position":0, - "memdev":"mem12", - "decoder":"decoder22.0" - } - ] -} -cxl region: cmd_create_region: created 1 region - -> -Did the commit_store() crash stop reproducing with latest cxl/preview -> -branch? -I missed the answer to this question. - -All of these changes are now in Linus' tree perhaps give that a try and -post the debug log again? - -On Thu, 11 Aug 2022 17:46:55 -0700 -Dan Williams wrote: - -> -Dan Williams wrote: -> -> Bobo WL wrote: -> -> > Hi Dan, -> -> > -> -> > Thanks for your reply! -> -> > -> -> > On Mon, Aug 8, 2022 at 11:58 PM Dan Williams -> -> > wrote: -> -> > > -> -> > > What is the output of: -> -> > > -> -> > > cxl list -MDTu -d decoder0.0 -> -> > > -> -> > > ...? It might be the case that mem1 cannot be mapped by decoder0.0, or -> -> > > at least not in the specified order, or that validation check is -> -> > > broken. -> -> > -> -> > Command "cxl list -MDTu -d decoder0.0" output: -> -> -> -> Thanks for this, I think I know the problem, but will try some -> -> experiments with cxl_test first. -> -> -Hmm, so my cxl_test experiment unfortunately passed so I'm not -> -reproducing the failure mode. This is the result of creating x4 region -> -with devices directly attached to a single host-bridge: -> -> -# cxl create-region -d decoder3.5 -w 4 -m -g 256 mem{12,10,9,11} -s $((1<<30)) -> -{ -> -"region":"region8", -> -"resource":"0xf1f0000000", -> -"size":"1024.00 MiB (1073.74 MB)", -> -"interleave_ways":4, -> -"interleave_granularity":256, -> -"decode_state":"commit", -> -"mappings":[ -> -{ -> -"position":3, -> -"memdev":"mem11", -> -"decoder":"decoder21.0" -> -}, -> -{ -> -"position":2, -> -"memdev":"mem9", -> -"decoder":"decoder19.0" -> -}, -> -{ -> -"position":1, -> -"memdev":"mem10", -> -"decoder":"decoder20.0" -> -}, -> -{ -> -"position":0, -> -"memdev":"mem12", -> -"decoder":"decoder22.0" -> -} -> -] -> -} -> -cxl region: cmd_create_region: created 1 region -> -> -> Did the commit_store() crash stop reproducing with latest cxl/preview -> -> branch? -> -> -I missed the answer to this question. -> -> -All of these changes are now in Linus' tree perhaps give that a try and -> -post the debug log again? -Hi Dan, - -I've moved onto looking at this one. -1 HB, 2RP (to make it configure the HDM decoder in the QEMU HB, I'll tidy that -up -at some stage), 1 switch, 4 downstream switch ports each with a type 3 - -I'm not getting a crash, but can't successfully setup a region. -Upon adding the final target -It's failing in check_last_peer() as pos < distance. -Seems distance is 4 which makes me think it's using the wrong level of the -heirarchy for -some reason or that distance check is wrong. -Wasn't a good idea to just skip that step though as it goes boom - though -stack trace is not useful. - -Jonathan - -On Wed, 17 Aug 2022 17:16:19 +0100 -Jonathan Cameron wrote: - -> -On Thu, 11 Aug 2022 17:46:55 -0700 -> -Dan Williams wrote: -> -> -> Dan Williams wrote: -> -> > Bobo WL wrote: -> -> > > Hi Dan, -> -> > > -> -> > > Thanks for your reply! -> -> > > -> -> > > On Mon, Aug 8, 2022 at 11:58 PM Dan Williams -> -> > > wrote: -> -> > > > -> -> > > > What is the output of: -> -> > > > -> -> > > > cxl list -MDTu -d decoder0.0 -> -> > > > -> -> > > > ...? It might be the case that mem1 cannot be mapped by decoder0.0, or -> -> > > > at least not in the specified order, or that validation check is -> -> > > > broken. -> -> > > -> -> > > Command "cxl list -MDTu -d decoder0.0" output: -> -> > -> -> > Thanks for this, I think I know the problem, but will try some -> -> > experiments with cxl_test first. -> -> -> -> Hmm, so my cxl_test experiment unfortunately passed so I'm not -> -> reproducing the failure mode. This is the result of creating x4 region -> -> with devices directly attached to a single host-bridge: -> -> -> -> # cxl create-region -d decoder3.5 -w 4 -m -g 256 mem{12,10,9,11} -s -> -> $((1<<30)) -> -> { -> -> "region":"region8", -> -> "resource":"0xf1f0000000", -> -> "size":"1024.00 MiB (1073.74 MB)", -> -> "interleave_ways":4, -> -> "interleave_granularity":256, -> -> "decode_state":"commit", -> -> "mappings":[ -> -> { -> -> "position":3, -> -> "memdev":"mem11", -> -> "decoder":"decoder21.0" -> -> }, -> -> { -> -> "position":2, -> -> "memdev":"mem9", -> -> "decoder":"decoder19.0" -> -> }, -> -> { -> -> "position":1, -> -> "memdev":"mem10", -> -> "decoder":"decoder20.0" -> -> }, -> -> { -> -> "position":0, -> -> "memdev":"mem12", -> -> "decoder":"decoder22.0" -> -> } -> -> ] -> -> } -> -> cxl region: cmd_create_region: created 1 region -> -> -> -> > Did the commit_store() crash stop reproducing with latest cxl/preview -> -> > branch? -> -> -> -> I missed the answer to this question. -> -> -> -> All of these changes are now in Linus' tree perhaps give that a try and -> -> post the debug log again? -> -> -Hi Dan, -> -> -I've moved onto looking at this one. -> -1 HB, 2RP (to make it configure the HDM decoder in the QEMU HB, I'll tidy -> -that up -> -at some stage), 1 switch, 4 downstream switch ports each with a type 3 -> -> -I'm not getting a crash, but can't successfully setup a region. -> -Upon adding the final target -> -It's failing in check_last_peer() as pos < distance. -> -Seems distance is 4 which makes me think it's using the wrong level of the -> -heirarchy for -> -some reason or that distance check is wrong. -> -Wasn't a good idea to just skip that step though as it goes boom - though -> -stack trace is not useful. -Turns out really weird corruption happens if you accidentally back two type3 -devices -with the same memory device. Who would have thought it :) - -That aside ignoring the check_last_peer() failure seems to make everything work -for this -topology. I'm not seeing the crash, so my guess is we fixed it somewhere along -the way. - -Now for the fun one. I've replicated the crash if we have - -1HB 1*RP 1SW, 4SW-DSP, 4Type3 - -Now, I'd expect to see it not 'work' because the QEMU HDM decoder won't be -programmed -but the null pointer dereference isn't related to that. - -The bug is straight forward. Not all decoders have commit callbacks... Will -send out -a possible fix shortly. - -Jonathan - - - -> -> -Jonathan -> -> -> -> -> -> - -On Thu, 18 Aug 2022 17:37:40 +0100 -Jonathan Cameron via wrote: - -> -On Wed, 17 Aug 2022 17:16:19 +0100 -> -Jonathan Cameron wrote: -> -> -> On Thu, 11 Aug 2022 17:46:55 -0700 -> -> Dan Williams wrote: -> -> -> -> > Dan Williams wrote: -> -> > > Bobo WL wrote: -> -> > > > Hi Dan, -> -> > > > -> -> > > > Thanks for your reply! -> -> > > > -> -> > > > On Mon, Aug 8, 2022 at 11:58 PM Dan Williams -> -> > > > wrote: -> -> > > > > -> -> > > > > What is the output of: -> -> > > > > -> -> > > > > cxl list -MDTu -d decoder0.0 -> -> > > > > -> -> > > > > ...? It might be the case that mem1 cannot be mapped by decoder0.0, -> -> > > > > or -> -> > > > > at least not in the specified order, or that validation check is -> -> > > > > broken. -> -> > > > -> -> > > > Command "cxl list -MDTu -d decoder0.0" output: -> -> > > -> -> > > Thanks for this, I think I know the problem, but will try some -> -> > > experiments with cxl_test first. -> -> > -> -> > Hmm, so my cxl_test experiment unfortunately passed so I'm not -> -> > reproducing the failure mode. This is the result of creating x4 region -> -> > with devices directly attached to a single host-bridge: -> -> > -> -> > # cxl create-region -d decoder3.5 -w 4 -m -g 256 mem{12,10,9,11} -s -> -> > $((1<<30)) -> -> > { -> -> > "region":"region8", -> -> > "resource":"0xf1f0000000", -> -> > "size":"1024.00 MiB (1073.74 MB)", -> -> > "interleave_ways":4, -> -> > "interleave_granularity":256, -> -> > "decode_state":"commit", -> -> > "mappings":[ -> -> > { -> -> > "position":3, -> -> > "memdev":"mem11", -> -> > "decoder":"decoder21.0" -> -> > }, -> -> > { -> -> > "position":2, -> -> > "memdev":"mem9", -> -> > "decoder":"decoder19.0" -> -> > }, -> -> > { -> -> > "position":1, -> -> > "memdev":"mem10", -> -> > "decoder":"decoder20.0" -> -> > }, -> -> > { -> -> > "position":0, -> -> > "memdev":"mem12", -> -> > "decoder":"decoder22.0" -> -> > } -> -> > ] -> -> > } -> -> > cxl region: cmd_create_region: created 1 region -> -> > -> -> > > Did the commit_store() crash stop reproducing with latest cxl/preview -> -> > > branch? -> -> > -> -> > I missed the answer to this question. -> -> > -> -> > All of these changes are now in Linus' tree perhaps give that a try and -> -> > post the debug log again? -> -> -> -> Hi Dan, -> -> -> -> I've moved onto looking at this one. -> -> 1 HB, 2RP (to make it configure the HDM decoder in the QEMU HB, I'll tidy -> -> that up -> -> at some stage), 1 switch, 4 downstream switch ports each with a type 3 -> -> -> -> I'm not getting a crash, but can't successfully setup a region. -> -> Upon adding the final target -> -> It's failing in check_last_peer() as pos < distance. -> -> Seems distance is 4 which makes me think it's using the wrong level of the -> -> heirarchy for -> -> some reason or that distance check is wrong. -> -> Wasn't a good idea to just skip that step though as it goes boom - though -> -> stack trace is not useful. -> -> -Turns out really weird corruption happens if you accidentally back two type3 -> -devices -> -with the same memory device. Who would have thought it :) -> -> -That aside ignoring the check_last_peer() failure seems to make everything -> -work for this -> -topology. I'm not seeing the crash, so my guess is we fixed it somewhere -> -along the way. -> -> -Now for the fun one. I've replicated the crash if we have -> -> -1HB 1*RP 1SW, 4SW-DSP, 4Type3 -> -> -Now, I'd expect to see it not 'work' because the QEMU HDM decoder won't be -> -programmed -> -but the null pointer dereference isn't related to that. -> -> -The bug is straight forward. Not all decoders have commit callbacks... Will -> -send out -> -a possible fix shortly. -> -For completeness I'm carrying this hack because I haven't gotten my head -around the right fix for check_last_peer() failing on this test topology. - -diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c -index c49d9a5f1091..275e143bd748 100644 ---- a/drivers/cxl/core/region.c -+++ b/drivers/cxl/core/region.c -@@ -978,7 +978,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, - rc = check_last_peer(cxled, ep, cxl_rr, - distance); - if (rc) -- return rc; -+ // return rc; - goto out_target_set; - } - goto add_target; --- - -I might find more bugs with more testing, but this is all the ones I've -seen so far + in Bobo's reports. Qemu fixes are now in upstream so -will be there in the release. - -As a reminder, testing on QEMU has a few corners... - -Need a patch to add serial number ECAP support. It is on list for revew, -but will have wait for after QEMU 7.1 release (which may be next week) - -QEMU still assumes HDM decoder on the host bridge will be programmed. -So if you want anything to work there should be at least -2 RP below the HB (no need to plug anything in to one of them). - -I don't want to add a commandline parameter to hide the decoder in QEMU -and detecting there is only one RP would require moving a bunch of static -stuff into runtime code (I think). - -I still think we should make the kernel check to see if there is a decoder, -but if not I might see how bad a hack it is to have QEMU ignore that decoder -if not committed in this one special case (HB HDM decoder with only one place -it can send stuff). Obviously that would be a break from specification -so less than idea! - -Thanks, - -Jonathan - -On Fri, 19 Aug 2022 09:46:55 +0100 -Jonathan Cameron wrote: - -> -On Thu, 18 Aug 2022 17:37:40 +0100 -> -Jonathan Cameron via wrote: -> -> -> On Wed, 17 Aug 2022 17:16:19 +0100 -> -> Jonathan Cameron wrote: -> -> -> -> > On Thu, 11 Aug 2022 17:46:55 -0700 -> -> > Dan Williams wrote: -> -> > -> -> > > Dan Williams wrote: -> -> > > > Bobo WL wrote: -> -> > > > > Hi Dan, -> -> > > > > -> -> > > > > Thanks for your reply! -> -> > > > > -> -> > > > > On Mon, Aug 8, 2022 at 11:58 PM Dan Williams -> -> > > > > wrote: -> -> > > > > > -> -> > > > > > What is the output of: -> -> > > > > > -> -> > > > > > cxl list -MDTu -d decoder0.0 -> -> > > > > > -> -> > > > > > ...? It might be the case that mem1 cannot be mapped by -> -> > > > > > decoder0.0, or -> -> > > > > > at least not in the specified order, or that validation check is -> -> > > > > > broken. -> -> > > > > -> -> > > > > Command "cxl list -MDTu -d decoder0.0" output: -> -> > > > -> -> > > > Thanks for this, I think I know the problem, but will try some -> -> > > > experiments with cxl_test first. -> -> > > -> -> > > Hmm, so my cxl_test experiment unfortunately passed so I'm not -> -> > > reproducing the failure mode. This is the result of creating x4 region -> -> > > with devices directly attached to a single host-bridge: -> -> > > -> -> > > # cxl create-region -d decoder3.5 -w 4 -m -g 256 mem{12,10,9,11} -s -> -> > > $((1<<30)) -> -> > > { -> -> > > "region":"region8", -> -> > > "resource":"0xf1f0000000", -> -> > > "size":"1024.00 MiB (1073.74 MB)", -> -> > > "interleave_ways":4, -> -> > > "interleave_granularity":256, -> -> > > "decode_state":"commit", -> -> > > "mappings":[ -> -> > > { -> -> > > "position":3, -> -> > > "memdev":"mem11", -> -> > > "decoder":"decoder21.0" -> -> > > }, -> -> > > { -> -> > > "position":2, -> -> > > "memdev":"mem9", -> -> > > "decoder":"decoder19.0" -> -> > > }, -> -> > > { -> -> > > "position":1, -> -> > > "memdev":"mem10", -> -> > > "decoder":"decoder20.0" -> -> > > }, -> -> > > { -> -> > > "position":0, -> -> > > "memdev":"mem12", -> -> > > "decoder":"decoder22.0" -> -> > > } -> -> > > ] -> -> > > } -> -> > > cxl region: cmd_create_region: created 1 region -> -> > > -> -> > > > Did the commit_store() crash stop reproducing with latest cxl/preview -> -> > > > branch? -> -> > > -> -> > > I missed the answer to this question. -> -> > > -> -> > > All of these changes are now in Linus' tree perhaps give that a try and -> -> > > post the debug log again? -> -> > -> -> > Hi Dan, -> -> > -> -> > I've moved onto looking at this one. -> -> > 1 HB, 2RP (to make it configure the HDM decoder in the QEMU HB, I'll tidy -> -> > that up -> -> > at some stage), 1 switch, 4 downstream switch ports each with a type 3 -> -> > -> -> > I'm not getting a crash, but can't successfully setup a region. -> -> > Upon adding the final target -> -> > It's failing in check_last_peer() as pos < distance. -> -> > Seems distance is 4 which makes me think it's using the wrong level of -> -> > the heirarchy for -> -> > some reason or that distance check is wrong. -> -> > Wasn't a good idea to just skip that step though as it goes boom - though -> -> > stack trace is not useful. -> -> -> -> Turns out really weird corruption happens if you accidentally back two -> -> type3 devices -> -> with the same memory device. Who would have thought it :) -> -> -> -> That aside ignoring the check_last_peer() failure seems to make everything -> -> work for this -> -> topology. I'm not seeing the crash, so my guess is we fixed it somewhere -> -> along the way. -> -> -> -> Now for the fun one. I've replicated the crash if we have -> -> -> -> 1HB 1*RP 1SW, 4SW-DSP, 4Type3 -> -> -> -> Now, I'd expect to see it not 'work' because the QEMU HDM decoder won't be -> -> programmed -> -> but the null pointer dereference isn't related to that. -> -> -> -> The bug is straight forward. Not all decoders have commit callbacks... -> -> Will send out -> -> a possible fix shortly. -> -> -> -For completeness I'm carrying this hack because I haven't gotten my head -> -around the right fix for check_last_peer() failing on this test topology. -> -> -diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c -> -index c49d9a5f1091..275e143bd748 100644 -> ---- a/drivers/cxl/core/region.c -> -+++ b/drivers/cxl/core/region.c -> -@@ -978,7 +978,7 @@ static int cxl_port_setup_targets(struct cxl_port *port, -> -rc = check_last_peer(cxled, ep, cxl_rr, -> -distance); -> -if (rc) -> -- return rc; -> -+ // return rc; -> -goto out_target_set; -> -} -> -goto add_target; -I'm still carrying this hack and still haven't worked out the right fix. - -Suggestions welcome! If not I'll hopefully get some time on this -towards the end of the week. - -Jonathan - diff --git a/results/classifier/016/arm/57195159 b/results/classifier/016/arm/57195159 deleted file mode 100644 index 563fc4ae..00000000 --- a/results/classifier/016/arm/57195159 +++ /dev/null @@ -1,342 +0,0 @@ -arm: 0.941 -virtual: 0.795 -debug: 0.764 -operating system: 0.321 -hypervisor: 0.160 -user-level: 0.133 -TCG: 0.131 -kernel: 0.088 -files: 0.061 -VMM: 0.035 -performance: 0.029 -PID: 0.026 -KVM: 0.022 -semantic: 0.017 -device: 0.015 -boot: 0.013 -assembly: 0.012 -risc-v: 0.011 -register: 0.008 -socket: 0.006 -architecture: 0.006 -alpha: 0.006 -network: 0.005 -ppc: 0.005 -x86: 0.004 -graphic: 0.003 -vnc: 0.003 -peripherals: 0.002 -permissions: 0.001 -i386: 0.001 -mistranslation: 0.001 - -[BUG Report] Got a use-after-free error while start arm64 VM with lots of pci controllers - -Hi, - -We got a use-after-free report in our Euler Robot Test, it is can be reproduced -quite easily, -It can be reproduced by start VM with lots of pci controller and virtio-scsi -devices. -You can find the full qemu log from attachment. -We have analyzed the log and got the rough process how it happened, but don't -know how to fix it. - -Could anyone help to fix it ? - -The key message shows bellow: -har device redirected to /dev/pts/1 (label charserial0) -==1517174==WARNING: ASan doesn't fully support makecontext/swapcontext -functions and may produce false positives in some cases! -================================================================= -==1517174==ERROR: AddressSanitizer: heap-use-after-free on address -0xfffc31a002a0 at pc 0xaaad73e1f668 bp 0xfffc319fddb0 sp 0xfffc319fddd0 -READ of size 8 at 0xfffc31a002a0 thread T1 - #0 0xaaad73e1f667 in memory_region_unref /home/qemu/memory.c:1771 - #1 0xaaad73e1f667 in flatview_destroy /home/qemu/memory.c:291 - #2 0xaaad74adc85b in call_rcu_thread util/rcu.c:283 - #3 0xaaad74ab31db in qemu_thread_start util/qemu-thread-posix.c:519 - #4 0xfffc3a1678bb (/lib64/libpthread.so.0+0x78bb) - #5 0xfffc3a0a616b (/lib64/libc.so.6+0xd616b) - -0xfffc31a002a0 is located 544 bytes inside of 1440-byte region -[0xfffc31a00080,0xfffc31a00620) -freed by thread T37 (CPU 0/KVM) here: - #0 0xfffc3c102e23 in free (/lib64/libasan.so.4+0xd2e23) - #1 0xfffc3bbc729f in g_free (/lib64/libglib-2.0.so.0+0x5729f) - #2 0xaaad745cce03 in pci_bridge_update_mappings hw/pci/pci_bridge.c:245 - #3 0xaaad745ccf33 in pci_bridge_write_config hw/pci/pci_bridge.c:271 - #4 0xaaad745ba867 in pci_bridge_dev_write_config -hw/pci-bridge/pci_bridge_dev.c:153 - #5 0xaaad745d6013 in pci_host_config_write_common hw/pci/pci_host.c:81 - #6 0xaaad73e2346f in memory_region_write_accessor /home/qemu/memory.c:483 - #7 0xaaad73e1d9ff in access_with_adjusted_size /home/qemu/memory.c:544 - #8 0xaaad73e28d1f in memory_region_dispatch_write /home/qemu/memory.c:1482 - #9 0xaaad73d7274f in flatview_write_continue /home/qemu/exec.c:3167 - #10 0xaaad73d72a53 in flatview_write /home/qemu/exec.c:3207 - #11 0xaaad73d7c8c3 in address_space_write /home/qemu/exec.c:3297 - #12 0xaaad73e5059b in kvm_cpu_exec /home/qemu/accel/kvm/kvm-all.c:2386 - #13 0xaaad73e07ac7 in qemu_kvm_cpu_thread_fn /home/qemu/cpus.c:1246 - #14 0xaaad74ab31db in qemu_thread_start util/qemu-thread-posix.c:519 - #15 0xfffc3a1678bb (/lib64/libpthread.so.0+0x78bb) - #16 0xfffc3a0a616b (/lib64/libc.so.6+0xd616b) - -previously allocated by thread T0 here: - #0 0xfffc3c1031cb in __interceptor_malloc (/lib64/libasan.so.4+0xd31cb) - #1 0xfffc3bbc7163 in g_malloc (/lib64/libglib-2.0.so.0+0x57163) - #2 0xaaad745ccb57 in pci_bridge_region_init hw/pci/pci_bridge.c:188 - #3 0xaaad745cd8cb in pci_bridge_initfn hw/pci/pci_bridge.c:385 - #4 0xaaad745baaf3 in pci_bridge_dev_realize -hw/pci-bridge/pci_bridge_dev.c:64 - #5 0xaaad745cacd7 in pci_qdev_realize hw/pci/pci.c:2095 - #6 0xaaad7439d9f7 in device_set_realized hw/core/qdev.c:865 - #7 0xaaad7485ed23 in property_set_bool qom/object.c:2102 - #8 0xaaad74868f4b in object_property_set_qobject qom/qom-qobject.c:26 - #9 0xaaad74863a43 in object_property_set_bool qom/object.c:1360 - #10 0xaaad742a53b7 in qdev_device_add /home/qemu/qdev-monitor.c:675 - #11 0xaaad742a9c7b in device_init_func /home/qemu/vl.c:2074 - #12 0xaaad74ad4d33 in qemu_opts_foreach util/qemu-option.c:1170 - #13 0xaaad73d60c17 in main /home/qemu/vl.c:4313 - #14 0xfffc39ff0b9f in __libc_start_main (/lib64/libc.so.6+0x20b9f) - #15 0xaaad73d6db33 -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x98db33) - -Thread T1 created by T0 here: - #0 0xfffc3c068f6f in __interceptor_pthread_create -(/lib64/libasan.so.4+0x38f6f) - #1 0xaaad74ab54ab in qemu_thread_create util/qemu-thread-posix.c:556 - #2 0xaaad74adc6a7 in rcu_init_complete util/rcu.c:326 - #3 0xaaad74bab2a7 in __libc_csu_init -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x17cb2a7) - #4 0xfffc39ff0b47 in __libc_start_main (/lib64/libc.so.6+0x20b47) - #5 0xaaad73d6db33 (/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x98db33) - -Thread T37 (CPU 0/KVM) created by T0 here: - #0 0xfffc3c068f6f in __interceptor_pthread_create -(/lib64/libasan.so.4+0x38f6f) - #1 0xaaad74ab54ab in qemu_thread_create util/qemu-thread-posix.c:556 - #2 0xaaad73e09b0f in qemu_dummy_start_vcpu /home/qemu/cpus.c:2045 - #3 0xaaad73e09b0f in qemu_init_vcpu /home/qemu/cpus.c:2077 - #4 0xaaad740d36b7 in arm_cpu_realizefn /home/qemu/target/arm/cpu.c:1712 - #5 0xaaad7439d9f7 in device_set_realized hw/core/qdev.c:865 - #6 0xaaad7485ed23 in property_set_bool qom/object.c:2102 - #7 0xaaad74868f4b in object_property_set_qobject qom/qom-qobject.c:26 - #8 0xaaad74863a43 in object_property_set_bool qom/object.c:1360 - #9 0xaaad73fe3e67 in machvirt_init /home/qemu/hw/arm/virt.c:1682 - #10 0xaaad743acfc7 in machine_run_board_init hw/core/machine.c:1077 - #11 0xaaad73d60b73 in main /home/qemu/vl.c:4292 - #12 0xfffc39ff0b9f in __libc_start_main (/lib64/libc.so.6+0x20b9f) - #13 0xaaad73d6db33 -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x98db33) - -SUMMARY: AddressSanitizer: heap-use-after-free /home/qemu/memory.c:1771 in -memory_region_unref - -Thanks -use-after-free-qemu.log -Description: -Text document - -Cc: address@hidden - -On 1/17/2020 4:18 PM, Pan Nengyuan wrote: -> -Hi, -> -> -We got a use-after-free report in our Euler Robot Test, it is can be -> -reproduced quite easily, -> -It can be reproduced by start VM with lots of pci controller and virtio-scsi -> -devices. -> -You can find the full qemu log from attachment. -> -We have analyzed the log and got the rough process how it happened, but don't -> -know how to fix it. -> -> -Could anyone help to fix it ? -> -> -The key message shows bellow: -> -har device redirected to /dev/pts/1 (label charserial0) -> -==1517174==WARNING: ASan doesn't fully support makecontext/swapcontext -> -functions and may produce false positives in some cases! -> -================================================================= -> -==1517174==ERROR: AddressSanitizer: heap-use-after-free on address -> -0xfffc31a002a0 at pc 0xaaad73e1f668 bp 0xfffc319fddb0 sp 0xfffc319fddd0 -> -READ of size 8 at 0xfffc31a002a0 thread T1 -> -#0 0xaaad73e1f667 in memory_region_unref /home/qemu/memory.c:1771 -> -#1 0xaaad73e1f667 in flatview_destroy /home/qemu/memory.c:291 -> -#2 0xaaad74adc85b in call_rcu_thread util/rcu.c:283 -> -#3 0xaaad74ab31db in qemu_thread_start util/qemu-thread-posix.c:519 -> -#4 0xfffc3a1678bb (/lib64/libpthread.so.0+0x78bb) -> -#5 0xfffc3a0a616b (/lib64/libc.so.6+0xd616b) -> -> -0xfffc31a002a0 is located 544 bytes inside of 1440-byte region -> -[0xfffc31a00080,0xfffc31a00620) -> -freed by thread T37 (CPU 0/KVM) here: -> -#0 0xfffc3c102e23 in free (/lib64/libasan.so.4+0xd2e23) -> -#1 0xfffc3bbc729f in g_free (/lib64/libglib-2.0.so.0+0x5729f) -> -#2 0xaaad745cce03 in pci_bridge_update_mappings hw/pci/pci_bridge.c:245 -> -#3 0xaaad745ccf33 in pci_bridge_write_config hw/pci/pci_bridge.c:271 -> -#4 0xaaad745ba867 in pci_bridge_dev_write_config -> -hw/pci-bridge/pci_bridge_dev.c:153 -> -#5 0xaaad745d6013 in pci_host_config_write_common hw/pci/pci_host.c:81 -> -#6 0xaaad73e2346f in memory_region_write_accessor /home/qemu/memory.c:483 -> -#7 0xaaad73e1d9ff in access_with_adjusted_size /home/qemu/memory.c:544 -> -#8 0xaaad73e28d1f in memory_region_dispatch_write /home/qemu/memory.c:1482 -> -#9 0xaaad73d7274f in flatview_write_continue /home/qemu/exec.c:3167 -> -#10 0xaaad73d72a53 in flatview_write /home/qemu/exec.c:3207 -> -#11 0xaaad73d7c8c3 in address_space_write /home/qemu/exec.c:3297 -> -#12 0xaaad73e5059b in kvm_cpu_exec /home/qemu/accel/kvm/kvm-all.c:2386 -> -#13 0xaaad73e07ac7 in qemu_kvm_cpu_thread_fn /home/qemu/cpus.c:1246 -> -#14 0xaaad74ab31db in qemu_thread_start util/qemu-thread-posix.c:519 -> -#15 0xfffc3a1678bb (/lib64/libpthread.so.0+0x78bb) -> -#16 0xfffc3a0a616b (/lib64/libc.so.6+0xd616b) -> -> -previously allocated by thread T0 here: -> -#0 0xfffc3c1031cb in __interceptor_malloc (/lib64/libasan.so.4+0xd31cb) -> -#1 0xfffc3bbc7163 in g_malloc (/lib64/libglib-2.0.so.0+0x57163) -> -#2 0xaaad745ccb57 in pci_bridge_region_init hw/pci/pci_bridge.c:188 -> -#3 0xaaad745cd8cb in pci_bridge_initfn hw/pci/pci_bridge.c:385 -> -#4 0xaaad745baaf3 in pci_bridge_dev_realize -> -hw/pci-bridge/pci_bridge_dev.c:64 -> -#5 0xaaad745cacd7 in pci_qdev_realize hw/pci/pci.c:2095 -> -#6 0xaaad7439d9f7 in device_set_realized hw/core/qdev.c:865 -> -#7 0xaaad7485ed23 in property_set_bool qom/object.c:2102 -> -#8 0xaaad74868f4b in object_property_set_qobject qom/qom-qobject.c:26 -> -#9 0xaaad74863a43 in object_property_set_bool qom/object.c:1360 -> -#10 0xaaad742a53b7 in qdev_device_add /home/qemu/qdev-monitor.c:675 -> -#11 0xaaad742a9c7b in device_init_func /home/qemu/vl.c:2074 -> -#12 0xaaad74ad4d33 in qemu_opts_foreach util/qemu-option.c:1170 -> -#13 0xaaad73d60c17 in main /home/qemu/vl.c:4313 -> -#14 0xfffc39ff0b9f in __libc_start_main (/lib64/libc.so.6+0x20b9f) -> -#15 0xaaad73d6db33 -> -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x98db33) -> -> -Thread T1 created by T0 here: -> -#0 0xfffc3c068f6f in __interceptor_pthread_create -> -(/lib64/libasan.so.4+0x38f6f) -> -#1 0xaaad74ab54ab in qemu_thread_create util/qemu-thread-posix.c:556 -> -#2 0xaaad74adc6a7 in rcu_init_complete util/rcu.c:326 -> -#3 0xaaad74bab2a7 in __libc_csu_init -> -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x17cb2a7) -> -#4 0xfffc39ff0b47 in __libc_start_main (/lib64/libc.so.6+0x20b47) -> -#5 0xaaad73d6db33 -> -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x98db33) -> -> -Thread T37 (CPU 0/KVM) created by T0 here: -> -#0 0xfffc3c068f6f in __interceptor_pthread_create -> -(/lib64/libasan.so.4+0x38f6f) -> -#1 0xaaad74ab54ab in qemu_thread_create util/qemu-thread-posix.c:556 -> -#2 0xaaad73e09b0f in qemu_dummy_start_vcpu /home/qemu/cpus.c:2045 -> -#3 0xaaad73e09b0f in qemu_init_vcpu /home/qemu/cpus.c:2077 -> -#4 0xaaad740d36b7 in arm_cpu_realizefn /home/qemu/target/arm/cpu.c:1712 -> -#5 0xaaad7439d9f7 in device_set_realized hw/core/qdev.c:865 -> -#6 0xaaad7485ed23 in property_set_bool qom/object.c:2102 -> -#7 0xaaad74868f4b in object_property_set_qobject qom/qom-qobject.c:26 -> -#8 0xaaad74863a43 in object_property_set_bool qom/object.c:1360 -> -#9 0xaaad73fe3e67 in machvirt_init /home/qemu/hw/arm/virt.c:1682 -> -#10 0xaaad743acfc7 in machine_run_board_init hw/core/machine.c:1077 -> -#11 0xaaad73d60b73 in main /home/qemu/vl.c:4292 -> -#12 0xfffc39ff0b9f in __libc_start_main (/lib64/libc.so.6+0x20b9f) -> -#13 0xaaad73d6db33 -> -(/home/qemu/aarch64-softmmu/qemu-system-aarch64+0x98db33) -> -> -SUMMARY: AddressSanitizer: heap-use-after-free /home/qemu/memory.c:1771 in -> -memory_region_unref -> -> -Thanks -> -use-after-free-qemu.log -Description: -Text document - diff --git a/results/classifier/016/assembly-arm/1809546 b/results/classifier/016/assembly-arm/1809546 deleted file mode 100644 index 405f29bd..00000000 --- a/results/classifier/016/assembly-arm/1809546 +++ /dev/null @@ -1,109 +0,0 @@ -arm: 0.987 -assembly: 0.979 -virtual: 0.462 -register: 0.123 -debug: 0.115 -kernel: 0.058 -peripherals: 0.042 -network: 0.042 -operating system: 0.032 -files: 0.031 -TCG: 0.023 -semantic: 0.020 -device: 0.020 -hypervisor: 0.015 -user-level: 0.010 -PID: 0.008 -socket: 0.006 -performance: 0.006 -VMM: 0.006 -ppc: 0.005 -risc-v: 0.005 -vnc: 0.005 -architecture: 0.004 -permissions: 0.003 -boot: 0.003 -alpha: 0.003 -x86: 0.003 -graphic: 0.002 -i386: 0.001 -KVM: 0.001 -mistranslation: 0.001 - -Writing a byte to a pl011 SFR overwrites the whole SFR - -The bug is present in QEMU 2.8.1 and, if my analysis is correct, also on master. - -I first noticed that a PL011 UART driver, which is fine on real hardware, fails to enable the RX interrupt in the IMSC register when running in QEMU. However, the problem only comes up if the code is compiled without optimizations. I think I've narrowed it down to a minimal example that will exhibit the problem if run as a bare-metal application. - -Given: - -pl011_addr: .word 0x10009000 - -The following snippet will be problematic: - - ldr r3, pl011_addr - ldrb r2, [r3, #0x38] // IMSC - mov r2, #0 - orr r2, r2, #0x10 // R2 == 0x10 - strb r2, [r3, #0x38] // Whole word reads correctly after this - ldrb r2, [r3, #0x39] - mov r2, #0 - strb r2, [r3, #0x39] // Problem here! Overwrites offset 0x38 as well - -After the first strb instruction, which writes to 0x10009038, everything is fine. It can be seen in the QEMU monitor: - -(qemu) xp 0x10009038 -0000000010009038: 0x00000010 - -After the second strb instruction, the write to 0x10009039 clears the entire word: - -(qemu) xp 0x10009038 -0000000010009038: 0x00000000 - -QEMU command-line, using the vexpress-a9 which has the PL011 at 0x10009000: - -qemu-system-arm -S -M vexpress-a9 -m 32M -no-reboot -nographic -monitor telnet:127.0.0.1:1234,server,nowait -kernel pl011-sfr.bin -gdb tcp::2159 -serial mon:stdio - -Compiling the original C code with optimizations makes the driver work. It compiles down to assembly that only does a single write: - - ldr r3, pl011_addr - mov r2, #0x10 - str r2, [r3, #0x38] - -Attached is the an assembly file, and linkscript, that shows the problem, and also includes the working code. - -I haven't debugged inside of QEMU itself but it seems to me that the problem is in pl011_write in pl011.c - the functions looks at which offset is being written, and then writes the entire SFR that offset falls under, which means that changing a single byte will change the whole SFR. - - - -Adding the link script. - -Yes, our PL011 implementation assumes that you only ever access the 32-bit registers with full width 32-bit word reads and writes. Don't try to do byte accesses to them. The PL011 data sheet doesn't specifically say that partial-width accesses to registers are permitted, so I think that trying to access offset 0x39 falls under the general note in section 3.1 that attempting to access reserved or unused address locations can result in unpredictable behaviour. - -You need to make sure you write your C code in a manner which enforces that accesses to device registers are done as single 32-bit accesses, and the compiler does not silently break them down into multiple reads and writes, or you will be in for a lot of pain trying to figure out what is going on if the compiler ever does it with registers that are write-to-clear or similar behaviour. Linux, for instance, does this by having readl() and writel() functions that end up doing inline asm of ldr/str instructions. - - -Thanks for the response. - -I don't think section 3.1 applies to 8-bit accesses. That is specifically about reserved locations, and neither offset 0x38 nor 0x39 are reserved, so I think it's a matter of whether 32-bit access is required or not. - -From what I usually see in ARM documentation, 32-bit access is explicitly mentioned when required. For the PL011, it's mentioned for the UARTPeriphID_n registers, for instance. In many other cases access size depends on the implementation and the corresponding memory mapping of that implementation. - -I understand that *in practice* you should ensure single-access writes unless doing otherwise is explicitly allowed. However, in cases like the PL011 it seems ambiguous whether that is actually required, so it seems like the best choice would be to explicitly document it for the QEMU implementation. That would save some guesswork. - -The QEMU project is currently considering to move its bug tracking to -another system. For this we need to know which bugs are still valid -and which could be closed already. Thus we are setting older bugs to -"Incomplete" now. - -If you still think this bug report here is valid, then please switch -the state back to "New" within the next 60 days, otherwise this report -will be marked as "Expired". Or please mark it as "Fix Released" if -the problem has been solved with a newer version of QEMU already. - -Thank you and sorry for the inconvenience. - - -[Expired for QEMU because there has been no activity for 60 days.] - diff --git a/results/classifier/016/assembly-x86/1156313 b/results/classifier/016/assembly-x86/1156313 deleted file mode 100644 index f3c30a2a..00000000 --- a/results/classifier/016/assembly-x86/1156313 +++ /dev/null @@ -1,147 +0,0 @@ -x86: 0.943 -assembly: 0.861 -operating system: 0.178 -debug: 0.028 -files: 0.011 -TCG: 0.007 -kernel: 0.006 -hypervisor: 0.006 -register: 0.005 -user-level: 0.004 -architecture: 0.004 -semantic: 0.003 -PID: 0.002 -device: 0.002 -performance: 0.002 -virtual: 0.002 -VMM: 0.001 -boot: 0.001 -KVM: 0.001 -risc-v: 0.001 -vnc: 0.001 -graphic: 0.001 -permissions: 0.000 -socket: 0.000 -ppc: 0.000 -network: 0.000 -peripherals: 0.000 -mistranslation: 0.000 -alpha: 0.000 -i386: 0.000 -arm: 0.000 - -X86-64 flags handling broken - -The current qemu sources cause improper handling of flags on x86-64. -This bug seems to have shown up a few weeks ago. - -A plain install of Debian GNU/Linux makes user processes catch -spurious signals. The kernel seems to run stably, though. - -The ADX feature works very poorly. It might be related; at least it -allows for reproducibly provoking invalid behaviour. - -Here is a test case: - -================================================================ -qemumain.c -#include -long adx(); -int -main () -{ - printf ("%lx\n", adx (0xffbeef, 17)); - return 0; -} -================================================================ -qemuadx.s: - .globl adx -adx: xor %rax, %rax -1: dec %rdi - jnz 1b - .byte 0xf3, 0x48, 0x0f, 0x38, 0xf6, 0xc0 # adox %rax, %rax - .byte 0x66, 0x48, 0x0f, 0x38, 0xf6, 0xc0 # adcx %rax, %rax - ret -================================================================ - -Compile and execute: -$ gcc -m64 qemumain.c qemuadx.s -$ a.out -ffffff8000378cd8 - -Expected output is simply "0". The garbage value varies between qemu -compiles and guest systems. - -Note that one needs a recent GNU assembler in order to handle adox and -adcx. For convenience I have supplied them as byte sequences. - -Exaplanation and feeble analysis: - -The 0xffbeef argument is a loop count. It is necessary to loop for a -while in order to trigger this bug. If the loop count is decreased, -the bug will seen intermittently; the lower the count, the less -frequent the invalid behaviour. - -It seems like a reasonable assumption that this bug is related to -flags handling at context switch. Presumably, qemu keeps flags state -in some internal format, then recomputes then when needing to form the -eflags register, as needed for example for context switching. - -I haven't tried to reproduce this bug using qemu-x86_64 and SYSROOT, -but I strongly suspect that to be impossible. I use -qemu-system-x86_64 and the guest Debian GNU/Linux x86_64 (version -6.0.6) . - -The bug happens also with the guest FreeBSD x86_64 version 9.1. (The -iteration count for triggering the problem 50% of the runs is not the -same when using the kernel Linux and FreeBSD's kernel, presumably due -to different ticks.) - -The bug happens much more frequently for a loaded system; in fact, the -loop count can be radically decreased if two instances of the trigger -program are run in parallel. - -Richard Henderson