1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
|
graphic: 0.866
user-level: 0.858
permissions: 0.842
peripherals: 0.842
mistranslation: 0.842
risc-v: 0.812
debug: 0.809
arm: 0.802
hypervisor: 0.797
virtual: 0.796
semantic: 0.794
KVM: 0.793
TCG: 0.791
register: 0.790
x86: 0.789
vnc: 0.787
architecture: 0.783
socket: 0.780
performance: 0.765
ppc: 0.762
network: 0.762
VMM: 0.754
device: 0.752
boot: 0.750
assembly: 0.721
PID: 0.712
kernel: 0.708
i386: 0.699
files: 0.694
Unable to pass-through PCIe devices from a ppc64le host to an x86_64 guest
Attempting to pass through a PCIe device from a ppc64le host to an x86_64 guest with QEMU v5.2.0-3031-g571d413b5d (built from git master) fails with the following error:
include/exec/memory.h:43:IOMMU_MEMORY_REGION: Object 0x10438eb00 is not an instance of type qemu:iommu-memory-region
To reproduce this issue, simply run the following command on a POWER9 system:
qemu-system-x86_64 -machine q35 -device vfio-pci,host=$DBSF
Where $DBSF is a domain:bus:slot.function PCIe device address.
This also fails with QEMU 3.1.0 (from Debian Buster), so I assume this has never worked. Helpfully, the error message it prints seems to indicate where the problem is:
hw/vfio/spapr.c:147:vfio_spapr_create_window: Object 0x164473510 is not an instance of type qemu:iommu-memory-region
My kernel (Linux v5.8.0 plus some small unrelated patches) is built with the page size set to 4k, so this issue shouldn't be due to a page size mismatch. And as I stated earlier, my host arch is ppc64le, so it shouldn't be an endianness issue, either.
I assume this should be possible (in theory) since I've seen reports of others getting PCIe passthrough working with aarch64 guests on x86_64 hosts, but of course that (passthrough to weird guest arch on x86) is somewhat the opposite of what I'm trying to do (passthrough to x86 on weird host arch) so I don't know for sure. If it is possible, I'm willing to develop a fix myself, but I'm almost completely unfamiliar with QEMU's internals so if anyone has any advice on where to start I'd greatly appreciate it.
I've done some more investigating, and have produced a backtrace of the error:
#0 0x00003ffff6b63228 in __libc_signal_restore_set (set=0x3fffffffcec8) at ../sysdeps/unix/sysv/linux/internal-signals.h:84
#1 0x00003ffff6b63228 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:48
#2 0x00003ffff6b4358c in __GI_abort () at abort.c:79
#3 0x000000010080d524 in object_dynamic_cast_assert
(obj=0x1016db860, typename=0x100bf9980 "qemu:iommu-memory-region", file=0x100bf9940 "/usr/src/qemu/include/exec/memory.h", line=<optimized out>, func=0x100c08a70 <__func__.21845> "IOMMU_MEMORY_REGION") at ../qom/object.c:883
#4 0x00000001006b6f84 in IOMMU_MEMORY_REGION (obj=<optimized out>) at /usr/src/qemu/include/exec/memory.h:42
#5 0x00000001006b6f84 in vfio_spapr_create_window (container=0x102357380, section=0x3fffffffd410, pgsize=0x3fffffffd368)
at ../hw/vfio/spapr.c:149
#6 0x00000001007a09a0 in vfio_listener_region_add (listener=0x102357390, section=0x3fffffffd410) at ../hw/vfio/common.c:709
#7 0x00000001006ea6f4 in listener_add_address_space (as=<optimized out>, listener=0x102357390) at ../softmmu/memory.c:2729
#8 0x00000001006ea6f4 in memory_listener_register (listener=0x102357390, as=<optimized out>) at ../softmmu/memory.c:2796
#9 0x00000001007a36f4 in vfio_connect_container (errp=0x3fffffffe818, as=<optimized out>, group=0x102357300)
at ../hw/vfio/common.c:1886
#10 0x00000001007a36f4 in vfio_get_group (groupid=<optimized out>, as=<optimized out>, errp=0x3fffffffe818)
at ../hw/vfio/common.c:2003
#11 0x000000010071a2a8 in vfio_realize (pdev=0x102350f80, errp=0x3fffffffe818) at ../hw/vfio/pci.c:2834
#12 0x0000000100488e20 in pci_qdev_realize (qdev=0x102350f80, errp=0x3fffffffe940) at ../hw/pci/pci.c:2113
#13 0x00000001008063e0 in device_set_realized (obj=0x102350f80, value=<optimized out>, errp=0x3fffffffea50) at ../hw/core/qdev.c:761
#14 0x000000010080afbc in property_set_bool
(obj=0x102350f80, v=<optimized out>, name=<optimized out>, opaque=0x1014f1930, errp=0x3fffffffea50) at ../qom/object.c:2257
#15 0x000000010080ee2c in object_property_set (obj=0x102350f80, name=0x100c023a0 "realized", v=
0x102351d30, errp=0x101450b30 <error_fatal>) at ../qom/object.c:1402
#16 0x000000010080a55c in object_property_set_qobject
(obj=0x102350f80, name=0x100c023a0 "realized", value=<optimized out>, errp=0x101450b30 <error_fatal>) at ../qom/qom-qobject.c:28
#17 0x000000010080f1b0 in object_property_set_bool
(obj=0x102350f80, name=0x100c023a0 "realized", value=<optimized out>, errp=0x101450b30 <error_fatal>) at ../qom/object.c:1472
#18 0x00000001008042bc in qdev_realize (dev=0x102350f80, bus=<optimized out>, errp=0x101450b30 <error_fatal>)
at ../hw/core/qdev.c:389
#19 0x000000010036cfac in qdev_device_add (opts=0x1014e9960, errp=0x101450b30 <error_fatal>)
at /usr/src/qemu/include/hw/qdev-core.h:17
#20 0x00000001006d5e68 in device_init_func (opaque=<optimized out>, opts=<optimized out>, errp=<optimized out>)
at ../softmmu/vl.c:1202
#21 0x0000000100abe070 in qemu_opts_foreach
(list=<optimized out>, func=0x1006d5e40 <device_init_func>, opaque=0x0, errp=0x101450b30 <error_fatal>)
at ../util/qemu-option.c:1167
#22 0x00000001006da110 in qemu_create_cli_devices () at ../softmmu/vl.c:2494
#23 0x00000001006da110 in qmp_x_exit_preconfig (errp=<optimized out>) at ../softmmu/vl.c:2542
#24 0x00000001006df87c in qemu_init (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ../softmmu/vl.c:3553
#25 0x000000010031d3c8 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ../softmmu/main.c:49
I also took a look at some of the arguments in frame #5 (where the type check is failing):
(gdb) f
#5 vfio_spapr_create_window (container=0x102357380, section=0x3fffffffd410, pgsize=0x3fffffffd368) at ../hw/vfio/spapr.c:149
149 IOMMUMemoryRegion *iommu_mr = IOMMU_MEMORY_REGION(section->mr);
(gdb) l
144 int vfio_spapr_create_window(VFIOContainer *container,
145 MemoryRegionSection *section,
146 hwaddr *pgsize)
147 {
148 int ret = 0;
149 IOMMUMemoryRegion *iommu_mr = IOMMU_MEMORY_REGION(section->mr);
150 uint64_t pagesize = memory_region_iommu_get_min_page_size(iommu_mr), pgmask;
151 unsigned entries, bits_total, bits_per_level, max_levels;
152 struct vfio_iommu_spapr_tce_create create = { .argsz = sizeof(create) };
153 long rampagesize = qemu_minrampagesize();
(gdb) p *section->mr
$5 = {parent_obj = {class = 0x101512f90, free = 0x0, Python Exception <class 'gdb.error'> There is no member named keys.:
properties = 0x101716b60, ref = 1, parent = 0x1016db800}, romd_mode = true,
ram = true, subpage = false, readonly = false, nonvolatile = false, rom_device = false, flush_coalesced_mmio = false,
dirty_log_mask = 0 '\000', is_iommu = false, ram_block = 0x101725c30, owner = 0x1016db800, ops = 0x101300b00 <unassigned_mem_ops>,
opaque = 0x0, container = 0x0, size = 134217728, addr = 0, destructor = 0x1006e0d50 <memory_region_destructor_ram>,
align = 2097152, terminates = true, ram_device = false, enabled = true, warning_printed = false, vga_logging_count = 0 '\000',
alias = 0x0, alias_offset = 0, priority = 0, subregions = {tqh_first = 0x0, tqh_circ = {tql_next = 0x0, tql_prev = 0x1016db908}},
subregions_link = {tqe_next = 0x0, tqe_circ = {tql_next = 0x0, tql_prev = 0x0}}, coalesced = {tqh_first = 0x0, tqh_circ = {
tql_next = 0x0, tql_prev = 0x1016db928}}, name = 0x101725b10 "pc.ram", ioeventfd_nb = 0, ioeventfds = 0x0}
So it seems this is failing because the memory region "pc.ram" is not an IOMMU ("is_iommu = false"). I'm not really sure what that means, and I still don't know how to fix this, but hopefully this information will help.
If there's any more information I should provide, please let me know.
I think cross-arch VFIO has already been discussed in:
https://bugs.launchpad.net/qemu/+bug/1869006
Perhaps you will have some answers.
> https://bugs.launchpad.net/qemu/+bug/1869006
Unfortunately, that's not the same issue I'm having, and the error I see happens regardless of how much or how little RAM I allocate to the VM.
The QEMU project is currently moving its bug tracking to another system.
For this we need to know which bugs are still valid and which could be
closed already. Thus we are setting the bug state to "Incomplete" now.
If the bug has already been fixed in the latest upstream version of QEMU,
then please close this ticket as "Fix released".
If it is not fixed yet and you think that this bug report here is still
valid, then you have two options:
1) If you already have an account on gitlab.com, please open a new ticket
for this problem in our new tracker here:
https://gitlab.com/qemu-project/qemu/-/issues
and then close this ticket here on Launchpad (or let it expire auto-
matically after 60 days). Please mention the URL of this bug ticket on
Launchpad in the new ticket on GitLab.
2) If you don't have an account on gitlab.com and don't intend to get
one, but still would like to keep this ticket opened, then please switch
the state back to "New" or "Confirmed" within the next 60 days (other-
wise it will get closed as "Expired"). We will then eventually migrate
the ticket automatically to the new system (but you won't be the reporter
of the bug in the new system and thus you won't get notified on changes
anymore).
Thank you and sorry for the inconvenience.
I've moved this bug over to GitLab here: https://gitlab.com/qemu-project/qemu/-/issues/391
Thanks for moving it over! ... so I'm closing this on Launchpad now.
|