1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
|
peripherals: 0.960
x86: 0.958
debug: 0.954
ppc: 0.953
arm: 0.953
graphic: 0.953
performance: 0.945
i386: 0.943
assembly: 0.939
risc-v: 0.939
register: 0.939
VMM: 0.937
PID: 0.934
user-level: 0.933
architecture: 0.932
permissions: 0.930
virtual: 0.927
TCG: 0.916
kernel: 0.916
KVM: 0.912
semantic: 0.911
vnc: 0.910
files: 0.907
device: 0.906
socket: 0.902
hypervisor: 0.888
mistranslation: 0.887
boot: 0.867
network: 0.859
shmat fails on 32-to-64 setup
I am trying to run a guest mips32 program (user mode) on a x86_64 host. The program fails on a call to shmat() reproducibly. when digging into this problem, I could make a small guest POC that fails when compiled as i386 (-m32) running on a x86_64 host, but pass when compiled as 64bit. The problem has to do with mmap flags.
From what I can understand, when running 32bits guests programs, qemu reserve the whole guest virtual space with an mmap call. That mmap call specifys MAP:PRIVATE flag. When shmat is called, it tries to make part of that region MAP_SHARED and that fails.
As a possible fix, it looks like it is possible to first unmap the shm region before calling shmat.
steps to reproduce:
1 - create a file shm.c with content below
2 - compile with: gcc -m32 shm.c -o shm32
3 - run on a x86_64 host: qemu-i386 ./shm32
4 - observe shmat fails, by returning ptr -1
5- compile without -m32: : gcc shm.c -o shm64
6 - observe it pass: qemu-x84_64 ./shm64
#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/mman.h>
#include <stdio.h>
int main()
{
struct shmid_ds shm_desc;
int err = 0;
int id = shmget(IPC_PRIVATE, 688128, IPC_CREAT|IPC_EXCL|0666);
err = shmctl(id, IPC_STAT, &shm_desc);
const void *at = 0x7f7df38ea000;
void* ptr = shmat(id, at, 0);
printf( "got err %d, ptr %p\n", err, ptr );
}
Which version of QEMU did you use here? Does it still reproduce with the latest version of QEMU?
[Expired for QEMU because there has been no activity for 60 days.]
I can confirm that this bug still exists in the current qemu master (short commit ID 0050f9978e):
~/qemu$ gcc -m32 shm_bug.c -o shm_bug32
shm_bug.c: In function ‘main’:
shm_bug.c:12:24: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
const void *at = 0x7f7df38ea000;
^~~~~~~~~~~~~~
~/qemu$ i386-linux-user/qemu-i386 ./shm_bug32
got err 0, ptr 0xffffffff
ari@ari-thinkpad:~/qemu$ gcc shm_bug.c -o shm_bug64
shm_bug.c: In function ‘main’:
shm_bug.c:12:24: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
const void *at = 0x7f7df38ea000;
^~~~~~~~~~~~~~
~/qemu$ x86_64-linux-user/qemu-x86_64 ./shm_bug64
got err 0, ptr 0x7f7df38ea000
ari@ari-thinkpad:~/qemu$
Additionally, running each executable directly on a 64-bit Ubuntu 18.04 system, we can see that the behavior of the 32-bit binary differs between qemu-i386 and native, while that of the 64-bit binary does not:
~/qemu$ ./shm_bug32
got err 0, ptr 0xf38ea000
~/qemu$ ./shm_bug64
got err 0, ptr 0x7f7df38ea000
~/qemu$
Erm isn't your 64bit pointer being truncated when you build for 32 bit?
Yes, that is correct, but the behavior is still different between "native" execution and execution through qemu-i386. Changing the pointer to be initialized with 0xf38ea000 (essentially truncating it manually) does not change anything:
~/qemu$ gcc -m32 shm_bug.c -o shm_bug32
shm_bug.c: In function ‘main’:
shm_bug.c:13:24: warning: initialization makes pointer from integer without a cast [-Wint-conversion]
const void *at = 0xf38ea000;
^~~~~~~~~~
~/qemu$ ./shm_bug32
got err 0, ptr 0xf38ea000
~/qemu$ i386-linux-user/qemu-i386 ./shm_bug32
got err 0, ptr 0xffffffff
~/qemu$
Here's the end of strace output for running through qemu-i386:
shmget(IPC_PRIVATE, 688128, IPC_CREAT|IPC_EXCL|0666) = 72810572
shmctl(72810572, IPC_STAT, {shm_perm={uid=1000, gid=1000, mode=0666, key=0, cuid=1000, cgid=1000}, shm_segsz=688128, shm_cpid=10438, shm_lpid=0, shm_nattch=0, shm_atime=0, shm_dtime=0, shm_ctime=1562340468}) = 0
shmctl(72810572, IPC_STAT, {shm_perm={uid=1000, gid=1000, mode=0666, key=0, cuid=1000, cgid=1000}, shm_segsz=688128, shm_cpid=10438, shm_lpid=0, shm_nattch=0, shm_atime=0, shm_dtime=0, shm_ctime=1562340468}) = 0
shmat(72810572, 0x7f27138eb000, 0) = -1 EINVAL (Invalid argument)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 12), ...}) = 0
mmap(0x7f271f46d000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f271f46d000
write(1, "got err 0, ptr 0xffffffff\n", 26got err 0, ptr 0xffffffff
) = 26
exit_group(0) = ?
+++ exited with 0 +++
~/qemu$
For comparison, the strace output when running natively:
shmget(IPC_PRIVATE, 688128, IPC_CREAT|IPC_EXCL|0666) = 72843341
shmctl(72843341, IPC_64|IPC_STAT, {shm_perm={uid=1000, gid=1000, mode=0666, key=0, cuid=1000, cgid=1000}, shm_segsz=688128, shm_cpid=10883, shm_lpid=0, shm_nattch=0, shm_atime=0, shm_dtime=0, shm_ctime=1562340846}) = 0
shmat(72843341, 0xf38ea000, 0) = 0xf38ea000
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 12), ...}) = 0
brk(NULL) = 0x58069000
brk(0x5808a000) = 0x5808a000
brk(0x5808b000) = 0x5808b000
write(1, "got err 0, ptr 0xf38ea000\n", 26got err 0, ptr 0xf38ea000
) = 26
exit_group(0) = ?
+++ exited with 0 +++
~/qemu$
I think analysis in comment #1 is correct: If I use "shmat(id, at, SHM_REMAP)" it works.
Mapping an arbitrary address in the memory takes the risk to have this address already in use by the loader or a library.
Where does this address come from?
Why the qemu-mips program doesn't call shmat() with NULL?
It's no less reasonable than doing an mmap() with a fixed address -- if the application knows what it's doing then it's fine. It's just that it bumps into our internal implementation details of (a) doing an mmap to reserve the full 32-bit space we want to allow the guest to do and (b) just passing guest mappings through to the kernel rather than tracking ourselves what memory the guest has allocated (which would allow us to implement the SHM_REMAP vs no-remap ourselves, modulo race conditions between threads).
(b) also prevents us from implementing the memory-related rlimits correctly, incidentally.
Right. I'd also like to point out that I carried out some additional experiments using variations of the program in the issue description, with bizarre results, mips64. I changed the value of the pointer to 0x7f7df38c0000 to increase the alignment a bit and then did the following experiments:
1) shmat(id, 0x7f7df38c0000, 0) fails, returning 0xffffffffffffffff, errno == 22 (EINVAL)
2) shmat(id, 0x7f7df38c0000, SHM_REMAP) fails, returning 0xffffffffffffffff, errno == 22 (EINVAL)
3) shmat(id, 0x7f7df38c0000, SHM_RND) succeeds. Additionally, the return value is exactly the pointer value passed to the host shmat(), that is, 0x7f7df38c0000.
The following thing bothers me:
With flags set to 0, the address 0x7f7df38c0000 is rejected. This could have been an alignment problem, but the call actually succeeds when the flags are set to SHM_RND (to align the returned address properly), with the pointer value remaining the same.
This looks bizarre to me, no matter the way you look at it.
This is an automated cleanup. This bug report has been moved to QEMU's
new bug tracker on gitlab.com and thus gets marked as 'expired' now.
Please continue with the discussion here:
https://gitlab.com/qemu-project/qemu/-/issues/115
|