summary refs log tree commit diff stats
path: root/accel/tcg/cputlb.c (follow)
Commit message (Collapse)AuthorAgeFilesLines
...
* accel/tcg: Add tlb_fill_flags to CPUTLBEntryFullRichard Henderson2024-03-051-4/+1
| | | | | | | | | | | | Allow the target to set tlb flags to apply to all of the comparators. Remove MemTxAttrs.byte_swap, as the bit is not relevant to memory transactions, only the page mapping. Adjust target/sparc to set TLB_BSWAP directly. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-id: 20240301204110.656742-4-richard.henderson@linaro.org Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
* tcg: Avoid double lock if page tables happen to be in mmio memory.Jonathan Cameron2024-02-291-24/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On i386, after fixing the page walking code to work with pages in MMIO memory (specifically CXL emulated interleaved memory), a crash was seen in an interrupt handling path. Useful part of backtrace 7 0x0000555555ab1929 in bql_lock_impl (file=0x555556049122 "../../accel/tcg/cputlb.c", line=2033) at ../../system/cpus.c:524 8 bql_lock_impl (file=file@entry=0x555556049122 "../../accel/tcg/cputlb.c", line=line@entry=2033) at ../../system/cpus.c:520 9 0x0000555555c9f7d6 in do_ld_mmio_beN (cpu=0x5555578e0cb0, full=0x7ffe88012950, ret_be=ret_be@entry=0, addr=19595792376, size=size@entry=8, mmu_idx=4, type=MMU_DATA_LOAD, ra=0) at ../../accel/tcg/cputlb.c:2033 10 0x0000555555ca0fbd in do_ld_8 (cpu=cpu@entry=0x5555578e0cb0, p=p@entry=0x7ffff4efd1d0, mmu_idx=<optimized out>, type=type@entry=MMU_DATA_LOAD, memop=<optimized out>, ra=ra@entry=0) at ../../accel/tcg/cputlb.c:2356 11 0x0000555555ca341f in do_ld8_mmu (cpu=cpu@entry=0x5555578e0cb0, addr=addr@entry=19595792376, oi=oi@entry=52, ra=0, ra@entry=52, access_type=access_type@entry=MMU_DATA_LOAD) at ../../accel/tcg/cputlb.c:2439 12 0x0000555555ca5f59 in cpu_ldq_mmu (ra=52, oi=52, addr=19595792376, env=0x5555578e3470) at ../../accel/tcg/ldst_common.c.inc:169 13 cpu_ldq_le_mmuidx_ra (env=0x5555578e3470, addr=19595792376, mmu_idx=<optimized out>, ra=ra@entry=0) at ../../accel/tcg/ldst_common.c.inc:301 14 0x0000555555b4b5fc in ptw_ldq (ra=0, in=0x7ffff4efd320) at ../../target/i386/tcg/sysemu/excp_helper.c:98 15 ptw_ldq (ra=0, in=0x7ffff4efd320) at ../../target/i386/tcg/sysemu/excp_helper.c:93 16 mmu_translate (env=env@entry=0x5555578e3470, in=0x7ffff4efd3e0, out=0x7ffff4efd3b0, err=err@entry=0x7ffff4efd3c0, ra=ra@entry=0) at ../../target/i386/tcg/sysemu/excp_helper.c:174 17 0x0000555555b4c4b3 in get_physical_address (ra=0, err=0x7ffff4efd3c0, out=0x7ffff4efd3b0, mmu_idx=0, access_type=MMU_DATA_LOAD, addr=18446741874686299840, env=0x5555578e3470) at ../../target/i386/tcg/sysemu/excp_helper.c:580 18 x86_cpu_tlb_fill (cs=0x5555578e0cb0, addr=18446741874686299840, size=<optimized out>, access_type=MMU_DATA_LOAD, mmu_idx=0, probe=<optimized out>, retaddr=0) at ../../target/i386/tcg/sysemu/excp_helper.c:606 19 0x0000555555ca0ee9 in tlb_fill (retaddr=0, mmu_idx=0, access_type=MMU_DATA_LOAD, size=<optimized out>, addr=18446741874686299840, cpu=0x7ffff4efd540) at ../../accel/tcg/cputlb.c:1315 20 mmu_lookup1 (cpu=cpu@entry=0x5555578e0cb0, data=data@entry=0x7ffff4efd540, mmu_idx=0, access_type=access_type@entry=MMU_DATA_LOAD, ra=ra@entry=0) at ../../accel/tcg/cputlb.c:1713 21 0x0000555555ca2c61 in mmu_lookup (cpu=cpu@entry=0x5555578e0cb0, addr=addr@entry=18446741874686299840, oi=oi@entry=32, ra=ra@entry=0, type=type@entry=MMU_DATA_LOAD, l=l@entry=0x7ffff4efd540) at ../../accel/tcg/cputlb.c:1803 22 0x0000555555ca3165 in do_ld4_mmu (cpu=cpu@entry=0x5555578e0cb0, addr=addr@entry=18446741874686299840, oi=oi@entry=32, ra=ra@entry=0, access_type=access_type@entry=MMU_DATA_LOAD) at ../../accel/tcg/cputlb.c:2416 23 0x0000555555ca5ef9 in cpu_ldl_mmu (ra=0, oi=32, addr=18446741874686299840, env=0x5555578e3470) at ../../accel/tcg/ldst_common.c.inc:158 24 cpu_ldl_le_mmuidx_ra (env=env@entry=0x5555578e3470, addr=addr@entry=18446741874686299840, mmu_idx=<optimized out>, ra=ra@entry=0) at ../../accel/tcg/ldst_common.c.inc:294 25 0x0000555555bb6cdd in do_interrupt64 (is_hw=1, next_eip=18446744072399775809, error_code=0, is_int=0, intno=236, env=0x5555578e3470) at ../../target/i386/tcg/seg_helper.c:889 26 do_interrupt_all (cpu=cpu@entry=0x5555578e0cb0, intno=236, is_int=is_int@entry=0, error_code=error_code@entry=0, next_eip=next_eip@entry=0, is_hw=is_hw@entry=1) at ../../target/i386/tcg/seg_helper.c:1130 27 0x0000555555bb87da in do_interrupt_x86_hardirq (env=env@entry=0x5555578e3470, intno=<optimized out>, is_hw=is_hw@entry=1) at ../../target/i386/tcg/seg_helper.c:1162 28 0x0000555555b5039c in x86_cpu_exec_interrupt (cs=0x5555578e0cb0, interrupt_request=<optimized out>) at ../../target/i386/tcg/sysemu/seg_helper.c:197 29 0x0000555555c94480 in cpu_handle_interrupt (last_tb=<synthetic pointer>, cpu=0x5555578e0cb0) at ../../accel/tcg/cpu-exec.c:844 Peter identified this as being due to the BQL already being held when the page table walker encounters MMIO memory and attempts to take the lock again. There are other examples of similar paths TCG, so this follows the approach taken in those of simply checking if the lock is already held and if it is, don't take it again. Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Suggested-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com> Message-Id: <20240219173153.12114-4-Jonathan.Cameron@huawei.com> [rth: Use BQL_LOCK_GUARD] Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* include/exec: Change cpu_mmu_index argument to CPUStateRichard Henderson2024-02-031-9/+13
| | | | | Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* Replace "iothread lock" with "BQL" in commentsStefan Hajnoczi2024-01-081-2/+2
| | | | | | | | | | | | | | | The term "iothread lock" is obsolete. The APIs use Big QEMU Lock (BQL) in their names. Update the code comments to use "BQL" instead of "iothread lock". Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Paul Durrant <paul@xen.org> Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com> Reviewed-by: Cédric Le Goater <clg@kaod.org> Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com> Message-id: 20240102153529.486531-5-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* system/cpus: rename qemu_mutex_lock_iothread() to bql_lock()Stefan Hajnoczi2024-01-081-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Big QEMU Lock (BQL) has many names and they are confusing. The actual QemuMutex variable is called qemu_global_mutex but it's commonly referred to as the BQL in discussions and some code comments. The locking APIs, however, are called qemu_mutex_lock_iothread() and qemu_mutex_unlock_iothread(). The "iothread" name is historic and comes from when the main thread was split into into KVM vcpu threads and the "iothread" (now called the main loop thread). I have contributed to the confusion myself by introducing a separate --object iothread, a separate concept unrelated to the BQL. The "iothread" name is no longer appropriate for the BQL. Rename the locking APIs to: - void bql_lock(void) - void bql_unlock(void) - bool bql_locked(void) There are more APIs with "iothread" in their names. Subsequent patches will rename them. There are also comments and documentation that will be updated in later patches. Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Paul Durrant <paul@xen.org> Acked-by: Fabiano Rosas <farosas@suse.de> Acked-by: David Woodhouse <dwmw@amazon.co.uk> Reviewed-by: Cédric Le Goater <clg@kaod.org> Acked-by: Peter Xu <peterx@redhat.com> Acked-by: Eric Farman <farman@linux.ibm.com> Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com> Acked-by: Hyman Huang <yong.huang@smartx.com> Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com> Message-id: 20240102153529.486531-2-stefanha@redhat.com Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* accel/tcg: Forward probe size on to notdirty_writeJessica Clarke2023-11-141-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Without this, we just dirty a single byte, and so if the caller writes more than one byte to the host memory then we won't have invalidated any translation blocks that start after the first byte and overlap those writes. In particular, AArch64's DC ZVA implementation uses probe_access (via probe_write), and so we don't invalidate the entire block, only the TB overlapping the first byte (and, in the unusual case an unaligned VA is given to the instruction, we also probe that specific address in order to get the right VA reported on an exception, so will invalidate a TB overlapping that address too). Since our IC IVAU implementation is a no-op for system emulation that relies on the softmmu already having detected self-modifying code via this mechanism, this means we have observably wrong behaviour when jumping to code that has been DC ZVA'ed. In practice this is an unusual thing for software to do, as in reality the OS will DC ZVA the page and the application will go and write actual instructions to it that aren't UDF #0, but you can write a test that clearly shows the faulty behaviour. For functions other than probe_access it's not clear what size to use when 0 is passed in. Arguably a size of 0 shouldn't dirty at all, since if you want to actually write then you should pass in a real size, but I have conservatively kept the implementation as dirtying the first byte in that case so as to avoid breaking any assumptions about that behaviour. Signed-off-by: Jessica Clarke <jrtc27@jrtc27.com> Message-Id: <20231104031232.3246614-1-jrtc27@jrtc27.com> [rth: Move the dirtysize computation next to notdirty_write.] Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Declare tcg_flush_jmp_cache() in 'exec/tb-flush.h'Philippe Mathieu-Daudé2023-11-071-0/+1
| | | | | | | | | | | | | | "exec/cpu-common.h" is meant to contain the declarations related to CPU usable with any accelerator / target combination. tcg_flush_jmp_cache() is specific to TCG, so restrict its declaration by moving it to "exec/tb-flush.h". Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230918104153.24433-2-philmd@linaro.org>
* accel/tcg: Fix condition for store_atom_insert_al16Richard Henderson2023-11-061-1/+1
| | | | | | | | Store bytes under a mask is fundamentally a cmpxchg, not a straight store. Use HAVE_CMPXCHG128 instead of HAVE_ATOMIC128_RW. Signed-off-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230916220151.526140-8-richard.henderson@linaro.org>
* accel/tcg: Move HMP info jit and info opcount codeRichard Henderson2023-11-061-15/+0
| | | | | | | | | | Move all of it into accel/tcg/monitor.c. This puts everything about tcg that is only used by the monitor in the same place. Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Make cpu-exec-common.c a target agnostic unitPhilippe Mathieu-Daudé2023-10-041-0/+1
| | | | | | | | | | | | | | | cpu_in_serial_context() is not target specific, move it declaration to "internal-common.h" (which we include in the 4 source files modified). Remove the unused "exec/exec-all.h" header from cpu-exec-common.c. There is no more target specific code in this file: make it target agnostic. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230914185718.76241-12-philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Rename target-specific 'internal.h' -> 'internal-target.h'Philippe Mathieu-Daudé2023-10-041-1/+1
| | | | | | | | | | | | accel/tcg/internal.h contains target specific declarations. Unit files including it become "target tainted": they can not be compiled as target agnostic. Rename using the '-target' suffix to make this explicit. Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230914185718.76241-9-philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: move ld/st helpers to ldst_common.c.incAnton Johansson2023-10-041-214/+0
| | | | | | | | | | | | | | | | A large chunk of ld/st functions are moved from cputlb.c and user-exec.c to ldst_common.c.inc as their implementation is the same between both modes. Eventually, ldst_common.c.inc could be compiled into a separate target-specific compilation unit, and be linked in with the targets. Keeping CPUArchState usage out of cputlb.c (CPUArchState is primarily used to access the mmu index in these functions). Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-12-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Unify user and softmmu do_[st|ld]*_mmu()Anton Johansson2023-10-041-5/+11
| | | | | | | | | | | | The prototype of do_[st|ld]*_mmu() is unified between system- and user-mode allowing a large chunk of helper_[st|ld]*() and cpu_[st|ld]*() functions to be expressed in same manner between both modes. These functions will be moved to ldst_common.c.inc in a following commit. Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-11-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Use CPUState in atomicity helpersAnton Johansson2023-10-041-10/+10
| | | | | | | | | | | Makes ldst_atomicity.c.inc almost target-independent, with the exception of TARGET_PAGE_MASK, which will be addressed in a future patch. Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-8-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Modify atomic_mmu_lookup() to use CPUStateAnton Johansson2023-10-041-13/+13
| | | | | | | | | | | | | The goal is to (in the future) allow for per-target compilation of functions in atomic_template.h whilst atomic_mmu_lookup() and cputlb.c are compiled once-per user- or system mode. Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-7-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> [rth: Use cpu->neg.tlb instead of cpu_tlb()] Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Modify memory access functions to use CPUStateAnton Johansson2023-10-041-177/+171
| | | | | | | | | | | | | do_[ld|st]*() and mmu_lookup*() are changed to use CPUState over CPUArchState, moving the target-dependence to the target-facing facing cpu_[ld|st] functions. Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-6-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> [rth: Use cpu->neg.tlb instead of cpu_tlb; cpu_env instead of env_ptr.] Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Modify probe_access_internal() to use CPUStateAnton Johansson2023-10-041-23/+23
| | | | | | | | | | | | | | probe_access_internal() is changed to instead take the generic CPUState over CPUArchState, in order to lessen the target-specific coupling of cputlb.c. Note: probe_access*() also don't need the full CPUArchState, but aren't touched in this patch as they are target-facing. Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-5-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> [rth: Use cpu->neg.tlb instead of cpu_tlb()] Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Modify tlb_*() to use CPUStateAnton Johansson2023-10-041-116/+104
| | | | | | | | | | | | | Changes tlb_*() functions to take CPUState instead of CPUArchState, as they don't require the full CPUArchState. This makes it easier to decouple target-(in)dependent code. Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230912153428.17816-4-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> [rth: Use cpu->neg.tlb instead of cpu_tlb()] Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Replace CPUState.env_ptr with cpu_env()Richard Henderson2023-10-041-10/+10
| | | | | Reviewed-by: Anton Johansson <anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Move can_do_io to CPUNegativeOffsetStateRichard Henderson2023-10-031-1/+1
| | | | | | | | | Minimize the displacement to can_do_io, since it may be touched at the start of each TranslationBlock. It fits into other padding within the substructure. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Introduce do_st16_mmio_leNRichard Henderson2023-09-161-30/+58
| | | | | | | | Split out int_st_mmio_leN, to be used by both do_st_mmio_leN and do_st16_mmio_leN. Move the locks down into the two functions, since each one now covers all accesses to once page. Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Introduce do_ld16_mmio_beNRichard Henderson2023-09-161-32/+59
| | | | | | | | Split out int_ld_mmio_beN, to be used by both do_ld_mmio_beN and do_ld16_mmio_beN. Move the locks down into the two functions, since each one now covers all accesses to once page. Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Merge io_writex into do_st_mmio_leNRichard Henderson2023-09-161-51/+29
| | | | | | | Avoid multiple calls to io_prepare for unaligned acceses. One call to do_st_mmio_leN will never cross pages. Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Merge io_readx into do_ld_mmio_beNRichard Henderson2023-09-161-54/+30
| | | | | | | Avoid multiple calls to io_prepare for unaligned acceses. One call to do_ld_mmio_beN will never cross pages. Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Replace direct use of io_readx/io_writex in do_{ld,st}_1Richard Henderson2023-09-161-2/+4
| | | | Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Merge cpu_transaction_failed into io_failedRichard Henderson2023-09-161-20/+13
| | | | | | | | Push computation down into the if statements to the point the data is used. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* plugin: Simplify struct qemu_plugin_hwaddrRichard Henderson2023-09-161-7/+9
| | | | | | | | | Rather than saving MemoryRegionSection and offset, save phys_addr and MemoryRegion. This matches up much closer with the plugin api. Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Use CPUTLBEntryFull.phys_addr in io_failedRichard Henderson2023-09-161-8/+4
| | | | | | | | | Since the introduction of CPUTLBEntryFull, we can recover the full cpu address space physical address without having to examine the MemoryRegionSection. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Split out io_prepare and io_failedRichard Henderson2023-09-161-31/+44
| | | | | | | These are common code from io_readx and io_writex. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Simplify tlb_plugin_lookupRichard Henderson2023-09-161-56/+23
| | | | | | | | | | | | | | | Now that we defer address space update and tlb_flush until the next async_run_on_cpu, the plugin run at the end of the instruction no longer has to contend with a flushed tlb. Therefore, delete SavedIOTLB entirely. Properly return false from tlb_plugin_lookup when we do not have a tlb match. Fixes a bug in which SavedIOTLB had stale data, because there were multiple i/o accesses within a single insn. Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Fix the comment for CPUTLBEntryFullLIU Zhiwei2023-09-151-4/+7
| | | | | | | | | | | | When memory region is ram, the lower TARGET_PAGE_BITS is not the physical section number. Instead, its value is always 0. Add comment and assert to make it clear. Signed-off-by: LIU Zhiwei <zhiwei_liu@linux.alibaba.com> Message-Id: <20230901060118.379-1-zhiwei_liu@linux.alibaba.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Update run_on_cpu_data static assertAnton Johansson2023-08-241-2/+3
| | | | | | | | | | As we are now using vaddr for representing guest addresses, update the static assert to check that vaddr fits in the run_on_cpu_data union. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230807155706.9580-10-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Widen address arg in tlb_compare_set()Anton Johansson2023-08-241-1/+1
| | | | | | | Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230807155706.9580-9-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* include/exec: Replace target_ulong with abi_ptr in cpu_[st|ld]*()Anton Johansson2023-08-241-5/+5
| | | | | | | | | | | | | | | Changes the address type of the guest memory read/write functions from target_ulong to abi_ptr. (abi_ptr is currently typedef'd to target_ulong but that will change in a following commit.) This will reduce the coupling between accel/ and target/. Note: Function pointers that point to cpu_[st|ld]*() in target/riscv and target/rx are also updated in this commit. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230807155706.9580-6-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Call save_iotlb_data from io_readx as wellMikhail Tyutin2023-08-061-15/+21
| | | | | | | | | | | | Apply save_iotlb_data() to io_readx() as well as to io_writex(). This fixes SEGFAULT on qemu_plugin_hwaddr_phys_addr() call plugins for addresses inside of MMIO region. Signed-off-by: Dmitriy Solovev <d.solovev@yadro.com> Signed-off-by: Mikhail Tyutin <m.tyutin@yadro.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230804110903.19968-1-m.tyutin@yadro.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Do not issue misaligned i/oRichard Henderson2023-08-051-46/+72
| | | | | | | | | | In the single-page case we were issuing misaligned i/o to the memory subsystem, which does not handle it properly. Split such accesses via do_{ld,st}_mmio_*. Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1800 Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Issue wider aligned i/o in do_{ld,st}_mmio_*Richard Henderson2023-08-051-7/+69
| | | | | | | | | If the address and size are aligned, send larger chunks to the memory subsystem. This will be required to make more use of these helpers. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Adjust parameters and locking with do_{ld,st}_mmio_*Richard Henderson2023-08-051-33/+34
| | | | | | | | | | | | Replace MMULookupPageData* with CPUTLBEntryFull, addr, size. Move QEMU_IOTHREAD_LOCK_GUARD to the caller. This simplifies the usage from do_ld16_beN and do_st16_leN, where we weren't locking the entire operation, and required hoop jumping for passing addr and size. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Zero-pad vaddr in tlb_debug outputAnton Johansson2023-07-241-10/+10
| | | | | | | | | | | | In replacing target_ulong with vaddr and TARGET_FMT_lx with VADDR_PRIx, the zero-padding of TARGET_FMT_lx got lost. Readd 16-wide zero-padding for logging consistency. Suggested-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Anton Johansson <anjo@rev.ng> Message-Id: <20230713120746.26897-1-anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* tcg: Use HAVE_CMPXCHG128 instead of CONFIG_CMPXCHG128Richard Henderson2023-07-151-1/+1
| | | | | | | | | | | | | | | | | | We adjust CONFIG_ATOMIC128 and CONFIG_CMPXCHG128 with CONFIG_ATOMIC128_OPT in atomic128.h. It is difficult to tell when those changes have been applied with the ifdef we must use with CONFIG_CMPXCHG128. So instead use HAVE_CMPXCHG128, which triggers -Werror-undef when the proper header has not been included. Improves tcg_gen_atomic_cmpxchg_i128 for s390x host, which requires CONFIG_ATOMIC128_OPT. Without this we fall back to EXCP_ATOMIC to single-step 128-bit atomics, which is slow enough to cause some tests to time out. Reported-by: Thomas Huth <thuth@redhat.com> Tested-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* plugins: force slow path when plugins instrument memory opsAlex Bennée2023-07-031-7/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The lack of SVE memory instrumentation has been an omission in plugin handling since it was introduced. Fortunately we can utilise the probe_* functions to force all all memory access to follow the slow path. We do this by checking the access type and presence of plugin memory callbacks and if set return the TLB_MMIO flag. We have to jump through a few hoops in user mode to re-use the flag but it was the desired effect: ./qemu-system-aarch64 -display none -serial mon:stdio \ -M virt -cpu max -semihosting-config enable=on \ -kernel ./tests/tcg/aarch64-softmmu/memory-sve \ -plugin ./contrib/plugins/libexeclog.so,ifilter=st1w,afilter=0x40001808 -d plugin gives (disas doesn't currently understand st1w): 0, 0x40001808, 0xe54342a0, ".byte 0xa0, 0x42, 0x43, 0xe5", store, 0x40213010, RAM, store, 0x40213014, RAM, store, 0x40213018, RAM And for user-mode: ./qemu-aarch64 \ -plugin contrib/plugins/libexeclog.so,afilter=0x4007c0 \ -d plugin \ ./tests/tcg/aarch64-linux-user/sha512-sve gives: 1..10 ok 1 - do_test(&tests[i]) 0, 0x4007c0, 0xa4004b80, ".byte 0x80, 0x4b, 0x00, 0xa4", load, 0x5500800370, load, 0x5500800371, load, 0x5500800372, load, 0x5500800373, load, 0x5500800374, load, 0x5500800375, load, 0x5500800376, load, 0x5500800377, load, 0x5500800378, load, 0x5500800379, load, 0x550080037a, load, 0x550080037b, load, 0x550080037c, load, 0x550080037d, load, 0x550080037e, load, 0x550080037f, load, 0x5500800380, load, 0x5500800381, load, 0x5500800382, load, 0x5500800383, load, 0x5500800384, load, 0x5500800385, load, 0x5500800386, lo ad, 0x5500800387, load, 0x5500800388, load, 0x5500800389, load, 0x550080038a, load, 0x550080038b, load, 0x550080038c, load, 0x550080038d, load, 0x550080038e, load, 0x550080038f, load, 0x5500800390, load, 0x5500800391, load, 0x5500800392, load, 0x5500800393, load, 0x5500800394, load, 0x5500800395, load, 0x5500800396, load, 0x5500800397, load, 0x5500800398, load, 0x5500800399, load, 0x550080039a, load, 0x550080039b, load, 0x550080039c, load, 0x550080039d, load, 0x550080039e, load, 0x550080039f, load, 0x55008003a0, load, 0x55008003a1, load, 0x55008003a2, load, 0x55008003a3, load, 0x55008003a4, load, 0x55008003a5, load, 0x55008003a6, load, 0x55008003a7, load, 0x55008003a8, load, 0x55008003a9, load, 0x55008003aa, load, 0x55008003ab, load, 0x55008003ac, load, 0x55008003ad, load, 0x55008003ae, load, 0x55008003af (4007c0 is the ld1b in the sha512-sve) Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Cc: Robert Henry <robhenry@microsoft.com> Cc: Aaron Lindsay <aaron@os.amperecomputing.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230630180423.558337-20-alex.bennee@linaro.org>
* accel/tcg: Move TLB_WATCHPOINT to TLB_SLOW_FLAGS_MASKRichard Henderson2023-06-261-4/+14
| | | | | | | | This frees up one bit of the primary tlb flags without impacting the TLB_NOTDIRTY logic. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Store some tlb flags in CPUTLBEntryFullRichard Henderson2023-06-261-39/+57
| | | | | | | | | | | | | | | We have run out of bits we can use within the CPUTLBEntry comparators, as TLB_FLAGS_MASK cannot overlap alignment. Store slow_flags[] in CPUTLBEntryFull, and merge with the flags from the comparator. A new TLB_FORCE_SLOW bit is set within the comparator as an indication that the slow path must be used. Move TLB_BSWAP to TLB_SLOW_FLAGS_MASK. Since we are out of bits, we cannot create a new bit without moving an old one. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* tcg: Add host memory barriers to cpu_ldst.h interfacesRichard Henderson2023-06-261-0/+10
| | | | | | | | Bring the helpers into line with the rest of tcg in respecting guest memory ordering. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Replace target_ulong with vaddr in *_mmu_lookup()Anton Johansson2023-06-261-3/+3
| | | | | | | | | | Update atomic_mmu_lookup() and cpu_mmu_lookup() to take the guest virtual address as a vaddr instead of a target_ulong. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230621135633.1649-10-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel: Replace target_ulong with vaddr in probe_*()Anton Johansson2023-06-261-6/+6
| | | | | | | | | | | Functions for probing memory accesses (and functions that call these) are updated to take a vaddr for guest virtual addresses over target_ulong. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230621135633.1649-9-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg: Widen pc to vaddr in CPUJumpCacheAnton Johansson2023-06-261-1/+1
| | | | | | | | | Related functions dealing with the jump cache are also updated. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230621135633.1649-8-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg/cputlb.c: Widen addr in MMULookupPageDataAnton Johansson2023-06-261-15/+15
| | | | | | | | | Functions accessing MMULookupPageData are also updated. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230621135633.1649-6-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel/tcg/cputlb.c: Widen CPUTLBEntry access functionsAnton Johansson2023-06-261-4/+4
| | | | | | | Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230621135633.1649-5-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
* accel: Replace target_ulong in tlb_*()Anton Johansson2023-06-261-90/+87
| | | | | | | | | | Replaces target_ulong with vaddr for guest virtual addresses in tlb_*() functions and auxilliary structs. Signed-off-by: Anton Johansson <anjo@rev.ng> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Message-Id: <20230621135633.1649-2-anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>