diff options
| -rw-r--r-- | README.md | 2 | ||||
| -rw-r--r-- | doc/README.md | 1 | ||||
| -rw-r--r-- | doc/jitter/jitter.ipynb | 543 | ||||
| -rw-r--r-- | miasm/arch/aarch64/arch.py | 9 | ||||
| -rw-r--r-- | miasm/jitter/vm_mngr.h | 2 | ||||
| -rw-r--r-- | setup.py | 2 | ||||
| -rw-r--r-- | test/arch/aarch64/arch.py | 3 | ||||
| -rw-r--r-- | test/jitter/jitcore.py | 12 |
8 files changed, 563 insertions, 11 deletions
diff --git a/README.md b/README.md index 50b8de59..63c1d4b9 100644 --- a/README.md +++ b/README.md @@ -516,7 +516,7 @@ instance to emulate library functions effects. Documentation ============= -TODO +Some documentation ressources are available in the [doc](doc) folder. An auto-generated documentation is available: * [Doxygen](http://miasm.re/miasm_doxygen) diff --git a/doc/README.md b/doc/README.md index e9c1c9d2..e125491c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -36,6 +36,7 @@ class LocationDB(builtins.object) - examples for the main features (see `/example`) - interactive tutorials (IPython Notebooks) on the following topics: + - Emulation API: [notebook](jitter/jitter.ipynb) - Miasm's IR bricks known as `Expr`: [notebook](expression/expression.ipynb) - Lifting from assembly to IR: [notebook](ir/lift.ipynb) - `LocationDB` usage, the database for locations: [notebook](locationdb/locationdb.ipynb) diff --git a/doc/jitter/jitter.ipynb b/doc/jitter/jitter.ipynb new file mode 100644 index 00000000..adab4c5b --- /dev/null +++ b/doc/jitter/jitter.ipynb @@ -0,0 +1,543 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Miasm Emulation Engine\n", + "\n", + "This short document provides an introduction to the APIs of the Miasm emulation engine. This emulation engine is commonly referred to as **jitter** throughout the project's code (as it is based on [JiT](https://en.wikipedia.org/wiki/Just-in-time_compilation) methods)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The first step is to obtain an instance of the jitter. To do this, we will go through an intermediate object, `Machine`. It will allow us to instantiate a set of elements linked to a given architecture supported by Miasm (x86, arm, mips, ...)." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['arml',\n", + " 'armb',\n", + " 'armtl',\n", + " 'armtb',\n", + " 'sh4',\n", + " 'x86_16',\n", + " 'x86_32',\n", + " 'x86_64',\n", + " 'msp430',\n", + " 'mips32b',\n", + " 'mips32l',\n", + " 'aarch64l',\n", + " 'aarch64b',\n", + " 'ppc32b',\n", + " 'mepl',\n", + " 'mepb']" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from miasm.analysis.machine import Machine\n", + "Machine.available_machine()" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "x86_32\n" + ] + } + ], + "source": [ + "machine = Machine(\"x86_32\")\n", + "print(machine.name)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `Machine` class in Miasm provides various interfaces:\n", + "\n", + "* `jitter`: an emulation engine\n", + "* `dis_engine`: a disassembly engine\n", + "* `lifter`: a lifting engine, to lift assembly code to the corresponding Miasm internal representation (IR)\n", + "* `lifter_model_call`: as `lifter`, but function calls are abstracted\n", + "* `mn`: low level object to interact with an architecture (assembly, disassembly of only a few bytes, etc.)\n", + "* `gdbserver`: a GDB remote debugging server (to link with a `jitter` instance)\n", + "\n", + "The IR related objects are already discussed in [this notebook](../ir/lift.ipynb)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To obtain an emulator instance, we first need to:\n", + "\n", + "1. instanciate a *symbol table* (`LocationDB` in Miasm, already explained in [this notebook](../locationdb/locationdb.ipynb)) ;\n", + "1. (optionnaly) choose a backed for the emulation (more specifically, to JiT the code):\n", + "\n", + " * \"python\": a backend entirely in Python, slow\n", + " * \"gcc\": a backend based on the compiler GCC (Assembly $\\rightarrow$ IR Miasm $\\rightarrow$ C $\\rightarrow$ GCC), fast\n", + " * \"llvm\": a backend based on the LLVM JiT engine (Assembly $\\rightarrow$ IR Miasm $\\rightarrow$ IR LLVM $\\rightarrow$ LLVM JiT), fast\n", + "\n", + "Note: using a JiT backend as `llvm` or `gcc` greatly improve the performance over the `python` one. As a result, without any argument, instanciating an emulator will automatically try to look for and choose the best suited backend. " + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "<miasm.arch.x86.jit.jitter_x86_32 object at 0x7f195fe07d60>\n" + ] + } + ], + "source": [ + "from miasm.core.locationdb import LocationDB\n", + "\n", + "loc_db = LocationDB()\n", + "jitter = machine.jitter(loc_db)\n", + "print(jitter)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For now, our emulator is an empty box. It has:\n", + "\n", + "* registers, reachable from the `.cpu` attribute. These are initialized to 0.\n", + "* a virtual memory, reachable from the `.vm` attribute. It starts empty." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### CPU state\n", + "\n", + "Let's manipulate a few registers:" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0\n", + "1\n" + ] + } + ], + "source": [ + "# Read a register\n", + "print(jitter.cpu.EAX)\n", + "\n", + "# Write a register\n", + "jitter.cpu.EAX = 1\n", + "print(jitter.cpu.EAX)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A few *helpers* are also available. For instance, one can get every registers using:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'RAX': 1, 'RBX': 0, 'RCX': 0, 'RDX': 0, 'RSI': 0, 'RDI': 0, 'RSP': 0, 'RBP': 0, 'R8': 0, 'R9': 0, 'R10': 0, 'R11': 0, 'R12': 0, 'R13': 0, 'R14': 0, 'R15': 0, 'RIP': 0, 'zf': 0, 'nf': 0, 'pf': 0, 'of': 0, 'cf': 0, 'af': 0, 'df': 0, 'ES': 0, 'CS': 0, 'SS': 0, 'DS': 0, 'FS': 0, 'GS': 0, 'MM0': 0, 'MM1': 0, 'MM2': 0, 'MM3': 0, 'MM4': 0, 'MM5': 0, 'MM6': 0, 'MM7': 0, 'XMM0': 0, 'XMM1': 0, 'XMM2': 0, 'XMM3': 0, 'XMM4': 0, 'XMM5': 0, 'XMM6': 0, 'XMM7': 0, 'XMM8': 0, 'XMM9': 0, 'XMM10': 0, 'XMM11': 0, 'XMM12': 0, 'XMM13': 0, 'XMM14': 0, 'XMM15': 0, 'tsc': 1234605616436508552}\n" + ] + } + ], + "source": [ + "# GPReg : General Purpose registers\n", + "regs = jitter.cpu.get_gpreg()\n", + "print(regs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note: in Miasm, `x86_32` implementation is merged with the `x86_64` one. As a result, the register names are the x64 ones. Still, the computation is made accordingly to the 32bits implementation, and registers can be accessed either using the x64 name or the 32bits name (ie `.EAX` and `.RAX`)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Virtual memory\n", + "\n", + "Initially, the virtual memory is empty. The *repr* output offers a summary of the availables pages: " + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Addr Size Access Comment" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "jitter.vm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Memory pages can then be added.\n", + "The **memory address is arbitrary** (no alignment requirement). **The size is also arbitrary** (byte precision).\n", + "\n", + "Optionnaly, a comment associated to the page can be provided." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Addr Size Access Comment\n", + "0x1000 0x1000 RWX test page\n", + "0x112233 0x666 RWX no alignment, byte precision" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from miasm.jitter.csts import PAGE_READ, PAGE_WRITE, PAGE_EXEC\n", + "jitter.vm.add_memory_page(0x1000, PAGE_READ | PAGE_WRITE | PAGE_EXEC, b\"\\x00\" * 0x1000, \"test page\")\n", + "jitter.vm.add_memory_page(0x112233, PAGE_READ | PAGE_WRITE | PAGE_EXEC, b\"\\x00\" * 0x666, \"no alignment, byte precision\")\n", + "jitter.vm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The memory can now be accessed, read and write:" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "b'\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "jitter.vm.get_mem(0x1000, 0x10)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "b'toto\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00'" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "jitter.vm.set_mem(0x1000, b\"toto\")\n", + "jitter.vm.get_mem(0x1000, 0x10)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note: having a byte-precision memory has some advantages. For instance:\n", + "* when mapping a partially known structure, instead of defaulting unknown fields to 0, we can instead only map in memory the known field. As a result, we obtain a sparse memory layout and if the programm try to read an unknown field, the execution will stop right on the responsible instruction, helping the reversing work\n", + "* allocation can be made byte-wise. Thus, it could be easier to detect overflows and underflows" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Emulation\n", + "\n", + "We will now run the emulator.\n", + "First, actual instructions are needed:" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "jitter.vm.set_mem(0x1000, bytes.fromhex(\"B844332211C3\"))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will also enable debug logging.\n", + "By default, the logger will logs executed instruction and register values." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [], + "source": [ + "jitter.set_trace_log()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "The emulation is initialized with `jitter.init_run(address)` and resumed with `jitter.continue_run()`.\n", + "For convenience, `jitter.run(address)` is usually used, wrapping these two API." + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[WARNING ]: [Errno cannot get mem ad] 0x1337beef\n", + "WARNING: address 0x1337BEEF is not mapped in virtual memory:\n", + "[WARNING ]: cannot disasm at 1337BEEF\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "00001000 MOV EAX, 0x11223344\n", + "EAX 11223344 EBX 00000000 ECX 00000000 EDX 00000000 ESI 00000000 EDI 00000000 ESP 88880FFC EBP 00000000 EIP 00001005 zf 0 nf 0 of 0 cf 0\n", + "00001005 RET \n", + "EAX 11223344 EBX 00000000 ECX 00000000 EDX 00000000 ESI 00000000 EDI 00000000 ESP 88881000 EBP 00000000 EIP 1337BEEF zf 0 nf 0 of 0 cf 0\n" + ] + } + ], + "source": [ + "jitter.vm.add_memory_page(0x88880000, PAGE_READ | PAGE_WRITE | PAGE_EXEC, b\"\\x00\" * 0x1000, \"stack\")\n", + "jitter.vm.set_mem(0x88880000 + 0x1000 - 4, b\"\\xef\\xbe\\x37\\x13\")\n", + "jitter.cpu.ESP = 0x88880000 + 0x1000 - 4\n", + "\n", + "jitter.run(0x1000)\n", + "\n", + "# The execution ends with an error, which is expected.\n", + "# Indeed, we RET on 0x1337beef, which is not mapped in memory,\n", + "# hence the \"WARNING: address 0x1337BEEF is not mapped in virtual memory\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Breakpoints\n", + "\n", + "One can register breakpoints to be raised just before a given address will be executed.\n", + "\n", + "As we control the CPU, the breakpoint implementation is invisible from the emulated environnement and we can use an arbitrary number of them." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": {}, + "outputs": [], + "source": [ + "# Breakpoints are actually callbacks which receive the emulation instance in argument\n", + "def hello_world(jitter):\n", + " print(\"Hello, world!\")\n", + " print(\"EAX value is %d\" % jitter.cpu.EAX)\n", + " # Stop execution right here\n", + " return False\n", + "\n", + "jitter.add_breakpoint(0x1005, hello_world)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If a breakpoint callback returns `True`, the emulation continues.\n", + "Otherwise, the emulation will stop and the value returned by the `jitter.continue_run()` call is the callback return value.\n", + "\n", + "Note: if one need to add more arguments to the callback, there are many Pythonic way to do it.\n", + "For instance, one can use `lambda` to capture an argument or object instance (ie. `jitter.add_breakpoint(address, obj.callback)` with a `def callback(self, jitter)` inside the `obj` definition).\n", + "\n", + "The breakpoint mechanism is used to implement several features. For instance:\n", + "* it can be used to hook a function, by breakpointing on the function address, emulating a few side effects then returning to the return address. Indeed, modifying the programm counter (usually using `jitter.pc = XXX`) in a breakpoint will resume the execution on this new address\n", + "* in Miasm examples, a breakpoint on a fake address is often used to properly stop the emulation. The callback is usually named `code_sentinelle` and allow to get back control after the emulation of a code snippet" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "### Exercices\n", + "\n", + "For the interested reader, this section introduce a few exercice to practice the API.\n", + "\n", + "#### Exercice 1\n", + "\n", + "Starting from a new instance:\n", + "1. Add a memory page which will act as a stack page\n", + "1. Update the stack pointer accordingly\n", + "1. Manually push a fake return address\n", + "1. Map a `RET` instruction (0xC3) in another page\n", + "1. Run the emulation and ensure it finish on your fake address" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Exercice 1 - Spoiler\n", + "\n", + "A few helpers are actually availables to reduce the burden of these manual actions:" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0x1000\n" + ] + } + ], + "source": [ + "# Init a stack and set stack pointer accordingly (architecture agnostic way of writing)\n", + "jitter.init_stack()\n", + "\n", + "# Push 0x1000 on the stack\n", + "jitter.push_uint32_t(0x1000)\n", + "\n", + "# Pop 0x1000 from the stack\n", + "print(hex(jitter.pop_uint32_t()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Exercice 2\n", + "\n", + "Same a Exercice 1, but instead of ending on an error, we will this time add a breakpoint on the fake return address to properly stop the execution." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "#### Exercice 2 - Spoiler\n", + "\n", + "The example `jitter/x86_32.py` is exactly doing this." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.12" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/miasm/arch/aarch64/arch.py b/miasm/arch/aarch64/arch.py index 0ade16bf..c642d201 100644 --- a/miasm/arch/aarch64/arch.py +++ b/miasm/arch/aarch64/arch.py @@ -1909,6 +1909,10 @@ adsu_name = {'ADD': 0, 'SUB': 1} bs_adsu_name = bs_name(l=1, name=adsu_name) +adsus_name = {'ADDS': 0, 'SUBS': 1} +bs_adsus_name = bs_name(l=1, name=adsus_name) + + offs19 = bs(l=19, cls=(aarch64_offs,), fname='off') offs19pc = bs(l=19, cls=(aarch64_offs_pc,), fname='off') @@ -1939,8 +1943,9 @@ aarch64op("CMN", [sf, bs('0'), bs('1'), bs('01011'), shift, bs('0'), rm_sft, imm aarch64op("cmp", [sf, bs('1'), bs('1'), bs('01011'), shift, bs('0'), rm_sft, imm6, rn, bs('11111')], [rn, rm_sft], alias=True) # add/sub (reg ext) -aarch64op("addsub", [sf, bs_adsu_name, modf, bs('01011'), bs('00'), bs('1'), rm_ext, option, imm3, rn, rd], [rd, rn, rm_ext]) -#aarch64op("cmp", [sf, bs('1'), bs('1'), bs('01011'), bs('00'), bs('1'), rm_ext, option, imm3, rn, bs('11111')], [rn, rm_ext], alias=True) +aarch64op("addsub", [sf, bs_adsu_name, bs('0'), bs('01011'), bs('00'), bs('1'), rm_ext, option, imm3, rn, rd], [rd, rn, rm_ext]) +aarch64op("addssubs", [sf, bs_adsus_name, bs('1'), bs('01011'), bs('00'), bs('1'), rm_ext, option, imm3, rn, rd_nosp], [rd_nosp, rn, rm_ext]) +aarch64op("cmp", [sf, bs('1'), bs('1'), bs('01011'), bs('00'), bs('1'), rm_ext, option, imm3, rn, bs('11111')], [rn, rm_ext], alias=True) aarch64op("neg", [sf, bs('1'), modf, bs('01011'), shift, bs('0'), rm_sft, imm6, bs('11111'), rd], [rd, rm_sft], alias=True) diff --git a/miasm/jitter/vm_mngr.h b/miasm/jitter/vm_mngr.h index 4c8383c4..f7aea5b8 100644 --- a/miasm/jitter/vm_mngr.h +++ b/miasm/jitter/vm_mngr.h @@ -39,7 +39,7 @@ #define __BYTE_ORDER __BYTE_ORDER__ #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN -#elif defined(__NetBSD__) || defined(__OpenBSD__) +#elif defined(__NetBSD__) || defined(__OpenBSD__) || defined(__FreeBSD__) #define __BYTE_ORDER _BYTE_ORDER #define __BIG_ENDIAN _BIG_ENDIAN #define __LITTLE_ENDIAN _LITTLE_ENDIAN diff --git a/setup.py b/setup.py index e1e54434..abecb1a1 100644 --- a/setup.py +++ b/setup.py @@ -416,7 +416,7 @@ _write_pkg_file_orig = DistributionMetadata.write_pkg_file def _write_pkg_file(self, file): - with TemporaryFile(mode="w+") as tmpfd: + with TemporaryFile(mode="w+", encoding="utf-8") as tmpfd: _write_pkg_file_orig(self, tmpfd) tmpfd.seek(0) for line in tmpfd: diff --git a/test/arch/aarch64/arch.py b/test/arch/aarch64/arch.py index 57ec9b14..4156e054 100644 --- a/test/arch/aarch64/arch.py +++ b/test/arch/aarch64/arch.py @@ -67,6 +67,9 @@ reg_tests_aarch64 = [ ("0000D5AC NEG W6, W6", "E603064B"), + ("XXXXXXXX CMP W11, W12 UXTB 0x0", + "7F012C6B"), + ("004028B8 CMP X0, XZR", "1F001FEB"), diff --git a/test/jitter/jitcore.py b/test/jitter/jitcore.py index 1e009d9a..95245855 100644 --- a/test/jitter/jitcore.py +++ b/test/jitter/jitcore.py @@ -9,16 +9,16 @@ jitter = machine.jitter(loc_db, sys.argv[1]) jitter.cpu.RAX = 16565615892967251934 assert jitter.cpu.RAX == 16565615892967251934 -jitter.cpu.RAX = -1 +jitter.cpu.RAX = -1 & 0xffffffffffffffff assert jitter.cpu.RAX == 0xffffffffffffffff -jitter.cpu.RAX = -2 +jitter.cpu.RAX = -2 & 0xffffffffffffffff assert jitter.cpu.RAX == 0xfffffffffffffffe -jitter.cpu.EAX = -2 +jitter.cpu.EAX = -2 & 0xffffffff assert jitter.cpu.EAX == 0xfffffffe -jitter.cpu.RAX = -0xffffffffffffffff +jitter.cpu.RAX = -0xffffffffffffffff & 0xffffffffffffffff assert jitter.cpu.RAX == 1 try: @@ -35,10 +35,10 @@ except TypeError: else: raise Exception("Should see that 0x10000000000000000 is too big for RAX") -jitter.cpu.EAX = -0xefffffff +jitter.cpu.EAX = -0xefffffff & 0xffffffff assert jitter.cpu.EAX == 0x10000001 -jitter.cpu.EAX = -0xFFFFFFFF +jitter.cpu.EAX = -0xFFFFFFFF & 0xffffffff assert jitter.cpu.EAX == 1 try: |