portugalfuturista/rizin
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Asm token documentation (#3711)

Browse source 
* Move compile_token_patterns to asm module.
* Simplify rz_asm_tokenize_asm_regex().
* Implement custom token parsing for bf.
* Add docs about asm token parsing and example implementation.
This commit is contained in:
Rot127 2023-08-02 03:00:18 +00:00 committed by GitHub
parent c0b02cf546
commit a3eefea988
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 393 additions and 60 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

233
doc/asm_strings.md Normal file
View file

@ -0,0 +1,233 @@
# Assembly Strings
## Colorizing asm strings
There are two ways we colorize our asm strings.
1. The generic method. For straight forward `<mnemonic> <op> <op>` syntax.
2. The custom method. Which allows for way more complicated assembly syntax.
In both methods we start by assigning tokens to sub-strings of the asm string.
### Tokenizing
A token itself simply is a reference a sub-string.
So it contains an offset into the asm string where a sub-string starts.
And the length of the sub-string in bytes as well as the token type.
```c
typedef struct {
size_t start; //< byte-offset into `str` where this token starts. Must be exactly at a utf-8 codepoint boundary.
size_t len; //< `str` length of token in bytes.
RzAsmTokenType type;
// ...
} RzAsmToken;
```
Here is a list of some token types:
```c
typedef enum {
RZ_ASM_TOKEN_UNKNOWN = 0, //< Does not fit to any token below.
RZ_ASM_TOKEN_MNEMONIC, //< Asm mnemonics like: mov, push, lea...
RZ_ASM_TOKEN_OPERATOR, //< Arithmetic operators: +,-,<< etc.
RZ_ASM_TOKEN_NUMBER, //< Numbers
RZ_ASM_TOKEN_REGISTER, //< Registers
RZ_ASM_TOKEN_SEPARATOR, //< Brackets, comma etc.
RZ_ASM_TOKEN_META, //< Meta information (e.g Hexagon packet prefix, ARM & Hexagon number prefix).
// If needed add one here.
} RzAsmTokenType;
```
Let's look at an example:
```asm
add r0, r1, 0x10
```
This asm string would be split into 7 tokens:
| Sub-String | start | length | type |
|------------|-------|--------|-----------|
| `add` | 0 | 3 | mnemonic |
| ` ` | 3 | 1 | separator |
| `r0` | 4 | 2 | register |
| `, ` | 6 | 2 | separator |
| `r1` | 8 | 2 | register |
| `, ` | 10 | 2 | separator |
| `0x10` | 12 | 4 | number |
A vector of those tokens now describes the whole asm string.
```c
typedef struct {
ut32 op_type; ///< RzAnalysisOpType. Mnemonic color depends on this.
RzStrBuf *str; ///< Contains the raw asm string
RzVector /*<RzAsmToken>*/ *tokens; ///< Contains only the tokenization meta-info without strings, ordered by start for log2(n) access
} RzAsmTokenString;
```
Note the `op_type` member. It should get a `RzAnalysisOpType` assigned to it.
Depending on this type, the coloring of the mnemonic will differ.
### Coloring
Coloring the token vector becomes very straight forward.
We simply append each token to each other and insert the color escape sequences before and after each sub-string.
### General tokenize method
The general method assumes that the asm string is roughly of the following form:
```
<mnemonic><separator><operand><separator><operand>...
```
This is the standard parsing pattern. It will be applied if no custom patterns are implemented by the asm module.
For parsing details refer to the documentation in the code (see: `tokenize_asm_generic()`).
### Custom tokenizing of an asm string.
Custom tokenizing of asm strings can be done for complicated or obscure asm strings.
To implement this for an asm module you need to:
1. Define regex patterns for each token type.
2. Parse the asm string in the `disassemble()` function of the asm module.
**Implementation**
In this example we implement tokenization for `bf` assembly.
If we disassemble `[->+<]` (adding two numbers) we get
```
while [ptr]
dec [ptr]
inc ptr
inc [ptr]
dec ptr
loop
```
First add the following function to `asm_bf.c`:
```c
static RZ_OWN RzPVector /*<RzAsmTokenPattern *>*/ *get_token_patterns() {
static RzPVector *pvec = NULL;
if (pvec) {
return pvec;
}
pvec = rz_pvector_new(rz_asm_token_pattern_free);
// Patterns get added here.
}
```
Now we can add the regex patterns for each token type and append them to the vector.
The mnemonics of the `bf` assembly are: `while`, `inc`, `dec`, `trap`, `nop`, `invalid`
and `loop`. We treat `ptr` as a register for simplicity.
We also have the separator ` ` and the operation `[]` (reference the value at `ptr`).
Now we add the regex patterns.
Note, the first patterns in the vector are matched first as well.
They have a higher priority. With the correct ordering, you can prevent conflicts between patterns.
You can also add multiple patterns of the same type. Each of which is further up or down in priority.
```c
static RZ_OWN RzPVector /*<RzAsmTokenPattern *>*/ *get_token_patterns() {
// ...
// Patterns get added here.
// Mnemonic pattern
RzAsmTokenPattern *pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_MNEMONIC;
pat->pattern = strdup(
"^((while)|(inc)|(dec)|(trap)|(nop)|(invalid)|(loop))"
);
rz_pvector_push(pvec, pat);
// ptr pattern
pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_REGISTER;
pat->pattern = strdup(
"(ptr)"
);
rz_pvector_push(pvec, pat);
// reference pattern
pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_OPERATOR;
pat->pattern = strdup(
"(\\[)|(\\])" // Matches a single bracket
);
rz_pvector_push(pvec, pat);
// Separator pattern
pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_SEPARATOR;
pat->pattern = strdup(
"([[:blank:]]+)"
);
rz_pvector_push(pvec, pat);
return pvec;
}
```
Now we can parse every disassembled asm string into tokens by passing it to `rz_asm_tokenize_asm_regex()`.
The result should be assigned to `RzAsm.asm_toks`.
In our example we do it in `disassemble()`.
```c
// ...
op_type = RZ_ANALYSIS_OP_TYPE_TRAP;
buf_asm = "trap";
break;
default:
op_type = RZ_ANALYSIS_OP_TYPE_NOP;
buf_asm = "nop";
break;
}
rz_strbuf_set(&op->buf_asm, buf_asm);
RzPVector *token_patterns = get_token_patterns();
op->asm_toks = rz_asm_tokenize_asm_regex(&op->buf_asm, token_patterns);
op->asm_toks->op_type = op_type;
// ...
}
```
If `RzAsm.asm_toks` is not `NULL` the tokens will be used to colorize the asm string.
The color of the mnemonic token is defined by the operation type.
So ensure that `RzAsmTokenString.op_type` is set correctly.
### What for?
For simple syntax the custom method might seem overengineered.
But defining properly which part of an asm string means what, has additional advantages.
- Numbers can be extracted from the string and used later.
- If some syntax needs to be manipulated, it is way easier to do this.
Selecting the token with the right type and change it. No raw string operations anymore.
- For complex syntax it will produce beautiful results.
Here is an example for the last point.
The general method:
![hexagon-syntax-general](img/hexagon-asm-syntax-general.png)
The custom method:
![hexagon-syntax-regex](img/hexagon-asm-syntax-regex.png)
As you see, the custom method is way more beautiful.
Check out `asm_hexagon.c` for an example of the complex patterns used for it.

BIN
doc/img/hexagon-asm-syntax-general.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 13 KiB

2
doc/img/hexagon-asm-syntax-general.png.license Normal file
View file

@ -0,0 +1,2 @@
SPDX-FileCopyrightText: 2023 Rot127 <unisono@quyllur.org>
SPDX-License-Identifier: LGPL-3.0-only

BIN
doc/img/hexagon-asm-syntax-regex.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 12 KiB

2
doc/img/hexagon-asm-syntax-regex.png.license Normal file
View file

@ -0,0 +1,2 @@
SPDX-FileCopyrightText: 2023 Rot127 <unisono@quyllur.org>
SPDX-License-Identifier: LGPL-3.0-only

103
librz/asm/asm.c
View file

@ -1431,6 +1431,16 @@ static bool overlaps_with_token(RZ_BORROW RzVector /*<RzAsmTokenString>*/ *toks,
return false;
}
/**
* \brief Compare two RzAsmTokens.
*
* \param a Token a to compare.
* \param b Token b to compare.
*
* \return -1 If a.start < b.start
* \return 1 If a.start > b.start
* \return 0 If a.start == b.start
*/
static int cmp_tokens(const RzAsmToken *a, const RzAsmToken *b) {
rz_return_val_if_fail(a && b, 0);
if (a->start < b->start) {
@ -1441,6 +1451,12 @@ static int cmp_tokens(const RzAsmToken *a, const RzAsmToken *b) {
return 0;
}
/**
* \brief Checks a token string if any token in it overlaps with another or a part of the asm string is not covered.
* It prints a warning if this is the case.
*
* \param toks The token string to check.
*/
static void check_token_coverage(RzAsmTokenString *toks) {
rz_return_if_fail(toks);
if (rz_vector_len(toks->tokens) == 0) {
@ -1480,6 +1496,27 @@ static void check_token_coverage(RzAsmTokenString *toks) {
}
}
/**
* \brief Compiles the regex patterns of a vector of RzAsmTokenPatterns.
*
* \param patterns The token patterns to compile the regex for.
*/
RZ_API void rz_asm_compile_token_patterns(RZ_INOUT RzPVector /*<RzAsmTokenPattern *>*/ *patterns) {
rz_return_if_fail(patterns);
void **it;
rz_pvector_foreach (patterns, it) {
RzAsmTokenPattern *pat = *it;
if (!pat->regex) {
pat->regex = rz_regex_new(pat->pattern, "e");
if (!pat->regex) {
RZ_LOG_WARN("Did not compile regex pattern %s.\n", pat->pattern);
rz_warn_if_reached();
}
}
}
}
/**
* \brief Splits an asm string into tokens by using the given regex patterns.
*
@ -1487,43 +1524,57 @@ static void check_token_coverage(RzAsmTokenString *toks) {
* \param patterns RzList<RzAsmTokenPattern> with the regex patterns describing each token type.
* \return RzAsmTokenString* The tokens.
*/
RZ_API RZ_OWN RzAsmTokenString *rz_asm_tokenize_asm_regex(RZ_BORROW RzStrBuf *asm_str, RzPVector /*<RzAsmTokenPattern *>*/ *patterns) {
rz_return_val_if_fail(asm_str && patterns, NULL);
RZ_API RZ_OWN RzAsmTokenString *rz_asm_tokenize_asm_regex(RZ_BORROW RzStrBuf *asm_string, RzPVector /*<RzAsmTokenPattern *>*/ *patterns) {
rz_return_val_if_fail(asm_string && patterns, NULL);
const char *asm_str = rz_strbuf_get(asm_string);
RzAsmTokenString *toks = rz_asm_token_string_new(asm_str);
const char *str = rz_strbuf_get(asm_str);
RzRegexMatch m[1];
size_t j = 0; // Offset into str. Regex patterns are only searched in substring str[j:].
st64 i = 0; // Start of token in str.
st64 s = 0; // Start of matched token in substring str[j:]
st64 l = 0; // Length of token.
RzAsmTokenString *toks = rz_asm_token_string_new(str);
void **it;
// Iterate over each pattern and search for it in str
rz_pvector_foreach (patterns, it) {
RzAsmTokenPattern *pat = *it;
if (!pat || !pat->regex) {
RzAsmTokenPattern *pattern = *it;
if (!pattern) {
rz_asm_token_string_free(toks);
return NULL;
}
j = 0;
if (!pat->regex) {
if (!pattern->regex) {
// Pattern was not compiled.
pattern->regex = rz_regex_new(pattern->pattern, "e");
if (!pattern->regex) {
rz_warn_if_reached();
return NULL;
}
}
/// Start pattern search from the beginning
size_t asm_str_off = 0;
if (!pattern->regex) {
continue;
}
while (rz_regex_exec(pat->regex, str + j, 1, m, 0) == 0) {
s = m[0].rm_so; // Token start in substring str[j:]
l = m[0].rm_eo - s; // (End in substring str[j:]) - (start in substring str[j:]) = Length of token.
i = j + s; // Start of token in str.
if (overlaps_with_token(toks->tokens, i, i + l - 1)) {
// Search for token pattern.
RzRegexMatch match[1];
while (rz_regex_exec(pattern->regex, asm_str + asm_str_off, 1, match, 0) == 0) {
st64 match_start = match[0].rm_so; // Token start
st64 match_end = match[0].rm_eo; // Token end
st64 len = match_end - match_start; // Length of token
st64 tok_offset = asm_str_off + match_start; // Token offset in str
if (overlaps_with_token(toks->tokens, tok_offset, tok_offset + len - 1)) {
// If this is true a token with higher priority was matched before.
j = i + l;
asm_str_off = tok_offset + len;
continue;
}
if (!is_num(str + i)) {
add_token(toks, i, l, pat->type, 0);
j = i + l;
// New token found, add it.
if (!is_num(asm_str + tok_offset)) {
add_token(toks, tok_offset, len, pattern->type, 0);
asm_str_off = tok_offset + len;
continue;
}
add_token(toks, i, l, pat->type, strtoull(str + i, NULL, 0));
j = i + l;
ut64 number = strtoull(asm_str + tok_offset, NULL, 0);
add_token(toks, tok_offset, len, pattern->type, number);
asm_str_off = tok_offset + len;
}
}
@ -1534,7 +1585,8 @@ RZ_API RZ_OWN RzAsmTokenString *rz_asm_tokenize_asm_regex(RZ_BORROW RzStrBuf *as
}
/**
* \brief Seeks to the end of the token at \p str + \p i and returns the length of it.
* \brief Seeks from \p str + \p i for a token of the given \p type.
* If any was found it returns the length of it. Or 0 if non was found.
*
* \param str The asm string.
* \param i Index into \p str where the token starts.
@ -1578,6 +1630,7 @@ static size_t seek_to_end_of_token(const char *str, size_t i, RzAsmTokenType typ
do {
++j;
} while (!isascii(*(str + j)) && !is_operator(str + j) && !is_separator(str + j) && !is_alpha_num(str + j));
break;
}
return j - i;
}

58
librz/asm/p/asm_bf.c
View file

@ -2,45 +2,103 @@
// SPDX-FileCopyrightText: 2009-2021 nibble <nibble.ds@gmail.com>
// SPDX-License-Identifier: LGPL-3.0-only
#include <rz_analysis.h>
#include <rz_asm.h>
static RZ_OWN RzPVector /*<RzAsmTokenPattern *>*/ *get_token_patterns() {
static RzPVector *pvec = NULL;
if (pvec) {
return pvec;
}
pvec = rz_pvector_new(rz_asm_token_pattern_free);
// Patterns get added here.
// Mnemonic pattern
RzAsmTokenPattern *pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_MNEMONIC;
pat->pattern = strdup(
"^((while)|(inc)|(dec)|(trap)|(nop)|(invalid)|(loop))");
rz_pvector_push(pvec, pat);
// ptr pattern
pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_REGISTER;
pat->pattern = strdup(
"(ptr)");
rz_pvector_push(pvec, pat);
// reference pattern
pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_OPERATOR;
pat->pattern = strdup(
"(\\[)|(\\])" // Matches a single bracket
);
rz_pvector_push(pvec, pat);
// Separator pattern
pat = RZ_NEW0(RzAsmTokenPattern);
pat->type = RZ_ASM_TOKEN_SEPARATOR;
pat->pattern = strdup(
"([[:blank:]]+)");
rz_pvector_push(pvec, pat);
return pvec;
}
static int disassemble(RzAsm *a, RzAsmOp *op, const ut8 *buf, int len) {
const char *buf_asm = "invalid";
ut32 op_type;
switch (*buf) {
case '[':
op_type = RZ_ANALYSIS_OP_TYPE_CJMP;
buf_asm = "while [ptr]";
break;
case ']':
op_type = RZ_ANALYSIS_OP_TYPE_UJMP;
buf_asm = "loop";
break;
case '>':
op_type = RZ_ANALYSIS_OP_TYPE_ADD;
buf_asm = "inc ptr";
break;
case '<':
op_type = RZ_ANALYSIS_OP_TYPE_SUB;
buf_asm = "dec ptr";
break;
case '+':
op_type = RZ_ANALYSIS_OP_TYPE_ADD;
buf_asm = "inc [ptr]";
break;
case '-':
op_type = RZ_ANALYSIS_OP_TYPE_SUB;
buf_asm = "dec [ptr]";
break;
case ',':
op_type = RZ_ANALYSIS_OP_TYPE_STORE;
buf_asm = "in [ptr]";
break;
case '.':
op_type = RZ_ANALYSIS_OP_TYPE_LOAD;
buf_asm = "out [ptr]";
break;
case 0xff:
case 0x00:
op_type = RZ_ANALYSIS_OP_TYPE_TRAP;
buf_asm = "trap";
break;
default:
op_type = RZ_ANALYSIS_OP_TYPE_NOP;
buf_asm = "nop";
break;
}
rz_strbuf_set(&op->buf_asm, buf_asm);
RzPVector *token_patterns = get_token_patterns();
op->asm_toks = rz_asm_tokenize_asm_regex(&op->buf_asm, token_patterns);
op->asm_toks->op_type = op_type;
op->size = 1;
return op->size;
}

18
librz/asm/p/asm_hexagon.c
View file

@ -111,22 +111,6 @@ static RZ_OWN RzPVector /*<RzAsmTokenPattern *>*/ *get_token_patterns() {
return pvec;
}
static void compile_token_patterns(RZ_INOUT RzPVector /*<RzAsmTokenPattern *>*/ *patterns) {
rz_return_if_fail(patterns);
void **it;
rz_pvector_foreach (patterns, it) {
RzAsmTokenPattern *pat = *it;
if (!pat->regex) {
pat->regex = rz_regex_new(pat->pattern, "e");
if (!pat->regex) {
RZ_LOG_WARN("Did not compile regex pattern %s.\n", pat->pattern);
rz_warn_if_reached();
}
}
}
}
/**
* \brief Setter for the plugins RzConfig nodes.
*
@ -173,7 +157,7 @@ static bool hexagon_init(void **user) {
SETCB("plugins.hexagon.reg.alias", "true", &hex_cfg_set, "Print the alias of registers (Alias from C0 = SA0).");
state->token_patterns = get_token_patterns();
compile_token_patterns(state->token_patterns);
rz_asm_compile_token_patterns(state->token_patterns);
return true;
}

1
librz/include/rz_asm.h
View file

@ -207,6 +207,7 @@ RZ_API RZ_OWN RzAsmTokenString *rz_asm_token_string_new(const char *asm_str);
RZ_API void rz_asm_token_string_free(RZ_OWN RzAsmTokenString *toks);
RZ_API RZ_OWN RzAsmTokenString *rz_asm_token_string_clone(RZ_OWN RZ_NONNULL RzAsmTokenString *toks);
RZ_API void rz_asm_token_pattern_free(void *p);
RZ_API void rz_asm_compile_token_patterns(RZ_INOUT RzPVector /*<RzAsmTokenPattern *>*/ *patterns);
RZ_API RZ_OWN RzAsmTokenString *rz_asm_tokenize_asm_regex(RZ_BORROW RzStrBuf *asm_str, RzPVector /*<RzAsmTokenPattern *>*/ *patterns);
RZ_API RZ_OWN RzAsmParseParam *rz_asm_get_parse_param(RZ_NULLABLE const RzReg *reg, ut32 ana_op_type);
RZ_DEPRECATE RZ_API RZ_OWN RzAsmTokenString *rz_asm_tokenize_asm_string(RZ_BORROW RzStrBuf *asm_str, RZ_NULLABLE const RzAsmParseParam *param);

36
librz/include/rz_util/rz_print.h
View file

@ -43,13 +43,13 @@ typedef const char *(*RzPrintColorFor)(void *user, ut64 addr, bool verbose);
typedef char *(*RzPrintHasRefs)(void *user, ut64 addr, int mode);
typedef enum {
RZ_ASM_TOKEN_UNKNOWN = 0, //< Does not fit to any token below.
RZ_ASM_TOKEN_MNEMONIC, //< Asm mnemonics like: mov, push, lea...
RZ_ASM_TOKEN_OPERATOR, //< Arithmetic operators: +,-,<< etc.
RZ_ASM_TOKEN_NUMBER, //< Numbers
RZ_ASM_TOKEN_REGISTER, //< Registers
RZ_ASM_TOKEN_SEPARATOR, //< Brackets, comma etc.
RZ_ASM_TOKEN_META, //< Meta information (e.g Hexagon packet prefix, ARM & Hexagon number prefix).
RZ_ASM_TOKEN_UNKNOWN = 0, ///< Does not fit to any token below.
RZ_ASM_TOKEN_MNEMONIC, ///< Asm mnemonics like: mov, push, lea...
RZ_ASM_TOKEN_OPERATOR, ///< Arithmetic operators: +,-,<< etc.
RZ_ASM_TOKEN_NUMBER, ///< Numbers
RZ_ASM_TOKEN_REGISTER, ///< Registers
RZ_ASM_TOKEN_SEPARATOR, ///< Brackets, comma etc.
RZ_ASM_TOKEN_META, ///< Meta information (e.g Hexagon packet prefix, ARM & Hexagon number prefix).
RZ_ASM_TOKEN_LAST,
} RzAsmTokenType;
@ -58,21 +58,21 @@ typedef enum {
* \brief A token of an asm string holding meta data.
*/
typedef struct {
size_t start; //< byte-offset into `str` where this token starts. Must be exactly at a utf-8 codepoint boundary.
size_t len; //< `str` length of token in bytes.
size_t start; ///< byte-offset into `str` where this token starts. Must be exactly at a utf-8 codepoint boundary.
size_t len; ///< `str` length of token in bytes.
RzAsmTokenType type;
union {
ut64 number; //< Number of RZ_ASM_TOKEN_NUMBER
ut64 number; ///< Number of RZ_ASM_TOKEN_NUMBER
} val;
} RzAsmToken;
/**
* \brief An tokenized asm string.
* \brief A tokenized asm string.
*/
typedef struct {
ut32 op_type; ///< RzAnalysisOpType. Mnemonic color depends on this.
RzStrBuf *str; //< Contains the raw asm string
RzVector /*<RzAsmToken>*/ *tokens; //< Contains only the tokenization meta-info without strings, ordered by start for log2(n) access
RzStrBuf *str; ///< Contains the raw asm string
RzVector /*<RzAsmToken>*/ *tokens; ///< Contains only the tokenization meta-info without strings, ordered by start for log2(n) access
} RzAsmTokenString;
typedef struct {
@ -84,9 +84,9 @@ typedef struct {
* \brief Pattern for a asm string token.
*/
typedef struct {
RzAsmTokenType type; //< Asm token type.
char *pattern; //< The regex pattern describing the tokens.
RzRegex *regex; //< Compiled regex pattern.
RzAsmTokenType type; ///< Asm token type.
char *pattern; ///< The regex pattern describing the tokens.
RzRegex *regex; ///< Compiled regex pattern.
} RzAsmTokenPattern;
/**
@ -94,8 +94,8 @@ typedef struct {
*
*/
typedef struct {
bool reset_bg; // Reset the background color?
ut64 hl_addr; // Address which should be highlighted. Usually the function address.
bool reset_bg; ///< Reset the background color?
ut64 hl_addr; ///< Address which should be highlighted. Usually the function address.
} RzPrintAsmColorOpts;
typedef struct rz_print_zoom_t {