The `rzil:` line printed by `ao` (and its `aoX` standard-mode variants)
was emitted without syntax highlighting, unlike `plf`/`aoi` which color
the RzIL effect body. Route the stringified body through the existing
`rz_core_il_colorize_body()` helper (the same one used by `plf` and the
bit editor) when `scr.color` is enabled, so parentheses, IL operations,
numbers and variables are colored consistently.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* util/vector: hoist quicksort scratch buffers out of the recursion
vector_quick_sort allocated its two element-sized scratch buffers (t and
pivot) with malloc/free on every recursive call. For a vector of n elements
the sort makes O(n) recursive calls, i.e. O(n) malloc/free pairs purely for
scratch space, and each call could also fail half-way through the sort.
Split the function into a small entry point that allocates the two buffers
once and a recursive worker that receives them as scratch. The buffers are
reused across the whole recursion (each partition step finishes using them
before recursing, and the recursion is sequential, so sharing one pair is
safe). Small elements -- the common case, including every RzPVector-backed
sort -- use stack buffers and allocate nothing at all; only elements larger
than 256 bytes fall back to a single heap allocation for the whole sort.
The element movement and rand()-based pivot selection are unchanged, so the
result is identical for any input (verified byte-for-byte against the previous
implementation for ascending and descending orders over many random arrays).
* util/vector: evaluate the comparator once per element in the quicksort
The partition loop tested the element against the pivot with two separate
calls to the comparator:
if ((cmp(VEC_INDEX(a, i), pivot, user) < 0 && !reverse) ||
(cmp(VEC_INDEX(a, i), pivot, user) > 0 && reverse)) {
Because cmp is an opaque function pointer the compiler cannot common up the
two calls, so depending on the result and the reverse flag the comparator was
invoked up to twice per element. Compute the result once into a local and test
that:
int c = cmp(VEC_INDEX(a, i), pivot, user);
if ((c < 0 && !reverse) || (c > 0 && reverse)) {
This halves comparator calls in the worst case and is a clear win whenever the
comparator is non-trivial (the common case for struct elements). Measured on a
shared host: ~12-14% faster for int sorting and ~30% faster with a moderately
expensive comparator. The ordering is unchanged (verified byte-for-byte).
* util/vector: simplify rz_pvector_remove_data index computation
The index of the located slot was computed as
size_t index = (el - (void **)vec->v.a) * sizeof(void **) / vec->v.elem_size;
For an RzPVector the element size is always sizeof(void *), so the
`* sizeof(void **) / vec->v.elem_size` factor is identically 1 and the pointer
difference `el - (void **)vec->v.a` already yields the index directly. Drop the
redundant scaling, which removes a multiply and a divide and makes the intent
clear. Behaviour is unchanged.
* test/unit: add RzVector sort and rz_pvector_remove_data regression tests
The existing sort tests only sort 4-5 small elements and there was no test for
rz_pvector_remove_data. Add coverage for the code paths exercised by the sort
changes and the remove_data cleanup:
- test_vector_sort_large sort 2000 heavily-duplicated ut32 values
ascending and descending, verifying the result
is ordered and a permutation of the input (vs a
reference qsort). Drives the recursion deeply
and the shared scratch buffers.
- test_vector_sort_large_elem sort 400 elements of 304 bytes each, taking the
heap-allocated scratch fallback, and check the
full payload (not just the key) stays consistent
through all the element moves.
- test_pvector_remove_data remove interior, first and last elements by
value while preserving order, and confirm
removing an absent value is a no-op.
All pass on both the previous and the optimized implementation (the sort and
remove_data changes are behaviour-preserving).
* test/bench: benchmark rz_vector_sort and rz_pvector_sort
bench_vector.c benchmarked only remove_at and swap. Add sort benchmarks so the
suite covers the functions touched by the sort optimizations and can be run
against the old and new librz for before/after numbers:
- rz_vector_sort over 4k ut64 with a cheap comparator
- rz_vector_sort over 4k ut64 with a deliberately expensive comparator
(shows the effect of evaluating the comparator once per element)
- rz_pvector_sort over 4k pointers (reference; pvector sort is unchanged)
Each iteration refills the buffer from an unsorted master copy via a single
memcpy before sorting; that overhead is identical across builds so the measured
delta reflects the sort.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
bin/dyldcache: dyldcache_classes() built the classes vector with
rz_pvector_new(free) and the per-class methods/fields lists with
rz_list_new() (no element destructor). Each RzBinClass was therefore
plain free()'d without releasing its name, methods and fields, leaking
memory.
debug/dmp (winkd): rz_debug_dmp_init() obtained the module list from
winkd_list_modules() in the non-triage branch, scanned it for
ntoskrnl.exe and then never freed it, leaking the list and all of its
WindModule entries.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
MIPS PIC code calls a function by loading its address from the GOT into
$t9 and then doing `jalr t9` (or `jr t9` for a tail call), e.g.:
lw v0, -sym._MIPS_STUBS(gp) ; v0 = *(gp + %call16(puts))
move t9, v0
jalr t9 ; -> puts
rizin did not turn this into a call to the imported function, so the
target was lost: rz-ghidra rendered it as an indirect `(*_data.XXXX)()`
instead of `puts(...)`, whereas a direct `jal sym.dummy` decompiled fine.
Root cause (two independent gaps):
1. For (R|U)CALL and RJMP ops the core creates the CALL/CODE xref from
op->ptr, not op->jump (see core_analysis_followptr() and the op-type
switch in librz/core/canalysis.c, RZ_ANALYSIS_OP_TYPE_RCALL/RJMP). The
MIPS plugin only ever set op->jump for `jalr`/`jr`, so no call xref was
produced even when $t9 was tracked, and the decompiler never saw a call
target.
2. $t9 was only tracked when it was the *direct* destination of a
gp-relative load (`lw t9, ...(gp)`). The very common sequence that loads
into another register first and then `move t9, vX` was not tracked, so
even op->jump was left unset there (this is the issue's binary). Note
that capstone emits `move t9, vX` as the 2-operand alias of `or` (and on
some toolchains `addu`/`daddu`), i.e. `or t9, vX, $zero`, so the move
must be recognised across MIPS_INS_MOVE *and* the 2-operand OR/ADDU
forms.
Additionally, the tracked value was the GOT *slot* address, while the call
target is the function the slot points to, so the slot has to be
dereferenced.
This commit:
- tracks the destination register and slot of every gp-relative load
(gp_load_reg/gp_load_ptr in MIPSContext), and propagates it to $t9 on a
register move (MIPS_INS_MOVE, or the 2-operand OR/ADDU alias), so the
PIC sequence above is recognised;
- adds mips_pic_call_target(), which dereferences the GOT slot via the
analysis IO bind (honouring word size and endianness) to obtain the
actual callee;
- sets op->ptr (and op->jump) to that resolved address for `jalr t9`
(RCALL) and `jr t9` (RJMP tail call), so the core emits the proper
CALL/CODE xref and the decompiler resolves the callee like it does for
`jal`.
The resolution is best-effort and fully guarded: if gp is unknown, the
slot cannot be read, or it holds 0, op->ptr is left unset and behaviour is
exactly as before. The core additionally validates the target
(is_valid_xref) before creating the xref, so a stale/garbage slot cannot
introduce a bogus call, and the propagation only fires when $t9 is the
destination, so ordinary moves (`move fp, sp`, ...) are unaffected.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Use rz_read_le32() in the x86/x64 egg emitter so 4-byte string immediates are
no longer byte-reversed on big-endian hosts (System Z/s390); little-endian
output is unchanged. Regenerate the stale 32-bit x86 (#1889) and 32-bit arm
rz-gg goldens to match the tool's actual output, drop BROKEN markers from the
simple_cmp tests that already pass, and annotate the remaining broken tests
(no AArch64 egg backend; compiler-dependent C output) with the reason.
Closes#3486Closes#1889
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* hash: add chi-square (vs uniform) rz-hash plugin
Adds a chi-square goodness-of-fit (vs a uniform byte distribution)
statistic as an rz-hash plugin and the rz_hash_chisquare() API.
Unlike Shannon entropy, chi-square separates high-entropy data that is
truly uniform (encryption/CSPRNG, ~255) from high-entropy data that is
merely compressed or packed (much larger values), which is a common
question when triaging firmware blobs.
* hash: add index-of-coincidence rz-hash plugin
Adds the index of coincidence as an rz-hash plugin and the
rz_hash_ioc() API. IoC is ~1/256 for uniform data and markedly higher
for text, padding, single-byte-XOR'd data and repeating-key regions;
computed at several strides it is the Friedman/Kasiski test for a
repeating-XOR period.
* hash: add min-entropy rz-hash plugin
Adds min-entropy H_inf = -log2(max_i p_i) as an rz-hash plugin and the
rz_hash_min_entropy() API. This is the conservative worst-case entropy
used by NIST SP 800-90B: 8.0 for a uniform block, dropping as soon as a
single byte value dominates.
* hash: add serial-correlation rz-hash plugin
Adds the lag-1 serial correlation coefficient (with wrap-around, as in
the classic `ent` tool) as an rz-hash plugin and the
rz_hash_serial_correlation() API. Near 0 for compressed/encrypted data
but clearly non-zero for executable code, counters and gradients - the
order-aware axis that the histogram-only metrics cannot see.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
When a function is relocated, each variable access offset (stored relative
to the function's entry point) is rebased by the relocation delta so that the
accesses keep pointing at the same absolute addresses. This was done as a
signed st64 subtraction:
st64 delta = addr - fcn->addr;
...
acc->offset -= delta;
Relocating to an address near the int64 boundary makes delta close to
INT64_MIN, and 'acc->offset - delta' then overflows st64. UBSAN aborts:
librz/arch/function.c:241:16: runtime error: signed integer overflow:
65568 - -9223372036854710512 cannot be represented in type 'long int'
(reproducible via test/unit/test_analysis_var, which relocates to
0x8000000000000010 and 0x7ffffffffffffe00).
Addresses and their differences are meant to wrap modulo 2^64, and the
offsets are only ever looked up as exact values (get_vars_used_at computes
op_addr - fcn->addr in ut64 as well), so perform the arithmetic in ut64.
delta becomes ut64 and the rebase is '(st64)((ut64)acc->offset - delta)';
the result is bit-identical for every non-overflowing case and well-defined
for the rest. The inst_vars rebase callback already subtracted in ut64.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
This rewrites the look of the interactive horizontal histogram in Rizin -
the output of `p==v`, `p==ev`, `p==mv`, `p==0v`, `p==fv`, `p==pv`, `p==zv`,
`p==Sv` and the analysis-based interactive variants. It closes#6365 and
#4431 in full, and now sits on top of the recently merged static-histogram
PR (#6427) so it reuses the new RzHistogramOptions struct and its helpers.
Five sub-tasks from #6365:
- Context-aware vertical ruler with up to 5 anchor labels (top, bottom,
quarter, half, three-quarter) honouring opts->value_min..value_max,
value_precision, value_scale and value_unit. Reuses the static
helpers compute_ruler_gutter, render_ruler_gutter, label_value_at and
select_ruler_label_rows so visual and static stay consistent.
- X-axis byte-offset ruler at the bottom: `^` ticks every (addr_w + 3)
cols followed by absolute byte offsets computed from opts->offpos +
realj * blocksize.
- Cursor offset + percentage shown right-aligned on the status line, with
"Index N data V" on the left.
- Top minimap: Unicode block characters (▁▂▃▄▅▆▇█) for the density of
each slice of the whole data, plus a ┏━━━┓ window indicator showing the
visible slice. Always rendered when the new `scr.hist.minimap` config
is true. When the terminal is wide (hist->w > 200) the minimap shrinks
by 43 columns to make room for a two-line `px 0x20`-style hex preview
panel on the right showing the 32 bytes at the cursor's file offset.
Each byte is wrapped in its px-style colour code (b0x00 / b0x7f /
b0xff / btext / other) when opts->color is set, matching rz_print's
rz_print_byte_color exactly. A trailing safety pad keeps the canvas
from clipping the last byte of the first hex row.
- Missing-half bug on `p==v` when the cursor is at offset 0 fixed by
clamping `adder` to [0, histogramwidth - span]. The old expression
`barnumber + 1 - histogramwidth/(zoom*2)` was always negative for the
default barnumber=0, causing the rendering loop to read `data[-N]`
(segfault on large files, missing left half on small ones).
Cursor visibility:
- The cursor column is drawn as a CONTINUOUS vertical line connecting two
plain markers at the top and bottom, ALWAYS exactly one character wide.
The line itself uses the dedicated `wordhl` palette colour (default
red background, configurable via `ec wordhl ...`), drawn on every
chart row so the cursor is always a full-height vertical strip. The
markers (`▼` at the top and `▲` at the bottom, or ASCII `v` / `^`
when scr.utf8=false) are intentionally left un-highlighted so they
read as a clean pair of arrows pointing at the cursor column.
- The cursor screen column is computed up-front (j_cursor) by inverting
the data-to-column map (rel * zoom * width / histogramwidth). Two
distinct widening bugs are avoided this way:
1. sizeofonebar > 1 (high zoom) - each data index spans several
screen columns; only j == j_cursor && kbar == 0 renders as the
cursor, the remaining kbar columns fall through to the gradient.
2. histogramwidth < width (chart much wider than data) - several
adjacent screen columns map to the same data index via integer
truncation; only the j_cursor column may render as the cursor.
Interactive keybindings & live config:
- The `:` hotkey drops into rz_core_visual_prompt_input, matching the
rest of Rizin's visual modes. Lets the user run arbitrary rizin
commands without leaving the histogram.
- The `?` help text now uses the same colour-coded format as the
visual / visual-bit-editor modes (rz_core_visual_append_help with
pal.args for keys and pal.help for descriptions), shown via
rz_cons_less_str.
- The config-driven opts (scr.hist.minimap, scr.hist.block, scr.utf8,
scr.color, hex.offset) are re-read on every redraw via
refresh_visual_opts_from_config, so `:` `e scr.hist.minimap=true`
<Enter> takes effect immediately without having to quit and re-enter.
The canvas's `color` field is refreshed alongside so `scr.color`
changes take effect on the same redraw.
Hex preview panel:
- When the terminal is wide (hist->w > 200) and the minimap is enabled,
the visual mode shows a two-line hex preview on the right of the
minimap rows: 32 bytes at the cursor's file offset, formatted as 8
pairs of 2 bytes separated by spaces (`abcd ef00 1234 5678 ...`),
matching `px 0x20` minus the header / offsets / ASCII column. The
bytes are fetched live via rz_io_read_at_mapped each redraw, so
moving the cursor (`h` / `l`) updates the preview.
- Each byte gets its px-style colour code: green for 0x00, red for
0xff, yellow for 0x7f, btext (white) for printable ASCII, "other"
(magenta) for non-printable. Mirrors rz_print_byte_color so the
histogram preview reads consistently with `px`.
- A trailing safety pad keeps the canvas from clipping the last hex
byte of the first row (a side effect of UTF-8 minimap glyphs
interacting with the canvas's width tracking when the row fills
the canvas exactly).
- Implemented via two new fields on RzHistogramInteractive
(`cursor_bytes`, `cursor_bytes_len`) that the caller fills in just
before the render call and clears right after. The minimap helper
grows two extra parameters that the visual function passes through;
when the panel is disabled (narrow terminal, no cursor_bytes, or
shrinking the minimap would leave it < 40 cols) the helper falls
back to the previous full-width minimap.
Closes#4431 in full:
- The negative-offset crash above is the immediate segfault from the
bug report.
- `print_histogram_bytes` now samples one byte per block instead of
reading nblocks contiguous bytes from core->offset. For an 8 GB file
shown across 80 bars the original code rendered the first 80 bytes
of the file; the new code samples at offsets brange->from + i *
blocksize so the chart represents the full span.
- The inner `int i` in the column-aggregation loop is renamed to `k`
to drop the shadow over the outer `size_t i`.
Refactor on the cmd_print.c side:
- New default_visual_opts(core, offset) returns an RZ_OWN
RzHistogramOptions* pre-populated for the visual commands (ruler=true,
minimap from scr.hist.minimap, offpos from caller, palette and
screen-mode toggles from config via refresh_visual_opts_from_config).
The nine print_visual_bytes call sites now build opts via this helper,
then pass it to print_visual_bytes which takes ownership. Entropy
sets value_max=8 / value_precision=1 / data_f=fdata so the visual
histogram shows the Shannon range matching the static side.
- `print_visual_bytes(core, opts, data, brange)` now propagates opts
cleanup along every error path; rz_histogram_interactive_new no
longer leaves a heap-allocated opts pointer dangling. The redraw loop
fetches 32 cursor bytes via rz_io_read_at_mapped, points
hist->cursor_bytes at a stack buffer for the call, then NULLs it
back so the next iteration's fetch is independent.
- RzHistogramInteractive gains `blocksize`, `cursor_bytes` and
`cursor_bytes_len` fields.
New config option:
- `scr.hist.minimap` (bool, default true) controls whether the top
minimap is shown for p==v / p==ev. Surfaces as `opts->minimap` and
is honoured by `rz_histogram_interactive_horizontal`. When true,
the minimap is ALWAYS rendered (provided there's room) - even when
the chart already shows the full data, in which case the window
indicator spans the whole map. Changes via `:` `e scr.hist.minimap=...`
<Enter> are picked up on the very next redraw.
Tests (33 total, 10 new for the visual side):
- test_histogram_interactive_horizontal_basic - smoke test with
barnumber=0 (pins the #4431 crash regression).
- test_histogram_interactive_horizontal_ruler_percent - fractional
labels with value_max=100 / value_scale=0.01 / unit="%".
- test_histogram_interactive_horizontal_ruler_default - the legacy
0..255 byte ruler.
- test_histogram_interactive_horizontal_no_negative_adder - covers
`p==v` at offset 0 on a small data set.
- test_histogram_interactive_horizontal_percent - status-line percent
indicator present.
- test_histogram_interactive_horizontal_cursor_markers - the ▼/▲
cursor markers (and ASCII v/^ fallback) appear on the cursor column,
left un-highlighted.
- test_histogram_interactive_horizontal_cursor_full_line - the cursor
bar is rendered on every chart row between the markers regardless
of the data threshold (continuous vertical line).
- test_histogram_interactive_horizontal_cursor_width - pins single-
char width across BOTH cursor-widening bugs: high zoom (sizeofonebar
> 1) AND chart wider than data (histogramwidth < width).
- test_histogram_interactive_horizontal_minimap_toggle - pins the
scr.hist.minimap gating across {zoomed, not zoomed} when
opts->minimap=true / =false.
- test_histogram_interactive_horizontal_hex_preview - 4 cases: wide
terminal + cursor_bytes shows the hex panel; narrow terminal
suppresses it; missing cursor_bytes suppresses it; colour mode
emits ANSI escape sequences for the bytes.
Both p== integration tests in test/db/cmd/cmd_print pass with their
regenerated EXPECT blocks (the per-block sampling change moves the
visible bars for small buffers).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Follow-up to the C bitfield support added in #6439 (#1240, #314): the PDB
type parser dropped struct/union bitfield members because an LF_BITFIELD
member type resolved to NULL in pdb_type_parse(), so "tc"/"ts"/"tp" on a
PDB-loaded type lost every bitfield field.
member_parse() now detects a TpiKind_BITFIELD field type, resolves the
member to the bitfield's underlying integer (base_type) and reports the
bit width (length) via an out-parameter. class_member_parse() and
union_member_parse() store it in RzTypeStructMember.size /
RzTypeUnionMember.size (the bitfield width in bits, 0 if not a bitfield),
matching the convention used by the C and DWARF member paths.
Update the PDB type expectations that previously asserted bitfield_typedef
had no members: db/cmd/cmd_idp (idpij) and db/tools/rz_bin (rz-bin -Pj)
now expect the three resolved "unsigned char" members of minimal.pdb.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* librz/core,cmd: add pf-aware autocompletion arg types
The `pf` family of commands (`pf`, `pf-`, `pfa`, `pfc`, `pfd`, `pf.`,
`pfn`, `pfo`, `pfs`, `pfv`, `pfw`) takes either a registered named
format, a `<format>.<field>[<idx>]...` path, or a Format Definition
File. None of these were autocompletable: the arg type was always
`RZ_CMD_ARG_TYPE_STRING`, so tab on `pf. <TAB>` did nothing and the
user had to remember every format and field name by hand.
Add three new arg types and wire them to the cmd_descs:
* `RZ_CMD_ARG_TYPE_PF_FORMAT_NAME` enumerates the named formats
from `rz_type_db_format_all()` and filters by the partial prefix.
Used by `pf-`, `pfa`, `pfc`, `pfd`, `pfn`, `pfs`, `pfv`.
* `RZ_CMD_ARG_TYPE_PF_FORMAT_PATH` is the path-aware completer for
`pf.` and `pfw`. Both accept arbitrarily deep paths of the form
`name[.field[<idx>]?]*`, walking through nested struct fields
(e.g. `pf. troll.str[1].two` follows the same path syntax that
`pf_path_navigate` accepts at runtime). The completer:
- Finds the last `.` in the partial input; everything before
it (inclusive) is the committed path, what follows is the
segment being completed.
- Walks each committed `name[N]?` segment in turn, looking up
STRUCT fields' `type_name` in the typedb and re-parsing the
referenced format. Descent past a scalar or an inline struct
(no `type_name`) returns no options.
- Offers the resolved format's field names for the tail.
- Returns nothing when the tail contains `[` or `]` -- the
user is mid-index or mid-descent, where identifier
completion would produce a syntax error if accepted.
- Suppresses the trailing space after each successful
completion so `.` can be typed next without an inserted
space getting in the way of descent.
- Rewrites `res->start` past the last `.` so completion only
replaces the tail; the committed path stays put.
* `RZ_CMD_ARG_TYPE_PF_FDF_FILE` lists the basenames of FDFs found
in the user's home formats dir and the system formats dir,
mirroring the search order used by `pfo` itself; files present
in both locations are reported once via the same `HtSU` de-dup
used in `cmd_print_format_file_handler`. Used by `pfo`.
All three new types are added to `CD_ARG_LAST_TYPES` in
cmd_descs_util.py. The generator sets `RZ_CMD_ARG_FLAG_LAST` on the
final arg of a command whenever that arg's type is in this set;
that flag tells the runtime arg-preprocessor to merge any trailing
whitespace-separated tokens into a single argv slot. This is the
same implicit-FLAG_LAST treatment `RZ_CMD_ARG_TYPE_STRING` already
gets, and it is what keeps invocations like `pfc zd4x8 foo bar cow`
(five tokens, one logical format-with-names argument) working --
without it the cmd parser would reject the extra tokens with "Wrong
number of arguments". The implicit merge does not interfere with
autocompletion: the completer still receives the partial input up
to the cursor and prefix-matches against it, and the format-path
completer's dot/bracket scan is unaffected by whitespace.
The path completer uses a small helper, `pf_path_seg_consume`, that
parses one `name[N]?` segment with safe handling of unterminated `[`,
empty `[]`, non-numeric indices, and end-of-input. `pf_resolve_path_format`
walks all committed segments and returns the RzPfFormat the caller
should complete against; the resolver is the same shape as the
`pf_path_navigate` walker in librz/type/pf/pf_parser.c, except it
operates on RzPfFormat trees (typedb names) rather than RzPfValue
trees (decoded data), so it can run before any read has happened.
The completers live in `cautocmpl.c` next to the existing type-name
completers (`autocmplt_cmd_arg_struct_type` and friends) and follow
the same loop+strncmp pattern. The dispatcher entry for
`RZ_CMD_ARG_TYPE_FOLDER` was missing an explicit `break;` and would
fall through to `default`; harmless, but fixed in passing so the
three new cases sit cleanly above `default:`.
Note on dot-search direction: `rz_sub_str_rchr` is `start..end`
range search returning the FIRST hit, not a right-to-left "find
last" -- the `r` is for "range", not "right". The path completer
needs the last dot, so it walks the buffer backwards itself.
* test/integration: cover pf autocompletion
Thirteen new tests in test_autocmplt.c exercise the three new pf arg
types, including the multi-segment path resolver:
Format name completion:
* `pf_format_name` -- `pfn ut_<TAB>` after registering two formats
confirms both are offered.
Single-segment path completion:
* `pf_format_path` -- three-phase walk through `pf. ut_path<TAB>`,
`pf. ut_path.<TAB>`, `pf. ut_path.cou<TAB>`, covering name-only,
dot-only, and dot-with-prefix. Verifies that `res->end_string`
is empty in the name phase (so `.` can be typed next without an
inserted space) and that `res->start` advances past the dot in
the field phase (so the completion only replaces the field
portion).
* `pfw_format_path` -- the same `<format>.<field>` syntax must
work on the write side too; confirms the PATH completer fires
for `pfw` and is not pf.-specific.
* `pf_format_path_empty` -- bare `pf. <TAB>` lists every
registered format. Snapshots the baseline count first so the
assertion stays robust against any default formats the type DB
might seed.
* `pf_format_path_unknown_name` -- `pf. nonexistent.<TAB>`
returns an empty option list rather than crashing or leaking
diagnostics.
* `pf_format_path_anon_field` -- formats whose fields don't all
have names (e.g. a `.` skip slot) must be iterated safely; the
named fields are offered and the anonymous slot is silently
dropped.
Multi-segment / nested-struct path completion:
* `pf_format_path_nested` -- two-level descent through a STRUCT
field whose `type_name` references another registered format,
parsing the child format and offering its fields.
* `pf_format_path_three_levels` -- three-level descent narrows
correctly: A -> B -> C, then filter C's fields by a prefix.
* `pf_format_path_array_index` -- `pf. troll.str[1].<TAB>` mirrors
the existing cmd_pf2 runtime test; the array index in the
middle segment is parsed and skipped (it doesn't change the
target type).
* `pf_format_path_inside_brackets` -- cursor inside an
unclosed `[` returns no options (mid-index).
* `pf_format_path_after_close_bracket` -- cursor right after `]`
without a trailing `.` also returns no options (mid-descent).
* `pf_format_path_through_scalar` -- descent past a scalar field
is meaningless and returns no options.
The tests use plain `rz_core_new()` (the real cmd_descs already
registers all `pf*` commands), matching the pattern used by
`test_autocmplt_eco_themes`. Format strings use the parser's
"specifier-then-names" form (no internal whitespace in the spec
region) so `rz_pf_parse` produces the expected field count.
* doc,librz/core: align pf docs and `pf?` help with the parser
The standalone reference doc/pf.md and the in-tree `pf?` help (driven
by the details: block in librz/core/cmd_descs/cmd_print.yaml) had
drifted from each other and from what the parser actually accepts.
Both are now consistent with librz/type/pf/pf_parser.c.
Specific corrections:
* `n` family. doc/pf.md claimed `N1`/`N2`/`N4`/`N8` existed as BE
counterparts to `n1`-`n8`, and that bare `n`/`N` defaulted to
`ctx.bits/8`. The parser handles only `n{1,2,4,8}`; all four
forms are context-endian (follow `ctx->big_endian`), and bare
`n` produces "unknown specifier". Rewrite the section to match,
and explain why context-endian is the right choice for header
readers like ELF.
* Deprecation list. The `pf?` "deprecation" note listed `c, s, z`
among the deprecated bare-letter codes -- they are not deprecated
(`c` is the current 1-byte-as-char specifier, `s` is the current
pointer-to-zstring, `z` is the current inline zstring). It was
missing `C, i, Z, X, F, T` which the parser does warn on.
doc/pf.md had `c` in its table marked "unchanged" (so it was
visibly inconsistent with itself) and was missing the `x` row.
Both lists now mirror the parser's PF_DIAG(DEPRECATED) call
sites: b, C, d, f, F, i, o, q, t, T, w, x, X, Z.
* TLV `h=`. `pf?` said "h=v/a (header inclusion)" (two options);
the parser accepts `v` (value only, default), `l` (length covers
len+value), and `a` (length covers tag+len+value). doc/pf.md
already listed all three; help now matches.
* `v(N)` bitvector. doc/pf.md documented this in detail (the
1..4096-bit-wide field type used for things like ELF
`DT_FLAGS_1`, PE characteristics, page-allocation maps), but the
`pf?` help didn't mention it at all. Added to the DSL extensions
section.
* Pointer widths. doc/pf.md documented `p2`/`p4`/`p8` explicitly;
`pf?` only mentioned bare `p` with a "size from ctx.bits"
parenthetical. Help now lists the four forms in one entry.
* GUID layouts. Both docs claimed `G(le)` was "all little-endian",
but the renderer treats `G(le)` and `G(ms)` identically -- D4
(the trailing 8 bytes) is always in buffer order regardless of
layout. Document the actual behaviour rather than the implied
one; this is a description fix, not a code change. Anyone who
wants the byte-reversed-D4 reading can still file it as a
follow-up bug against the renderer in pf_render.c.
cmd_descs.[ch] is regenerated automatically by the custom_target rule
when cmd_print.yaml changes; the .c diff in this commit is the result
of that regeneration (5 lines of comment text inside the existing
detail entries).
Expose the typeclasses of types through the new `tk` commands:
- `tk <type>` shows the typeclass of a type;
- `tkl` lists the available typeclasses, while `tkll` (verbose), `tklt`
(table) and `tklj` (JSON) additionally list the types belonging to each
typeclass;
- `tks <type> <typeclass>` sets the typeclass of a type.
A new rz_base_type_set_typeclass() API backs the `tks` command. Since a
typedef without its own typeclass inherits the one of the type it points
to, setting the typeclass of an atomic type is automatically reflected on
the typedefs resolving to it.
The `tk` help also explains what typeclasses are and lists the available
ones.
Closes#3371
An enum type hint set on an operand (ahie) only replaced the immediate when its
value matched an enum member exactly, via rz_type_db_enum_member_by_val(). A
value that is the bitwise OR of several flag members -- e.g. the access(2) mode
R_OK|W_OK == 6 from the issue -- matched no single member and was left as a raw
number.
replace_enum_hint() in rz_parse now falls back to rz_type_db_enum_get_bitfield()
when there is no exact member, so the value is rendered as the OR of the
matching members, e.g. "access_def.W_OK | access_def.R_OK".
rz_type_db_enum_get_bitfield() is reworked along the way: it was unused and
buggy (capped at 32 bits, reused a stale match for bits without a member, and
emitted a "0x.. : " debug prefix). It now walks all 64 bits, returns the
matching members qualified with the enum name and joined by " | ", and returns
NULL when the value is 0, the type is not an enum, or any set bit has no member
(so a value that is not cleanly a combination of flags stays a number).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* Fix revert if adding a node failed.
* Reduce allocations by keeping only a single RzGraphEdge object per edge around.
* Add benchmark for graph deletion and addition of nodes/edges
* Use rz_pvector_remove_at_unsorted to save some runtime.
* Use realloc and memmove for matrix graphs on capacity increase.
* Add helper to determine memory usage.
* Add benchmark
* Revert matrix capacity extension to simple and jsut as fast loop.
* Missing type annotations
Introduce rz_type_db_rename_base_type() and expose it through the new
`tr` command to rename a base type. In addition to renaming the type
itself, every reference to it is updated so the analysis state stays
consistent after the rename:
- other base types and function types (callables) that use the type,
including self-references such as a linked-list struct that contains a
pointer to itself;
- the `pf` formats that mention the type, both the format stored under
the type's own name and the "(name)" references inside other stored
formats;
- the types of analysis global variables (as reported by `avg`), the
function signatures (return type and arguments) and the function local
variables.
The whole update is orchestrated by rz_core_types_rename(), and the
recursive RzType reference renamer is exposed as
rz_type_rename_references() so it can be reused for type usages that live
outside the type database.
Closes#1078
The C grammar already exposes a struct/union member's bitfield width as a
"bitfield_clause" node and the parser already recognized "int a : 4;", but
it threw the width away (the member was stored with size 0 and a FIXME). As
a result "tc" rendered the member as a plain "int a;", "ts" produced a
full-int "pf" format ("pf d4d4d4 a b c") and "tp" mis-read every field as a
whole integer.
RzTypeStructMember and RzTypeUnionMember already carry a "size" field
documented as "in bits"; it is now used as the bitfield width, where 0 means
the member is not a bitfield:
- The struct and union member parsers store the parsed bit width on the
member instead of discarding it.
- The pretty printer emits " : <width>" before the member's trailing ';',
so "tc"/"tcd" round-trip "struct qwe { int a : 4; int b : 16; int c : 3; }".
- rz_base_type_as_format() (used by "ts"/"tp") now emits the pf packed-bits
spec ":N" for a bitfield member, with the bit order matching the target
endianness ("<" little-endian / ">" big-endian), instead of the member
type's full format. "ts qwe" becomes 'pf ":4<:16<:3< a b c"' and "tp"
unpacks the fields correctly (a=5, b=0x1234, c=0 for 0x00012345 on x86).
Serialization keeps the width in the 3rd comma-field of the existing
"struct.<name>.<member>" / "union.<name>.<member>" value, i.e.
"type,offset,bitsize". That field was already written (always as 0) and
ignored on load, so the on-disk layout is unchanged and only its meaning is
now honored. The project version is bumped to 24 with an additive no-op
migration: an absent or 0 bitsize deserializes as a regular, non-bitfield
member.
Tests: a unit test parses a bitfield struct and union and checks the member
widths, the rendered string and the "pf" format; a db/cmd/types test covers
the full "td"/"tcd"/"ts"/"tp" flow; a project-migration test with a new
v23-bitfield.rzdb fixture covers v23->v24, and the "migrate info" db test is
updated for the new version.
The PDB type parser (librz/arch/pdb_process.c) still drops bitfield members
because their LF_BITFIELD member type resolves to NULL; wiring LF_BITFIELD up
to the new member "size" field, with tests in test/unit/test_pdb.c, is left
as a follow-up.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* Update Capstone and add m68k ColdFire support
* Fix test 'core regs linux m68k'
* Add M68K ColdFire integration coverage
* Update Capstone v6 to version 6.0.0-Alpha9
* Refactor CPU mode detection to use case-insensitive string comparison
* Update Capstone next revision to 3df6ff0
The ELF section classifier in sections_obj() only marked a section as
data when its name contained "data". As a result .dynstr (type
SHT_STRTAB, SHF_ALLOC) was flagged neither as data nor as containing
strings, so is_data_section() rejected it. Two problems followed:
- the string search (iz, and the default AUTO scan used by -A) skipped
.dynstr, so its strings were never listed (only izz, which scans the
whole file regardless of section flags, showed them);
- because no RZ_META_TYPE_STRING metadata was applied over the region,
the disassembler rendered the NUL-terminated symbol names as code,
e.g. on MIPS "_GLOBAL_OFFSET_TABLE_" decoded to bgtzl/ldr/jalx/...
Mark a section as containing strings when it is mapped into memory
(SHF_ALLOC) and is either a string table (SHT_STRTAB) or carries the
explicit SHF_STRINGS flag. The SHF_ALLOC restriction keeps the loaded
string tables (.dynstr) while leaving the non-allocated .strtab and
.shstrtab to izz, matching the "iz lists the loaded image" semantics.
This is architecture independent; MIPS was simply where the bad
disassembly was first noticed.
Closes#5182
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
C23 lets an enum fix its underlying type, e.g. "enum E : long long { ... }".
The C grammar already exposes it as the "underlying_type" field of an
enum_specifier, and RzBaseType already has a "type" slot documented as
used by enums, but the parser ignored the field and always left it NULL.
- parse_enum_node() now reads the "underlying_type" field and stores the
parsed type on RzBaseType::type (reusing parse_type_node_single(), so
primitive, sized and typedef'd integer types are all handled). Classic
enums keep a NULL underlying type.
- The pretty printer emits " : <type>" between the enum name and its body
when an underlying type is present, so "tc"/"tcd"/"tec" round-trip it.
- enum_bitsize() now derives the width from the underlying type instead of
the hardcoded 32-bit default (resolving the long-standing FIXME); it
still falls back to 32 for a classic enum.
Single-token underlying types (int, uint64_t, char, ...) work end-to-end
with the bundled grammar revision: the existing "enhanced enum" db test is
updated to round-trip "enum v : int" and a unit test covers
"enum EU : uint64_t". Multi-word underlying types (long long, unsigned int)
are added as BROKEN db tests; the bundled grammar parses them to an ERROR
node and drops the underlying type, so these tests fail for now and will
pass once rizin-grammar-c accepts sized type specifiers as the enum
underlying type. No further rizin change is needed for that step.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Close two long-standing issues against the static horizontal histogram
(`p==`, `p==e`, `p==0` and the rest of the `p==X` family). Add the
`scr.hist.width` / `scr.hist.height` config options (closes#6372),
both clamped via RZ_MIN() against the current terminal size with a
defensive sanity floor. Make the Y-axis ruler context-aware (closes
#5290) via new `value_min` / `value_max` / `value_unit` fields on
`RzHistogramOptions` and rescale each datum from its ut8 storage into
the vmin..vmax display range before thresholding, which also fixes
cer-0's follow-up about the chart's top rows staying blank for
low-range data.
Tighten the look in the same step. Y-axis labels are now sparse (top,
~25 %, ~50 %, ~75 %, bottom) instead of one per row, in the style of
tokio-console / Granite, with an inclusive label range (top = vmax,
bottom = vmin) decoupled from the threshold formula so the existing
`_` baseline behaviour still appears. The bottom of the chart grows
an X-axis ruler with `^` tick markers and `0x...` start / middle /
end offsets (driven by a new `opts->blocksize` field). Two more
options, `value_scale` and `value_precision`, let callers format
ruler labels as fractional values; the `p==e` (entropy) command opts
in with `value_max = 1, value_precision = 2` so the ruler reads
0.00..1.00 by default, matching Shannon-entropy convention.
A new `opts->data_f` field lets callers feed double-precision data
directly into the renderer (and engages double-precision arithmetic
for the row thresholds), so entropy keeps full precision end-to-end
instead of losing ~1/255 of resolution to the `(ut8)(255 * fraction)`
quantisation step. This captures the precision win sketched in PR
#6355 without introducing a separate `rz_histogram_horizontal_f64()`
twin function. Ten unit tests in `test/unit/test_cons_histogram.c`
(seven new) cover sparse labels, fractional labels, the X-axis offset
ruler, the cer-0 regression, `opts->cols`, the legacy 0..255 ruler,
and the fp data path; all pass.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
With bin.demangle=true the call/jump operand is replaced by the full
demangled symbol name and then colorized token-by-token before being
filtered. For long C++ symbols (e.g. libc++ STL names) the colorized
operand easily exceeds the 1024-byte operand buffer (ds->str): in
truecolor each token is wrapped in a ~19-byte "\x1b[38;2;R;G;Bm" escape,
so a ~110-char name grows past 1700 bytes.
The generic filter() copied that operand with
strncpy(str, data, len);
which has two problems when strlen(data) >= len:
- it does not NUL-terminate, so reading str ran past the buffer into
the adjacent strsub[] buffer, printing stale bytes (the "call 0x..."
bleed); and
- the byte-bounded cut can land in the middle of a color escape,
leaving a partial "\x1b[38;2;.." that, followed by the bled bytes,
forms a complete but bogus "\x1b[..c" sequence. Terminals read that
as a Device-Attributes query and reply with e.g. "62;4c", which is
the garbage the reporter saw at the prompt.
This only triggers with color enabled and a symbol long enough that its
colorized form overflows the buffer, which is why it showed up on some
x86 files only and had no portable reproducer (no-color = 149 bytes,
16-color = 763, 256-color = 1170, truecolor = 1713 for the sample
symbol; the threshold is 1024).
Add test/unit/test_parse.c driving rz_parse_filter() with over-long
colored operands (a deterministic case whose cut lands inside an escape,
and a realistic libc++-style colorized operand) plus a short-operand
passthrough case. The two overflow tests fail on the old strncpy and
pass with the fix.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
format.c can already turn an RzType into a pf format string. Add the
inverse in the same file, next to its counterparts:
rz_type_format_to_c_declaration() parses a pf format string with
rz_pf_parse() and emits an equivalent C struct/union declaration built
from the standard fixed-width types (uint8_t, int32_t, float, ...).
The conversion is structural: it consumes only the parsed RzPfFormat
(field kinds, widths, array counts, pointer flags), never a byte buffer,
so it runs without any target data. Every field kind the engine produces
is mapped; a handful of specifiers with no exact static C form are mapped
best-effort (documented inline): @N alignment is dropped, an
unknown-length inline z string becomes char *, LEB128 widens to its
largest decoded integer, and ?(Name) / E(Name) emit struct/enum
references that must themselves be defined to parse.
Added new 'tdf' command that exposes that conversion to the user
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Most of the Windows types reported in issue #3728 were stub entries of
the form
NAME=type
type.NAME=p
i.e. opaque atomic pointers with no actual body. rz-ghidra rejects
these as 'Invalid atomic type'. Convert them to proper typedef entries
backed by either real struct definitions (for well-known types) or
forward-declared opaque structs (for architecture-specific ones like
CONTEXT and KNONVOLATILE_CONTEXT_POINTERS, whose layout varies across
x86/x64/ARM/ARM64 and which the shared types-windows.sdb.txt cannot
commit to).
Specifically, this commit:
* converts the stubs for PSID, LPOLESTR, LPCLSID, PCONTEXT,
PKNONVOLATILE_CONTEXT_POINTERS, LPSYSTEM_INFO, LPWIN32_FIND_DATAW,
PCACTCTXW, PSID_AND_ATTRIBUTES, PLUID_AND_ATTRIBUTES, PTRACEHANDLE
and the duplicate LPMSG/PLUID into proper typedefs;
* adds new top-level typedefs for CLSID, OLECHAR, LPCOLESTR
(corrected to const OLECHAR *), REFCLSID, LPBC, MSG, PMSG, NPMSG,
LPMSG, SYSTEM_INFO, WIN32_FIND_DATAW, PWIN32_FIND_DATAW,
OSVERSIONINFOW, POSVERSIONINFOW, LPOSVERSIONINFOW,
RTL_OSVERSIONINFOW, PRTL_OSVERSIONINFOW, OSVERSIONINFOEXW (plus its
POSVERSIONINFOEXW/LPOSVERSIONINFOEXW siblings), ACTCTXW, PACTCTXW,
LUID, PLUID, LUID_AND_ATTRIBUTES and SID_AND_ATTRIBUTES;
* adds struct bodies for tagMSG, _OSVERSIONINFOW, _OSVERSIONINFOEXW,
_SYSTEM_INFO (with the union flattened to wProcessorArchitecture +
wReserved per modern usage), _WIN32_FIND_DATAW, tagACTCTXW,
_SID_AND_ATTRIBUTES, _LUID, _LUID_AND_ATTRIBUTES and the empty
opaque tags _CONTEXT, _KNONVOLATILE_CONTEXT_POINTERS and _IBindCtx;
* fixes a small typo in struct._EVENT_DATA_DESCRIPTOR where the
field list said 'Usize' while the per-field entry was 'Size'.
Member layouts were taken from the Microsoft Win32 documentation
(winnt.h, sysinfoapi.h, winuser.h, evntrace.h, minwinbase.h,
winbase.h) and cross-checked with the Wine include/winnt.h,
wtypes.idl and guiddef.h headers.
Offsets follow the 32-bit Windows ABI used by the rest of this file
(this matches the existing _OBJECT_ATTRIBUTES, _FILETIME etc. layouts).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
