rz_reg_profile_to_cc() only emitted the first four argument registers
(A0-A3), so architectures that pass more arguments in registers -- the
C6000 EABI uses ten, and x86-64/riscv/ppc all declare more than four --
got a truncated convention. Walk the whole A0-A9 role range, stopping at
the first role the profile leaves undefined, and build the cc string with
RzStrBuf. Covered by a new test_reg unit test.
Co-authored-by agent: Claude/claude-opus-4-8
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Warning: this also swaps the arguments of the old rz_bv_append() to be
consistend with the new inplace variant.
The reason why the inplace function has the low as the first operand is
that it can be more efficient to append to an existing vector inplace
than to prepend to it. Then, the first argument is being used as the
in-out one in all other inplace functions.
- Introduced md1img.h and md1img.c for parsing MediaTek md1img container format.
- Implemented mtk.h and mtk.c for parsing MediaTek GFH firmware images (md1rom).
- Added plugin support for md1img and mtk formats in bin_md1img.c and bin_mtk.c.
- Updated meson.build to include new source files and plugins.
- Enhanced RzBuffer utility with LZMA alone decompression support.
---------
Co-authored-by: Giovanni <561184+wargio@users.noreply.github.com>
Squash of rizinorg/rizin PR #6434 ("improve TMS320C55x+ analysis and RzIL")
rebased onto dev, with the PR head's doubled c55plus_il.c (every symbol defined
twice, failing to compile) de-duplicated to a single clean copy.
Substantially extends the C55x/C55x+ RzIL lifter over the existing structured-
operand helpers: mov/copy (immediate, register, memory load/store, half-register
read-modify-write), the full addressing-mode set with post-modify side effects,
control and system-register moves, 40-bit accumulator ALU with shifted sources,
16-bit and dual-memory add/sub, the ST0_55 status-flag model (cmp/cmpand/rol/ror,
named-bit bset/bclr), a documented psh/pop stack model, address-unit amov/aadd/
asub and amar, and bcc/callcc control transfer. The multiply/MAC family and satr
are lifted with explicit, documented integer-mode approximations (not verified
DSP semantics); irreducibly multi-output primitives (bit counts, Viterbi, FIR,
distance) are left correct-or-NULL. The register file covers ac0-7 and xar, found
by validating on real Motorola Wrigley C55x+ firmware whose prologues save 40-bit
accumulators as dbl(acN)+acN.g pairs.
Adds RzIL-VM emulation tests (including the C55x and C55x+ _decrypt emulateme
binaries), per-instruction IL assertions, and ~95% instruction-class disassembly
coverage per corpus; pins little-endian in the VM tests for big-endian hosts; and
regenerates the analysis expectations against current dev.
* util/vector: hoist quicksort scratch buffers out of the recursion
vector_quick_sort allocated its two element-sized scratch buffers (t and
pivot) with malloc/free on every recursive call. For a vector of n elements
the sort makes O(n) recursive calls, i.e. O(n) malloc/free pairs purely for
scratch space, and each call could also fail half-way through the sort.
Split the function into a small entry point that allocates the two buffers
once and a recursive worker that receives them as scratch. The buffers are
reused across the whole recursion (each partition step finishes using them
before recursing, and the recursion is sequential, so sharing one pair is
safe). Small elements -- the common case, including every RzPVector-backed
sort -- use stack buffers and allocate nothing at all; only elements larger
than 256 bytes fall back to a single heap allocation for the whole sort.
The element movement and rand()-based pivot selection are unchanged, so the
result is identical for any input (verified byte-for-byte against the previous
implementation for ascending and descending orders over many random arrays).
* util/vector: evaluate the comparator once per element in the quicksort
The partition loop tested the element against the pivot with two separate
calls to the comparator:
if ((cmp(VEC_INDEX(a, i), pivot, user) < 0 && !reverse) ||
(cmp(VEC_INDEX(a, i), pivot, user) > 0 && reverse)) {
Because cmp is an opaque function pointer the compiler cannot common up the
two calls, so depending on the result and the reverse flag the comparator was
invoked up to twice per element. Compute the result once into a local and test
that:
int c = cmp(VEC_INDEX(a, i), pivot, user);
if ((c < 0 && !reverse) || (c > 0 && reverse)) {
This halves comparator calls in the worst case and is a clear win whenever the
comparator is non-trivial (the common case for struct elements). Measured on a
shared host: ~12-14% faster for int sorting and ~30% faster with a moderately
expensive comparator. The ordering is unchanged (verified byte-for-byte).
* util/vector: simplify rz_pvector_remove_data index computation
The index of the located slot was computed as
size_t index = (el - (void **)vec->v.a) * sizeof(void **) / vec->v.elem_size;
For an RzPVector the element size is always sizeof(void *), so the
`* sizeof(void **) / vec->v.elem_size` factor is identically 1 and the pointer
difference `el - (void **)vec->v.a` already yields the index directly. Drop the
redundant scaling, which removes a multiply and a divide and makes the intent
clear. Behaviour is unchanged.
* test/unit: add RzVector sort and rz_pvector_remove_data regression tests
The existing sort tests only sort 4-5 small elements and there was no test for
rz_pvector_remove_data. Add coverage for the code paths exercised by the sort
changes and the remove_data cleanup:
- test_vector_sort_large sort 2000 heavily-duplicated ut32 values
ascending and descending, verifying the result
is ordered and a permutation of the input (vs a
reference qsort). Drives the recursion deeply
and the shared scratch buffers.
- test_vector_sort_large_elem sort 400 elements of 304 bytes each, taking the
heap-allocated scratch fallback, and check the
full payload (not just the key) stays consistent
through all the element moves.
- test_pvector_remove_data remove interior, first and last elements by
value while preserving order, and confirm
removing an absent value is a no-op.
All pass on both the previous and the optimized implementation (the sort and
remove_data changes are behaviour-preserving).
* test/bench: benchmark rz_vector_sort and rz_pvector_sort
bench_vector.c benchmarked only remove_at and swap. Add sort benchmarks so the
suite covers the functions touched by the sort optimizations and can be run
against the old and new librz for before/after numbers:
- rz_vector_sort over 4k ut64 with a cheap comparator
- rz_vector_sort over 4k ut64 with a deliberately expensive comparator
(shows the effect of evaluating the comparator once per element)
- rz_pvector_sort over 4k pointers (reference; pvector sort is unchanged)
Each iteration refills the buffer from an unsorted master copy via a single
memcpy before sorting; that overhead is identical across builds so the measured
delta reflects the sort.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* hash: add chi-square (vs uniform) rz-hash plugin
Adds a chi-square goodness-of-fit (vs a uniform byte distribution)
statistic as an rz-hash plugin and the rz_hash_chisquare() API.
Unlike Shannon entropy, chi-square separates high-entropy data that is
truly uniform (encryption/CSPRNG, ~255) from high-entropy data that is
merely compressed or packed (much larger values), which is a common
question when triaging firmware blobs.
* hash: add index-of-coincidence rz-hash plugin
Adds the index of coincidence as an rz-hash plugin and the
rz_hash_ioc() API. IoC is ~1/256 for uniform data and markedly higher
for text, padding, single-byte-XOR'd data and repeating-key regions;
computed at several strides it is the Friedman/Kasiski test for a
repeating-XOR period.
* hash: add min-entropy rz-hash plugin
Adds min-entropy H_inf = -log2(max_i p_i) as an rz-hash plugin and the
rz_hash_min_entropy() API. This is the conservative worst-case entropy
used by NIST SP 800-90B: 8.0 for a uniform block, dropping as soon as a
single byte value dominates.
* hash: add serial-correlation rz-hash plugin
Adds the lag-1 serial correlation coefficient (with wrap-around, as in
the classic `ent` tool) as an rz-hash plugin and the
rz_hash_serial_correlation() API. Near 0 for compressed/encrypted data
but clearly non-zero for executable code, counters and gradients - the
order-aware axis that the histogram-only metrics cannot see.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
This rewrites the look of the interactive horizontal histogram in Rizin -
the output of `p==v`, `p==ev`, `p==mv`, `p==0v`, `p==fv`, `p==pv`, `p==zv`,
`p==Sv` and the analysis-based interactive variants. It closes#6365 and
#4431 in full, and now sits on top of the recently merged static-histogram
PR (#6427) so it reuses the new RzHistogramOptions struct and its helpers.
Five sub-tasks from #6365:
- Context-aware vertical ruler with up to 5 anchor labels (top, bottom,
quarter, half, three-quarter) honouring opts->value_min..value_max,
value_precision, value_scale and value_unit. Reuses the static
helpers compute_ruler_gutter, render_ruler_gutter, label_value_at and
select_ruler_label_rows so visual and static stay consistent.
- X-axis byte-offset ruler at the bottom: `^` ticks every (addr_w + 3)
cols followed by absolute byte offsets computed from opts->offpos +
realj * blocksize.
- Cursor offset + percentage shown right-aligned on the status line, with
"Index N data V" on the left.
- Top minimap: Unicode block characters (▁▂▃▄▅▆▇█) for the density of
each slice of the whole data, plus a ┏━━━┓ window indicator showing the
visible slice. Always rendered when the new `scr.hist.minimap` config
is true. When the terminal is wide (hist->w > 200) the minimap shrinks
by 43 columns to make room for a two-line `px 0x20`-style hex preview
panel on the right showing the 32 bytes at the cursor's file offset.
Each byte is wrapped in its px-style colour code (b0x00 / b0x7f /
b0xff / btext / other) when opts->color is set, matching rz_print's
rz_print_byte_color exactly. A trailing safety pad keeps the canvas
from clipping the last byte of the first hex row.
- Missing-half bug on `p==v` when the cursor is at offset 0 fixed by
clamping `adder` to [0, histogramwidth - span]. The old expression
`barnumber + 1 - histogramwidth/(zoom*2)` was always negative for the
default barnumber=0, causing the rendering loop to read `data[-N]`
(segfault on large files, missing left half on small ones).
Cursor visibility:
- The cursor column is drawn as a CONTINUOUS vertical line connecting two
plain markers at the top and bottom, ALWAYS exactly one character wide.
The line itself uses the dedicated `wordhl` palette colour (default
red background, configurable via `ec wordhl ...`), drawn on every
chart row so the cursor is always a full-height vertical strip. The
markers (`▼` at the top and `▲` at the bottom, or ASCII `v` / `^`
when scr.utf8=false) are intentionally left un-highlighted so they
read as a clean pair of arrows pointing at the cursor column.
- The cursor screen column is computed up-front (j_cursor) by inverting
the data-to-column map (rel * zoom * width / histogramwidth). Two
distinct widening bugs are avoided this way:
1. sizeofonebar > 1 (high zoom) - each data index spans several
screen columns; only j == j_cursor && kbar == 0 renders as the
cursor, the remaining kbar columns fall through to the gradient.
2. histogramwidth < width (chart much wider than data) - several
adjacent screen columns map to the same data index via integer
truncation; only the j_cursor column may render as the cursor.
Interactive keybindings & live config:
- The `:` hotkey drops into rz_core_visual_prompt_input, matching the
rest of Rizin's visual modes. Lets the user run arbitrary rizin
commands without leaving the histogram.
- The `?` help text now uses the same colour-coded format as the
visual / visual-bit-editor modes (rz_core_visual_append_help with
pal.args for keys and pal.help for descriptions), shown via
rz_cons_less_str.
- The config-driven opts (scr.hist.minimap, scr.hist.block, scr.utf8,
scr.color, hex.offset) are re-read on every redraw via
refresh_visual_opts_from_config, so `:` `e scr.hist.minimap=true`
<Enter> takes effect immediately without having to quit and re-enter.
The canvas's `color` field is refreshed alongside so `scr.color`
changes take effect on the same redraw.
Hex preview panel:
- When the terminal is wide (hist->w > 200) and the minimap is enabled,
the visual mode shows a two-line hex preview on the right of the
minimap rows: 32 bytes at the cursor's file offset, formatted as 8
pairs of 2 bytes separated by spaces (`abcd ef00 1234 5678 ...`),
matching `px 0x20` minus the header / offsets / ASCII column. The
bytes are fetched live via rz_io_read_at_mapped each redraw, so
moving the cursor (`h` / `l`) updates the preview.
- Each byte gets its px-style colour code: green for 0x00, red for
0xff, yellow for 0x7f, btext (white) for printable ASCII, "other"
(magenta) for non-printable. Mirrors rz_print_byte_color so the
histogram preview reads consistently with `px`.
- A trailing safety pad keeps the canvas from clipping the last hex
byte of the first row (a side effect of UTF-8 minimap glyphs
interacting with the canvas's width tracking when the row fills
the canvas exactly).
- Implemented via two new fields on RzHistogramInteractive
(`cursor_bytes`, `cursor_bytes_len`) that the caller fills in just
before the render call and clears right after. The minimap helper
grows two extra parameters that the visual function passes through;
when the panel is disabled (narrow terminal, no cursor_bytes, or
shrinking the minimap would leave it < 40 cols) the helper falls
back to the previous full-width minimap.
Closes#4431 in full:
- The negative-offset crash above is the immediate segfault from the
bug report.
- `print_histogram_bytes` now samples one byte per block instead of
reading nblocks contiguous bytes from core->offset. For an 8 GB file
shown across 80 bars the original code rendered the first 80 bytes
of the file; the new code samples at offsets brange->from + i *
blocksize so the chart represents the full span.
- The inner `int i` in the column-aggregation loop is renamed to `k`
to drop the shadow over the outer `size_t i`.
Refactor on the cmd_print.c side:
- New default_visual_opts(core, offset) returns an RZ_OWN
RzHistogramOptions* pre-populated for the visual commands (ruler=true,
minimap from scr.hist.minimap, offpos from caller, palette and
screen-mode toggles from config via refresh_visual_opts_from_config).
The nine print_visual_bytes call sites now build opts via this helper,
then pass it to print_visual_bytes which takes ownership. Entropy
sets value_max=8 / value_precision=1 / data_f=fdata so the visual
histogram shows the Shannon range matching the static side.
- `print_visual_bytes(core, opts, data, brange)` now propagates opts
cleanup along every error path; rz_histogram_interactive_new no
longer leaves a heap-allocated opts pointer dangling. The redraw loop
fetches 32 cursor bytes via rz_io_read_at_mapped, points
hist->cursor_bytes at a stack buffer for the call, then NULLs it
back so the next iteration's fetch is independent.
- RzHistogramInteractive gains `blocksize`, `cursor_bytes` and
`cursor_bytes_len` fields.
New config option:
- `scr.hist.minimap` (bool, default true) controls whether the top
minimap is shown for p==v / p==ev. Surfaces as `opts->minimap` and
is honoured by `rz_histogram_interactive_horizontal`. When true,
the minimap is ALWAYS rendered (provided there's room) - even when
the chart already shows the full data, in which case the window
indicator spans the whole map. Changes via `:` `e scr.hist.minimap=...`
<Enter> are picked up on the very next redraw.
Tests (33 total, 10 new for the visual side):
- test_histogram_interactive_horizontal_basic - smoke test with
barnumber=0 (pins the #4431 crash regression).
- test_histogram_interactive_horizontal_ruler_percent - fractional
labels with value_max=100 / value_scale=0.01 / unit="%".
- test_histogram_interactive_horizontal_ruler_default - the legacy
0..255 byte ruler.
- test_histogram_interactive_horizontal_no_negative_adder - covers
`p==v` at offset 0 on a small data set.
- test_histogram_interactive_horizontal_percent - status-line percent
indicator present.
- test_histogram_interactive_horizontal_cursor_markers - the ▼/▲
cursor markers (and ASCII v/^ fallback) appear on the cursor column,
left un-highlighted.
- test_histogram_interactive_horizontal_cursor_full_line - the cursor
bar is rendered on every chart row between the markers regardless
of the data threshold (continuous vertical line).
- test_histogram_interactive_horizontal_cursor_width - pins single-
char width across BOTH cursor-widening bugs: high zoom (sizeofonebar
> 1) AND chart wider than data (histogramwidth < width).
- test_histogram_interactive_horizontal_minimap_toggle - pins the
scr.hist.minimap gating across {zoomed, not zoomed} when
opts->minimap=true / =false.
- test_histogram_interactive_horizontal_hex_preview - 4 cases: wide
terminal + cursor_bytes shows the hex panel; narrow terminal
suppresses it; missing cursor_bytes suppresses it; colour mode
emits ANSI escape sequences for the bytes.
Both p== integration tests in test/db/cmd/cmd_print pass with their
regenerated EXPECT blocks (the per-block sampling change moves the
visible bars for small buffers).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
An enum type hint set on an operand (ahie) only replaced the immediate when its
value matched an enum member exactly, via rz_type_db_enum_member_by_val(). A
value that is the bitwise OR of several flag members -- e.g. the access(2) mode
R_OK|W_OK == 6 from the issue -- matched no single member and was left as a raw
number.
replace_enum_hint() in rz_parse now falls back to rz_type_db_enum_get_bitfield()
when there is no exact member, so the value is rendered as the OR of the
matching members, e.g. "access_def.W_OK | access_def.R_OK".
rz_type_db_enum_get_bitfield() is reworked along the way: it was unused and
buggy (capped at 32 bits, reused a stale match for bits without a member, and
emitted a "0x.. : " debug prefix). It now walks all 64 bits, returns the
matching members qualified with the enum name and joined by " | ", and returns
NULL when the value is 0, the type is not an enum, or any set bit has no member
(so a value that is not cleanly a combination of flags stays a number).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* Fix revert if adding a node failed.
* Reduce allocations by keeping only a single RzGraphEdge object per edge around.
* Add benchmark for graph deletion and addition of nodes/edges
* Use rz_pvector_remove_at_unsorted to save some runtime.
* Use realloc and memmove for matrix graphs on capacity increase.
* Add helper to determine memory usage.
* Add benchmark
* Revert matrix capacity extension to simple and jsut as fast loop.
* Missing type annotations
The C grammar already exposes a struct/union member's bitfield width as a
"bitfield_clause" node and the parser already recognized "int a : 4;", but
it threw the width away (the member was stored with size 0 and a FIXME). As
a result "tc" rendered the member as a plain "int a;", "ts" produced a
full-int "pf" format ("pf d4d4d4 a b c") and "tp" mis-read every field as a
whole integer.
RzTypeStructMember and RzTypeUnionMember already carry a "size" field
documented as "in bits"; it is now used as the bitfield width, where 0 means
the member is not a bitfield:
- The struct and union member parsers store the parsed bit width on the
member instead of discarding it.
- The pretty printer emits " : <width>" before the member's trailing ';',
so "tc"/"tcd" round-trip "struct qwe { int a : 4; int b : 16; int c : 3; }".
- rz_base_type_as_format() (used by "ts"/"tp") now emits the pf packed-bits
spec ":N" for a bitfield member, with the bit order matching the target
endianness ("<" little-endian / ">" big-endian), instead of the member
type's full format. "ts qwe" becomes 'pf ":4<:16<:3< a b c"' and "tp"
unpacks the fields correctly (a=5, b=0x1234, c=0 for 0x00012345 on x86).
Serialization keeps the width in the 3rd comma-field of the existing
"struct.<name>.<member>" / "union.<name>.<member>" value, i.e.
"type,offset,bitsize". That field was already written (always as 0) and
ignored on load, so the on-disk layout is unchanged and only its meaning is
now honored. The project version is bumped to 24 with an additive no-op
migration: an absent or 0 bitsize deserializes as a regular, non-bitfield
member.
Tests: a unit test parses a bitfield struct and union and checks the member
widths, the rendered string and the "pf" format; a db/cmd/types test covers
the full "td"/"tcd"/"ts"/"tp" flow; a project-migration test with a new
v23-bitfield.rzdb fixture covers v23->v24, and the "migrate info" db test is
updated for the new version.
The PDB type parser (librz/arch/pdb_process.c) still drops bitfield members
because their LF_BITFIELD member type resolves to NULL; wiring LF_BITFIELD up
to the new member "size" field, with tests in test/unit/test_pdb.c, is left
as a follow-up.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
C23 lets an enum fix its underlying type, e.g. "enum E : long long { ... }".
The C grammar already exposes it as the "underlying_type" field of an
enum_specifier, and RzBaseType already has a "type" slot documented as
used by enums, but the parser ignored the field and always left it NULL.
- parse_enum_node() now reads the "underlying_type" field and stores the
parsed type on RzBaseType::type (reusing parse_type_node_single(), so
primitive, sized and typedef'd integer types are all handled). Classic
enums keep a NULL underlying type.
- The pretty printer emits " : <type>" between the enum name and its body
when an underlying type is present, so "tc"/"tcd"/"tec" round-trip it.
- enum_bitsize() now derives the width from the underlying type instead of
the hardcoded 32-bit default (resolving the long-standing FIXME); it
still falls back to 32 for a classic enum.
Single-token underlying types (int, uint64_t, char, ...) work end-to-end
with the bundled grammar revision: the existing "enhanced enum" db test is
updated to round-trip "enum v : int" and a unit test covers
"enum EU : uint64_t". Multi-word underlying types (long long, unsigned int)
are added as BROKEN db tests; the bundled grammar parses them to an ERROR
node and drops the underlying type, so these tests fail for now and will
pass once rizin-grammar-c accepts sized type specifiers as the enum
underlying type. No further rizin change is needed for that step.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Close two long-standing issues against the static horizontal histogram
(`p==`, `p==e`, `p==0` and the rest of the `p==X` family). Add the
`scr.hist.width` / `scr.hist.height` config options (closes#6372),
both clamped via RZ_MIN() against the current terminal size with a
defensive sanity floor. Make the Y-axis ruler context-aware (closes
#5290) via new `value_min` / `value_max` / `value_unit` fields on
`RzHistogramOptions` and rescale each datum from its ut8 storage into
the vmin..vmax display range before thresholding, which also fixes
cer-0's follow-up about the chart's top rows staying blank for
low-range data.
Tighten the look in the same step. Y-axis labels are now sparse (top,
~25 %, ~50 %, ~75 %, bottom) instead of one per row, in the style of
tokio-console / Granite, with an inclusive label range (top = vmax,
bottom = vmin) decoupled from the threshold formula so the existing
`_` baseline behaviour still appears. The bottom of the chart grows
an X-axis ruler with `^` tick markers and `0x...` start / middle /
end offsets (driven by a new `opts->blocksize` field). Two more
options, `value_scale` and `value_precision`, let callers format
ruler labels as fractional values; the `p==e` (entropy) command opts
in with `value_max = 1, value_precision = 2` so the ruler reads
0.00..1.00 by default, matching Shannon-entropy convention.
A new `opts->data_f` field lets callers feed double-precision data
directly into the renderer (and engages double-precision arithmetic
for the row thresholds), so entropy keeps full precision end-to-end
instead of losing ~1/255 of resolution to the `(ut8)(255 * fraction)`
quantisation step. This captures the precision win sketched in PR
#6355 without introducing a separate `rz_histogram_horizontal_f64()`
twin function. Ten unit tests in `test/unit/test_cons_histogram.c`
(seven new) cover sparse labels, fractional labels, the X-axis offset
ruler, the cer-0 regression, `opts->cols`, the legacy 0..255 ruler,
and the fp data path; all pass.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
With bin.demangle=true the call/jump operand is replaced by the full
demangled symbol name and then colorized token-by-token before being
filtered. For long C++ symbols (e.g. libc++ STL names) the colorized
operand easily exceeds the 1024-byte operand buffer (ds->str): in
truecolor each token is wrapped in a ~19-byte "\x1b[38;2;R;G;Bm" escape,
so a ~110-char name grows past 1700 bytes.
The generic filter() copied that operand with
strncpy(str, data, len);
which has two problems when strlen(data) >= len:
- it does not NUL-terminate, so reading str ran past the buffer into
the adjacent strsub[] buffer, printing stale bytes (the "call 0x..."
bleed); and
- the byte-bounded cut can land in the middle of a color escape,
leaving a partial "\x1b[38;2;.." that, followed by the bled bytes,
forms a complete but bogus "\x1b[..c" sequence. Terminals read that
as a Device-Attributes query and reply with e.g. "62;4c", which is
the garbage the reporter saw at the prompt.
This only triggers with color enabled and a symbol long enough that its
colorized form overflows the buffer, which is why it showed up on some
x86 files only and had no portable reproducer (no-color = 149 bytes,
16-color = 763, 256-color = 1170, truecolor = 1713 for the sample
symbol; the threshold is 1024).
Add test/unit/test_parse.c driving rz_parse_filter() with over-long
colored operands (a deterministic case whose cut lands inside an escape,
and a realistic libc++-style colorized operand) plus a short-operand
passthrough case. The two overflow tests fail on the old strncpy and
pass with the fix.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
format.c can already turn an RzType into a pf format string. Add the
inverse in the same file, next to its counterparts:
rz_type_format_to_c_declaration() parses a pf format string with
rz_pf_parse() and emits an equivalent C struct/union declaration built
from the standard fixed-width types (uint8_t, int32_t, float, ...).
The conversion is structural: it consumes only the parsed RzPfFormat
(field kinds, widths, array counts, pointer flags), never a byte buffer,
so it runs without any target data. Every field kind the engine produces
is mapped; a handful of specifiers with no exact static C form are mapped
best-effort (documented inline): @N alignment is dropped, an
unknown-length inline z string becomes char *, LEB128 widens to its
largest decoded integer, and ?(Name) / E(Name) emit struct/enum
references that must themselves be defined to parse.
Added new 'tdf' command that exposes that conversion to the user
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* librz/type: rewrite pf format parser
Replace the legacy print-format engine in librz/type/format.c with a
clean three-stage pipeline (parse -> read -> render) and a properly
typed DSL.
The new pipeline:
rz_pf_parse(str) source string -> RzPfFormat
rz_pf_read(fmt, buf, len, ctx) RzPfFormat -> RzPfValue[]
rz_pf_render(vals, mode, opts) RzPfValue[] -> printable string
rz_pf_format() one-shot for all of the above
Public API lives in <rz_pf.h> (pulled in transitively via <rz_type.h>).
DSL improvements over the legacy scheme:
* Sized integers with explicit endianness via case:
x4 d4 u4 o4 b4 (LE) vs X4 D4 U4 O4 B4 (BE).
* Sized floats: f2 / f4 / f8 (and BE F2 / F4 / F8).
* Encoding-aware strings: z(utf8) / z(utf16le) / z(utf16be) /
z(utf32le) / z(utf32be) / z(latin1) / z(ebcdic).
* Length-prefixed strings: z[N] reads a N-byte LE prefix then body.
* Parameterized timestamps: t(unix32) / t(unix64) / t(unixms) /
t(unixus) / t(unixns) / t(filetime) / t(dos) / t(hfs) /
t(oletime) / t(webkit) / t(cocoa). ntfs alias maps to filetime.
* GUID: G(ms) (Microsoft mixed-endian, default), G(be), G(le).
* Inline bitfields: B4(FLAG_A=1,FLAG_B=2,FLAG_C=4).
* Alignment: @N advances to next multiple of N.
* Bit fields: :N with optional < (LSB-first) or > (MSB-first).
* Length-by-reference arrays: [@earlier_field]T.
* TLV records: V(t=u1,l=u2,e=le,h=v|l|a,d=table). Dispatch table
uses tlv.<name>.<hex-tag> in the typedb formats hash.
Architecture highlights:
* Per-instance ReadState (bit_cursor + sibling lookup window).
The bit cursor snap-flushes to the next byte when a non-bit
field follows mid-byte.
* Recursion bound via RzPfCtx::max_depth (default 32).
* Five render modes: text (default), json, cstruct, quiet, dot.
The dot renderer emits a column-aligned record with offset/type/
name/value rows, suitable for direct rendering through Graphviz.
* Positioned diagnostics: RzPfError carries severity, category,
column position, and a human-readable message. Each is also
emitted via RZ_LOG_WARN for backward compatibility.
* Optional RzPfPalette and RzPfRenderOpts threaded through render
for inline ANSI colorisation. NULL palette = canonical text.
* Pointer dereference via RzPfCtx::read_at callback.
The integration test expectations in test/db/cmd/cmd_pf* are
updated in the same commit to match the new text-output format and
the new DOT renderer's column-aligned record layout. Bundling these
keeps the test suite green at this commit.
A transitional rz_type_format_data() shim at the bottom of
pf_parser.c forwards legacy callers (cprint.c, disasm.c) to the new
API; it is removed in the librz/core migration commit.
* librz/arch/types: migrate type DB to new pf format codes
Map every legacy single-character format code in the bundled type
SDB files to its new-DSL equivalent:
b / C -> x1 (1-byte hex)
c -> c (signed char, unchanged)
w -> x2 (2-byte hex LE)
i -> d4 (signed 32-bit decimal LE)
d -> x4 (4-byte hex LE)
q -> x8 (8-byte hex LE)
f -> f4 (32-bit float LE)
F -> F8 (64-bit float BE)
Z -> z(utf16le)
x -> x4
o -> o4
X -> r (hexdump)
n / N -> d4 / u4
Also add new typedef-only types for explicit display formats:
bin{8,16,32,64}_t -> b{1,2,4,8} (binary)
hex{8,16,32,64}_t -> x{1,2,4,8} (hex)
oct{8,16,32,64}_t -> o{1,2,4,8} (octal)
These let users opt into a display representation per field without
changing the underlying scalar size.
Update the corresponding integration test expectations in
test/db/cmd/cmd_avg and test/db/cmd/types: the 'tu', 'tuc', 'tp',
and 'avgp' commands report formats via the SDB, so the migrated
codes appear in their text output (e.g. 'pf 0f4x4 a b' instead of
the legacy 'pf 0fd a b' for a union of float + int).
* librz/core: rewrite pf integration on the new parser API
Remove the legacy rz_type_format_data() bridge and rewrite every
caller to use the new <rz_pf.h> API directly.
librz/core/cprint.c
core_print_format() constructs an RzPfCtx populated from the
RzCore (typedb, big_endian, bits, max_depth) and calls
rz_pf_format() in one shot. The bridge between RzCore I/O and the
pf reader is cprint_pf_read_at(), which forwards to rz_io_nread_at.
The bitmask-to-RzPfMode mapping is cprint_pf_mode().
For DOT mode the format name is passed via RzPfRenderOpts::graph_label
so the dot renderer uses it as the top-level record label.
When scr.color > 0 in TEXT mode, an RzPfPalette is populated from
the active color theme via RzConsPrintablePalette so `pf` output
blends with the rest of Rizin's UI and respects user theme
choices (eco / ec). The palette slots map to theme fields as:
pf field offset <- pal->offset (address column)
pf field name <- pal->fname (symbolic name)
pf endian marker <- pal->meta (metadata tag)
pf hex/number literal <- pal->num (numeric literal)
pf typedb label <- pal->flag (resolved symbol)
reset <- pal->reset
Each slot falls back to its previous hardcoded ANSI escape if the
theme field is unset, preserving prior behaviour for stripped-down
contexts. Other modes (json/cstruct/quiet/dot) ignore the palette.
Closes rizinorg/rizin#782.
librz/core/disasm.c
RZ_META_TYPE_FORMAT case rewritten to resolve via rz_pf_resolve_name
and decode via rz_pf_format() directly. Palette is supplied from
the same theme palette when ds->show_color is true so that
pd @ <struct> blends with the surrounding listing.
librz/include/rz_type.h
Drop the rz_type_format_data() forward declaration. Callers must
use the public <rz_pf.h> surface instead.
librz/core/cmd_descs/cmd_print.yaml
Rewrite the pf command help for the new DSL: sized integers
(case-discriminated endian), special scalars (G GUID, V TLV,
: bits, @ align), strings with encodings, parameterized timestamps,
typed composites (E enum, B bitfield, ? struct), DSL extensions
(length-prefix strings, length-by-reference arrays, inline
bitfields), skip/repeat/pointers, example invocations.
cmd_descs.c is regenerated from the YAML.
* test: pf parser unit and DSL-extension integration tests
Add fresh test coverage for the new pf parser. These are purely
additive: existing integration tests were updated in the parser
and SDB-migration commits so the suite stayed green throughout.
test/unit/test_pf.c (new, 131 tests)
Field-size tables (every fixed-size type) and ctype mapping, every
parse path (sized integers / floats / strings / timestamps /
pointers / structs / enums / bitfields / GUIDs / TLV / alignment /
bits / arrays), every read path (with both fixed and length-by-
reference array counts), every render mode (text / json / cstruct /
quiet / dot), every DSL extension (align, bits MSB/LSB, GUID layout,
length-ref array, length-prefix string, inline bitfield, TLV with
dispatch table), pointer dereference via an in-memory I/O callback,
recursion safety on self-referential structs, lifecycle / NULL
safety, diagnostics (positioned errors, all categories, caret-line
formatter, source capture, verbose parse), ambiguity (each
superficially similar DSL form parses unambiguously), the palette /
colorisation API (issue #782) including a regression test surfaced
by the property-based test harness for input bytes containing ESC,
and DOT-mode coverage (column-aligned layout, single field,
empty/filtered, timestamp values, TLV records).
test/db/cmd/cmd_pf_dsl (new)
8 integration tests targeting the new DSL extensions specifically:
@N alignment, :N bits in MSB and LSB ordering, default-layout GUID,
z[N] length-prefixed string, B4(K=V) inline bitfield, bare V TLV
with defaults, V(t=u2,l=u2,e=be) configured TLV.
test/db/cmd/types_format (new)
Verifies that the new bin{N}_t / hex{N}_t / oct{N}_t typedefs apply
the expected display representation when used inside struct fields
via the 'tp' command, demonstrating the per-base rendering
(binary / decimal / octal / hex).
* doc: pf DSL reference and librz/type README
doc/pf.md (new)
User-facing reference for the pf format DSL covering quick
examples, spec grammar (sized integers with case-discriminated
endian, floats, strings with encodings and length prefixes,
parameterized timestamps, composites -- struct ?, enum E, bitfield B
inline and typed, GUID G, TLV V, raw hexdump r), repetition and
arrays ([N], [@name], {N}, leading 0 for union), padding and
alignment (. skip, @N align, :N bits MSB / LSB), pointer
dereference (*<type>), names grammar (plain vs (typename)name),
output modes (text, json, cstruct, quiet, dot), and diagnostics
(severity / category / position / caret-line formatter; backward-
compatible RZ_LOG_WARN channel).
librz/type/README.md (new)
Subsystem architecture doc for librz/type. Covers public headers,
file map, the pf three-stage pipeline (parse -> read -> render) with
a diagram and entry-point table, per-stage explanations (parser
shape, ReadState scoping rules including bit cursor and sibling
lookup window and snap-flush rule, render mode dispatch including
the DOT column-aligned layout and the graph_label parameter),
typedb integration and recursion bound, error-reporting model,
test coverage summary, and pointer to doc/pf.md as the user-facing
reference.
* librz/type: remove legacy DSL conversion shim
All in-tree callers -- librz/bin/d/, librz/bin/format/, and the
integration tests in test/db/cmd/ -- have been migrated to the new
DSL in the earlier commits of this series. The compatibility shim
that translated bare legacy specifiers (x -> x4, b -> x1, w -> x2,
nN -> uN, NN -> UN, etc.) into the new DSL at the entry of
rz_pf_parse() is no longer load-bearing; this commit deletes it.
- librz/type/pf_parser.c: drop the 267-line
convert_legacy_to_new_dsl() function, the WARN_LEGACY macro,
and the per-parse allocation + free of the converted string.
rz_pf_parse() now walks the caller's spec directly.
- test/db/cmd/cmd_pf,cmd_pf2,cmd_pf_write,cmd_pfd,cmd_pf_new,
metadata: migrate legacy specifiers in CMDS blocks to the new
DSL (b -> x1, w -> x2, q -> x8, i -> d4, bare x -> x4, etc.)
and regenerate EXPECT blocks against the new render where
affected.
The parser now only accepts the new DSL. Any caller still passing
bare legacy specifiers will get "unknown specifier" warnings.
* librz/core,type: add scr.pf.short for delta-offset rendering
When scr.pf.short is enabled, MODE_TEXT rendering shows offsets as
deltas (+<n>) from the format's base address instead of the absolute
hex address. Useful for self-contained struct dumps where the
absolute load address is noise:
$ rizin -e scr.pf.short=true -qc 'wx 00007a452a4b9a02
pf fcb1d4 a b c d' =
0 : a = 4000 [LE]
+4 : b = '*'
+5 : c = 0b0100_1011 [LE]
+6 : d = 666 [LE]
Nested structs reset their delta from the same base, so a child of
`head` at offset +8 still prints as +0 in its own struct block.
The base is derived from the offset of the first top-level value;
callers can override it via RzPfRenderOpts::base_offset.
Includes 5 integration tests in test/db/cmd/cmd_pf_short covering
basic struct, off-mode parity (control), nested struct, concatenated
TLV records, and @-offset usage.
* librz/type,test,doc: add v(N) bitvector type for forensics use
Adds a new specifier `v(N)` to the pf DSL that reads N individual
bits (1..4096) from `ceil(N/8)` bytes and exposes them as N
separate 0/1 scalars. Forensics targets: page-frame bitmaps, NTFS
$Bitmap clusters, ext4 block/inode bitmaps, ELF DT_FLAGS_1, PE
characteristics, ACL bitmasks -- anywhere you want to *see* a
bitmap rather than collapse it to a hex number.
Grammar:
v(N) N-bit bitvector, default MSB-first per byte
v(N,lsb) LSB-first per byte (Intel order)
v(N,msb) explicit MSB-first (DWARF / network order)
Rendering:
text: [ 1 0 1 0 1 0 1 1 | 1 1 0 0 ] (12-bit)
json: {"bit_width":12,"value":"101010111100"}
quiet: 1 0 1 0 1 0 1 1 1 1 0 0
Bitvec fields do not participate in the packed-bit cursor used by
:N; each v(N) reads whole bytes and stands alone. A v(N) field
adjacent to a :N field flushes the partial bit cursor first.
Width is clamped to [1, 4096] with a RANGE diagnostic. The reader
consumes exactly ceil(N/8) bytes, leaving the cursor positioned for
the next field. Verified by the existing tail-field test pattern.
Tests:
test/unit/test_pf.c +10 unit tests (146 total OK)
test/db/cmd/cmd_pf_new +9 cmd tests (440 OK / 9 BR / 0 XX / 4 FX
across all 19 pf suites)
property harness +7 properties (140 unique props, 1000
trials each, 0 failures)
Docs:
doc/pf.md bitvector section with examples
librz/type/README.md cursor-interaction note for v(N) vs :N
* librz/core,type,test: address PR CI feedback for pf rewrite
Five fixes prompted by CI feedback and reviewer comments on the pf
rewrite series:
* librz/core/cmd_descs/cmd_print.yaml: two help-text lines exceeded the
yamllint 120-char ceiling (n1/n2/n4/n8 comment and the deprecation
notes entry). Both now use the same folded-scalar (comment: >) form
already used in cmd_analysis.yaml and cmd_descs.yaml. cmd_descs.c is
regenerated.
* test/db/cmd/cmd_pf_write: drop the BROKEN 'pf xxd print' test. It
relied on '.pf*' (run pf output as rizin commands) which the new
parser no longer emits -- analogous to the '.pfw' removal done
earlier. The companion 'pf xxd print happy' test exercises the same
path through 'pf.' and stays.
* librz/type/pf_parser.c: thread the RzTypeDB through render_cstruct
and render_dot (and their inner helpers) so RZ_PF_ENUM resolution
works in pfc and pfd mode the same way it already does in text mode.
Without this, 'pfc elf_header' emitted '/* 0x00000002 */' and 'pfd'
emitted '|value|0x0000ffff|'; with it the comments and cells carry
the symbolic name (ELFCLASS64, ET_HIPROC, etc).
scalar_text() RZ_PF_BITFIELD: when no inline B4(K=V,...) flags are
present but val->type_name names a typedb enum, walk that enum and
treat each case as a settable bit name. This makes the typedb form
'B (pe_characteristics) flags' decode to
'0x00008140 : IMAGE_DLLCHARACTERISTICS_DYNAMIC_BASE | ...' instead
of just the raw hex -- previously the bitname decoding only worked
on inline B4(...) forms.
* librz/core/cprint.c: cprint_pf_mode() had no mapping for
RZ_PRINT_VALUE, so 'pfv' fell through to TEXT mode and emitted
'0x00000004 = 0x11111111 [LE]' instead of the bare value '0x11111111'
expected by db/esil/arm_16 and existing pfv users. Added an explicit
RZ_PRINT_VALUE -> RZ_PF_MODE_QUIET mapping; QUIET already emits one
bare scalar per field, which is exactly the pfv contract.
* test EXPECTs: cmd_pfd unbreaks its bitfield case (also fixes the
data buffer -- the original used 'wx 0x00008140' which is parsed
byte-by-byte and yields LE u32 0x40810000, not 0x00008140, so no
flags matched). cmd_pf 'PE test' picks up the now-decoded
characteristics and dllCharacteristics bitfields. cmd_pf 'Print
value only' updates to the bare-value pfv contract. test/db/cmd/print
'elf 64bit ls pfc.elf_header' picks up the decoded enum symbols.
Also picks up clang-format-20 reflows in cprint.c, cmd_print.c,
pf_parser.c, and rz_pf.h that the CI's clang-format check flagged.
Also bumps the offset-delta buffer in render_val_text from 16 to 24
bytes so a worst-case ut64 decimal (20 digits + sign + NUL = 22) fits
without tripping -Werror=format-truncation on gcc with -O0.
Also fixes three swapped-argument mu_assert calls in test_pf.c (the
v(N) bitvector tests added in commit 8): the macro signature is
mu_assert(message, test) but those three calls passed (test, message),
which tripped -Werror=format= under -O3 because the int 'test' was
fed to the format string's %s slot.
Adds a second round of regression fixes prompted by a manual audit
against pristine upstream (review feedback after the first pass):
* render_quiet: handle raw byte-sequential types (RZ_PF_UINT128,
RZ_PF_HEXDUMP). scalar_text() has no per-byte case for these so the
default-clause '?' was emitted, making 'pfq Q' and 'pfv Q' print '?'.
render_quiet now mirrors text mode and emits a space-separated hex
stream for raw types.
* Sized pointers (p{2,4,8}): the legacy parser accepted 'p2', 'p4',
'p8' to force a 16/32/64-bit pointer width regardless of ctx.bits;
the new parser only handled bare 'p'. Restored via a new dispatch
branch ahead of bare 'p', recording the override in
RzPfField::bit_width (overloaded -- documented in rz_pf.h). A new
fld_ptr_size() helper consults the override first; read_ptr, the
pointer-deref read path, the STRPTR read path, and
pf_struct_size_impl all use it. 'pf p2p4p8pp2' now consumes
2+4+8+4+2=20 bytes and renders the right value at each position.
* Numeric pointer dereference: '*d4', '*x2', '*u8', etc. (pointer to
fixed-size scalar) only read the pointer itself and never followed
it. The reader now also calls ctx->read_at to fetch the target word
and populate scalars[0]; the text renderer prints both the pointer
literal and the dereferenced value -- '(*0x20) 42' instead of bare
'(*0x20)'. Affects 'pf *d4 ...' and the 'Pointers' / '32 bit twice
then string' tests.
* Pointer-to-struct dereference (*?): pointer-to-struct fields read
the pointer value but stopped there, so 'pf *?' (or typedb forms like
'(troll)Bah' marked with '*') showed only '(*0x30)'. The reader now
recurses through ctx->read_at into a worst-case 4 KiB buffer and
re-invokes read_nested_struct; the renderer prints the nested struct
body after the '(*ptr)' annotation. Recursion is bounded by
ctx->max_depth so cyclic Bah->Bah pointer chains terminate at the
documented limit instead of unbounded recursion. Affects 'nested
struct', 'complex nested struct', and 'flag for nested struct'.
* String pointer (bare 's'): RZ_PF_STRPTR rendering didn't show the
dereferenced target -- output was bare '"hello"' with no pointer
context. Now emits '(*ptr) "string"' (mirroring '*z') when the
deref produced a non-empty body; falls back to bare '""' on
unmapped or empty targets to avoid advertising a phantom pointer.
* render_val_text is_pointer branch: extended to render the nested
struct body for *? pointers and the dereferenced value for numeric
*d4/*x2/*u8 forms.
* Top-level read loop: rz_pf_read passed both 'cur_off + off' (as
read_field's off parameter) AND 'base_addr + cur_off' (as base_addr),
but read_field computes 'val->offset = base_addr + off'. This
double-added cur_off on every iteration past the first, so 'pf 2ic'
reading 5 bytes per iter reported iter 1's nb at offset 0x0a instead
of 0x05, and 'pf 2F' reported iter 1's double at offset 0x10 instead
of 0x08. Data reads were correct; only the displayed/JSON offsets
drifted. The fix: pass base_addr unchanged so val->offset becomes
base_addr + (cur_off + off). Affects 'array obj', 'print n-times a
format', 'pf', 'pf field name', 'JSON output'.
* Bare 'C' (legacy 1-byte unsigned decimal): the legacy parser accepted
'C' as 'print byte as decimal'; the new parser had dropped it. The
'types' test format 'pf fcb1d4C foo bar fool beer plop' was therefore
truncating to 4 fields. Restored as a deprecated alias for 'u1' (with
the standard 'use u1' note).
* pfw dotted-path navigation: the write-mode lookup only matched
top-level field names, so 'pfw gobelin.Buh.first=42' through nested
structs failed with 'field not found'. Replaced with a segment-by-
segment walker that descends through children at each dot. Affects
'write specific element through nested struct'.
* Drop '.pfw' usage from tests. '.pfw' (execute pfw output as rizin
commands) was a legacy convention -- the new pfw writes directly via
rz_io_write_at and emits a human-readable confirmation line, so the
'.' prefix evaluates that confirmation line as a command and fails.
Same direction as the earlier '.pf*' removal. All cmd_pf,
cmd_pf_write, cmd_pf2 tests updated.
* test/db/cmd/cmd_pf 'Register' marked BROKEN with a tracking comment:
the legacy 'r (regname)' looked up CPU registers via
RzPrint::get_register; the new parser repurposes 'r' as raw hex byte
dump and the register-fetch path is gone. Restoring would mean
wiring a new register-lookup hook through RzPfCtx, which is outside
the parser-rewrite scope.
* test/db/cmd/cmd_pf2 'pf F max precision (#13027)' annotated: not a
precision regression. Values differ from the original #13027 expected
output because the legacy 'F' specifier read bytes as LE-then-
reinterpret while the new parser follows the documented 'UPPERCASE
= BE' rule literally; the 17-digit precision -- the actual subject
of #13027 -- is preserved.
EXPECTs updated to match the corrected output across cmd_pf, cmd_pf2,
cmd_pf_write.
* doc/pf.md updated to reflect all DSL and rendering changes:
documents the sized pointers (p/p2/p4/p8), pointer dereference
semantics (numeric, struct, and string), the missing composites
(Q, U, L, n/N), the case-as-endian rule (no standalone endian
directive), the pfw write mode (dotted-path navigation, direct I/O,
no '.pfw'/'.pf*'), and the table of deprecated single-letter
aliases (b, d, o, q, u, i, f, F, w, Z, t, T, X, C).
* librz/type/pf_render.c (new): extracted the render-mode code paths
from pf_parser.c into their own translation unit. Hosts the text /
quiet / JSON / cstruct / DOT renderers, the per-mode helpers
(scalar_text, scalar_json, render_binary, render_guid,
compute_name_width, field_matches, emit_colored*), the RenderCtx
bag, and the public rz_pf_render() + rz_pf_render_json() entry
points. pf_parser.c keeps the parse + read pipeline and shrinks
from 4831 to 3280 lines. Helpers used by both translation units
(is_string_type, is_raw_type, endian_str, pf_vasprintf) move to
pf_parser.h as static inlines.
* librz/type/pf/ (new directory): moved every pf source out of the
librz/type/ top level into a dedicated pf/ subdirectory; only the
legacy format.c stays top-level. meson.build updated with the new
paths and the pf/ include dir.
* Split the monolithic parser into focused sub-grammar TUs, wired
together by the new pf/pf_internal.h (which centralizes the PF_DIAG
diagnostic macro, the shared ReadState, and all cross-TU
declarations):
- pf_parser_string.c string/encoding specs (z / s / Z)
- pf_parser_bitfield.c inline + typed bitfields
- pf_parser_bitvec.c bitvectors v(N)
- pf_parser_array.c array-count resolution
- pf_parser_struct.c nested struct / union reading
pf_parser.c shrinks from 3280 to ~2620 lines and now hosts only the
parse driver, type-spec dispatcher, reader core, context, and the
public utility surface. The TLV TU was migrated off its bespoke
extern/TLV_DIAG block onto the shared header.
* Add an RzStructuredData renderer: pf/pf_render_sd.c with the public
rz_pf_render_sd() (declared in rz_pf.h). (Named _sd, not _sdb: SDB is
Rizin's key/value database, unrelated to RzStructuredData.) It maps the decoded value
vector to the generic key/value document model -- scalars to typed
entries, arrays/bitvectors to arrays, nested structs to sub-maps
with a _type tag, raw/GUID payloads to byte blocks, timestamps to a
formatted string plus a <name>_raw sibling -- so callers get JSON,
YAML, or iterator access for free. Unit and property tests cover it
(see the consistency-pass note below).
* doc/pf.md and librz/type/README.md updated for the new pf/ layout
and the structured-data render mode.
* Consistency pass over librz/type/pf/: deduplicated the structured-data
scalar dispatch (one classifier feeding thin map/array emitters instead
of two parallel ~60-line switches), dropped unused includes
(pf_parser_time.h from the TLV TU, string.h from the struct TU,
rz_endian.h from the SD renderer) and a redundant extern decl now
covered by pf_internal.h, and fixed a dangling-pointer bug in the SD
char path (the classifier returned a pointer into its own by-value
temporary). Expanded coverage: 11 SD unit tests (157 total) and 6 new
theft properties (SD tree non-NULL, JSON structural well-formedness,
top-level object shape, YAML safety, determinism, filter-narrows).
* Split pf_render.c (1548 lines) into per-mode renderer TUs, mirroring
the earlier pf_render_sd.c extraction: pf_render_text.c (text+quiet),
pf_render_json.c (+rz_pf_render_json), pf_render_cstruct.c, and
pf_render_dot.c. A new pf_render.h carries the shared RenderCtx record
plus the cross-TU helper (pf_field_matches, pf_scalar_text,
pf_render_guid) and per-mode entry-point declarations. pf_render.c now
holds only those shared helpers and the rz_pf_render() dispatcher
(~372 lines). meson.build, the file-doc layout listings, and
librz/type/README.md are updated accordingly.
* Second consistency pass: ensured every public RZ_API entry point has a
Doxygen \brief block (added ones for rz_type_format_struct_size and
rz_pf_render_sd) and every file a \file description (added to
pf_parser.h and pf_parser_time.h). Deduplicated the field-size logic:
pf_struct_size_impl's sum and union loops now share one
pf_field_static_size() helper, the enum/bitfield byte-width override is
a single pf_enum_bitfield_width() helper (was inlined 3x), sized-pointer
width routes through the existing fld_ptr_size(), and the inline-bitflag
test shared by the JSON and DOT renderers became the
pf_value_has_inline_bitflags() predicate in pf_render.h. Dropped unused
includes left over from the render split. No behaviour change (157 unit
tests, 146 theft properties, full cmd sweep all green).
* Fix a CodeQL High alert (cpp/integer-multiplication-cast-to-long) in
the scalar-array path selector: the per-element byte delta was computed
as `idx * width` in 32-bit int before being added to the ut64 offset,
so a large array index could overflow int prior to the widening. The
multiplication is now done in ut64 ((ut64)idx * width). No behaviour
change for in-range indices; verified by the path-selection cmd tests.
* Fix a big-endian bug in the structured-data renderer: the bitvector
reader stores each bit in the RzPfScalar union's v_u8 member, but the
SD classifier read it back through v_u64. On little-endian those alias
to the same low byte so it worked by luck; on big-endian (s390x) v_u8
is the high byte of the 64-bit slot, so every set bit read as a large
number and the test_pf_render_sd_bitvec assertion failed. The
classifier now reads v_u8 for RZ_PF_BITVEC, matching the writer; the
other scalar types already read the same width the reader wrote.
* Clean up leftover refactoring-era comments in the renderer TUs: drop
the "Split out of pf_render.c" changelog phrasing from the file-doc
headers (the docs now describe the current layout) and update two
stale references to the pre-rename helper names (field_matches ->
pf_field_matches, render_guid -> pf_render_guid) in comments.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* Add option to replace (geometric) invalid values with another value.
* Add geometric Mean and Std Dev to benchmarks.
* Add Doxygen documentation
* Add a README to the bench dir as intro.
empty_cond was never signalled and the termination depended only on
the timeout in rz_th_queue_close_when_empty() causing a re-check of
emptiness.
However, the rz_th_cond_timed_wait() implementation, which was used
there, was flawed because it expected a relative timeout but passed that
directly to pthread_cond_timedwait() which expected an absolute time
value, practically causing it to time out immediately. Depending on the
pthread_cond implementation, this possibly created a situation where the
mutex could never be acquired by another thread, effectively causing a
deadlock. This behavior was observed on Mac OS X 10.5 (ppc) when running
the test_core_bin test.
We solve this by not using a timeout at all and signalling the condition
variable for all waiting threads at the appropriate time.
* Add Welfords square of sums algorithm for variance and std deviation calculations.
* Add standard deviation to benchmarks
* Simplify Welford
* Add geometric mean and standard deviation to Welford Sums
* Add order ignoring remove_at version with better performance.
* Optimize rz_vector_swap by using stack memory for small elements.
* Add benchmark for rz_vector_swap
* Refactor del_edges to use RzGraphStatus.
* Refactor del_edge() to use RzGraphStatus.
* Refactor update_edge() to use RzGraphStatus.
* Refactor has_edge() to use RzGraphStatus.
* Refactor add_edge() to use RzGraphStatus.
* Fix leak of b
* Fix leak of xref list
* Fix leaks of analysis ops
* Address review comments
* Inlcude clean up
* Fix invalid free
* Add tests with node and edge data.
* Extend tests
Various issues related to pointer size and RzVector behavior.
Testing rz_pvector_shrink() at the place removed here is not necessary
as it has its own dedicated tests.
* GRAPH: Add rz_graph_add_get_node() and refine API.
* GRAPH: Remove unused, and rename parameters.
* Improve doxygen.
* Implement rz_graph_del_edges()
* Rename rz_graph_node_get_hash_id() to rz_graph_node_get_hash_id() to make clear what identifier is returned.
* Simplify node identification.
Removes the option to have two sources of identifiers (node data or other identifier data).
Changes the API to use the hash id instead of a pointer to data.
* Fix type annotations.
* Remove duplicate function.
* Fix invalid asserts.
* Set flag if node was present.
* Grow by a factor of 1.25. Exponential growth quickly leads to OOM.
* Remove the edge index again to not remove reduce the main advantage of an adjacency matrix
* Refactor list based graph to use vectors instead of hash maps for edges.
* Fix heap-use-after-free
* Add an rz_graph_update_edge function.
* Add function to print graph as dot graph.
* Add warning about del_edges runtime.
* Refactor add_node to use RzGraphStatus.
* Refactor del_node to use RzGraphStatus.
* Use cast-macro to prevent ASAN issues.
Register roles (RzRegisterId) are now associated directly with an
RzRegItem in RzReg rather than mapping to name strings, which previously
needed an additional hashtable lookup to get more information about the
register. Conversely, if the name is needed from an RzRegItem, it is
available directly as a member.
This is not a pure refactor as there were cases before where a register
name was assigned to a role in the register profile, but no register
actually existed under that name. Such cases will now cause a warning to
be printed during profile load and the role association will be ignored.
Changes in register profiles in this commit are for fixing such cases.
* Remove JOP and COP from ROP search, Unify terminal instruction checks and Fix conditional terminator filter
* Fix detail search for delay slot archs
* Add regression test for consistency check b/w /R and /Rg
* Remove redundant analysis from test, cmd_rop test time reduced to 18sec from 22 sec
* Add getter for vector capacity.
* Make rz_(p)vector_assign_at consistent in behavior.
* Document rz_vector_clear
* Move doxygen for rz_vector_flush to definition.
* Add rz_vector_purge for clearing, but capacity keeping.
* Rename flush -> take_array to signal ownership transfer.
