Information about the final register binding and memories is needed
independently of a stateful vm, specifically for analysis.
This is a pure refactor.
* arch/tms320: add TMS320C54x disassembly support
Add a C54x instruction decoder that reuses the shared C55x decode engine
(c55_decode/c55_format) via the C55ArchDesc plug-in interface, rather than
duplicating the matcher/formatter. Disassembly only for now (.lift = NULL).
Engine changes (c55_ir.c/.h):
- add C55ArchDesc.words_le so the decoder can byte-swap the little-endian
16-bit instruction words used by the C54x COFF object format;
- add a self-contained C54x memory-operand renderer (direct @dma, MMR,
indirect *ARx with all post-modify modes, *ARx(lk) const-index, *(lk)
ABS16 absolute and circular '%' addressing) and bare-hex immediates;
- add C55Operand.circular for the '%' suffix and C55Operand.space_join
for the space-separated second half of a C54x parallel instruction;
- extend the data-memory operand-field analysis (register, base pointer,
displacement, direction, referenced size) to the LOAD/STORE op types the
C54x ld/st family uses, in addition to the C55x MOV form.
The C54x decoder (isa/tms320/c54x/c54x.c) covers the complete documented
instruction set - all 117 mnemonics of the SPRU172 opcode map, in every
documented encoding form:
- load/store/move, integer and logical ALU ops in every addressing form
(Smem, #lk, dual-accumulator, Xmem/Ymem, TS/ASM/SHIFT-shifted, the
shift-by-16 and #lk,16 long-immediate forms, and the two-word
Smem,SHIFT form whose operation selector lives in the second word);
- the full multiply/MAC family: Smem, #lk, program-memory, squaring,
multiply-by-A, signed-unsigned and the dual-operand MAC[R]/MAS[R]
Xmem,Ymem forms;
- the parallel (dual-operation) class rendered "op1 .. || op2 .." -
ST||ADD/SUB/LD/MPY/MAC[R]/MAS[R], ST||LD T and LD||MAC[R]/MAS[R];
- double/long-word (Lmem) add/subtract, the unary accumulator ops
(exp/norm/abs/neg/rnd/sat/min/max/rol/ror/sftc/cmpl/...);
- control flow with the separate delayed (bd/calld/bcd/banzd/fcalad/...)
variants, conditional return/execute (rc[d]/xc) and the multi-condition
"tc, c"-style combinable condition fields, repeats (incl. rpt #lk),
conditional stores, I/O port access, status-bit set/clear and the
non-linear idle encoding.
Operands resolve to their architectural names - the full memory-mapped
register file (AR0-AR7, the accumulator AL/AH/AG/BL/BH/BG halves, T, TRN,
SP, BK, BRC/RSA/REA, IMR/IFR, PMST, XPC), the ST0/ST1 status bits and the
named condition codes; the memory-mapped-register operand is kept single
word (its long-offset modes are not legal). The analyzer classifies every
instruction (op->type, op->id), resolves branch/call targets and the stack
effect of calls/returns/pushes, and exposes operand details: the register,
base pointer, displacement and access direction of data-memory loads and
stores, and the target register of indirect branches/calls.
All encodings were verified byte-exact against the TI asm500 assembler,
and every decoded instruction re-assembles to an identical encoding (a
full-opcode-space disassemble/reassemble round-trip is stable). A 297-case
disasm test suite and an analysis test suite (opcode classification, branch
and call targets, stack effects, memory-operand fields, data-immediate values, the register
profile, named instruction ids and COFF binary-fixture function discovery)
are added, and the real-world emulateme C54x .text decodes cleanly.
* arch/tms320: add TMS320C54x RzIL lifting
Lift the C54x integer core to RzIL so emulation and IL-based analysis work
for C54x as they already do for C55x/C55x+.
- Register profile: C54x previously fell through to the C64x profile
(a0-a31, =PC pce1), wrong for the A/B accumulator core. Add a proper
C54x profile: the two 40-bit accumulators A/B (with the L/H 16-bit and
G 8-bit guard slices overlapping their parent), AR0-AR7, T/TRN, SP, DP,
BK, ST0/ST1/PMST, BRC/RSA/REA, IMR/IFR, XPC and a 24-bit PC.
- IL VM config: tms320_c54x_il_config() binds the canonical registers; the
accumulator slices stay unbound, the lifter expresses them as bit-slices
of A/B so they never desynchronise.
- Lifter (C55ArchDesc::lift hook, dispatched by c55_lift): the no-shift
forms of LD/LDU/LDR/LDM, ADD/SUB/AND/OR/XOR, STL/STH/STLM/STM, the mvd*
memory-to-memory moves, the DLD/DST 32-bit double-word load/store (high
word at the lower address), PSHM/POPM and RET. Shift/round/saturate
variants are left unlifted (their shift count is carried only as a
display string); the engine's generic EA/read/write/post-modify helpers
are reused for the addressing modes.
Tested via two new RzIL VM blocks in test/db/rzil/tms320: a register/
immediate/memory execute test, and an end-to-end emulation of the
emulateme binary's _decrypt (a UART hex-writer) showing the IL VM emits
the hex digits and advances the write position.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* Cast operands for AND and OR instructions to the correct width
* Add missing operand casts for SBB and ADC.
* Add flawed instructions to asm tests
---------
Co-authored-by: Dhruv Maroo <dhruvmaru007@gmail.com>
Verified against the TI dis55 disassembler and the C55x+ Algebraic
Instruction Set (SWPU104) encoding tables.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Builds on the shared decode-IR engine to lift a broad set of common C55x and
C55x+ (Ryujin) instruction forms that were previously left without RzIL,
deriving the exact semantics from the TI C55x+ references (SWPU104 / SWPU086).
Control flow and addressing:
- ret / reti / retcc: return address read from the top of stack, SP popped by
two words, control transferred to it (retcc guarded by its predicate).
- 24-bit XAR correctness on C55x+ (XAR0-15, XSP, XSSP, XDP, XCDP) in the
register table, the IL-VM profile and the pointer-arithmetic width, enabling
the 24-bit amov address immediate; classic C55x keeps its 23-bit file.
- long constant-index and 16-bit-absolute addressing modes, memory-to-memory
copy, shifted memory loads (uns()/signed, immediate shift), and the
memory-mapped-register moves mov reg,mmap(@reg) / mov mmap(@reg),reg.
- push/pop of accumulator sub-register halves and dbl(xarN) pointer pairs.
Arithmetic, logical, shift and bit operations:
- sub-register add/sub on a 16-bit accumulator slice, for the immediate and
register forms: a .L destination updates [15:0], a .H destination updates
[39:16] sign-extended through the guard (SWPU104 1.5.1).
- bitwise and/or/xor with a 16-bit source (half or AR/T) into a full
accumulator, zero-extending the operands to 40 bits (SWPU104 6.6.1); the
memory-source bitwise forms into full and half destinations; and the
shift-ALU forms <op> ACx.<sub> << #S6, ACy.<sub> on the 16-bit slice.
- memory-source add (including a 16-bit half addend and the reverse-subtract
sub ACx.<sub>, Smem, ACy.<sub>), the 32-bit dbl(Lmem) add/sub forms
(ACy = ACx +/- dbl(Lmem) and the reversed ACy = dbl(Lmem) - ACx), and the
memory-destination immediate RMW add/sub #k, Smem.
- register bit ops bclr / bset / bnot @#k, ACx[.h/.l] / ARx: clear, set or
toggle bit k of the register, the bit number taken relative to the
addressed sub-register (a .h operand targets bit k+16, the guard k+32).
- bitwise not ACx.<sub>/ARx into an accumulator half or a 16-bit register
(not ACx.l, ARy), the half-register and short (0x7b #1/#-1) sftl/sfts
including register-count shifts, and btst @#k, ACx.l/.h, TCy.
This also corrects an op_type-fallback mis-lift: on C55x+ several non-move
instructions (round, sat/satr, the mant/nexp helper) and the bit-field
extract/expand bfxtr/bfxpa were typed as a move and so were lifted as a plain
register copy. round and sat/satr now carry their lops and reach the existing
rounding / saturation handlers (matching C55x); mant/nexp, bfxtr/bfxpa, the
operand-less sat and the register-indexed memory bit ops (bclr/bset/bnot Baddr)
are marked decode-only (no modelled data effect) so the fallback can no longer
guess at them.
Also fixes a FIRSADD/FIRSSUB lifting bug: the Cmem operand was not converted
from AR to XAR before emitting its pointer post-modify, so a post-modified Cmem
(e.g. firssub *ar3-, *ar5-, *ar6-, ...) produced an invalid 16-bit-vs-24-bit
subtract that failed IL validation; Cmem is now widened like Xmem/Ymem.
The multiply/MAC family stays unlifted by design, as do the flag-predicate
retcc forms (whose condition register field is not represented by the decoder)
and the software-interrupt intr (which would need the interrupt-vector base);
the correct-or-NULL contract test asserts a representative deferred form.
The disassembler tests for both variants are extended with the expected IL for
every form that lifts -- previously the optional IL field was omitted on many
lines whose lift already existed, leaving the lifting unchecked -- so the asm
suite now validates RzIL for all lifted C55x / C55x+ instructions, not just a
subset.
Measured on a 5 MB C55x+ firmware image (16k-instruction sample) RzIL coverage
rises from 82% to 99.4% of decoded instructions, the remaining tail being the
multiply/MAC family and the deferred forms above.
Replace the per-variant, table/token-driven disassembly and lifting for the
TMS320C55x and C55x+ DSPs with a single shared decode-IR layer, c55_ir, that
both variants drive through a C55ArchDesc descriptor. The shared engine
table-walks an instruction (c55_decode), formats it (c55_format), fills the
analysis op (c55_fill_analysis), and lifts to RzIL (c55_lift) arch-agnostically,
with the opcode tables, register files, operand extractors, and op-type/lift
mappings supplied per variant (c55x and c55x_plus).
Both test corpora decode and lift entirely through the shared engine: forcing
the shared-only decode path and, separately, the shared-only lift path each pass
the full suite -- 1288 tests including the three RzIL VM emulation tests, whose
decrypt loops execute every instruction through the shared IL. The legacy
decoder and lifter are retained only as a fallback for a complex long-tail (the
parallel dual-MAC group, dual-memory addsub/subadd, the absolute-k24 and dbl
multi-operand loads/stores, and the bit-field forms); the shared path is used
when it can decode an instruction and control falls through otherwise, so
behaviour is preserved at every step.
Several forms are extended beyond the old engine: the full C55x+ register file
(ac0-31, ar/xar0-15, CPU-gated in the profile and IL VM) so extended-register
forms lift where the legacy lifter bound only the low eight; pc-relative and
compare-and-branch control transfer; the bcc flag predicate; and the
single-data-memory bitwise and address-arithmetic forms (and/or #k16,Smem and
register-mode amar Smem,xar) decoded cleanly with RzIL rather than inherited from
the legacy tables.
Squash of rizinorg/rizin PR #6434 ("improve TMS320C55x+ analysis and RzIL")
rebased onto dev, with the PR head's doubled c55plus_il.c (every symbol defined
twice, failing to compile) de-duplicated to a single clean copy.
Substantially extends the C55x/C55x+ RzIL lifter over the existing structured-
operand helpers: mov/copy (immediate, register, memory load/store, half-register
read-modify-write), the full addressing-mode set with post-modify side effects,
control and system-register moves, 40-bit accumulator ALU with shifted sources,
16-bit and dual-memory add/sub, the ST0_55 status-flag model (cmp/cmpand/rol/ror,
named-bit bset/bclr), a documented psh/pop stack model, address-unit amov/aadd/
asub and amar, and bcc/callcc control transfer. The multiply/MAC family and satr
are lifted with explicit, documented integer-mode approximations (not verified
DSP semantics); irreducibly multi-output primitives (bit counts, Viterbi, FIR,
distance) are left correct-or-NULL. The register file covers ac0-7 and xar, found
by validating on real Motorola Wrigley C55x+ firmware whose prologues save 40-bit
accumulators as dbl(acN)+acN.g pairs.
Adds RzIL-VM emulation tests (including the C55x and C55x+ _decrypt emulateme
binaries), per-instruction IL assertions, and ~95% instruction-class disassembly
coverage per corpus; pins little-endian in the VM tests for big-endian hosts; and
regenerates the analysis expectations against current dev.
Add a new LGPL-3.0 VAX-11 architecture plugin that replaces the removed
binutils-derived GPL one. It is written from scratch from the documented
VAX operand-specifier encoding and does not reuse any GPL code.
Drop the old GPL-licensed VAX disassembler (vax_gnu), which was derived
from GNU binutils and gated behind the use_gpl build option. It is being
replaced by a new, clean-room LGPL implementation in a following commit.
Removed:
- librz/arch/isa_gnu/vax/vax-dis.c
- librz/arch/isa_gnu/vax/vax.h
- librz/arch/p_gnu/arch_vax.c
- librz/arch/p_gnu/asm/asm_vax_gnu.c
- librz/arch/p_gnu/analysis/analysis_vax_gnu.c
and their entries from the use_gpl plugin/source lists in
librz/arch/meson.build.
location_by_biggest_range() computed each location-list entry's PC-range
size as (begin - end). For a normal [begin, end) range (begin < end) this
underflows and wraps to a huge ut64, so the entry with the *smallest* span
was always chosen as a variable's single representative storage instead of
the largest.
This breaks functions whose register arguments and locals are described by
location lists, e.g.
item: [low, X): DW_OP_reg0 ; [X, high): DW_OP_reg8
input_buffer: [low, Y): DW_OP_reg1 ; [Y, high): DW_OP_reg10
with DW_AT_frame_base = DW_OP_call_frame_cfa (.debug_loc + DW_AT_GNU_locviews,
no .debug_loclists). The wrongly-picked short entry is frequently one that
does not resolve to a valid RzAnalysisVarStorage (e.g. an implicit
DW_OP_stack_value piece), leaving the variable with EVAL_PENDING storage. The
affected variables then fail to materialize and 'afv'/'afvl' reports nothing
for the whole function -- even though the arguments live plainly in registers
and need no CFA computation.
Computing the span as (end - begin) selects the genuinely largest range, so
each variable resolves to the register it occupies for most of the function
and the register arguments load correctly.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
MIPS PIC code calls a function by loading its address from the GOT into
$t9 and then doing `jalr t9` (or `jr t9` for a tail call), e.g.:
lw v0, -sym._MIPS_STUBS(gp) ; v0 = *(gp + %call16(puts))
move t9, v0
jalr t9 ; -> puts
rizin did not turn this into a call to the imported function, so the
target was lost: rz-ghidra rendered it as an indirect `(*_data.XXXX)()`
instead of `puts(...)`, whereas a direct `jal sym.dummy` decompiled fine.
Root cause (two independent gaps):
1. For (R|U)CALL and RJMP ops the core creates the CALL/CODE xref from
op->ptr, not op->jump (see core_analysis_followptr() and the op-type
switch in librz/core/canalysis.c, RZ_ANALYSIS_OP_TYPE_RCALL/RJMP). The
MIPS plugin only ever set op->jump for `jalr`/`jr`, so no call xref was
produced even when $t9 was tracked, and the decompiler never saw a call
target.
2. $t9 was only tracked when it was the *direct* destination of a
gp-relative load (`lw t9, ...(gp)`). The very common sequence that loads
into another register first and then `move t9, vX` was not tracked, so
even op->jump was left unset there (this is the issue's binary). Note
that capstone emits `move t9, vX` as the 2-operand alias of `or` (and on
some toolchains `addu`/`daddu`), i.e. `or t9, vX, $zero`, so the move
must be recognised across MIPS_INS_MOVE *and* the 2-operand OR/ADDU
forms.
Additionally, the tracked value was the GOT *slot* address, while the call
target is the function the slot points to, so the slot has to be
dereferenced.
This commit:
- tracks the destination register and slot of every gp-relative load
(gp_load_reg/gp_load_ptr in MIPSContext), and propagates it to $t9 on a
register move (MIPS_INS_MOVE, or the 2-operand OR/ADDU alias), so the
PIC sequence above is recognised;
- adds mips_pic_call_target(), which dereferences the GOT slot via the
analysis IO bind (honouring word size and endianness) to obtain the
actual callee;
- sets op->ptr (and op->jump) to that resolved address for `jalr t9`
(RCALL) and `jr t9` (RJMP tail call), so the core emits the proper
CALL/CODE xref and the decompiler resolves the callee like it does for
`jal`.
The resolution is best-effort and fully guarded: if gp is unknown, the
slot cannot be read, or it holds 0, op->ptr is left unset and behaviour is
exactly as before. The core additionally validates the target
(is_valid_xref) before creating the xref, so a stale/garbage slot cannot
introduce a bogus call, and the propagation only fires when $t9 is the
destination, so ordinary moves (`move fp, sp`, ...) are unaffected.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
When a function is relocated, each variable access offset (stored relative
to the function's entry point) is rebased by the relocation delta so that the
accesses keep pointing at the same absolute addresses. This was done as a
signed st64 subtraction:
st64 delta = addr - fcn->addr;
...
acc->offset -= delta;
Relocating to an address near the int64 boundary makes delta close to
INT64_MIN, and 'acc->offset - delta' then overflows st64. UBSAN aborts:
librz/arch/function.c:241:16: runtime error: signed integer overflow:
65568 - -9223372036854710512 cannot be represented in type 'long int'
(reproducible via test/unit/test_analysis_var, which relocates to
0x8000000000000010 and 0x7ffffffffffffe00).
Addresses and their differences are meant to wrap modulo 2^64, and the
offsets are only ever looked up as exact values (get_vars_used_at computes
op_addr - fcn->addr in ut64 as well), so perform the arithmetic in ut64.
delta becomes ut64 and the rebase is '(st64)((ut64)acc->offset - delta)';
the result is bit-identical for every non-overflowing case and well-defined
for the rest. The inst_vars rebase callback already subtracted in ut64.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Follow-up to the C bitfield support added in #6439 (#1240, #314): the PDB
type parser dropped struct/union bitfield members because an LF_BITFIELD
member type resolved to NULL in pdb_type_parse(), so "tc"/"ts"/"tp" on a
PDB-loaded type lost every bitfield field.
member_parse() now detects a TpiKind_BITFIELD field type, resolves the
member to the bitfield's underlying integer (base_type) and reports the
bit width (length) via an out-parameter. class_member_parse() and
union_member_parse() store it in RzTypeStructMember.size /
RzTypeUnionMember.size (the bitfield width in bits, 0 if not a bitfield),
matching the convention used by the C and DWARF member paths.
Update the PDB type expectations that previously asserted bitfield_typedef
had no members: db/cmd/cmd_idp (idpij) and db/tools/rz_bin (rz-bin -Pj)
now expect the three resolved "unsigned char" members of minimal.pdb.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
An enum type hint set on an operand (ahie) only replaced the immediate when its
value matched an enum member exactly, via rz_type_db_enum_member_by_val(). A
value that is the bitwise OR of several flag members -- e.g. the access(2) mode
R_OK|W_OK == 6 from the issue -- matched no single member and was left as a raw
number.
replace_enum_hint() in rz_parse now falls back to rz_type_db_enum_get_bitfield()
when there is no exact member, so the value is rendered as the OR of the
matching members, e.g. "access_def.W_OK | access_def.R_OK".
rz_type_db_enum_get_bitfield() is reworked along the way: it was unused and
buggy (capped at 32 bits, reused a stale match for bits without a member, and
emitted a "0x.. : " debug prefix). It now walks all 64 bits, returns the
matching members qualified with the enum name and joined by " | ", and returns
NULL when the value is 0, the type is not an enum, or any set bit has no member
(so a value that is not cleanly a combination of flags stays a number).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
The C grammar already exposes a struct/union member's bitfield width as a
"bitfield_clause" node and the parser already recognized "int a : 4;", but
it threw the width away (the member was stored with size 0 and a FIXME). As
a result "tc" rendered the member as a plain "int a;", "ts" produced a
full-int "pf" format ("pf d4d4d4 a b c") and "tp" mis-read every field as a
whole integer.
RzTypeStructMember and RzTypeUnionMember already carry a "size" field
documented as "in bits"; it is now used as the bitfield width, where 0 means
the member is not a bitfield:
- The struct and union member parsers store the parsed bit width on the
member instead of discarding it.
- The pretty printer emits " : <width>" before the member's trailing ';',
so "tc"/"tcd" round-trip "struct qwe { int a : 4; int b : 16; int c : 3; }".
- rz_base_type_as_format() (used by "ts"/"tp") now emits the pf packed-bits
spec ":N" for a bitfield member, with the bit order matching the target
endianness ("<" little-endian / ">" big-endian), instead of the member
type's full format. "ts qwe" becomes 'pf ":4<:16<:3< a b c"' and "tp"
unpacks the fields correctly (a=5, b=0x1234, c=0 for 0x00012345 on x86).
Serialization keeps the width in the 3rd comma-field of the existing
"struct.<name>.<member>" / "union.<name>.<member>" value, i.e.
"type,offset,bitsize". That field was already written (always as 0) and
ignored on load, so the on-disk layout is unchanged and only its meaning is
now honored. The project version is bumped to 24 with an additive no-op
migration: an absent or 0 bitsize deserializes as a regular, non-bitfield
member.
Tests: a unit test parses a bitfield struct and union and checks the member
widths, the rendered string and the "pf" format; a db/cmd/types test covers
the full "td"/"tcd"/"ts"/"tp" flow; a project-migration test with a new
v23-bitfield.rzdb fixture covers v23->v24, and the "migrate info" db test is
updated for the new version.
The PDB type parser (librz/arch/pdb_process.c) still drops bitfield members
because their LF_BITFIELD member type resolves to NULL; wiring LF_BITFIELD up
to the new member "size" field, with tests in test/unit/test_pdb.c, is left
as a follow-up.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* Update Capstone and add m68k ColdFire support
* Fix test 'core regs linux m68k'
* Add M68K ColdFire integration coverage
* Update Capstone v6 to version 6.0.0-Alpha9
* Refactor CPU mode detection to use case-insensitive string comparison
* Update Capstone next revision to 3df6ff0
With bin.demangle=true the call/jump operand is replaced by the full
demangled symbol name and then colorized token-by-token before being
filtered. For long C++ symbols (e.g. libc++ STL names) the colorized
operand easily exceeds the 1024-byte operand buffer (ds->str): in
truecolor each token is wrapped in a ~19-byte "\x1b[38;2;R;G;Bm" escape,
so a ~110-char name grows past 1700 bytes.
The generic filter() copied that operand with
strncpy(str, data, len);
which has two problems when strlen(data) >= len:
- it does not NUL-terminate, so reading str ran past the buffer into
the adjacent strsub[] buffer, printing stale bytes (the "call 0x..."
bleed); and
- the byte-bounded cut can land in the middle of a color escape,
leaving a partial "\x1b[38;2;.." that, followed by the bled bytes,
forms a complete but bogus "\x1b[..c" sequence. Terminals read that
as a Device-Attributes query and reply with e.g. "62;4c", which is
the garbage the reporter saw at the prompt.
This only triggers with color enabled and a symbol long enough that its
colorized form overflows the buffer, which is why it showed up on some
x86 files only and had no portable reproducer (no-color = 149 bytes,
16-color = 763, 256-color = 1170, truecolor = 1713 for the sample
symbol; the threshold is 1024).
Add test/unit/test_parse.c driving rz_parse_filter() with over-long
colored operands (a deterministic case whose cut lands inside an escape,
and a realistic libc++-style colorized operand) plus a short-operand
passthrough case. The two overflow tests fail on the old strncpy and
pass with the fix.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Most of the Windows types reported in issue #3728 were stub entries of
the form
NAME=type
type.NAME=p
i.e. opaque atomic pointers with no actual body. rz-ghidra rejects
these as 'Invalid atomic type'. Convert them to proper typedef entries
backed by either real struct definitions (for well-known types) or
forward-declared opaque structs (for architecture-specific ones like
CONTEXT and KNONVOLATILE_CONTEXT_POINTERS, whose layout varies across
x86/x64/ARM/ARM64 and which the shared types-windows.sdb.txt cannot
commit to).
Specifically, this commit:
* converts the stubs for PSID, LPOLESTR, LPCLSID, PCONTEXT,
PKNONVOLATILE_CONTEXT_POINTERS, LPSYSTEM_INFO, LPWIN32_FIND_DATAW,
PCACTCTXW, PSID_AND_ATTRIBUTES, PLUID_AND_ATTRIBUTES, PTRACEHANDLE
and the duplicate LPMSG/PLUID into proper typedefs;
* adds new top-level typedefs for CLSID, OLECHAR, LPCOLESTR
(corrected to const OLECHAR *), REFCLSID, LPBC, MSG, PMSG, NPMSG,
LPMSG, SYSTEM_INFO, WIN32_FIND_DATAW, PWIN32_FIND_DATAW,
OSVERSIONINFOW, POSVERSIONINFOW, LPOSVERSIONINFOW,
RTL_OSVERSIONINFOW, PRTL_OSVERSIONINFOW, OSVERSIONINFOEXW (plus its
POSVERSIONINFOEXW/LPOSVERSIONINFOEXW siblings), ACTCTXW, PACTCTXW,
LUID, PLUID, LUID_AND_ATTRIBUTES and SID_AND_ATTRIBUTES;
* adds struct bodies for tagMSG, _OSVERSIONINFOW, _OSVERSIONINFOEXW,
_SYSTEM_INFO (with the union flattened to wProcessorArchitecture +
wReserved per modern usage), _WIN32_FIND_DATAW, tagACTCTXW,
_SID_AND_ATTRIBUTES, _LUID, _LUID_AND_ATTRIBUTES and the empty
opaque tags _CONTEXT, _KNONVOLATILE_CONTEXT_POINTERS and _IBindCtx;
* fixes a small typo in struct._EVENT_DATA_DESCRIPTOR where the
field list said 'Usize' while the per-field entry was 'Size'.
Member layouts were taken from the Microsoft Win32 documentation
(winnt.h, sysinfoapi.h, winuser.h, evntrace.h, minwinbase.h,
winbase.h) and cross-checked with the Wine include/winnt.h,
wtypes.idl and guiddef.h headers.
Offsets follow the 32-bit Windows ABI used by the rest of this file
(this matches the existing _OBJECT_ATTRIBUTES, _FILETIME etc. layouts).
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
* librz/type: rewrite pf format parser
Replace the legacy print-format engine in librz/type/format.c with a
clean three-stage pipeline (parse -> read -> render) and a properly
typed DSL.
The new pipeline:
rz_pf_parse(str) source string -> RzPfFormat
rz_pf_read(fmt, buf, len, ctx) RzPfFormat -> RzPfValue[]
rz_pf_render(vals, mode, opts) RzPfValue[] -> printable string
rz_pf_format() one-shot for all of the above
Public API lives in <rz_pf.h> (pulled in transitively via <rz_type.h>).
DSL improvements over the legacy scheme:
* Sized integers with explicit endianness via case:
x4 d4 u4 o4 b4 (LE) vs X4 D4 U4 O4 B4 (BE).
* Sized floats: f2 / f4 / f8 (and BE F2 / F4 / F8).
* Encoding-aware strings: z(utf8) / z(utf16le) / z(utf16be) /
z(utf32le) / z(utf32be) / z(latin1) / z(ebcdic).
* Length-prefixed strings: z[N] reads a N-byte LE prefix then body.
* Parameterized timestamps: t(unix32) / t(unix64) / t(unixms) /
t(unixus) / t(unixns) / t(filetime) / t(dos) / t(hfs) /
t(oletime) / t(webkit) / t(cocoa). ntfs alias maps to filetime.
* GUID: G(ms) (Microsoft mixed-endian, default), G(be), G(le).
* Inline bitfields: B4(FLAG_A=1,FLAG_B=2,FLAG_C=4).
* Alignment: @N advances to next multiple of N.
* Bit fields: :N with optional < (LSB-first) or > (MSB-first).
* Length-by-reference arrays: [@earlier_field]T.
* TLV records: V(t=u1,l=u2,e=le,h=v|l|a,d=table). Dispatch table
uses tlv.<name>.<hex-tag> in the typedb formats hash.
Architecture highlights:
* Per-instance ReadState (bit_cursor + sibling lookup window).
The bit cursor snap-flushes to the next byte when a non-bit
field follows mid-byte.
* Recursion bound via RzPfCtx::max_depth (default 32).
* Five render modes: text (default), json, cstruct, quiet, dot.
The dot renderer emits a column-aligned record with offset/type/
name/value rows, suitable for direct rendering through Graphviz.
* Positioned diagnostics: RzPfError carries severity, category,
column position, and a human-readable message. Each is also
emitted via RZ_LOG_WARN for backward compatibility.
* Optional RzPfPalette and RzPfRenderOpts threaded through render
for inline ANSI colorisation. NULL palette = canonical text.
* Pointer dereference via RzPfCtx::read_at callback.
The integration test expectations in test/db/cmd/cmd_pf* are
updated in the same commit to match the new text-output format and
the new DOT renderer's column-aligned record layout. Bundling these
keeps the test suite green at this commit.
A transitional rz_type_format_data() shim at the bottom of
pf_parser.c forwards legacy callers (cprint.c, disasm.c) to the new
API; it is removed in the librz/core migration commit.
* librz/arch/types: migrate type DB to new pf format codes
Map every legacy single-character format code in the bundled type
SDB files to its new-DSL equivalent:
b / C -> x1 (1-byte hex)
c -> c (signed char, unchanged)
w -> x2 (2-byte hex LE)
i -> d4 (signed 32-bit decimal LE)
d -> x4 (4-byte hex LE)
q -> x8 (8-byte hex LE)
f -> f4 (32-bit float LE)
F -> F8 (64-bit float BE)
Z -> z(utf16le)
x -> x4
o -> o4
X -> r (hexdump)
n / N -> d4 / u4
Also add new typedef-only types for explicit display formats:
bin{8,16,32,64}_t -> b{1,2,4,8} (binary)
hex{8,16,32,64}_t -> x{1,2,4,8} (hex)
oct{8,16,32,64}_t -> o{1,2,4,8} (octal)
These let users opt into a display representation per field without
changing the underlying scalar size.
Update the corresponding integration test expectations in
test/db/cmd/cmd_avg and test/db/cmd/types: the 'tu', 'tuc', 'tp',
and 'avgp' commands report formats via the SDB, so the migrated
codes appear in their text output (e.g. 'pf 0f4x4 a b' instead of
the legacy 'pf 0fd a b' for a union of float + int).
* librz/core: rewrite pf integration on the new parser API
Remove the legacy rz_type_format_data() bridge and rewrite every
caller to use the new <rz_pf.h> API directly.
librz/core/cprint.c
core_print_format() constructs an RzPfCtx populated from the
RzCore (typedb, big_endian, bits, max_depth) and calls
rz_pf_format() in one shot. The bridge between RzCore I/O and the
pf reader is cprint_pf_read_at(), which forwards to rz_io_nread_at.
The bitmask-to-RzPfMode mapping is cprint_pf_mode().
For DOT mode the format name is passed via RzPfRenderOpts::graph_label
so the dot renderer uses it as the top-level record label.
When scr.color > 0 in TEXT mode, an RzPfPalette is populated from
the active color theme via RzConsPrintablePalette so `pf` output
blends with the rest of Rizin's UI and respects user theme
choices (eco / ec). The palette slots map to theme fields as:
pf field offset <- pal->offset (address column)
pf field name <- pal->fname (symbolic name)
pf endian marker <- pal->meta (metadata tag)
pf hex/number literal <- pal->num (numeric literal)
pf typedb label <- pal->flag (resolved symbol)
reset <- pal->reset
Each slot falls back to its previous hardcoded ANSI escape if the
theme field is unset, preserving prior behaviour for stripped-down
contexts. Other modes (json/cstruct/quiet/dot) ignore the palette.
Closes rizinorg/rizin#782.
librz/core/disasm.c
RZ_META_TYPE_FORMAT case rewritten to resolve via rz_pf_resolve_name
and decode via rz_pf_format() directly. Palette is supplied from
the same theme palette when ds->show_color is true so that
pd @ <struct> blends with the surrounding listing.
librz/include/rz_type.h
Drop the rz_type_format_data() forward declaration. Callers must
use the public <rz_pf.h> surface instead.
librz/core/cmd_descs/cmd_print.yaml
Rewrite the pf command help for the new DSL: sized integers
(case-discriminated endian), special scalars (G GUID, V TLV,
: bits, @ align), strings with encodings, parameterized timestamps,
typed composites (E enum, B bitfield, ? struct), DSL extensions
(length-prefix strings, length-by-reference arrays, inline
bitfields), skip/repeat/pointers, example invocations.
cmd_descs.c is regenerated from the YAML.
* test: pf parser unit and DSL-extension integration tests
Add fresh test coverage for the new pf parser. These are purely
additive: existing integration tests were updated in the parser
and SDB-migration commits so the suite stayed green throughout.
test/unit/test_pf.c (new, 131 tests)
Field-size tables (every fixed-size type) and ctype mapping, every
parse path (sized integers / floats / strings / timestamps /
pointers / structs / enums / bitfields / GUIDs / TLV / alignment /
bits / arrays), every read path (with both fixed and length-by-
reference array counts), every render mode (text / json / cstruct /
quiet / dot), every DSL extension (align, bits MSB/LSB, GUID layout,
length-ref array, length-prefix string, inline bitfield, TLV with
dispatch table), pointer dereference via an in-memory I/O callback,
recursion safety on self-referential structs, lifecycle / NULL
safety, diagnostics (positioned errors, all categories, caret-line
formatter, source capture, verbose parse), ambiguity (each
superficially similar DSL form parses unambiguously), the palette /
colorisation API (issue #782) including a regression test surfaced
by the property-based test harness for input bytes containing ESC,
and DOT-mode coverage (column-aligned layout, single field,
empty/filtered, timestamp values, TLV records).
test/db/cmd/cmd_pf_dsl (new)
8 integration tests targeting the new DSL extensions specifically:
@N alignment, :N bits in MSB and LSB ordering, default-layout GUID,
z[N] length-prefixed string, B4(K=V) inline bitfield, bare V TLV
with defaults, V(t=u2,l=u2,e=be) configured TLV.
test/db/cmd/types_format (new)
Verifies that the new bin{N}_t / hex{N}_t / oct{N}_t typedefs apply
the expected display representation when used inside struct fields
via the 'tp' command, demonstrating the per-base rendering
(binary / decimal / octal / hex).
* doc: pf DSL reference and librz/type README
doc/pf.md (new)
User-facing reference for the pf format DSL covering quick
examples, spec grammar (sized integers with case-discriminated
endian, floats, strings with encodings and length prefixes,
parameterized timestamps, composites -- struct ?, enum E, bitfield B
inline and typed, GUID G, TLV V, raw hexdump r), repetition and
arrays ([N], [@name], {N}, leading 0 for union), padding and
alignment (. skip, @N align, :N bits MSB / LSB), pointer
dereference (*<type>), names grammar (plain vs (typename)name),
output modes (text, json, cstruct, quiet, dot), and diagnostics
(severity / category / position / caret-line formatter; backward-
compatible RZ_LOG_WARN channel).
librz/type/README.md (new)
Subsystem architecture doc for librz/type. Covers public headers,
file map, the pf three-stage pipeline (parse -> read -> render) with
a diagram and entry-point table, per-stage explanations (parser
shape, ReadState scoping rules including bit cursor and sibling
lookup window and snap-flush rule, render mode dispatch including
the DOT column-aligned layout and the graph_label parameter),
typedb integration and recursion bound, error-reporting model,
test coverage summary, and pointer to doc/pf.md as the user-facing
reference.
* librz/type: remove legacy DSL conversion shim
All in-tree callers -- librz/bin/d/, librz/bin/format/, and the
integration tests in test/db/cmd/ -- have been migrated to the new
DSL in the earlier commits of this series. The compatibility shim
that translated bare legacy specifiers (x -> x4, b -> x1, w -> x2,
nN -> uN, NN -> UN, etc.) into the new DSL at the entry of
rz_pf_parse() is no longer load-bearing; this commit deletes it.
- librz/type/pf_parser.c: drop the 267-line
convert_legacy_to_new_dsl() function, the WARN_LEGACY macro,
and the per-parse allocation + free of the converted string.
rz_pf_parse() now walks the caller's spec directly.
- test/db/cmd/cmd_pf,cmd_pf2,cmd_pf_write,cmd_pfd,cmd_pf_new,
metadata: migrate legacy specifiers in CMDS blocks to the new
DSL (b -> x1, w -> x2, q -> x8, i -> d4, bare x -> x4, etc.)
and regenerate EXPECT blocks against the new render where
affected.
The parser now only accepts the new DSL. Any caller still passing
bare legacy specifiers will get "unknown specifier" warnings.
* librz/core,type: add scr.pf.short for delta-offset rendering
When scr.pf.short is enabled, MODE_TEXT rendering shows offsets as
deltas (+<n>) from the format's base address instead of the absolute
hex address. Useful for self-contained struct dumps where the
absolute load address is noise:
$ rizin -e scr.pf.short=true -qc 'wx 00007a452a4b9a02
pf fcb1d4 a b c d' =
0 : a = 4000 [LE]
+4 : b = '*'
+5 : c = 0b0100_1011 [LE]
+6 : d = 666 [LE]
Nested structs reset their delta from the same base, so a child of
`head` at offset +8 still prints as +0 in its own struct block.
The base is derived from the offset of the first top-level value;
callers can override it via RzPfRenderOpts::base_offset.
Includes 5 integration tests in test/db/cmd/cmd_pf_short covering
basic struct, off-mode parity (control), nested struct, concatenated
TLV records, and @-offset usage.
* librz/type,test,doc: add v(N) bitvector type for forensics use
Adds a new specifier `v(N)` to the pf DSL that reads N individual
bits (1..4096) from `ceil(N/8)` bytes and exposes them as N
separate 0/1 scalars. Forensics targets: page-frame bitmaps, NTFS
$Bitmap clusters, ext4 block/inode bitmaps, ELF DT_FLAGS_1, PE
characteristics, ACL bitmasks -- anywhere you want to *see* a
bitmap rather than collapse it to a hex number.
Grammar:
v(N) N-bit bitvector, default MSB-first per byte
v(N,lsb) LSB-first per byte (Intel order)
v(N,msb) explicit MSB-first (DWARF / network order)
Rendering:
text: [ 1 0 1 0 1 0 1 1 | 1 1 0 0 ] (12-bit)
json: {"bit_width":12,"value":"101010111100"}
quiet: 1 0 1 0 1 0 1 1 1 1 0 0
Bitvec fields do not participate in the packed-bit cursor used by
:N; each v(N) reads whole bytes and stands alone. A v(N) field
adjacent to a :N field flushes the partial bit cursor first.
Width is clamped to [1, 4096] with a RANGE diagnostic. The reader
consumes exactly ceil(N/8) bytes, leaving the cursor positioned for
the next field. Verified by the existing tail-field test pattern.
Tests:
test/unit/test_pf.c +10 unit tests (146 total OK)
test/db/cmd/cmd_pf_new +9 cmd tests (440 OK / 9 BR / 0 XX / 4 FX
across all 19 pf suites)
property harness +7 properties (140 unique props, 1000
trials each, 0 failures)
Docs:
doc/pf.md bitvector section with examples
librz/type/README.md cursor-interaction note for v(N) vs :N
* librz/core,type,test: address PR CI feedback for pf rewrite
Five fixes prompted by CI feedback and reviewer comments on the pf
rewrite series:
* librz/core/cmd_descs/cmd_print.yaml: two help-text lines exceeded the
yamllint 120-char ceiling (n1/n2/n4/n8 comment and the deprecation
notes entry). Both now use the same folded-scalar (comment: >) form
already used in cmd_analysis.yaml and cmd_descs.yaml. cmd_descs.c is
regenerated.
* test/db/cmd/cmd_pf_write: drop the BROKEN 'pf xxd print' test. It
relied on '.pf*' (run pf output as rizin commands) which the new
parser no longer emits -- analogous to the '.pfw' removal done
earlier. The companion 'pf xxd print happy' test exercises the same
path through 'pf.' and stays.
* librz/type/pf_parser.c: thread the RzTypeDB through render_cstruct
and render_dot (and their inner helpers) so RZ_PF_ENUM resolution
works in pfc and pfd mode the same way it already does in text mode.
Without this, 'pfc elf_header' emitted '/* 0x00000002 */' and 'pfd'
emitted '|value|0x0000ffff|'; with it the comments and cells carry
the symbolic name (ELFCLASS64, ET_HIPROC, etc).
scalar_text() RZ_PF_BITFIELD: when no inline B4(K=V,...) flags are
present but val->type_name names a typedb enum, walk that enum and
treat each case as a settable bit name. This makes the typedb form
'B (pe_characteristics) flags' decode to
'0x00008140 : IMAGE_DLLCHARACTERISTICS_DYNAMIC_BASE | ...' instead
of just the raw hex -- previously the bitname decoding only worked
on inline B4(...) forms.
* librz/core/cprint.c: cprint_pf_mode() had no mapping for
RZ_PRINT_VALUE, so 'pfv' fell through to TEXT mode and emitted
'0x00000004 = 0x11111111 [LE]' instead of the bare value '0x11111111'
expected by db/esil/arm_16 and existing pfv users. Added an explicit
RZ_PRINT_VALUE -> RZ_PF_MODE_QUIET mapping; QUIET already emits one
bare scalar per field, which is exactly the pfv contract.
* test EXPECTs: cmd_pfd unbreaks its bitfield case (also fixes the
data buffer -- the original used 'wx 0x00008140' which is parsed
byte-by-byte and yields LE u32 0x40810000, not 0x00008140, so no
flags matched). cmd_pf 'PE test' picks up the now-decoded
characteristics and dllCharacteristics bitfields. cmd_pf 'Print
value only' updates to the bare-value pfv contract. test/db/cmd/print
'elf 64bit ls pfc.elf_header' picks up the decoded enum symbols.
Also picks up clang-format-20 reflows in cprint.c, cmd_print.c,
pf_parser.c, and rz_pf.h that the CI's clang-format check flagged.
Also bumps the offset-delta buffer in render_val_text from 16 to 24
bytes so a worst-case ut64 decimal (20 digits + sign + NUL = 22) fits
without tripping -Werror=format-truncation on gcc with -O0.
Also fixes three swapped-argument mu_assert calls in test_pf.c (the
v(N) bitvector tests added in commit 8): the macro signature is
mu_assert(message, test) but those three calls passed (test, message),
which tripped -Werror=format= under -O3 because the int 'test' was
fed to the format string's %s slot.
Adds a second round of regression fixes prompted by a manual audit
against pristine upstream (review feedback after the first pass):
* render_quiet: handle raw byte-sequential types (RZ_PF_UINT128,
RZ_PF_HEXDUMP). scalar_text() has no per-byte case for these so the
default-clause '?' was emitted, making 'pfq Q' and 'pfv Q' print '?'.
render_quiet now mirrors text mode and emits a space-separated hex
stream for raw types.
* Sized pointers (p{2,4,8}): the legacy parser accepted 'p2', 'p4',
'p8' to force a 16/32/64-bit pointer width regardless of ctx.bits;
the new parser only handled bare 'p'. Restored via a new dispatch
branch ahead of bare 'p', recording the override in
RzPfField::bit_width (overloaded -- documented in rz_pf.h). A new
fld_ptr_size() helper consults the override first; read_ptr, the
pointer-deref read path, the STRPTR read path, and
pf_struct_size_impl all use it. 'pf p2p4p8pp2' now consumes
2+4+8+4+2=20 bytes and renders the right value at each position.
* Numeric pointer dereference: '*d4', '*x2', '*u8', etc. (pointer to
fixed-size scalar) only read the pointer itself and never followed
it. The reader now also calls ctx->read_at to fetch the target word
and populate scalars[0]; the text renderer prints both the pointer
literal and the dereferenced value -- '(*0x20) 42' instead of bare
'(*0x20)'. Affects 'pf *d4 ...' and the 'Pointers' / '32 bit twice
then string' tests.
* Pointer-to-struct dereference (*?): pointer-to-struct fields read
the pointer value but stopped there, so 'pf *?' (or typedb forms like
'(troll)Bah' marked with '*') showed only '(*0x30)'. The reader now
recurses through ctx->read_at into a worst-case 4 KiB buffer and
re-invokes read_nested_struct; the renderer prints the nested struct
body after the '(*ptr)' annotation. Recursion is bounded by
ctx->max_depth so cyclic Bah->Bah pointer chains terminate at the
documented limit instead of unbounded recursion. Affects 'nested
struct', 'complex nested struct', and 'flag for nested struct'.
* String pointer (bare 's'): RZ_PF_STRPTR rendering didn't show the
dereferenced target -- output was bare '"hello"' with no pointer
context. Now emits '(*ptr) "string"' (mirroring '*z') when the
deref produced a non-empty body; falls back to bare '""' on
unmapped or empty targets to avoid advertising a phantom pointer.
* render_val_text is_pointer branch: extended to render the nested
struct body for *? pointers and the dereferenced value for numeric
*d4/*x2/*u8 forms.
* Top-level read loop: rz_pf_read passed both 'cur_off + off' (as
read_field's off parameter) AND 'base_addr + cur_off' (as base_addr),
but read_field computes 'val->offset = base_addr + off'. This
double-added cur_off on every iteration past the first, so 'pf 2ic'
reading 5 bytes per iter reported iter 1's nb at offset 0x0a instead
of 0x05, and 'pf 2F' reported iter 1's double at offset 0x10 instead
of 0x08. Data reads were correct; only the displayed/JSON offsets
drifted. The fix: pass base_addr unchanged so val->offset becomes
base_addr + (cur_off + off). Affects 'array obj', 'print n-times a
format', 'pf', 'pf field name', 'JSON output'.
* Bare 'C' (legacy 1-byte unsigned decimal): the legacy parser accepted
'C' as 'print byte as decimal'; the new parser had dropped it. The
'types' test format 'pf fcb1d4C foo bar fool beer plop' was therefore
truncating to 4 fields. Restored as a deprecated alias for 'u1' (with
the standard 'use u1' note).
* pfw dotted-path navigation: the write-mode lookup only matched
top-level field names, so 'pfw gobelin.Buh.first=42' through nested
structs failed with 'field not found'. Replaced with a segment-by-
segment walker that descends through children at each dot. Affects
'write specific element through nested struct'.
* Drop '.pfw' usage from tests. '.pfw' (execute pfw output as rizin
commands) was a legacy convention -- the new pfw writes directly via
rz_io_write_at and emits a human-readable confirmation line, so the
'.' prefix evaluates that confirmation line as a command and fails.
Same direction as the earlier '.pf*' removal. All cmd_pf,
cmd_pf_write, cmd_pf2 tests updated.
* test/db/cmd/cmd_pf 'Register' marked BROKEN with a tracking comment:
the legacy 'r (regname)' looked up CPU registers via
RzPrint::get_register; the new parser repurposes 'r' as raw hex byte
dump and the register-fetch path is gone. Restoring would mean
wiring a new register-lookup hook through RzPfCtx, which is outside
the parser-rewrite scope.
* test/db/cmd/cmd_pf2 'pf F max precision (#13027)' annotated: not a
precision regression. Values differ from the original #13027 expected
output because the legacy 'F' specifier read bytes as LE-then-
reinterpret while the new parser follows the documented 'UPPERCASE
= BE' rule literally; the 17-digit precision -- the actual subject
of #13027 -- is preserved.
EXPECTs updated to match the corrected output across cmd_pf, cmd_pf2,
cmd_pf_write.
* doc/pf.md updated to reflect all DSL and rendering changes:
documents the sized pointers (p/p2/p4/p8), pointer dereference
semantics (numeric, struct, and string), the missing composites
(Q, U, L, n/N), the case-as-endian rule (no standalone endian
directive), the pfw write mode (dotted-path navigation, direct I/O,
no '.pfw'/'.pf*'), and the table of deprecated single-letter
aliases (b, d, o, q, u, i, f, F, w, Z, t, T, X, C).
* librz/type/pf_render.c (new): extracted the render-mode code paths
from pf_parser.c into their own translation unit. Hosts the text /
quiet / JSON / cstruct / DOT renderers, the per-mode helpers
(scalar_text, scalar_json, render_binary, render_guid,
compute_name_width, field_matches, emit_colored*), the RenderCtx
bag, and the public rz_pf_render() + rz_pf_render_json() entry
points. pf_parser.c keeps the parse + read pipeline and shrinks
from 4831 to 3280 lines. Helpers used by both translation units
(is_string_type, is_raw_type, endian_str, pf_vasprintf) move to
pf_parser.h as static inlines.
* librz/type/pf/ (new directory): moved every pf source out of the
librz/type/ top level into a dedicated pf/ subdirectory; only the
legacy format.c stays top-level. meson.build updated with the new
paths and the pf/ include dir.
* Split the monolithic parser into focused sub-grammar TUs, wired
together by the new pf/pf_internal.h (which centralizes the PF_DIAG
diagnostic macro, the shared ReadState, and all cross-TU
declarations):
- pf_parser_string.c string/encoding specs (z / s / Z)
- pf_parser_bitfield.c inline + typed bitfields
- pf_parser_bitvec.c bitvectors v(N)
- pf_parser_array.c array-count resolution
- pf_parser_struct.c nested struct / union reading
pf_parser.c shrinks from 3280 to ~2620 lines and now hosts only the
parse driver, type-spec dispatcher, reader core, context, and the
public utility surface. The TLV TU was migrated off its bespoke
extern/TLV_DIAG block onto the shared header.
* Add an RzStructuredData renderer: pf/pf_render_sd.c with the public
rz_pf_render_sd() (declared in rz_pf.h). (Named _sd, not _sdb: SDB is
Rizin's key/value database, unrelated to RzStructuredData.) It maps the decoded value
vector to the generic key/value document model -- scalars to typed
entries, arrays/bitvectors to arrays, nested structs to sub-maps
with a _type tag, raw/GUID payloads to byte blocks, timestamps to a
formatted string plus a <name>_raw sibling -- so callers get JSON,
YAML, or iterator access for free. Unit and property tests cover it
(see the consistency-pass note below).
* doc/pf.md and librz/type/README.md updated for the new pf/ layout
and the structured-data render mode.
* Consistency pass over librz/type/pf/: deduplicated the structured-data
scalar dispatch (one classifier feeding thin map/array emitters instead
of two parallel ~60-line switches), dropped unused includes
(pf_parser_time.h from the TLV TU, string.h from the struct TU,
rz_endian.h from the SD renderer) and a redundant extern decl now
covered by pf_internal.h, and fixed a dangling-pointer bug in the SD
char path (the classifier returned a pointer into its own by-value
temporary). Expanded coverage: 11 SD unit tests (157 total) and 6 new
theft properties (SD tree non-NULL, JSON structural well-formedness,
top-level object shape, YAML safety, determinism, filter-narrows).
* Split pf_render.c (1548 lines) into per-mode renderer TUs, mirroring
the earlier pf_render_sd.c extraction: pf_render_text.c (text+quiet),
pf_render_json.c (+rz_pf_render_json), pf_render_cstruct.c, and
pf_render_dot.c. A new pf_render.h carries the shared RenderCtx record
plus the cross-TU helper (pf_field_matches, pf_scalar_text,
pf_render_guid) and per-mode entry-point declarations. pf_render.c now
holds only those shared helpers and the rz_pf_render() dispatcher
(~372 lines). meson.build, the file-doc layout listings, and
librz/type/README.md are updated accordingly.
* Second consistency pass: ensured every public RZ_API entry point has a
Doxygen \brief block (added ones for rz_type_format_struct_size and
rz_pf_render_sd) and every file a \file description (added to
pf_parser.h and pf_parser_time.h). Deduplicated the field-size logic:
pf_struct_size_impl's sum and union loops now share one
pf_field_static_size() helper, the enum/bitfield byte-width override is
a single pf_enum_bitfield_width() helper (was inlined 3x), sized-pointer
width routes through the existing fld_ptr_size(), and the inline-bitflag
test shared by the JSON and DOT renderers became the
pf_value_has_inline_bitflags() predicate in pf_render.h. Dropped unused
includes left over from the render split. No behaviour change (157 unit
tests, 146 theft properties, full cmd sweep all green).
* Fix a CodeQL High alert (cpp/integer-multiplication-cast-to-long) in
the scalar-array path selector: the per-element byte delta was computed
as `idx * width` in 32-bit int before being added to the ut64 offset,
so a large array index could overflow int prior to the widening. The
multiplication is now done in ut64 ((ut64)idx * width). No behaviour
change for in-range indices; verified by the path-selection cmd tests.
* Fix a big-endian bug in the structured-data renderer: the bitvector
reader stores each bit in the RzPfScalar union's v_u8 member, but the
SD classifier read it back through v_u64. On little-endian those alias
to the same low byte so it worked by luck; on big-endian (s390x) v_u8
is the high byte of the 64-bit slot, so every set bit read as a large
number and the test_pf_render_sd_bitvec assertion failed. The
classifier now reads v_u8 for RZ_PF_BITVEC, matching the writer; the
other scalar types already read the same width the reader wrote.
* Clean up leftover refactoring-era comments in the renderer TUs: drop
the "Split out of pf_render.c" changelog phrasing from the file-doc
headers (the docs now describe the current layout) and update two
stale references to the pre-rename helper names (field_matches ->
pf_field_matches, render_guid -> pf_render_guid) in comments.
---------
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
Gives the C55x / C55x+ disassembler a Capstone-style per-instruction
identifier and threads it through to analysis, so the analyzers dispatch
on named operations instead of raw opcode bytes.
What lands
==========
* librz/arch/isa/tms320/tms320c55x_insn.{c,h} -- a shared TMS320C55InsID enum
(TMS320C55_INS_AADD, TMS320C55_INS_B, TMS320C55_INS_CALL, ...) covering
C55x and C55x+, plus tms320c55x_insn_name(), tms320c55x_insn_id_from_syntax()
and tms320c55x_insn_optype(). The prefix is TMS320C55_ / tms320c55x_
rather than TMS320_ / tms320_ because the other TMS320 families (C54x,
C64x, C28x) have substantially different instruction sets; the enum,
the file and these helpers are C55x/C55x+ specific.
* insn_head_t gains an .id field; every head entry in c55x/table.h is
tagged with its TMS320C55InsID. The disassembler resolves the decoded
instruction ID from the emitted mnemonic (so multi-form leading bytes
such as 0x95 -> intr/trap, 0x48 -> ret/reti/rpt, 0x50 -> sftl/popboth
resolve to the exact instruction, which a static byte->head map cannot).
tms320_dasm_t gains an insn_id field and tms320_dasm_insn_id() accessor.
* tms320_c55x_insn_id_decode() / tms320_c55x_plus_insn_id_decode() let the
analyzers resolve the named ID for a byte sequence via a cached decoder
instance.
* The C55x analyzer's dispatch is rewritten from switch(opcode_byte) to
switch(TMS320C55InsID). Both analyzers set op->id to the named ID, the
same way the Capstone-based plugins (e.g. c64x) populate op->id.
* The C55x+ analyzer keeps its byte-level dispatch for control flow,
stack deltas and operands, but takes the final arithmetic/logical/move/
multiply/stack type from tms320c55x_insn_optype(op->id). A leading byte on
C55x+ encodes several instructions (selected by operand bits), so the
byte switch alone cannot tell ADD from SUB, AND from OR/XOR, or AMOV
from ASUB; the decoded id can.
Bugs fixed
==========
Driving dispatch / typing from the decoded id fixes a number of latent
mis-classifications the raw-byte switch had masked, verified against the
TI dis55 reference disassembler and the C55x+ documentation:
- 0x50 0x66 (psh Tx) was typed SHL; now a stack push (corrects
sym._main's computed stackframe in the rel.stripped.coff test).
- 0x48 0x05 (reti) was typed REP; now RET.
- 0x95 0x0F / 0x8F (intr vs trap) distinguished by the decoder.
- C55x+ 0x7B: byte1 bit7 selects LD (mov) vs add/sub, and within
add/sub byte2 bit7 selects sub; was always typed add/mov by nibble.
(TI SWPU104 Table 7-2, opcode 01111011.)
- C55x+ 0xD2 mar(XDAa op k24): address-register modify, now LEA like
AADD / AMOV; was typed SUB. (Table 7-2, opcode 11010010.)
- 57 further C55x+ arith/logical/move/stack contradictions found by
cross-referencing a Motorola Droid (Wrigley3G) C55x+ baseband dump
against the decoder are resolved by the optype override.
* DELAY is a memory-delay MOVE per TI SWPU104 sec.6.7.1 (Memory Delay,
grouped under Move Operations): it copies Smem to Smem+1. Both
analyzers now mark it as a memory access (width 2, write) and drop the
spurious FAMILY_CPU (CPU is already the default family).
Tests
=====
New checks assert op->id carries the right TMS320C55_INS_* value on each
CPU, plus regression tests for every byte/decoded-id mismatch fixed above
(c55x: 0x50/0x48/0x95/0xb6; c55x+: 0x7b/0xd2). The c55x opcode-
classification and stackframe expectations are updated to the corrected
output.
Cross-reference
===============
TI SPRU374 'TMS320C55x DSP Mnemonic Instruction Set Reference Guide'
TI SWPU086 'TMS320C55x+ DSP Algebraic Instruction Set Reference Guide'
TI SWPU104 'TMS320C55x+ DSP Mnemonic Instruction Set Reference Guide'
Short summary of the three cpu= values supported by the tms320 arch
plugin (c55x, c55x+, c64x), the silicon they map to, and a note on
the two distinct meanings of 'c55x+': TI's publicly-named C55x DSP
Core+ shipping in current low-power C55x parts (C5504-C5545), and
the pre-release Ryujin revision documented in TI SWPU086 / SWPU104
that lives in older modem silicon. The rizin plugin handles both.
Rewrites the C55x and C55x+ analysis classifiers as pure byte-level
dispatch -- no mnemonic-string matching, no round-trip through the
disassembler -- and adds the supporting infrastructure they need to
produce useful RzAnalysisOp metadata.
What lands
==========
* librz/arch/isa/tms320/c55x/c55x_analysis.{c,h} -- C55x baseline
classifier, ~360 lines, 256-entry size table extracted from the
decoder's table.h.
* librz/arch/isa/tms320/c55x_plus/c55plus_analysis.c -- C55x+
classifier rewritten in the same shape, ~470 lines covering 90+
opcodes with byte-level disambiguation for 0x02 / 0x03 / 0x74 /
0x76 / 0x7B / 0xC5.
* librz/arch/isa/tms320/tms320_dwarf_regnum_table.h plus a hook in
librz/arch/dwarf_process.c -- TI cgt55 ABI DWARF register-number
mapping, so the cl55 compiler's .debug_info variable locations
resolve into rizin register names instead of returning the dummy
"?" placeholder.
* librz/arch/p/analysis/analysis_tms320.c -- thin dispatcher that
picks the per-cpu classifier and stops carrying the tms320_dasm_t
engine in analysis state.
Why a byte-driven classifier
============================
The old classifier round-tripped through the disassembler and did
strncasecmp() on the mnemonic string. Three problems:
1. It kept a tms320_dasm_t engine alive in the analysis context
just to read its 'syntax' buffer after every classify call.
Removing it shrinks the per-analysis state and removes a
tms320_dasm_init/_fini pair from the analysis_init/_fini path.
2. It only set op->type -- never op->jump, op->fail, op->stackop,
op->stackptr, op->val, op->eob. Basic-block formation followed
only the most obvious control flow, and call/ret/push/pop
semantics were invisible to higher-level analysis.
3. It couldn't disambiguate predicated versus unconditional calls:
the disassembler emits 'callcc' vs 'call', but the substring
match missed the conditional fail-path for CALLCC.
The new classifiers fix all three:
- Read the leading byte (and second-byte refinements where the
encoding family is shared) directly from buf.
- Resolve jump and call targets from BE-stored displacement and
absolute fields, with correct sign extension for the 8-bit and
16-bit relative forms.
- Read 24-bit absolute targets via rz_read_at_be24().
- Set op->fail = addr + size for every conditional jump/call,
op->eob = true for unconditional branches and RET so basic-block
walkers terminate correctly.
- Track the stack: PSH/POP per ISA cluster, CALL/CALLCC +2,
RET/RETI -2.
- Capture INTR/TRAP immediates in op->val via set_imm().
- Disambiguate sub-opcodes that share a leading byte by reading
the relevant bits of the second byte. For C55x, the most
notable case is 0x48 (RPT/RPTADD/RPTSUB/RET/RETI) which uses
bits 0-2 of byte 1; for C55x+ the disambiguations are 0x02,
0x03, 0x74, 0x76, 0x7B and 0xC5.
- Handle parallel-prefix bytes (odd-valued leading bytes below
0x80 in C55x like 0x03, 0x05, 0x07, 0x11, ...) by treating
them as a 1-byte prefix and dispatching on byte 1 so paired
'|| retcc', '|| bcc', etc. classify correctly.
Both classifiers ship analyzer helpers (set_cjmp, set_call, set_jmp,
set_ret, set_cret, set_push, set_pop, set_imm, set_mem_width,
set_dst_reg, set_ireg, set_dir, set_disp) so each opcode entry fills
the RzAnalysisOp ptr / val / stackop / stackptr / fail / eob fields
uniformly across both architectures.
DWARF register mapping
======================
Loading any cl55-compiled TI COFF v2 with debug info (every
emulateme*.ticoff2.dbg.coff in rizin-testbins) used to fire:
ERROR: No DWARF register mapping function defined for tms320 32 bits
per variable, because dwarf_process.c had no entry for arch=tms320.
The new tms320_dwarf_regnum_table.h covers the cgt55 ABI numbering:
AC0-AC3, T0-T3, AR0-AR7, SP/SSP/CDP, BK03/BK47/BKC, DP/PDP, CSR,
BRC0/BRC1, TRN0/TRN1, RPTC, IER0/IER1, IFR0/IFR1, DBIER0/DBIER1,
IVPD/IVPH, ST0_55..ST3_55 (42 entries). Reach into the table is
guarded; out-of-range numbers fall back to NULL so the caller
surfaces the dummy "?" instead of confidently picking the wrong
register.
Wrigley3G coverage
==================
Validation against a 3.1 MB Wrigley3G baseband firmware (Motorola
Droid A855, MSG39UPEU_A1.19_1.80, partition CG45.img) found 31
leading-byte values producing real instructions classified as NULL.
The c55x+ classifier here covers those:
0x50-0x5F MOV memory/register cluster
0x88, 0x8A MOV ACx <-> mem high/low halves
0x8C ADD with carry, mem -> ACx
0x97 Dual-memory MOV (parallel)
0xA0 MOV with parallel dual addressing
0xAC, 0xAD MOV #k16, ACx (long immediate)
0xB4, 0xB5 MOV with rounding and shift
0xB6, 0xB7 ADD with shift (T-register or immediate)
0xC0, 0xC2, 0xC4 ADD #k16 with shift slots
0xCC Packed ADD :: MOV dual-instruction encoding
0xD0 MOV ACx, dbl(*(#abs24))
0x2E, 0x2F XCCPART predicated execute
0x0B, 0x23 Wrigley silicon pseudo-ops (TRAP)
0xC6 BFXTR / BFXPA bit-field extract (MOV)
The 0x03 family classifier extends from a 4-bit (0xF0) to a 6-bit
(0xC0) mask so the full encoded range resolves:
0x03 0x00-0x3F INTR #k5
0x03 0x40-0x7F TRAP #k5
0x03 0x80-0xBF SWAP register pairs
0x03 0xC0-0xFF SIM_TRIG (Wrigley-specific simulator trigger)
Coverage on Wrigley3G rises from 94.4% to 97.4% (2000-sample
random survey).
Tests
=====
Two new test suites land alongside the classifiers:
test/db/analysis/tms320.c55x_32 11 tests (batched)
test/db/analysis/tms320.c55x+_32 13 tests (batched + binary
fixtures)
Tests are intentionally batched -- each test bundles 10-12 opcode
checks behind one rizin process spawn instead of one per check.
That brings both suites down to under 0.5 seconds combined.
The c55x+ suite includes six binary-fixture tests against the
companion rizin-testbins drop-in tms320/coff2/*.obj corpus,
covering function discovery (afl), stack-pointer tracking
(afvs / afS), data-section walk (iS), and globals enumeration
(is). The c55x suite covers tms320/emulateme_nostd.ccsv5.c55x
.ticoff2.dbg.coff from the existing rizin-testbins tree.
Cross-reference
===============
TI SPRU374 'TMS320C55x DSP Mnemonic Instruction Set Reference
Guide' (publicly available) -- C55x baseline.
TI SWPU086 'TMS320C55x+ DSP Algebraic Instruction Set Reference
Guide' (May 2005) -- C55x+ instruction encodings.
TI SWPU104 'TMS320C55x+ DSP Mnemonic Instruction Set Reference
Guide' (December 2006) -- C55x+ mnemonic forms.
The c55x_plus disassembler glue layer (c55plus.c) is rewritten as a
thin shim over the th0rpe c55plus_decode() walker:
- Drop the ad-hoc ctype tolower() loop; use rz_str_case() to
lower-case the walker's mixed-case mnemonics in one call.
- Reorder the includes; drop the unused ones.
- Scope local variables to where they are actually used.
- Move the global setup (ins_buff / ins_buff_len) into the same
block as the c55plus_decode() call so the dataflow is obvious.
While here, fix a pre-existing off-by-one in utils.c. The hex-digit
lookup table was declared as a 17-character string with a duplicated
leading '0':
static char hex_str[] = "01234567890abcdef";
This shifted every nibble >= 0xA by one position in the table:
hex_str[10] = '0' (should be 'a')
hex_str[11] = 'a' (should be 'b')
...
hex_str[15] = 'e' (should be 'f')
hex_str[16] = 'f' (unreachable)
Result: get_hex_str(0xff) returned "ee", get_hex_str(0xab) returned
"0a", and every disassembled '.byte 0xNN' for an unknown opcode with
nibbles >= A came out wrong. The function is used from
c55plus_decode.c on the unknown-opcode fallback path
(hash_code == 0x223), so the bug surfaces whenever the decoder bails
out and emits a raw byte.
Fix: use the correct 16-character table "0123456789abcdef" and rename
the static to hex_digits to make the role obvious. While here, tidy
strcat_dup() to use bitwise tests on the n_free bitmask so the
'3 = free both' contract is enforced uniformly, add docstrings to both
helpers, and drop the redundant memcpy length guard that was a no-op
for non-NULL length-zero strings.
No behavioural change for either piece beyond the bug fix above.
