* Add ht benchmarks
* Initial implementation
* Finalize native per-group lookup support
* Lookup SSE2 implementation
* Improve hashing
* Add support for custom elem_size
* Avoid double h2 hashing when reserving slot
* Make custom elem_size support conditional
* Fix issues with bitwise and default lookup implementations
* Implement deletion trick optimization
* Track growth_size instead of deleted_slots
* Refactor SDB to access ht via API instead of internals
* Modify SDB tests which rely on hashtable order
* Fix SDB build warnings
* Fix bug with finding next power of two
* foreach_kv to return a bool result
* Change SDB diff order expected by serialize_analysis unit test
* Fix bug in the bitwise lookup implementation
* Remove second call to rz_core_init() which causes memory leaks
* Update some regression tests to accept reordered output
* Adapt ht clear to new implementation
* Use fini_kv_pair and fix 1 potential leak on malloc failure
* Fix cmd/types test after merge
* Avoid second call to calsize_key and avoid iter leaks on malloc failure
* Improve hash distribution
* Extend benchmark suite
* Fix bug with string hashing
* Branchless write to mirrored ctrl bytes
* Simplify string hash and remove potential UB
* Move RZ_PREFETCH macro to rz_types.h
* Add SSE2 discovery in Meson
* Forward SDB string hash function to ht string hash
* Try to revert test_cpu_profiles() to avoid relying on a baked SDB file
* Revert SDB/CDB hash function change
* Fix SDB reference to HT hash function instead of CDB hash
* Change calloc to malloc
* Avoid storing/checking key_len and key_value if they are ut64
* Improve string hash function
* Rename default hash functions
* Improve bench code
* linter.yml: set clang-path to point to llvm-18
The getter is useful if a module outputs debug logs buts also some updated status update in a single line with rz_cons_printf("\r...").
The status update should not happen if the log level is below warning.
This brings these copy functions in line with the more common order of
dst, src as used in memcpy and rz_bv_*_inplace functions.
This is a breaking change and callers of rz_bv_copy() and
rz_bv_copy_nbits() must adjust to the new order.
* Add rz_list_safe_prev/next macros
* Remove rz_list_iter_get_prev/next() functions
* Use rz_list_safe_prev() macro instead of rz_list_iter_get_prev() function
* Use rz_list_safe_next() macro instead of rz_list_iter_get_next() function
* Add an in-place addition of bitvectors.
* Decouple _elem_len from assumption it has just enough bytes to hold len in bits.
* Add in-place copy of bits.
* Use default copy bits bitvector function.
* Mark stack allocated bit vector.
* Add inplace variant of complement_1
* Implement inplace bitvecotr not.
* Add inplace variant of rz_bv_and
* Implement inplace variant of rz_bv_or
* Implement inplace variant of rz_bv_xor
* Constify rz_bv_lsb/msb
* Implement inplace variant of rz_bv_neg
* Fix inplace add. Add inplace SUB
* Remove prefix
* Remove invalid const
* Implement in-place casting of bit vectors.
* Add a hash test
* Fix unnecessary &
* Prevent OOB reads & writes by copying only the minimum of bytes.
* Add inplace MUL
* Add ble version for bitvector set/get with bytes.
* Add documentation for inplace bitvector functions.
* Add bench of add/sub() and sub/add_inplace().
* Remove rz_bv_copy_nbits_inplace because it was the same as rz_bv_copy_nbits.
* Implement optimized functions for small to large and large to small
* Add benchmark instructions in testREADME.md
* Use the term unaligned consistently instead of nonaligned
* Slightly improve bitvector test error message
* Remove unnecessary annotations
* Edit readme
* Add explicit fallthrough comments
* Return statement for rz_bv_toggle_all()
* Extend tests
* Add benchmark for rz_bv_copy_nbits
* Improve performance for large to large and small to small bitvector copy
* Fix bug with nbit=64 and simplify code
* Add test for same bitvector copy
* Move bit copy logic to separate function in rz_bits.h + improve comments
* Support same vector copy for unaligned case
* Expect non-null RzTable in bench utils and add comments
* Test against reference implementation instead of hardcoded values
* fix: format with clang-format-20
* fix: fix clang-format linter with rewrite '^#define.*/\*.*\\$'
* fix: update clang-format to version 20 in workflows and documentation
* fix: add SPDX license information to .git-blame-ignore-revs
* Clarify docs of raw_alignment
* Set Go and Rust string encoding for binary string search.
* Add detailed doxygen about string search in bin plugins.
* Add the number string code points as detail to string search hits.
* Add helper to check if string encoding needs scanning.
* Add helper to generate a string wildcard pattern.
The pattern is supposed to match any human readable string.
Before the string search refactor there was no explicit definition
what characters mark the end of a string and which ones don't.
This pattern should match strings of the old interpretation.
* Enable the RzBin plugin to search strings with direct matching instead of scanning.
This significantly speeds up the string search when a binary is opened.
The prerequisite is that the plugin specifies the string encoding.
* Enforce UTF-8 for initial string search.
* Add custom string search for PE files.
* Clean up and document rz_utf8_encode
* Don't demote explicit UTF-8 encoding to ASCII.
* Increase min JIT stack size due to not reproducable search results.
* Simplify wildcard regex pattern.
* Rename `maxDescLength` to `max_desc_len`
* Rename `descLength` to `desc_len`
* Rename `maxFlagAndArgLength` to `max_flag_and_arg_len`
* Rename `flagAndArgLength` to `flag_and_arg_len`
* Rename `flagLength` to `flag_len`
* Rename `argLength` to `arg_len`
* Rename `maxOptSpaces` to `max_opt_spaces`
* Rename `remainingSpaces` to `remaining_spaces`
* Rename `maxDescSpaces` to `max_desc_spaces`
* Change `flag_and_arg_len` to `flag_n_arg_len`
* Remove unused declaration.
* Fix integration test to have a real scan example.
This also fixes a bug that the last NUL/invalid byte in a string
was not included in the UTF-8 bytes -> memory offset map.
* Add speed up search for unaligned buffers and string encodings.
The new search mode adjusts the buffer's data (swap endianness, fix alignment)
so the buffer can be searched with PCRE2 without scanning.
It also enables RzRegexMulti opbjects to have their own JIT stack.
This makes them usable in a thread safe manner.
* Fix regex compilation on big endian systems
* Update the scanning check (UTF needs no scanning anymore).
* Enable string search collection to search for multiple encodings.
This moves the encoding and alignment properties out of the collection
object into the RzDetectedStr object.
It essentially removes duplicates of this info.
Because of this move, rz_search_collection_string_add() will now
check for invalid collection configurations.
* Rename type -> encoding
* Add function to memcpy memory with offset.
* Add count trailing zeros function.
* Add function to determine pointer alignment.
* Add helper to swap sequences of 2 bytes.
* Add helper to swap sequences of 4 bytes.
* Print pointer in warning
* Fix tests
* Use uintptr_t type to prevent undefined behavior.
* Remove alignment requirement and rely on malloc promises.
* Add in place byte swap functions.
* Move the endianness swapping into util/rz_endian
* Use GNU builtin __builtin_ctzll/clzll.
Also fixes the include guards of __builtin_clzll
and handles the undefined case of x == 0 of it.
* Add doxygen
* Add integration test for a simple string search.
* Clean up rz_str_is_false
* Doxygen
* Add rz_buf_type
* Sort search hits not just by address but also by size.
* Add a paranoid regex Hindi test.
* Add an integration test using RzCore and searching for UTF-16LE.
* Revert "Temporarily remove codecov workflow due to https://github.com/codecov/codecov-action/issues/1877 (#5366)"
This reverts commit dc5d50ce58.
* Make checks against defined Unicode points optional
* Allow to decode UTF-16 without writing the result.
* Remove PCRE2_NO_UTF_CHECK as default, since it can lead to undefined behavior.
* First refactor regex to support utf16 and utf32
* Add UTF-16-BE encoding function
* Add UTF-8 counting helper functions.
- One for counting the number of Unicode code points.
- The other to get the number of bytes required to represent the given UTF-8 string in UTF-16.
* Remove unused code
* Add UTF-8 to UTF-16 conversion function.
* Add type annotations
* Implement utf8 to utf32 string conversion
* Add UTF-16/32 versions of all other necessary regex functions for str search.
* Another regex refactor for utf16/32
* Add UTF16/32 regex matching tests.
* Implement still segfaulting (possibly JIT double usage) regex search.
* Duplicate match_first functions to reduce necessary branch predictions.
* Reduce number of required branches for encoding UTF16/32 to one.
* Duplicate match_all_internal functions to reduce necessary branch predictions.
* Fix too early free
* Only allocate match vector when needed.
* Fix: use code point size of buffer.
* Add missing return
* Normalize pointers to UTF16/32 strings to use proper code point with
* Also replace spaces with in utf16/32
* Add an additional host endian tests for UTF16/32 string encodings.
* Ensure thread savety.
JIT compiled patterns need to be owned by a single thread.
For the search we need to clone it.
JIT matching structures are optionally cloned as well.
* Enforce NO_UTF_CHECK in regex search.
This improves performance and currently is
default because we always match on binary data.
* Fix matching of UTF strings which are not suported by direct buffer matching.
PCRE2 only supports matching against memory which is aligned
to a code point width of the encoding.
These changes prevent taking the fast (direct matching with PCRE2) path
and use the slow string search path if the alignment doesn't match
the UTF string encoding.
To not complicate the change and additional alignment member
is added to each searched string in the search collection.
* Create search hit description on the stack
* Unset complete JIT matching if user provided custom jflags.
* Document what passing NULL to copy function pointers does.
* Replace the retarded idea of tracking offsets with a hashmap with a linear buffer.
This improves performance something like 10x.
* Remove const for the non-JIT builds.
* Remove additional flags for skip checking.
It is not needed because each decoded character is checked for printablity below anyways.
* Enfore no setup of IO mem with 0xff
* Fix: Set JIT complete flag for multi regex patterns
* Fix heap.
* Add note about worsed performance path.
* Fix unit test with string terminated by undefined code point.
* Run clang-format
* Fix order of arguments
* Add warning about string search with encoding=guess to tests.
* Fix string lengths, they no longer count the final invalid code point.
* Fix endian macro on Windows
* Fix NULL dereference
* Fix number tests
* Use endianness check not dependent on stdbit
* Fix type annotations.
* Reintroduce rz_str_len_utf8char
* Fix command description.
* Fix and unify RZ_SYS_ENDIAN macros.
- Don't allow unhandled architectures anymore.
- Check endianness for Sparc and PPC using non GCC/Clang compilers.
- Fix several endianness checks using the value instead of the macros.
* Fix tests
* Apply review comments.
* Update magic implementation from openbsd to 0e59d0d commit
* Various refactoring of the original code to use RzUtil primitives instead of the custom implementation
* Remove static variables from the original code
* Remove Queue, regex_t and use RzList, RzRegex instead
* Remove unneeded code from the original implementation
* Use RZ_LOG_* macroses instead of the custom logging of the original code
* Use rz_endian primitives where appropriate
* Use os-specific functions where appropriate
* Move magic functions to internal header
* Rename librz/util/ubase64.c into librz/util/base64.c for consistency #5216
* Add Doxygen documentation to function in librz/util/base85.c #5216
* Add unit-tests for librz/util/base85.c #5216
* Refactor base85 API and internal functions #5216
base85 was using a FILE as an input and it used to print its result to
STDOUT, which was very bad for having unit-testing or to use it to
extend rz-hash functionality, since the solution I really had was
through using pipes which isn't ideal as much as the refactoring one.
* Extend rz-hash with base85 #5126
* Fix formatting #5216
* Extract base36 decoding function from subprojects/rzwinkd/iob_net.c to librz/util/base36.c #5216
- Also, added Doxygen docs to rz_base36_decode function
* Added rz_base36_encode_dyn #5216
- this function encode a u64 value to it's char[13+1] base36 counterpart
- later, it will be helpful in expanding the funcitionality of rz-hash
- added its Doxygen docs
* Expand rz-hash with base36 encoding/decoding #5216
* Refactor rz_base36_decode to return a suitable error code like -1 #5216
- that should help testing it later for valid and invalid decodings
- keeping the original behaviour as it was in
subprojects/rzwinkd/iob_net.c
* Add unit-tests for base36 encoding & decoding functions #5216
* Fix formatting #5216
* Fix base36.h header file guards #5216
* Add base32 encoding & decoding #5216
added their Doxygen Docs as well
* Add base32 encoding and decoding unti-tests #5216
* Expand rz-hash with base32 encoding/decoding functionality #5216
* Fix formatting #5216
* Fix add base32 to codec_name_bytes #5216
* Fix base32 docs #5216
* Add base16 encoding/decoding functions #5216
- added their Doxygen docs as well.
* Add base16 unit-tests #5216
* Fix test_base85 conversions warnings #5216
* Expand rz-hash with base16 encoding/decoding functionality #5216
* Fix formatting #5216
* Fix doxygen docs #5216
* Restore subprojects/rizin-shell-parser/parser.c to match origin/dev
* Fix the order by baseXX
* Fix base16 - invert the logic in calculate_src_length
* Fix base16 - intialize variables in rz_base16_encode
* Fix base16 - null terminate the output buffer of rz_base16_encode_dyn
* Fix base16 - get rid of unnecessary else in rz_base16_decode
* Fix base16 - use `len & 1` instead of `(len % 2) != 0`
* Fix base32 - invert the logic in calculate_src_length
* Fix base32 - compress two return statments by using `rz_return_val_if_fail(src && dest, 0);`
* Fix base32 - null terminate the output of rz_base32_encode
* Fix base32 - get rid of unnecessary else in rz_base32_decode
add more parentheses to split addition from mult for better clarity
* Fix base32 - intialize variables in rz_base32_encode_dyn
* Fix base36 - intialize `tmp` variable in rz_base36_encode_dyn
* Fix base36 - use RZ_LOG_ERROR instead of eprintf
* Fix base36 - use RZ_NULLABLE for the API interface
* Fix base85 - use RZ_OUT & RZ_NULLABLE for the API interface
* Fix base85 - use RZ_LOG_ERROR instead of eprintf
* Fix base85 - use size_t instead of int for decode_tuple_buf
* Fix base85 - remove unused varaible, `out_len`, in rz_base85_encode_dyn
* Fix base85 - correct decode buffer size calculation
The previous rz_base85_dec_buflen() underestimated the worst‑case output
size (3 bytes per 4 input chars), causing overflows when using ‘z’/’y’
abbreviations. Update it to allocate 4 bytes per input character plus
one for the NUL terminator, eliminating heap-buffer-overflow errors.
* Fix formatting
* Fix base85-test - use `newlines` variable to make sure line breaks were inserted as expected
* Fix base85-test - remove unnecessary includes
* Fix rz-hash test - updated rz-hash -L expected result
* Fix crypto_base36 - use RZ_LOG_ERROR instead of eprintf
* Fix base16 - compress rz_return_val_if_fail statements and move before locals
* Update base16 - use hex.c existing encoding/decoding logic
- base16.c encoding/decoding functions are now wrappers for hex.c
`rz_hex_bin2str` and `rz_hex_str2bin` functions
- updated related base16 unit testing and documentation
- updated the integraion with rz-hash binary (or crypto_base16)
* Fix formatting
* Add documentation to rz_hex_bin2str
* Fix base85 - add missing `reutrn` docs to rz_base85_encode function
* Fix base85 - use RZ_NONNULL for dest and src function instead of RZ_NULLABLE
* Fix base36 - use RZ_NONNULL for dest and src function instead of RZ_NULLABLE
* Fix base16 - use RZ_NONNULL for dest and src function instead of RZ_NULLABLE
* Fix base32 - use RZ_NONNULL for dest and src function instead of RZ_NULLABLE
* Fix base16 - add missig checks for invalid inputs in rz_base16_encode & rz_base16_encode_dyn
* Fix base32 - add missing checks for bad encoding
* Fix - add missing SPDX
* Fix Formatting
* Add rz_base36_encode & rz_base36_decode_dyn functions to base36 API
* Enable initialization of negative zero and add float neg tests.
* Use argument instead of another function to init negative zero.
* Use platform agnostic negative inifinity value.
