* Clarify docs of raw_alignment
* Set Go and Rust string encoding for binary string search.
* Add detailed doxygen about string search in bin plugins.
* Add the number string code points as detail to string search hits.
* Add helper to check if string encoding needs scanning.
* Add helper to generate a string wildcard pattern.
The pattern is supposed to match any human readable string.
Before the string search refactor there was no explicit definition
what characters mark the end of a string and which ones don't.
This pattern should match strings of the old interpretation.
* Enable the RzBin plugin to search strings with direct matching instead of scanning.
This significantly speeds up the string search when a binary is opened.
The prerequisite is that the plugin specifies the string encoding.
* Enforce UTF-8 for initial string search.
* Add custom string search for PE files.
* Clean up and document rz_utf8_encode
* Don't demote explicit UTF-8 encoding to ASCII.
* Increase min JIT stack size due to not reproducable search results.
* Simplify wildcard regex pattern.
* Remove unused declaration.
* Fix integration test to have a real scan example.
This also fixes a bug that the last NUL/invalid byte in a string
was not included in the UTF-8 bytes -> memory offset map.
* Enable string search collection to search for multiple encodings.
This moves the encoding and alignment properties out of the collection
object into the RzDetectedStr object.
It essentially removes duplicates of this info.
Because of this move, rz_search_collection_string_add() will now
check for invalid collection configurations.
* Rename type -> encoding
* Make checks against defined Unicode points optional
* Allow to decode UTF-16 without writing the result.
* Remove PCRE2_NO_UTF_CHECK as default, since it can lead to undefined behavior.
* First refactor regex to support utf16 and utf32
* Add UTF-16-BE encoding function
* Add UTF-8 counting helper functions.
- One for counting the number of Unicode code points.
- The other to get the number of bytes required to represent the given UTF-8 string in UTF-16.
* Remove unused code
* Add UTF-8 to UTF-16 conversion function.
* Add type annotations
* Implement utf8 to utf32 string conversion
* Add UTF-16/32 versions of all other necessary regex functions for str search.
* Another regex refactor for utf16/32
* Add UTF16/32 regex matching tests.
* Implement still segfaulting (possibly JIT double usage) regex search.
* Duplicate match_first functions to reduce necessary branch predictions.
* Reduce number of required branches for encoding UTF16/32 to one.
* Duplicate match_all_internal functions to reduce necessary branch predictions.
* Fix too early free
* Only allocate match vector when needed.
* Fix: use code point size of buffer.
* Add missing return
* Normalize pointers to UTF16/32 strings to use proper code point with
* Also replace spaces with in utf16/32
* Add an additional host endian tests for UTF16/32 string encodings.
* Ensure thread savety.
JIT compiled patterns need to be owned by a single thread.
For the search we need to clone it.
JIT matching structures are optionally cloned as well.
* Enforce NO_UTF_CHECK in regex search.
This improves performance and currently is
default because we always match on binary data.
* Fix matching of UTF strings which are not suported by direct buffer matching.
PCRE2 only supports matching against memory which is aligned
to a code point width of the encoding.
These changes prevent taking the fast (direct matching with PCRE2) path
and use the slow string search path if the alignment doesn't match
the UTF string encoding.
To not complicate the change and additional alignment member
is added to each searched string in the search collection.
* Create search hit description on the stack
* Unset complete JIT matching if user provided custom jflags.
* Document what passing NULL to copy function pointers does.
* Replace the retarded idea of tracking offsets with a hashmap with a linear buffer.
This improves performance something like 10x.
* Remove const for the non-JIT builds.
* Remove additional flags for skip checking.
It is not needed because each decoded character is checked for printablity below anyways.
* Enfore no setup of IO mem with 0xff
* Fix: Set JIT complete flag for multi regex patterns
* Fix heap.
* Add note about worsed performance path.
* Fix unit test with string terminated by undefined code point.
* Run clang-format
* Fix order of arguments
* Add warning about string search with encoding=guess to tests.
* Fix string lengths, they no longer count the final invalid code point.
* Fix endian macro on Windows
* Fix NULL dereference
* Fix number tests
* Use endianness check not dependent on stdbit
* Fix type annotations.
* Reintroduce rz_str_len_utf8char
* Fix command description.
* Fix and unify RZ_SYS_ENDIAN macros.
- Don't allow unhandled architectures anymore.
- Check endianness for Sparc and PPC using non GCC/Clang compilers.
- Fix several endianness checks using the value instead of the macros.
* Fix tests
* Apply review comments.
Adds the new string search implementation, fixes many bugs and makes performance improvements.
- Adds support to search reliably for all supported encodings (fixes non-ASCII string search).
- Fixes some wrong assumptions what valid code points are (e.g. 0x000000ff is a valid code point in UTF-32/UTF-16 BE).
- Adds several '/z' command options how to perform the string search (literal, regex, extended regex, caseless).
- Checks every decoded code point for validity to improve correctness.
- Improves performance of string decoding by not writing to the heap in all cases.
Part 8/9. Likely won't build in between parts.
Co-authored-by: wargio <deroad@kumo.xn--q9jyb4c>
Adds the core implementation of the new search.
The rough architecture is the following:
A search for a certain type of information (strings, bytes, keys etc.)
creates a collections of items to search for (byte patterns, regular expressions etc.).
Then specifies some settings how the search (number of threads, maxum hits...)
and the finding is performed (string length, inverse match etc.).
It also defines a search space, which is currently only the IO buffer.
But can be anything in the future, like a graphs or the knowledge base.
The search splits up the search space into windows (for IO: address ranges)
and dispatches each search window into a 'find()' thread.
The 'find()' handler (provided by a specific search implementation)
checks the given window and produces search hits matching the elements in the search collection.
The main search handler collects the hits of the dispatched workers
and returns them to the user.
Note: The byte and string search implementations are added in the next two commits.
Part 6/9. Likely won't build in between parts.
Co-authored-by: wargio <deroad@kumo.xn--q9jyb4c>
The previous description was confusing. Because a Rune is not a defined term for Unicode encodings (except of course the ancient runes).
Because the type seems to be always used as "Unicode code point" this rename should bring clarity.
* Remove old string detection methods.
* Allow changing encoding based on `bin.str.enc`
* Cleanup for optimization regarding RzAnalysisData and its kind
* Allow changing endianness.
Adds /*<type>*/ comments and a linter check from rz-bindgen to enforce
their existence and consistency
Also includes the following fixes made when adding the annotations:
* removed unused intern_table arguments in pyc_dis.c, pyc_dis.h, asm_pyc.c
* removed unused classes argument from place_nodes in agraph.c
* removed unused recurse and recurse_bb functions in canalysis.c
* removed unused vars field from RzPrint struct
* removed unused RzAnalysisType* structs from rz_analysis.h
* removed unused list field from RzEgg struct
* fixed bug in bp_plugin.c where duplication-checking logic iterates over the wrong list
* removed unused q_regs field from RzDebug struct
* removed unused backtrace field from RzDebugPlugin struct
* removed unused classes_list field from RzBinNXOObj struct
* removed unused methods_list and classes_list fields from RzBinZimgObj struct
* Improved false-positive detection in str_search
This commit adds the following features:
- Extend the false-positive check on ASCII frequencies to all UTF strings
- Add a global option to activate/deactivate di check
- Improve the false-positive heuristic by adding a special case for extended-ASCII strings
* Moved rz_print_2bpp_* under rz_core
* Moved rz_print_string to rz_str_stringify_raw_buffer
* Added test for print urlencode and added pu0
* Removed RzCoreString and implemented utf32be/utf16be
* Do not escape strings during the search
* Port `C` (metainformation) commands to the rzshell
* Improve string autodetection
* Remove `Cr` and RZ_META_TYPE_RUN
* Escape `\x1b` as `\e` character in strings
* Move string encoding options to `RzStrEncOptions` struct
* Add `esc_double_quotes` option to `RzStrEncOptions`
* Do not filter out unprintable characters for uppercase check
* Use `esc_double_quotes` when printing the string between '"' in disasm
* Use signed chars on all platforms/architectures
Co-authored-by: Riccardo Schirone <sirmy15@gmail.com>
* Refactored string search
string search logic moved from librz/bin/bfile.c to
librz/util/str_search.c
* UTF-16BE and UTF-32BE support
* Decoupled rz_str_search with rz_bin
* Changed parameter order in rz_scan_strings
* Fixed needle reset in rz_string_search
Now when the string search fails, needle is restored to original_needle+1,
i.e., we continue the searching process starting from the next byte.
* Changed heuristic for UTF BE detection
one more byte in for the detection (6 instead of 5)
* Fixed unit test in str_search
* Refactored rz_scan_strings and better UTF16-LE and UTF16-BE detection
Now if there is an ambiguity between UTF16-LE and UTF16-BE, we take the
longest one.
* Changed UTF BE heuristic
If there is any ambiguity about big-endian or little-endian, always
choose little endian
* Small format fixes in str_search
* Added preferred endianess switch in RzUtilStrScanOptions
* Fixed str_search regression tests
