* Make checks against defined Unicode points optional
* Allow to decode UTF-16 without writing the result.
* Remove PCRE2_NO_UTF_CHECK as default, since it can lead to undefined behavior.
* First refactor regex to support utf16 and utf32
* Add UTF-16-BE encoding function
* Add UTF-8 counting helper functions.
- One for counting the number of Unicode code points.
- The other to get the number of bytes required to represent the given UTF-8 string in UTF-16.
* Remove unused code
* Add UTF-8 to UTF-16 conversion function.
* Add type annotations
* Implement utf8 to utf32 string conversion
* Add UTF-16/32 versions of all other necessary regex functions for str search.
* Another regex refactor for utf16/32
* Add UTF16/32 regex matching tests.
* Implement still segfaulting (possibly JIT double usage) regex search.
* Duplicate match_first functions to reduce necessary branch predictions.
* Reduce number of required branches for encoding UTF16/32 to one.
* Duplicate match_all_internal functions to reduce necessary branch predictions.
* Fix too early free
* Only allocate match vector when needed.
* Fix: use code point size of buffer.
* Add missing return
* Normalize pointers to UTF16/32 strings to use proper code point with
* Also replace spaces with in utf16/32
* Add an additional host endian tests for UTF16/32 string encodings.
* Ensure thread savety.
JIT compiled patterns need to be owned by a single thread.
For the search we need to clone it.
JIT matching structures are optionally cloned as well.
* Enforce NO_UTF_CHECK in regex search.
This improves performance and currently is
default because we always match on binary data.
* Fix matching of UTF strings which are not suported by direct buffer matching.
PCRE2 only supports matching against memory which is aligned
to a code point width of the encoding.
These changes prevent taking the fast (direct matching with PCRE2) path
and use the slow string search path if the alignment doesn't match
the UTF string encoding.
To not complicate the change and additional alignment member
is added to each searched string in the search collection.
* Create search hit description on the stack
* Unset complete JIT matching if user provided custom jflags.
* Document what passing NULL to copy function pointers does.
* Replace the retarded idea of tracking offsets with a hashmap with a linear buffer.
This improves performance something like 10x.
* Remove const for the non-JIT builds.
* Remove additional flags for skip checking.
It is not needed because each decoded character is checked for printablity below anyways.
* Enfore no setup of IO mem with 0xff
* Fix: Set JIT complete flag for multi regex patterns
* Fix heap.
* Add note about worsed performance path.
* Fix unit test with string terminated by undefined code point.
* Run clang-format
* Fix order of arguments
* Add warning about string search with encoding=guess to tests.
* Fix string lengths, they no longer count the final invalid code point.
* Fix endian macro on Windows
* Fix NULL dereference
* Fix number tests
* Use endianness check not dependent on stdbit
* Fix type annotations.
* Reintroduce rz_str_len_utf8char
* Fix command description.
* Fix and unify RZ_SYS_ENDIAN macros.
- Don't allow unhandled architectures anymore.
- Check endianness for Sparc and PPC using non GCC/Clang compilers.
- Fix several endianness checks using the value instead of the macros.
* Fix tests
* Apply review comments.
699c7d2dff