Apply a Regex on Stream?

Question

I'm searching for fast and safe way to apply Regular Expressions on Streams.

I found some examples over the internet that talking about converting each buffer to String and then apply the Regex on the string.

This approach have two problems:

• Performance: converting to strings and GC'ing the strings is waste of time and CPU and sure can be avoided if there was a more native way to apply Regex on Streams.
• Pure Regex support: Regex pattern sometimes can match only if combining two buffers together (buffer 1 ends with the first part of the match, and buffer 2 starts with the second part of the match). The convert-to-string way cannot handle this type of matching natively, I have to provide more information like the maximum length that the pattern can match, this does not support the + and * regex signs at all and will never support (unlimited match length).

So, the convert-to-string way is not fast, and doesn't fully support Regex.

Is there any way / Library that can be used to apply Regex on Streams without converting to strings and with full Regex support?

Answer 1

Intel has recently open sourced hyperscan library under BSD license. It's a high-performance non-backtracking NFA-based regex engine.

Features: ability to work on streams of input data and simultaneous multiple patterns matching. The last one differs from (pattern1|pattern2|...) approach, it actually matches patterns concurrently.

It also utilizes Intel's SIMD instructions sets like SSE4.2, AVX2 and BMI. The summary of the design and explanation of work can be found here. It also has great developer's reference guide with a lot of explanations as well as performance and usage considerations. Small article about using it in the wild (in russian).