Top "Tokenize" questions

Tokenizing strings using regular expression in Javascript

Suppose I've a long string containing newlines and tabs as: var x = "This is a long string.\n\t This …

javascript regex string tokenize stringtokenizer

tokenize a string keeping delimiters in Python

Is there any equivalent to str.split in Python that also returns the delimiters? I need to preserve the whitespace …

python string split tokenize

Nested strtok function problem in C

I have a string like this: a;b;c;d;e f;g;h;i;j 1;2;3;4;5 and i want to …

c nested token tokenize strtok

implicit declaration of function ‘strtok_r’ [-Wimplicit-function-declaration] inspite including <string.h>

I have the following code to tokenize a string containing lines separated by \n and each line has integers separated …

c string tokenize strtok gcc-warning

Writing a tokenizer in Python

I want to design a custom tokenizer module in Python that lets users specify what tokenizer(s) to use for …

python regex token tokenize nltk

Tokenize problem in Java with separator ". "

I need to split a text using the separator ". ". For example I want this string : Washington is the U.S …

java string tokenize stringtokenizer

Boost::tokenizer comma separated (c++)

Should be an easy one for you guys..... I'm playing around with tokenizers using Boost and I want create a …

c++ boost tokenize boost-tokenizer

how to convert csv to table in oracle

How can I make a package that returns results in table format when passed in csv values. select * from table(…

string oracle csv plsql tokenize

Pythonic way to implement a tokenizer

I'm going to implement a tokenizer in Python and I was wondering if you could offer some style advice? I've …

python coding-style tokenize

ElasticSearch Analyzer and Tokenizer for Emails

I could not find a perfect solution either in Google or ES for the following situation, hope someone could help …

email elasticsearch lucene tokenize analyzer