Extracting whole words

python regex word alphabetical text-extraction
orlade picture orlade · Apr 19, 2011 · Viewed 33.6k times · Source

I have a large set of real-world text that I need to pull words out of to input into a spell checker. I'd like to extract as many meaningful words as possible without too much noise. I know there's plenty of regex ninjas around here, so hopefully someone can help me out.

Currently I'm extracting all alphabetical sequences with '[a-z]+'. This is an okay approximation, but it drags a lot of rubbish out with it.

Ideally I would like some regex (doesn't have to be pretty or efficient) that extracts all alphabetical sequences delimited by natural word separators (such as [/-_,.: ] etc.), and ignores any alphabetical sequences with illegal bounds.

However I'd also be happy to just be able to get all alphabetical sequences that ARE NOT adjacent to a number. So for instance 'pie21' would NOT extract 'pie', but 'http://foo.com' would extract ['http', 'foo', 'com'].

I tried lookahead and lookbehind assertions, but they were applied per-character (so for example re.findall('(?<!\d)[a-z]+(?!\d)', 'pie21') would return 'pi' when I want it to return nothing). I tried wrapping the alpha part as a term ((?:[a-z]+)) but it didn't help.

More detail: The data is an email database, so it's mostly plain English with normal numbers, but occasionally there's rubbish strings like GIHQ4NWL0S5SCGBDD40ZXE5IDP13TYNEA and AC7A21C0 that I'd like to ignore completely. I'm assuming any alphabetical sequence with a number in it is rubbish.

Answer

Tim Pietzcker picture Tim Pietzcker · Apr 19, 2011

If you restrict yourself to ASCII letters, then use (with the re.I option set)

\b[a-z]+\b

\b is a word boundary anchor, matching only at the start and end of alphanumeric "words". So \b[a-z]+\b matches pie, but not pie21 or 21pie.

To also allow other non-ASCII letters, you can use something like this:

\b[^\W\d_]+\b

which also allows accented characters etc. You may need to set the re.UNICODE option, especially when using Python 2, in order to allow the \w shorthand to match non-ASCII letters.

[^\W\d_] as a negated character class allows any alphanumeric character except for digits and underscore.

Related questions

How to extract numbers from a string in Python?

I would extract all the numbers contained in a string. Which is the better suited for the purpose, regular expressions or the isdigit() method? Example: line = "hello 12 hi 89" Result: [12, 89]

Read More
Python string.replace regular expression

I have a parameter file of the form: parameter-name parameter-value Where the parameters may be in any order but there is only one parameter per line. I want to replace one parameter's parameter-value with a new value. I am using …

Read More
Split string on whitespace in Python

I'm looking for the Python equivalent of String str = "many fancy word \nhello \thi"; String whiteSpaceRegex = "\\s"; String[] words = str.split(whiteSpaceRegex); ["many", "fancy", "word", "hello", "hi"]

Read More