Raw string and regular expression in Python

Question

I have some confusions regarding raw string in the following code:

import re

text2 = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
text2_re = re.sub(r'(\d+)/(\d+)/(\d+)', r'\3-\1-\2', text2)
print (text2_re) #output: Today is 2012-11-27. PyCon starts 2013-3-13.

print (r'(\d+)/(\d+)/(\d+)') #output: (\d+)/(\d+)/(\d+)

As how I understand the raw string, without r, the \ is treated as escape character; with r, the backslash \ is treated as itself literally.

However, what I cannot understand in the above code is that: In the regular expression Line 5, even though there is a r, the "\d" inside is treated as one number [0-9] instead of one backslash \ plus one letter d.

In the second print Line 8, all characters are treated as raw strings.

What is the difference?

Additional Edition:

I made the following Four variations, with or without r:

import re

text2 = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
text2_re = re.sub(r'(\d+)/(\d+)/(\d+)', r'\3-\1-\2', text2)
text2_re1 = re.sub('(\d+)/(\d+)/(\d+)', r'\3-\1-\2', text2)
text2_re2 = re.sub(r'(\d+)/(\d+)/(\d+)', '\3-\1-\2', text2)
text2_re3 = re.sub('(\d+)/(\d+)/(\d+)', '\3-\1-\2', text2)

print (text2_re)
print (text2_re1)
print (text2_re2)
print (text2_re3)

And get the following output:

Could you explain these Four situations specifically?

Answer 1

You're getting confused by the difference between a string and a string literal.

A string literal is what you put between " or ' and the python interpreter parses this string and puts it into memory. If you mark your string literal as a raw string literal (using r') then the python interpreter will not change the representation of that string before putting it into memory but once they've been parsed they are stored exactly the same way.

This means that in memory there is no such thing as a raw string. Both the following strings are stored identically in memory with no concept of whether they were raw or not.

r'a regex digit: \d'  # a regex digit: \d
'a regex digit: \\d'  # a regex digit: \d

Both these strings contain \d and there is nothing to say that this came from a raw string. So when you pass this string to the re module it sees that there is a \d and sees it as a digit because the re module does not know that the string came from a raw string literal.

In your specific example, to get a literal backslash followed by a literal d you would use \\d like so:

import re

text2 = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
text2_re = re.sub(r'(\\d+)/(\\d+)/(\\d+)', r'\3-\1-\2', text2)
print (text2_re) #output: Today is 11/27/2012. PyCon starts 3/13/2013.

Alternatively, without using raw strings:

import re

text = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
text_re = re.sub('(\\d+)/(\\d+)/(\\d+)', '\\3-\\1-\\2', text2)
print (text_re) #output: Today is 2012-11-27. PyCon starts 2013-3-13.

text2 = 'Today is 11/27/2012. PyCon starts 3/13/2013.'
text2_re = re.sub('(\\\\d+)/(\\\\d+)/(\\\\d+)', '\\3-\\1-\\2', text2)
print (text2_re) #output: Today is 11/27/2012. PyCon starts 3/13/2013.

I hope that helps somewhat.

Edit: I didn't want to complicate things but because \d is not a valid escape sequence python does not change it, so '\d' == r'\d' is true. Since \\ is a valid escape sequence it gets changed to \, so you get the behaviour '\d' == '\\d' == r'\d'. Strings get confusing sometimes.

Edit2: To answer your edit, let's look at each line specifically:

text2_re = re.sub(r'(\d+)/(\d+)/(\d+)', r'\3-\1-\2', text2)

re.sub receives the two strings (\d+)/(\d+)/(\d+) and \3-\1-\2. Hopefully this behaves as you expect now.

text2_re1 = re.sub('(\d+)/(\d+)/(\d+)', r'\3-\1-\2', text2)

Again (because \d is not a valid string escape it doesn't get changed, see my first edit) re.sub receives the two strings (\d+)/(\d+)/(\d+) and \3-\1-\2. Since \d doesn't get changed by the python interpreter r'(\d+)/(\d+)/(\d+)' == '(\d+)/(\d+)/(\d+)'. If you understand my first edit then hopefully you should understand why these two cases behave the same.

text2_re2 = re.sub(r'(\d+)/(\d+)/(\d+)', '\3-\1-\2', text2)

This case is a bit different because \1, \2 and \3 are all valid escape sequences, they are replaced with the unicode character whose decimal representation is given by the number. That's quite complex but it basically boils down to:

\1  # stands for the ascii start-of-heading character
\2  # stands for the ascii start-of-text character
\3  # stands for the ascii end-of-text character

This means that re.sub receives the first string as it has done in the first two examples ((\d+)/(\d+)/(\d+)) but the second string is actually <start-of-heading>/<start-of-text>/<end-of-text>. So re.sub replaces the match with that second string exactly but since none of the three (\1, \2 or \3) are printable characters python just prints a stock place-holder character instead.

text2_re3 = re.sub('(\d+)/(\d+)/(\d+)', '\3-\1-\2', text2)

This behaves like the third example because r'(\d+)/(\d+)/(\d+)' == '(\d+)/(\d+)/(\d+)', as explained in the second example.