About the changing id of an immutable string

Question

Something about the id of objects of type str (in python 2.7) puzzles me. The str type is immutable, so I would expect that once it is created, it will always have the same id. I believe I don't phrase myself so well, so instead I'll post an example of input and output sequence.

>>> id('so')
140614155123888
>>> id('so')
140614155123848
>>> id('so')
140614155123808

so in the meanwhile, it changes all the time. However, after having a variable pointing at that string, things change:

>>> so = 'so'
>>> id('so')
140614155123728
>>> so = 'so'
>>> id(so)
140614155123728
>>> not_so = 'so'
>>> id(not_so)
140614155123728

So it looks like it freezes the id, once a variable holds that value. Indeed, after del so and del not_so, the output of id('so') start changing again.

This is not the same behaviour as with (small) integers.

I know there is not real connection between immutability and having the same id; still, I am trying to figure out the source of this behaviour. I believe that someone whose familiar with python's internals would be less surprised than me, so I am trying to reach the same point...

Update

Trying the same with a different string gave different results...

>>> id('hello')
139978087896384
>>> id('hello')
139978087896384
>>> id('hello')
139978087896384

Now it is equal...

Answer 1

CPython does not promise to intern all strings by default, but in practice, a lot of places in the Python codebase do reuse already-created string objects. A lot of Python internals use (the C-equivalent of) the sys.intern() function call to explicitly intern Python strings, but unless you hit one of those special cases, two identical Python string literals will produce different strings.

Python is also free to reuse memory locations, and Python will also optimize immutable literals by storing them once, at compile time, with the bytecode in code objects. The Python REPL (interactive interpreter) also stores the most recent expression result in the _ name, which muddles up things some more.

As such, you will see the same id crop up from time to time.

Running just the line id(<string literal>) in the REPL goes through several steps:

1. The line is compiled, which includes creating a constant for the string object:

   >>> compile("id('foo')", '<stdin>', 'single').co_consts
   ('foo', None)
   

   This shows the stored constants with the compiled bytecode; in this case a string 'foo' and the None singleton. Simple expressions consisting of that produce an immutable value may be optimised at this stage, see the note on optimizers, below.

2. On execution, the string is loaded from the code constants, and id() returns the memory location. The resulting int value is bound to _, as well as printed:

   >>> import dis
   >>> dis.dis(compile("id('foo')", '<stdin>', 'single'))
     1           0 LOAD_NAME                0 (id)
                 3 LOAD_CONST               0 ('foo')
                 6 CALL_FUNCTION            1
                 9 PRINT_EXPR          
                10 LOAD_CONST               1 (None)
                13 RETURN_VALUE        
   

3. The code object is not referenced by anything, reference count drops to 0 and the code object is deleted. As a consequence, so is the string object.

Python can then perhaps reuse the same memory location for a new string object, if you re-run the same code. This usually leads to the same memory address being printed if you repeat this code. This does depend on what else you do with your Python memory.

ID reuse is not predictable; if in the meantime the garbage collector runs to clear circular references, other memory could be freed and you'll get new memory addresses.

Next, the Python compiler will also intern any Python string stored as a constant, provided it looks enough like a valid identifier. The Python code object factory function PyCode_New will intern any string object that contains only ASCII letters, digits or underscores, by calling intern_string_constants(). This function recurses through the constants structures and for any string object v found there executes:

if (all_name_chars(v)) {
    PyObject *w = v;
    PyUnicode_InternInPlace(&v);
    if (w != v) {
        PyTuple_SET_ITEM(tuple, i, v);
        modified = 1;
    }
}

where all_name_chars() is documented as

/* all_name_chars(s): true iff s matches [a-zA-Z0-9_]* */

Since you created strings that fit that criterion, they are interned, which is why you see the same ID being used for the 'so' string in your second test: as long as a reference to the interned version survives, interning will cause future 'so' literals to reuse the interned string object, even in new code blocks and bound to different identifiers. In your first test, you don't save a reference to the string, so the interned strings are discarded before they can be reused.

Incidentally, your new name so = 'so' binds a string to a name that contains the same characters. In other words, you are creating a global whose name and value are equal. As Python interns both identifiers and qualifying constants, you end up using the same string object for both the identifier and its value:

>>> compile("so = 'so'", '<stdin>', 'single').co_names[0] is compile("so = 'so'", '<stdin>', 'single').co_consts[0]
True

If you create strings that are either not code object constants, or contain characters outside of the letters + numbers + underscore range, you'll see the id() value not being reused:

>>> some_var = 'Look ma, spaces and punctuation!'
>>> some_other_var = 'Look ma, spaces and punctuation!'
>>> id(some_var)
4493058384
>>> id(some_other_var)
4493058456
>>> foo = 'Concatenating_' + 'also_helps_if_long_enough'
>>> bar = 'Concatenating_' + 'also_helps_if_long_enough'
>>> foo is bar
False
>>> foo == bar
True

The Python compiler either uses the peephole optimizer (Python versions < 3.7) or the more capable AST optimizer (3.7 and newer) to pre-calculate (fold) the results of simple expressions involving constants. The peepholder limits it's output to a sequence of length 20 or less (to prevent bloating code objects and memory use), while the AST optimizer uses a separate limit for strings of 4096 characters. This means that concatenating shorter strings consisting only of name characters can still lead to interned strings if the resulting string fits within the optimizer limits of your current Python version.

E.g. on Python 3.7, 'foo' * 20 will result in a single interned string, because constant folding turns this into a single value, while on Python 3.6 or older only 'foo' * 6 would be folded:

>>> import dis, sys
>>> sys.version_info
sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0)
>>> dis.dis("'foo' * 20")
  1           0 LOAD_CONST               0 ('foofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoo')
              2 RETURN_VALUE

and

>>> dis.dis("'foo' * 6")
  1           0 LOAD_CONST               2 ('foofoofoofoofoofoo')
              2 RETURN_VALUE
>>> dis.dis("'foo' * 7")
  1           0 LOAD_CONST               0 ('foo')
              2 LOAD_CONST               1 (7)
              4 BINARY_MULTIPLY
              6 RETURN_VALUE