I'm trying to understand the exact mechanism behind updating a python dictionary using d[key] += diff
. I have some helper classes to trace magic method invocations:
class sdict(dict):
def __setitem__(self, *args, **kargs):
print "sdict.__setitem__"
return super(sdict, self).__setitem__(*args, **kargs)
def __delitem__(self, *args, **kargs):
print "sdict.__delitem__"
return super(sdict, self).__delitem__(*args, **kargs)
def __getitem__(self, *args, **kargs):
print "sdict.__getitem__"
return super(sdict, self).__getitem__(*args, **kargs)
def __iadd__(self, *args, **kargs):
print "sdict.__iadd__"
return super(sdict, self).__iadd__(*args, **kargs)
def __add__(self, *args, **kargs):
print "sdict.__add__"
return super(sdict, self).__add__(*args, **kargs)
class mutable(object):
def __init__(self, val=0):
self.value = val
def __iadd__(self, val):
print "mutable.__iadd__"
self.value = self.value + val
return self
def __add__(self, val):
print "mutable.__add__"
return mutable(self.value + val)
With these tools, let's go diving:
>>> d = sdict()
>>> d["a"] = 0
sdict.__setitem__
>>> d["a"] += 1
sdict.__getitem__
sdict.__setitem__
>>> d["a"]
sdict.__getitem__
1
We don't see any __iadd__
operation invoked here, which makes sense because the left-hand side expression d["a"]
returns an integer that does not implement the __iadd__
method. We do see python magically converting the +=
operator into __getitem__
and __setitem__
calls.
Continuing:
>>> d["m"] = mutable()
sdict.__setitem__
>>> d["m"] += 1
sdict.__getitem__
mutable.__iadd__
sdict.__setitem__
>>> d["m"]
sdict.__getitem__
<__main__.mutable object at 0x106c4b710>
Here the +=
operator successfully invokes an __iadd__
method. It looks like the +=
operator is actually being used twice:
__getitem__
and __setitem__
calls__iadd__
call.Where I need help is the following:
+=
operator into __getitem__
and __setitem__
calls?+=
operator used twice? Doesn't python translate the statement to d["m"] = d["m"] + 1
(In which case wouldn't we see __add__
be invoked instead of __iadd__
?)In the first example, you didn't apply the +=
operator to the dictionary. You applied it to the value stored in the d['a']
key, and that's a different object altogether.
In other words, Python will retrieve d['m']
(a __getitem__
call), apply the +=
operator to that, then set the result of that expression back to d['m']
(the __setitem__
call).
The __iadd__
method either returns self
mutated in-place, or a new object, but Python cannot know for sure what the method returned. So it has to call d.__setitem__('m', <return_value_from_d['m'].__iadd__(1)>)
, always.
The exact same thing happens if you did:
m = d['m']
m += 1
d['m'] = m
but without the extra name m
in the global namespace.
If the mutable()
instance was not stored in a dictionary but in the global namespace instead, the exact same sequence of events takes place, but directly on the globals()
dictionary, and you wouldn't get to see the __getitem__
and __setitem__
calls.
This is documented under the augmented assignment reference documentation:
An augmented assignment evaluates the target (which, unlike normal assignment statements, cannot be an unpacking) and the expression list, performs the binary operation specific to the type of assignment on the two operands, and assigns the result to the original target.
where d['m']
is the target; evaluating the target here involves __getitem__
, assigning the result back to the original target invokes __setitem__
.