Creating a class within a function and access a function defined in the containing function's scope
Edit:
See my full answer at the bottom of this question.
tl;dr answer: Python has statically nested scopes. The static aspect can interact with the implicit variable declarations, yielding non-obvious results.
(This can be especially surprising because of the language's generally dynamic nature).
I thought I had a pretty good handle on Python's scoping rules, but this problem has me thoroughly stymied, and my google-fu has failed me (not that I'm surprised - look at the question title ;)
I'm going to start with a few examples that work as expected, but feel free to skip to example 4 for the juicy part.
Example 1.
>>> x = 3
>>> class MyClass(object):
... x = x
...
>>> MyClass.x
3
Straightforward enough: during class definition we're able to access the variables defined in the outer (in this case global) scope.
Example 2.
>>> def mymethod(self):
... return self.x
...
>>> x = 3
>>> class MyClass(object):
... x = x
... mymethod = mymethod
...
>>> MyClass().mymethod()
3
Again (ignoring for the moment why one might want to do this), there's nothing unexpected here: we can access functions in the outer scope.
Note: as Frédéric pointed out below, this function doesn't seem to work. See Example 5 (and beyond) instead.
Example 3.
>>> def myfunc():
... x = 3
... class MyClass(object):
... x = x
... return MyClass
...
>>> myfunc().x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in myfunc
File "<stdin>", line 4, in MyClass
NameError: name 'x' is not defined
This is essentially the same as example 1: we're accessing the outer scope from within the class definition, just this time that scope isn't global, thanks to myfunc().
Edit 5: As @user3022222 pointed out below, I botched this example in my original posting. I believe this fails because only functions (not other code blocks, like this class definition) can access variables in the enclosing scope. For non-function code blocks, only local, global and built-in variables are accessible. A more thorough explanation is available in this question
One more:
Example 4.
>>> def my_defining_func():
... def mymethod(self):
... return self.y
... class MyClass(object):
... mymethod = mymethod
... y = 3
... return MyClass
...
>>> my_defining_func()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in my_defining_func
File "<stdin>", line 5, in MyClass
NameError: name 'mymethod' is not defined
Um...excuse me?
What makes this any different from example 2?
I'm completely befuddled. Please sort me out. Thanks!
P.S. on the off-chance that this isn't just a problem with my understanding, I've tried this on Python 2.5.2 and Python 2.6.2. Unfortunately those are all I have access to at the moment, but they both exhibit the same behaviour.
Edit According to http://docs.python.org/tutorial/classes.html#python-scopes-and-namespaces: at any time during execution, there are at least three nested scopes whose namespaces are directly accessible:
- the innermost scope, which is searched first, contains the local names
- the scopes of any enclosing functions, which are searched starting with the nearest enclosing scope, contains non-local, but also non-global names
- the next-to-last scope contains the current module’s global names
- the outermost scope (searched last) is the namespace containing built-in names
#4. seems to be a counter-example to the second of these.
Edit 2
Example 5.
>>> def fun1():
... x = 3
... def fun2():
... print x
... return fun2
...
>>> fun1()()
3
Edit 3
As @Frédéric pointed out the assignment of to a variable of the same name as it has in the outer scope seems to "mask" the outer variable, preventing the assignment from functioning.
So this modified version of Example 4 works:
def my_defining_func():
def mymethod_outer(self):
return self.y
class MyClass(object):
mymethod = mymethod_outer
y = 3
return MyClass
my_defining_func()
However this doesn't:
def my_defining_func():
def mymethod(self):
return self.y
class MyClass(object):
mymethod_temp = mymethod
mymethod = mymethod_temp
y = 3
return MyClass
my_defining_func()
I still don't fully understand why this masking occurs: shouldn't the name binding occur when the assignment happens?
This example at least provides some hint (and a more useful error message):
>>> def my_defining_func():
... x = 3
... def my_inner_func():
... x = x
... return x
... return my_inner_func
...
>>> my_defining_func()()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in my_inner_func
UnboundLocalError: local variable 'x' referenced before assignment
>>> my_defining_func()
<function my_inner_func at 0xb755e6f4>
So it appears that the local variable is defined at function creation (which succeeds), resulting in the local name being "reserved" and thus masking the outer-scope name when the function is called.
Interesting.
Thanks Frédéric for the answer(s)!
For reference, from the python docs:
It is important to realize that scopes are determined textually: the global scope of a function defined in a module is that module’s namespace, no matter from where or by what alias the function is called. On the other hand, the actual search for names is done dynamically, at run time — however, the language definition is evolving towards static name resolution, at “compile” time, so don’t rely on dynamic name resolution! (In fact, local variables are already determined statically.)
Edit 4
The Real Answer
This seemingly confusing behaviour is caused by Python's statically nested scopes as defined in PEP 227. It actually has nothing to do with PEP 3104.
From PEP 227:
The name resolution rules are typical for statically scoped languages [...] [except] variables are not declared. If a name binding operation occurs anywhere in a function, then that name is treated as local to the function and all references refer to the local binding. If a reference occurs before the name is bound, a NameError is raised.
[...]
An example from Tim Peters demonstrates the potential pitfalls of nested scopes in the absence of declarations:
i = 6 def f(x): def g(): print i # ... # skip to the next page # ... for i in x: # ah, i *is* local to f, so this is what g sees pass g()The call to g() will refer to the variable i bound in f() by the for loop. If g() is called before the loop is executed, a NameError will be raised.
Lets run two simpler versions of Tim's example:
>>> i = 6
>>> def f(x):
... def g():
... print i
... # ...
... # later
... # ...
... i = x
... g()
...
>>> f(3)
3
when g() doesn't find i in its inner scope, it dynamically searches outwards, finding the i in f's scope, which has been bound to 3 through the i = x assignment.
But changing the order the final two statements in f causes an error:
>>> i = 6
>>> def f(x):
... def g():
... print i
... # ...
... # later
... # ...
... g()
... i = x # Note: I've swapped places
...
>>> f(3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in f
File "<stdin>", line 3, in g
NameError: free variable 'i' referenced before assignment in enclosing scope
Remembering that PEP 227 said "The name resolution rules are typical for statically scoped languages", lets look at the (semi-)equivalent C version offer:
// nested.c
#include <stdio.h>
int i = 6;
void f(int x){
int i; // <--- implicit in the python code above
void g(){
printf("%d\n",i);
}
g();
i = x;
g();
}
int main(void){
f(3);
}
compile and run:
$ gcc nested.c -o nested
$ ./nested
134520820
3
So while C will happily use an unbound variable (using whatever happens to have been stored there before: 134520820, in this case), Python (thankfully) refuses.
As an interesting side-note, statically nested scopes enable what Alex Martelli has called "the single most important optimization the Python compiler does: a function's local variables are not kept in a dict, they're in a tight vector of values, and each local variable access uses the index in that vector, not a name lookup."
Answer
That's an artifact of Python's name resolution rules: you only have access to the global and the local scopes, but not to the scopes in-between, e.g. not to your immediate outer scope.
EDIT: The above was poorly worded, you do have access to the variables defined in outer scopes, but by doing x = x or mymethod = mymethod from a non-global namespace, you're actually masking the outer variable with the one you're defining locally.
In example 2, your immediate outer scope is the global scope, so MyClass can see mymethod, but in example 4 your immediate outer scope is my_defining_func(), so it can't, because the outer definition of mymethod is already masked by its local definition.
See PEP 3104 for more details about nonlocal name resolution.
Also note that, for the reasons explained above, I can't get example 3 to run under either Python 2.6.5 or 3.1.2:
>>> def myfunc():
... x = 3
... class MyClass(object):
... x = x
... return MyClass
...
>>> myfunc().x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in myfunc
File "<stdin>", line 4, in MyClass
NameError: name 'x' is not defined
But the following would work:
>>> def myfunc():
... x = 3
... class MyClass(object):
... y = x
... return MyClass
...
>>> myfunc().y
3