"MetaClass", "__new__", "cls" and "super" - what is the mechanism exactly?

Question

I have read posts like these:

1. What is a metaclass in Python?
2. What are your (concrete) use-cases for metaclasses in Python?
3. Python's Super is nifty, but you can't use it

But somehow I got confused. Many confusions like:

When and why would I have to do something like the following?

# Refer link1
return super(MyType, cls).__new__(cls, name, bases, newattrs)

or

# Refer link2
return super(MetaSingleton, cls).__call__(*args, **kw)

or

# Refer link2
return type(self.__name__ + other.__name__, (self, other), {})

How does super work exactly?

What is class registry and unregistry in link1 and how exactly does it work? (I thought it has something to do with singleton. I may be wrong, being from C background. My coding style is still a mix of functional and OO).

What is the flow of class instantiation (subclass, metaclass, super, type) and method invocation (

metaclass->__new__, metaclass->__init__, super->__new__, subclass->__init__ inherited from metaclass

) with well-commented working code (though the first link is quite close, but it does not talk about cls keyword and super(..) and registry). Preferably an example with multiple inheritance.

P.S.: I made the last part as code because Stack Overflow formatting was converting the text metaclass->__new__ to metaclass->new

Answer 1

OK, you've thrown quite a few concepts into the mix here! I'm going to pull out a few of the specific questions you have.

In general, understanding super, the MRO and metclasses is made much more complicated because there have been lots of changes in this tricky area over the last few versions of Python.

Python's own documentation is a very good reference, and completely up to date. There is an IBM developerWorks article which is fine as an introduction and takes a more tutorial-based approach, but note that it's five years old, and spends a lot of time talking about the older-style approaches to meta-classes.

super is how you access an object's super-classes. It's more complex than (for example) Java's super keyword, mainly because of multiple inheritance in Python. As Super Considered Harmful explains, using super() can result in you implicitly using a chain of super-classes, the order of which is defined by the Method Resolution Order (MRO).

You can see the MRO for a class easily by invoking mro() on the class (not on an instance). Note that meta-classes are not in an object's super-class hierarchy.

Thomas' description of meta-classes here is excellent:

A metaclass is the class of a class. Like a class defines how an instance of the class behaves, a metaclass defines how a class behaves. A class is an instance of a metaclass.

In the examples you give, here's what's going on:

1. The call to __new__ is being bubbled up to the next thing in the MRO. In this case, super(MyType, cls) would resolve to type; calling type.__new__ lets Python complete it's normal instance creation steps.

2. This example is using meta-classes to enforce a singleton. He's overriding __call__ in the metaclass so that whenever a class instance is created, he intercepts that, and can bypass instance creation if there already is one (stored in cls.instance). Note that overriding __new__ in the metaclass won't be good enough, because that's only called when creating the class. Overriding __new__ on the class would work, however.

3. This shows a way to dynamically create a class. Here's he's appending the supplied class's name to the created class name, and adding it to the class hierarchy too.

I'm not exactly sure what sort of code example you're looking for, but here's a brief one showing meta-classes, inheritance and method resolution:

class MyMeta(type):
    def __new__(cls, name, bases, dct):
        print "meta: creating %s %s" % (name, bases)
        return type.__new__(cls, name, bases, dct)

    def meta_meth(cls):
        print "MyMeta.meta_meth"

    __repr__ = lambda c: c.__name__

class A(object):
    __metaclass__ = MyMeta
    def __init__(self):
        super(A, self).__init__()
        print "A init"

    def meth(self):
        print "A.meth"

class B(object):
    __metaclass__ = MyMeta
    def __init__(self):
        super(B, self).__init__()
        print "B init"

    def meth(self):
        print "B.meth"

class C(A, B):
    __metaclass__ = MyMeta
    def __init__(self):
        super(C, self).__init__()
        print "C init"

>>> c_obj = C()
meta: creating A (<type 'object'>,)
meta: creating B (<type 'object'>,)
meta: creating C (A, B)
B init
A init
C init
>>> c_obj.meth()
A.meth
>>> C.meta_meth()
MyMeta.meta_meth
>>> c_obj.meta_meth()
Traceback (most recent call last):
  File "mro.py", line 38, in <module>
    c_obj.meta_meth()
AttributeError: 'C' object has no attribute 'meta_meth'