Dynamically created method and decorator, got error 'functools.partial' object has no attribute '__module__'

Question

I am currently using EndpointsModel to create a RESTful API for all my models on AppEngine. Since it is RESTful, these api have a lot of repeat code which I want to avoid.

For example:

class Reducer(EndpointsModel):
    name = ndb.StringProperty(indexed=False)

@endpoints.api(
    name="bigdata",
    version="v1",
    description="""The BigData API""",
    allowed_client_ids=ALLOWED_CLIENT_IDS,
)
class BigDataApi(remote.Service):
    @Reducer.method(
        path="reducer",
        http_method="POST",
        name="reducer.insert",
        user_required=True,
    )
    def ReducerInsert(self, obj):
        pass

    ## and GET, POST, PUT, DELETE
    ## REPEATED for each model

I want to make them become generic. So I try to dynamic add method to the class. What I have tried so far:

from functools import partial, wraps

def GenericInsert(self, obj, cls):
    obj.owner = endpoints.get_current_user()
    obj.put()
    return obj

# Ignore GenericDelete, GenericGet, GenericUpdate ...

import types
from functools import partial

def register_rest_api(api_server, endpoint_cls):
    name = endpoint_cls.__name__

    # create list method 
    query_method = types.MethodType(
    endpoint_cls.query_method(
        query_fields=('limit', 'pageToken'),
        path="%ss" % name,
        http_method="GET",
        name="%s.list" % name,
        user_required=True
    )(partial(GenericList, cls=endpoint_cls)))

    setattr(api_server, "%sList", query_method)

    # create insert method
    # ...

register_rest_api(BigDataApi, Reducer)

But I got 'functools.partial' object has no attribute '__module__' exception. I think it is because there are some conflicts between endpoints.method's decorator and partial. But no idea how to avoid it.

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 239, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 298, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 84, in LoadObject
    obj = __import__(path[0])
  File "/Users/Sylvia/gcdc2013/apis.py", line 795, in <module>
    register_rest_api(BigDataApi, Reducer)
  File "/Users/Sylvia/gcdc2013/apis.py", line 788, in register_rest_api
    )(partial(GenericList, cls=endpoint_cls)))
  File "/Users/Sylvia/gcdc2013/endpoints_proto_datastore/ndb/model.py", line 1544, in RequestToQueryDecorator
    @functools.wraps(api_method)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", line 33, in update_wrapper
    setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'functools.partial' object has no attribute '__module__'

related articles:

• Class method differences in Python: bound, unbound and static
• Python - can I programmatically decorate class methods from a class instance?
• Programmatically generate methods for a class
• Adding a Method to an Existing Object Instance

Answer 1

I also stumbled upon this, I was really surprised, for me the issue was that partial objects are missing certain attributes, specifically __module__ and __name__

Being that wraps by default uses functools.WRAPPER_ASSIGNMENTS to update attributes, which defaults to ('__module__', '__name__', '__doc__') in python 2.7.6 anyway, there are a couple ways of dealing with this ...

Update only present attributes ...

import functools
import itertools

def wraps_safely(obj, attr_names=functools.WRAPPER_ASSIGNMENTS):
    return wraps(obj, assigned=itertools.ifilter(functools.partial(hasattr, obj), attr_names))

>>> def foo():
...     """ Ubiquitous foo function ...."""
... 
>>> functools.wraps(partial(foo))(foo)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", line 33, in update_wrapper
setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'functools.partial' object has no attribute '__module__'
>>> wraps_safely(partial(foo))(foo)()
>>> 

Here we simply filter out all those attribute which aren't present.

Another approach would be to strictly deal with partial objects only, you could fold wraps with singledispatch and create wrapped partial objects whose attributes would be taken from the deepest func attribute.

Something along the lines:

import functools

def wraps_partial(wrapper, *args, **kwargs):
    """ Creates a callable object whose attributes will be set from the partials nested func attribute ..."""
    wrapper = wrapper.func
    while isinstance(wrapper, functools.partial):
        wrapper = wrapper.func
    return functools.wraps(wrapper, *args, **kwargs)

def foo():
    """ Foo function.
    :return: None """
    pass

>>> wraps_partial(partial(partial(foo)))(lambda : None).__doc__
' Foo Function, returns None '
>>> wraps_partial(partial(partial(foo)))(lambda : None).__name__
'foo'
>>> wraps_partial(partial(partial(foo)))(lambda : None)()
>>> pfoo = partial(partial(foo))
>>> @wraps_partial(pfoo)
... def not_foo():
...     """ Not Foo function ... """
... 
>>> not_foo.__doc__
' Foo Function, returns None '
>>> not_foo.__name__
'foo'
>>>

This is slightly better since now we can get the original functions docs which before defaulted to using the partial objects doc string.

This can be modified to only search if the current partial object doesn't already have the set attribute, which should be slightly faster when nesting many partial objects ...

UPDATE

It seems that python(CPython) 3 (at least 3.4.3) doesn't have this issue, since I don't know nor should I assume all versions of python 3 or other implementations such as Jython also share this issue here is another future ready approach

from functools import wraps, partial, WRAPPER_ASSIGNMENTS

try:
    wraps(partial(wraps))(wraps)
except AttributeError:
    @wraps(wraps)
    def wraps(obj, attr_names=WRAPPER_ASSIGNMENTS, wraps=wraps):
        return wraps(obj, assigned=(name for name in attr_names if hasattr(obj, name))) 

a couple things to note:

• we define a new wraps function only if we fail to wrap a partial, in case future versions of python2 or other versions fix this issue.
• we use the original wraps to copy the docs and other info
• we don't use ifilter since it was removed in python3, I've timeit with and without ifilter but the results where inconclusive, at least in python (CPython) 2.7.6, the difference was marginal at best either way...