Calling python method from C++ (or C) callback

Question

I am trying to call methods in a python class from C++. The C++ method from which this is called is a C++ callback.

Within this method when I am trying to call python method, it was giving segmentation fault.

I have saved an instance of python function in a global variable like

// (pFunc is global variable of type PyObject*)
pFunc = PyDict_GetItemString(pDict, "PlxMsgWrapper");

where PlxMsgWrapper is a python method, which will be used in the callback.

In the callback, the arguments are created as

PyObject* args = PyTuple_Pack(2, PyString_FromString(header.c_str()),
                                 PyString_FromString(payload.c_str()));

When creating the

PyObject * pInstance = PyObject_CallObject(pFunc, args);

In this line its giving segmentation fault. After this the actual python method is called as

PyObject* recv_msg_func = PyObject_GetAttrString(module, (char *)"recvCallback");
args = PyTuple_Pack(1, pInstance);
PyObject_CallObject(recv_msg_func, args);

Answer 1

There are a few things you need to do if you are invoking a Python function from a C/C++ callback. First when you save off your python function object, you need to increment the reference count with:

Py_INCREF(pFunc)

Otherwise Python has no idea you are holding onto an object reference, and it may garbage collect it, resulting in a segmentation fault when you try to use it from your callback.

Then next thing you need to be concerned about is what thread is running when your C/C++ callback is invoked. If you are getting called back from another non-Python created thread (i.e. a C/C++ thread receiving data on a socket), then you MUST acquire Python's Global Interpreter Lock (GIL) before calling any Python API functions. Otherwise your program's behavior is undefined. To acquire the GIL you do:

void callback() {
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();

    // Get args, etc.

    // Call your Python function object
    PyObject * pInstance = PyObject_CallObject(pFunc, args);

    // Do any other needed Python API operations

    // Release the thread. No Python API allowed beyond this point.
    PyGILState_Release(gstate);
}

Also, in your extension module's init function, you should do the following to ensure that threading is properly initialized:

// Make sure the GIL has been created since we need to acquire it in our
// callback to safely call into the python application.
if (! PyEval_ThreadsInitialized()) {
    PyEval_InitThreads();
}

Otherwise, crashes and strange behavior may ensue when you attempt to acquire the GIL from a non-Python thread.

See Non-Python Created Threads for more detail on this.