Giving C++ ownership of class object in pybind11

[VaradDerivative]

I have an application in which I want to expose a C++ class MyClass to python using pybind. Since I want python to be embedded within c++, I use PYBIND11_EMBEDDED_MODULE to expose my c++ class.

class MyClass
{
	void method1();
	void method2();
}

PYBIND11_EMBEDDED_MODULE(myModule, m)
{
	py::class_<MyClass>(m, "MyPyClass")
	.def(py::init())
	.def("method1()", &MyClass::method1)
	.def("method2()", &MyClass::method2)
}

Now, I want to it to work in such a way that I take an already instantiated instance of MyClass, say myInstance - and create the python instance from this. To achieve this, I use py::cast()

void doPython(MyClass* myInstance)
{
	py::module_ myModule = py::module_::import("myModule");

	py::object myPyInstance = py::cast(myInstance);

	... // work with myPyInstance further
}

Now, the issue that I'm trying to solve is - I want my c++ application to have sole ownership of myInstance. I want myPyInstance to have nothing to do with the lifetime of myInstance. And that if myInstance is destructed at some point, myPyInstance should know about it and handle that case.

Is there some kind of weak ref workflow that I can use? pybind natively allows for unique_ptr or shared_ptr workflow, but none of those work for my case.

The custom smart pointer workflow sounds promising, and I pursued a solution where the smart pointer held a holder to myInstance. I hoped that .get() on the smart pointer would be called when the individual methods are called, allowing the smart pointer to check whether myInstance is alive or not. But it looks like .get() is called when cast() is called and the result is held onto.

There is return_value_policy::reference but it doesn't handle the case of the object being deleted by the c++ application while still being referenced by python, and this is a crucial case for me.

[smurfix]

How do you want that to work? the Python reference isn't going to go away magically.

You need to add a delete function to your object that tells Python to free its reference. Maybe convert your Python reference into a weakref, or some derivative thereof, and figure out how to tell it that its referent is gone.

[VaradDerivative]

Hey, thanks for the pointers, can you elaborate a bit on these ideas? a weakref workflow does sound the best, wherein C++ holds on to the strong ref, and pybind is given the weakref, and I tried to get something to work via the Custom Smart Pointer workflow that pybind offers, but that didn't work.

[smurfix]

Well, when you pass your object to Python, you need to pass the Python object back. Then your destructor can simply call Py_DECREF on it (and raise an error when the refcount wasn't one).

On the Python side you create a weakref for hanging on to the object. You need to make very sure that any dereferences you do are short-lived. That does include frames that get saved when you raise exceptions.

You might investigate switching to nanobind which has a few more elaborate schemes that are worth looking into.