How to use COM from python win32com or comtypes to access an IRTDServer?

Question

Environment

Windows 10 + python 3.6.3 64 bit (also tried 32 bit). I am a python developer trying to use COM for (nearly) the first time and hit this huge blocker.

Problem

I have had various errors when trying to use an IRTDServer implemented in a dll (not written by me), via either win32com or comtypes. Using win32com turned out to be more difficult. I have an included an example unittest for both libraries below.

Accessing the server from Excel 2016 works as expected; this returns the expected value:

=RTD("foo.bar", , "STAT1", "METRIC1")

---

Code using win32com library

Here is a simple test case which should connect to the server but doesn't. (This is just one version, as I have changed it many times trying to debug the problem.)

from unittest import TestCase

class COMtest(TestCase):
    def test_win32com(self):
        import win32com.client
        from win32com.server.util import wrap

        class RTDclient:
            # are these only required when implementing the server?
            _com_interfaces_ = ["IRTDUpdateEvent"]
            _public_methods_ = ["Disconnect", "UpdateNotify"]
            _public_attrs_ = ["HeartbeatInterval"]

            def __init__(self, *args, **kwargs):
                self._comObj = win32com.client.Dispatch(*args, **kwargs)
            def connect(self):
                self._rtd = win32com.client.CastTo(self._comObj, 'IRtdServer')
                result = self._rtd.ServerStart(wrap(self))
                assert result > 0

            def UpdateNotify(self):
                print("UpdateNotify() callback")
            def Disconnect(self):
                print("Disconnect() called")
            HeartbeatInterval = -1

_rtd = RTDclient("foo.bar")
_rtd.connect()

Result:

Traceback (most recent call last):
  File "env\lib\site-packages\win32com\client\gencache.py", line 532, in EnsureDispatch
    ti = disp._oleobj_.GetTypeInfo()
pywintypes.com_error: (-2147467263, 'Not implemented', None, None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test\test.py", line 23, in test_win32com
    _rtd.connect()
  File "test\test.py", line 16, in connect
    self._rtd = win32com.client.CastTo(dispatch, 'IRtdServer')
  File "env\lib\site-packages\win32com\client\__init__.py", line 134, in CastTo
    ob = gencache.EnsureDispatch(ob)
  File "env\lib\site-packages\win32com\client\gencache.py", line 543, in EnsureDispatch
    raise TypeError("This COM object can not automate the makepy process - please run makepy manually for this object")
TypeError: This COM object can not automate the makepy process - please run makepy manually for this object

Following those directions, I ran the makepy script successfully:

> env\Scripts\python.exe env\lib\site-packages\win32com\client\makepy.py "foo.bar"
Generating to C:\Users\user1\AppData\Local\Temp\gen_py\3.5\longuuid1x0x1x0.py
Building definitions from type library...
Generating...
Importing module

(I replaced the UUID on stackoverflow for privacy. This UUID is the same as the typelib UUID for "foo.bar".)

The generated file contains various the function and type definitions of both IRtdServer and IRTDUpdateEvent. But in this file, both interfaces are subclasses of win32com.client.DispatchBaseClass, while according to OleViewDotNet, they should be subclasses of IUnknown?

However, when I attempted to run the unittest again, I received the exact same error as before. It is as if the lookup mechanism is not finding the generated module?

Also, GetTypeInfo returning Not implemented is alarming me. From my understanding, win32com uses that method (part of IDispatch COM interface) to determine the argument and return types for all other functions in other interfaces, including IRtdServer. If it's not implemented, it would be unable to determine the types correctly. Yet, the generated file seems to include this information, which is also perplexing.

---

Code using comtypes library

from unittest import TestCase

class COMtest(TestCase):
    def test_comtypes(self):
        import comtypes.client

        class RTDclient:
            # are these for win32com only?
            _com_interfaces_ = ["IRTDUpdateEvent"]
            _public_methods_ = ["Disconnect", "UpdateNotify"]
            _public_attrs_ = ["HeartbeatInterval"]

            def __init__(self, clsid):
                self._comObj = comtypes.client.CreateObject(clsid)
            def connect(self):
                self._rtd = self._comObj.IRtdServer()
                result = self._rtd.ServerStart(self)
                assert result > 0

            def UpdateNotify(self):
                print("UpdateNotify() callback")
            def Disconnect(self):
                print("Disconnect() called")
            HeartbeatInterval = -1

_rtd = RTDclient("foo.bar")
_rtd.connect()

Result:

  File "test\test.py", line 27, in test_comtypes
    _rtd.connect()
  File "test\test.py", line 16, in connect
    self._rtd = self._comObj.IRTDServer()
  File "env\lib\site-packages\comtypes\client\dynamic.py", line 110, in __getattr__
    dispid = self._comobj.GetIDsOfNames(name)[0]
  File "env\lib\site-packages\comtypes\automation.py", line 708, in GetIDsOfNames
    self.__com_GetIDsOfNames(riid_null, arr, len(names), lcid, ids)
_ctypes.COMError: (-2147352570, 'Unknown name.', (None, None, None, 0, None))

---

Some other solutions I've tried

(Based on googling and answers in the comments below)

• (Re-)Registered the DLL
• Registered the 32 bit version of the DLL and tried python 32 bit
• Set compatibility mode of python.exe to Windows XP SP3

• Tried not instantiating IRtdServer, that is, replacing these two lines:

  self._rtd = self._comObj.IRtdServer()
  result = self._rtd.ServerStart(self)
  

  with:

  result = self._comObj.ServerStart(self)
  

  The error this time is:

  TypeError: 'NoneType' object is not callable
  

  That would seem to indicate that the ServerStart function exists, but is undefined? (Seems really weird. There must be more to this mystery.)

• Tried passing interface="IRtdServer" parameter to CreateObject:

  def __init__(self, clsid):
      self._comObj = comtypes.client.CreateObject(clsid, interface="IRtdServer")
  def connect(self):
      result = self._comObj.ServerStart(self)
      ...
  

  The error received is:

    File "test\test.py", line 13, in __init__
      self._comObj = comtypes.client.CreateObject(clsid, interface="IRtdServer")
    File "env\lib\site-packages\comtypes\client\__init__.py", line 238, in CreateObject
      obj = comtypes.CoCreateInstance(clsid, clsctx=clsctx, interface=interface)
    File "env\lib\site-packages\comtypes\__init__.py", line 1223, in CoCreateInstance
      p = POINTER(interface)()
  TypeError: Cannot create instance: has no _type_
  

  Tracing code in the comtypes library, that would seem to indicate that the interface parameter wants an interface class, not a string. I found various interfaces defined in the comtypes library: IDispatch, IPersist, IServiceProvider. All are subclasses of IUnknown. According to OleViewDotNet, IRtdServer is also a subclass of IUnknown. This leads me to believe that I need to similarly write an IRtdServer class in python in order to use the interface, but I don't know how to do that.

• I noticed the dynamic parameter of CreateObject. The code indicates this is mutually exclusive to the interface parameter, so I tried that:

  def __init__(self, clsid):
      self._comObj = comtypes.client.CreateObject(clsid, dynamic=True)
  def connect(self):
      self._rtd = self._comObj.IRtdServer()
      result = self._rtd.ServerStart(self)
  

  But the error is the same as my original error: IRtdServer has _ctypes.COMError: (-2147352570, 'Unknown name.', (None, None, None, 0, None))

Any help or clues would be greatly be appreciated. Thank you in advance.

---

(Not really knowing what I'm doing,) I tried to use OleViewDotNet to look at the DLL:

---

Answer 1

I ran into same problem.

I also tried using win32com to get excel run that for me, that's a bit unstable to be honest...I cannot even touch my Excel.

Therefore I spent some time looking into this. The problem lies with CastTo. Think that COM object you (and I) loaded just does not contain enough information to be casted (some methods like GetTypeInfo are not implemented etc...)

Therefore I created a wrapper that makes methods of those COM objects callable...not obvious. And this seems working for me.

Client code is modified from a project called pyrtd, which didn't work for various reasons (think due to change of RTD model...return of RefreshData is just completely different now).

import functools

import pythoncom
import win32com.client
from win32com import universal
from win32com.client import gencache
from win32com.server.util import wrap


EXCEL_TLB_GUID = '{00020813-0000-0000-C000-000000000046}'
EXCEL_TLB_LCID = 0
EXCEL_TLB_MAJOR = 1
EXCEL_TLB_MINOR = 4

gencache.EnsureModule(EXCEL_TLB_GUID, EXCEL_TLB_LCID, EXCEL_TLB_MAJOR, EXCEL_TLB_MINOR)

universal.RegisterInterfaces(EXCEL_TLB_GUID,
                             EXCEL_TLB_LCID, EXCEL_TLB_MAJOR, EXCEL_TLB_MINOR,
                             ['IRtdServer', 'IRTDUpdateEvent'])


# noinspection PyProtectedMember
class ObjectWrapperCOM:
    """
    This object can act as a wrapper for an object dispatched using win32com.client.Dispatch
    Sometimes the object written by 3rd party is not well constructed that win32com will not be able to obtain
    type information etc in order to cast the object to a certain interface. win32com.client.CastTo will fail.

    This wrapper class will enable the object to call its methods in this case, even if we do not know what exactly
    the wrapped object is.
    """
    LCID = 0x0

    def __init__(self, obj):
        self._impl = obj  # type: win32com.client.CDispatch

    def __getattr__(self, item):
        flags, dispid = self._impl._find_dispatch_type_(item)
        if dispid is None:
            raise AttributeError("{} is not a valid property or method for this object.".format(item))
        return functools.partial(self._impl._oleobj_.Invoke, dispid, self.LCID, flags, True)


# noinspection PyPep8Naming
class RTDUpdateEvent:
    """
    Implements interface IRTDUpdateEvent from COM imports
    """
    _com_interfaces_ = ['IRTDUpdateEvent']
    _public_methods_ = ['Disconnect', 'UpdateNotify']
    _public_attrs_ = ['HeartbeatInterval']

    # Implementation of IRTDUpdateEvent.
    HeartbeatInterval = -1

    def __init__(self, event_driven=True):
        self.ready = False
        self._event_driven = event_driven

    def UpdateNotify(self):
        if self._event_driven:
            self.ready = True

    def Disconnect(self):
        pass


class RTDClient:
    """
    Implements a Real-Time-Data (RTD) client for accessing COM data sources that provide an IRtdServer interface.
    """

    MAX_REGISTERED_TOPICS = 1024

    def __init__(self, class_id):
        """
        :param classid: can either be class ID or program ID
        """
        self._class_id = class_id
        self._rtd = None
        self._update_event = None

        self._topic_to_id = {}
        self._id_to_topic = {}
        self._topic_values = {}
        self._last_topic_id = 0

    def connect(self, event_driven=True):
        """
        Connects to the RTD server.

        Set event_driven to false if you to disable update notifications.
        In this case you'll need to call refresh_data manually.
        """

        dispatch = win32com.client.Dispatch(self._class_id)
        self._update_event = RTDUpdateEvent(event_driven)
        try:
            self._rtd = win32com.client.CastTo(dispatch, 'IRtdServer')
        except TypeError:
            # Automated makepy failed...no detailed construction available for the class
            self._rtd = ObjectWrapperCOM(dispatch)

        self._rtd.ServerStart(wrap(self._update_event))

    def update(self):
        """
        Check if there is data waiting and call RefreshData if necessary. Returns True if new data has been received.
        Note that you should call this following a call to pythoncom.PumpWaitingMessages(). If you neglect to
        pump the message loop you'll never receive UpdateNotify callbacks.
        """
        # noinspection PyUnresolvedReferences
        pythoncom.PumpWaitingMessages()
        if self._update_event.ready:
            self._update_event.ready = False
            self.refresh_data()
            return True
        else:
            return False

    def refresh_data(self):
        """
        Grabs new data from the RTD server.
        """

        (ids, values) = self._rtd.RefreshData(self.MAX_REGISTERED_TOPICS)
        for id_, value in zip(ids, values):
            if id_ is None and value is None:
                # This is probably the end of message
                continue
            assert id_ in self._id_to_topic, "Topic ID {} is not registered.".format(id_)
            topic = self._id_to_topic[id_]
            self._topic_values[topic] = value

    def get(self, topic: tuple):
        """
        Gets the value of a registered topic. Returns None if no value is available. Throws an exception if
        the topic isn't registered.
        """
        assert topic in self._topic_to_id, 'Topic %s not registered.' % (topic,)
        return self._topic_values.get(topic)

    def register_topic(self, topic: tuple):
        """
        Registers a topic with the RTD server. The topic's value will be updated in subsequent data refreshes.
        """
        if topic not in self._topic_to_id:
            id_ = self._last_topic_id
            self._last_topic_id += 1

            self._topic_to_id[topic] = id_
            self._id_to_topic[id_] = topic

            self._rtd.ConnectData(id_, topic, True)

    def unregister_topic(self, topic: tuple):
        """
        Un-register topic so that it will not get updated.
        :param topic:
        :return:
        """
        assert topic in self._topic_to_id, 'Topic %s not registered.' % (topic,)
        self._rtd.DisconnectData(self._topic_to_id[topic])

    def disconnect(self):
        """
        Closes RTD server connection.
        :return:
        """
        self._rtd.ServerTerminate()