I have an installation package (MSI based) which attempts to register a dll file for use as a Windows Explorer Bar (Internet Explorer Toolbar).
The custom action is defined as follows:
[SystemFolder]\regsvr32.exe /s "pathtodllhere"
Output from MSI log:
CustomAction SystemFolder_2 returned actual error code 5. Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Action SystemFolder_2, location: C:\Windows\SysWOW64\, command: regsvr32.exe /s "C:\Program Files (x86)\Test Install\test.dll"
MSI (s) (10:F4): Product: Test Install -- Error 1722. There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Action SystemFolder_2, location: C:\Windows\SysWOW64\, command: regsvr32.exe /s "C:\Program Files (x86)\Test Install\test.dll"
A brief run-down of my tests/thoughts thus-far:
Hopefully I have shown my research here and don't waste anyone's time!
The best practise solution (as you've alluded to) is to extract the registry entries from the component and write them to the registry using the msi.
Background
Regsvr32 requires admin rights and elevated privilege confirmation to register COM components under 64-bit Windows 7 but msiexec doesn't know to request elevated privileges for the custom action.
You can test this requirement by using "Open with..." to run C:\Windows\SysWOW64\regsvr32.exe on the component (which will fail). Whereas if you create a batch file for the registration and then "Run as Administrator" the component will register successfully.