When you self-host "launch as Eclipse application" are you sure that you have the p2 bundles selected? This includes the p2 bundles, the needed ecf bundles, DS and the *equinox.util. DS must also be started.
In Eclipse 3.6 there is a checky (on the configuration tab in the launch configuration) that says "Support software installation in the launched application". Can you make sure this is checked?