What exactly does the VS project option "Register for COM interop" actually do? Because when I build my library with this option enabled I can call CreateObject on my library from VBScript. But if I build without this and then run regasm manually CreateObject fails. So I'm wondering -- what does VS2010 do that I'm not doing?
It does the same thing as running Regasm.exe with the /tlb and /codebase options. The /codebase option is probably the one you forgot. Regasm likes assuming you put the DLL in the GAC and generates a warning when you don't. The GAC is indeed a very good way to avoid DLL Hell, always a COM problem. But not appropriate on your dev machine, you don't want to pollute the GAC while developing and testing the code. It only matters on your user's machine, the one that's likely to be exposed to multiple versions.
Using the wrong version of Regasm.exe on a 64-bit machine is another way to get in trouble, there are usually 4 versions on your machine. Be sure to distinguish the 32-bit and 64-bit versions (c:\windows\microsoft\framework vs framework64), they write different registry keys. You want to pick the one that's compatible with the client app. Using both is okay too, .NET code can run in either mode, but pretty unusual. And distinguish between the v2.0.50727 (.NET 2.0 through 3.5SP1) and the v4.0 versions. Picking the right Visual Studio Command Prompt is half the battle.