When I create and compile a "Hello, World!" application in C#, I get three files in the Debug folder apart from the main exe (e.g. HelloWorld.exe)
What purpose do these files serve?
The vshost.exe feature was introduced with Visual Studio 2005 (to answer your comment).
The purpose of it is mostly to make debugging launch quicker - basically there's already a process with the framework running, just ready to load your application as soon as you want it to.
See this MSDN article and this blog post for more information.