How to decrease MSBuild times

Question

My situation

In the C# project I am now working on we have a fairly big solution (80+ projects). Now rebuild times of 5 minutes+ are really becoming quite a problem using MSBuild from Visual Studio 2008.

In an analysis I did last week it turned out that my build time was spent as follows:

1. Copying files to the projects and recopying it to the projects that depend on it (CopyToLocal), etc. (60%)

2. Invoking a postbuild to decompile/compile. (20%)

3. Doing the actual compilation, etc. (20%)

Apart from the 'normal' project bin\debug folders output is also copied to an external directory to set up the main 'loader' program. The main program structure is a bit like this:

\loader\bin\loader.exe

\loader\plugin\plugin1\plugin1.dll

\loader\plugin\plugin1\somedependency.dll

What I did

In an attempt to make things go a little faster I thought of the following:

1. Copy all the files to one a big bin directory and don't use CopyTolocal. I don't like this because we can no longer use different versions of the same DLL files and my bin directory is getting quite a mess.

2. Use parallelism (/m) for MSBuild. This helps only very little in build times.

3. Try to reduce dependencies between projects which is always a good thing of course.

4. Invest in hardware. I found some research on solid-state drives, but this does not seem promising.

My question

I also noticed that when I make a change to a project that is at the root of my dependency tree everything gets rebuild. Even if the change was only in the 'private' part and the interface of the project did not change.

Does MSBuild use a timestamp of dependent projects to determine if a project needs a rebuild?

Can this be changed to a different condition? For example, the checksum of the file?

Apart from this specific suggestion I would sure appreciate all suggestions to make build times faster.

Answer 1

I'm working on 500+ C# application projects. Projects are compiled in parallel and copylocal set to false. Compile time is about 37 min without unit tests and code coverage. 13 min for incremental build without any change in the code.

If I turn off parallel compilation and set copylocal to true, compile time is than 1 h 40 min.

I have different configuration for local build, gated check-in build and server builds with deploy phase (night builds).

Here are my experiences:

1. Copying output files to one directory is not good idea if you want to build your projects in parallel without CopyLocal set to false. My assemblies were sometimes locked when multiple projects referenced the same assembly and MSBuild tried to copy this reference to the output folder at the same time. This solution was very helpful for me. I set copylocal to false for all references and my build directory size was lowered 10x (10 times less I/O). I have a different setup for local build and for server build. Different setup for gated check-in build and for full deploy build.
2. If I enable parallel build, builds are faster, much faster. If you have a strong build server, your /m:2 build should be 2x faster as a /m:1 build. It has nothing to do with dependencies between projects (if copylocal is set to false).
3. You should reduce dependencies between the projects if you want to have a fast incremental build. It has no impact on a full build (copylocal false). Incremental compile time depends on the changed project location in the build tree.

Yes, MSBuild uses a timestamp of dependent projects to determine if a project needs a rebuild. It compares input files (code files, referenced assemblies, temporary files,..) timestamp with output assembly. If something is changed, your project is recompiled. Try to reduce the number of dependencies between projects to minimize recompilation. If your change was only in the 'private' part of the project, your output assembly will be changed, assembly timestamp will be changed and all related projects will be rebuild also. You cannot do much with this.

Run your build two times with diagnostic verbosity without any change in your code and check for "Building target "CoreCompile" completely" like I described here. You can have something wrong in your project files and your projects are recompiled every time. If you don't change anything your build log should not contain "Building target "CoreCompile" completely" logs.

Our build server is a virtual machine, not a real piece of hardware. It is not good idea to use a VM for a build server, but it was not my decision.

If you have multi GB RAM try to use part of it as a in-memory hard drive. Your build should be much faster :)

SSD drives are sensitive to high I/O per day. It has an impact on warranty.

I hope it helps someone ... ;)