TFS2017 - The build server is running more tests than we wrote. Wait, what?!?

There are a few cases in which the build server will run more tests than exist in a project. In all cases it is because one or more tests are being run more than once. This most often is easily understood and happens due to the test code being compiled more than once. This can happen when compiling for:

• Multiple configurations such as Debug and Release
• Multiple frameworks, see my previous post

All of these cases are easy to detect from looking at the build definition alone. I recently ran into a case where a team wrote 119 tests and 135 where being run during the build. They were building a single configuration and targeting a single framework so the code was only compiled once. This anomaly was therefore not due to any of the situations mentioned above.

To start to debug this situation we downloaded the trx file from the TFS Server for one of their builds and noticed there were 16 tests with the same name and same ID. Looking through the testing step output from the build, we saw the same set of 16 tests duplicated, so now we knew which 16 were being run twice but we still needed to determine the cause of the duplication. 

Looking at the code provided the next clue. The 16 tests were all in one test assembly and they were the only tests in that assembly. Also in that assembly were some abstract classes used by the tests.

Looking at the build output we found that the assembly containing those 16 tests was in two directories, which explained why those 16 tests were being re-run, because the testing step was configured to find any assembly that matched the pattern *Test.dll and execute it, so it found both instances of the assembly and executed the tests in each copy.

The final piece of the puzzle came when we found that the dll with the 16 duplicated tests in it was referenced by another unit test project. On compilation this caused that dll to be in its bin folder and in the bin folder of the project that referenced it. Those abstract classes I mentioned earlier were used by both projects. We did a bit of refactoring, pulled the common non-test code into a new assembly and the duplicate tests no longer showed up in the build.

The moral of the story: Assemblies containing unit tests should not reference other assemblies with Unit Tests. Be mindful of dependencies. While it is easy to reference other projects, it isn't always a good idea. This reminds me of a project that had circular build dependencies, that that is a story for another time.