Search This Blog

December 20, 2012

TFS Build: When Automated Tests Fail, TFS Build Fails

Yesterday I saw one guy asking one interesting question in MSDN forum. The question can be extended: If automated tests (unit tests, web tests, DB tests, WCF tests…) via MSTEST fail in TFS build, TFS build needs to fail.  It is easy to configure in the TFS build definition:

Open TFS build definition

In Process tab, expand Automated Test

Set Fail Build On Test Failure = True (False by default)

When you queue a new build, TFS build fails as long as one test fails .

In TFS Build definition, I like to Build 2 projects with default platforms and configurations. One is to build Developer project and the other one is to build TEST project.

If TFS can build 2 projects successfully and then run my automated tests, sometimes tests fail due to the code change (service change, DB change) and you are not aware of it. In this case, you can’t make TFS build fail because you need to modify your automated tests.

Therefore, I don’t encourage people to Set Fail Build On Test Failure = True.

If TFS can’t build Developer or TEST project successfully, TFS build fails.

If TFS can build Developer or TEST project successfully, and some automated tests fail, TFS will show Build partially succeeded.

In this case, you can investigate the issue and log the bug or modify your automated tests if required.


Unknown said...

Tests are your guarantee. I think that if tests fail, it isn't acceptable that the build was partially succeeded.

Ray said...

From my experience, the tests fail due to the new change of spec and developers modify the code, but testers are not aware of. In this case, do you think making TFS build fail is a good idea? :)