Yesterday I
saw one guy asking one interesting question in MSDN forum. The question
can be extended: If automated tests (unit tests, web tests, DB tests, WCF tests…)
via MSTEST fail in TFS build, TFS build needs to fail. It is easy
to configure in the TFS build definition:
Open TFS
build definition
In Process
tab, expand Automated Test
Set
Fail Build On Test Failure = True (False by default)
When you
queue a new build, TFS build fails as long as one test fails .
In TFS Build
definition, I like to Build 2 projects with default
platforms and configurations. One is to build Developer project and the
other one is to build TEST project.
If TFS can
build 2 projects successfully and then run my automated tests, sometimes tests
fail due to the code change (service change, DB change) and you are not aware
of it. In this case, you can’t make TFS build fail because you need to modify
your automated tests.
Therefore, I
don’t encourage people to Set Fail Build On Test
Failure = True.
If TFS can’t
build Developer or TEST project successfully, TFS build fails.
If TFS can build
Developer or TEST project successfully, and some automated tests fail, TFS will
show Build partially succeeded.
In this
case, you can investigate the issue and log the bug or modify your automated tests if
required.
2 comments:
Tests are your guarantee. I think that if tests fail, it isn't acceptable that the build was partially succeeded.
From my experience, the tests fail due to the new change of spec and developers modify the code, but testers are not aware of. In this case, do you think making TFS build fail is a good idea? :)
Post a Comment