Search This Blog

June 21, 2019

Set maximum # of attempts for Selenium UI automation tests in Azure release pipeline

Our Azure Devops engineer set up the Selenium UI Automation Tests in the Azure release pipeline. I asked him to set maximum # of attempts =3.

The idea is that UI Automation tests are not as robust as API tests. If the test fails 3 times, it means there is something wrong with this test.

Maybe there is a bug.

Maybe developers update the UI element names without letting me know.

Maybe there is the resolution issue (1024*768) in Azure devops.

Maybe there is the account conflict when doing the parallel runs.

When my colleagues see the Pipeline tab , they always feel confused. Why it shows the error? But in the Logs tab, it shows succeeded. So the test result should show error or succeeded?

I always explain that it is the design issue in the release pipeline. We set maximum # of attempts =3. For example, this test has been run 2 times and it shows the Test Run Failed messages twice. But the final test result is succeeded.

Finally they understand they need to look at succeeded text.

No comments: