Search This Blog

June 24, 2019

How to debug the Selenium test failures against React app in Azure Devops release pipeline?

I have been working on UI Automation Tests against React App with Selenium WebDriver in my current Microsoft Project since last December, 2018. 

Our team sets up the release pipelines in Azure Devops and runs the tests including our Selenium UI Automation tests.

Sometimes some UI Automation tests failed in Azure release pipeline, but passed in my local machine. How to debug the failed tests is the challenge?

Here are my lessons learned.

Since we use Azure Pipelines Hosted VS2017 image to run the tests, we need to know the setup details of hosted VS 2017.

Chrome Driver:

If your Chrome browser is automatically updated to the latest one in your local machine, it does not mean Azure Devops will use the latest one. It takes 1-3 months to update the latest version in Azure Devops.

I remember when Chrome was automatically updated to v74 in my machine, Azure Devops still used Chrome v71. The tests passed in v74, but failed in v71. Therefore, downgrading the right Chrome version to mimic the one in Azure Devops is more important.

Resolution: 1024*768

Azure Devops is using 1024*768 resolution by default to run UI automation tests.  It is not possible for us to change the resolution in Azure Devops.

Some people in the online forum mentioned screen resolution utility task and they implemented it in the release pipeline to fix the resolution issue.

However, in our team at Microsoft , we do not have direct control over what DevOps tasks are installed and there is an approval process for onboarding. I also submitted the request, but declined unfortunately.

Therefore, if the tests fail in the release pipelines, the easiest way is to change the resolution to 1024*768 in the local machine and run the tests. Most of the time I can repro the issue.

Adding more logs in the tests:

I remember when I looked at the test failures in the release pipeline for the first time, I did not know why the tests failed because there was no log information. Therefore, I added more in each test. It is easy to track which step causes the test failure.

Set maximum # of attempts =3.
I asked the Azure Devops engineer to set maximum # of attempts =3. The idea is that UI Automation tests are not as robust as API tests. If the test fails 3 times, it means there is something wrong with this test.(Please see this post: Setmaximum # of attempts for Selenium UI automation tests in Azure releasepipeline)

Add wait time when the page loads to wait for elements to appear
Sometimes UI automation tests run too fast to capture the UI element name. Therefore, using WebDriverWait to add the wait time is a good idea.

WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(seconds));

Selenium Simple Click is not really reliable in React App in Chrome

When Chrome was upgraded from v71 to v74, a lot of UI automation tests against React App failed because the element.Click() was not working properly. There was no error because clicking the element had no response. I filed the bug in Chromium project , but they closed bug with no repro.

In the latest Chrome version, Element.Click() is not working properly in React App. Using Actions class to hover over element is not always working either. The only way I found out was that using IJavaScriptExecutor() class handled the click easily. It took us one week to fix all failed tests and it was the great achievement!

Next time if you would like to debug the test failures in Selenium against React App in Azure Devops, the information above may be helpful to you.

June 21, 2019

Set maximum # of attempts for Selenium UI automation tests in Azure release pipeline

Our Azure Devops engineer set up the Selenium UI Automation Tests in the Azure release pipeline. I asked him to set maximum # of attempts =3.

The idea is that UI Automation tests are not as robust as API tests. If the test fails 3 times, it means there is something wrong with this test.

Maybe there is a bug.

Maybe developers update the UI element names without letting me know.

Maybe there is the resolution issue (1024*768) in Azure devops.

Maybe there is the account conflict when doing the parallel runs.

When my colleagues see the Pipeline tab , they always feel confused. Why it shows the error? But in the Logs tab, it shows succeeded. So the test result should show error or succeeded?

I always explain that it is the design issue in the release pipeline. We set maximum # of attempts =3. For example, this test has been run 2 times and it shows the Test Run Failed messages twice. But the final test result is succeeded.

Finally they understand they need to look at succeeded text.

June 13, 2019

Fixed the Visual Studio 2017 IDE Error when trying to use debug coded web performance test.

Today I used the Visual Studio Coded Web Performance test to record the web request and I generated the code to coded.cs file.

The interesting part was that when using 'debug Web Performance test' feature, the Visual Studio IDE showed the following error.

After looking into this issue, it is defenitely the product bug. How to fix it?

Go to Local.testsettings. In General Tab, just change it to “Run tests using local computer or a test controller” and you are all set.