I have
been working on UI Automation Tests against React App with Selenium WebDriver in my current
Microsoft Project since last December, 2018.
Our team sets up the release
pipelines in Azure Devops and runs the tests including our Selenium UI
Automation tests.
Sometimes
some UI Automation tests failed in Azure release pipeline, but passed in my
local machine. How to debug the failed tests is the challenge?
Here
are my lessons learned.
Since
we use Azure
Pipelines Hosted VS2017 image to run the tests, we need to know the
setup details of hosted VS 2017.
Chrome
Driver:
If your Chrome browser is
automatically updated to the latest one in your local machine, it does not mean Azure Devops will use
the latest one. It takes 1-3 months to update the latest version in
Azure Devops.
I remember when Chrome was automatically
updated to v74 in my machine, Azure Devops still used Chrome v71. The tests
passed in v74, but failed in v71. Therefore, downgrading the right Chrome
version to mimic the one in Azure Devops is more important.
Resolution: 1024*768
Azure Devops is using 1024*768
resolution by default to run UI automation tests. It is not possible for
us to change the resolution in Azure Devops.
Some people in the online forum
mentioned screen resolution utility task and they
implemented it in the release pipeline to fix the resolution issue.
However, in our team at
Microsoft , we do not have direct control over what DevOps tasks are installed and
there is an approval process for onboarding. I also submitted the request, but
declined unfortunately.
Therefore, if the tests fail
in the release pipelines, the easiest way is to change the resolution to 1024*768 in the
local machine and run the tests. Most of the time I can repro the issue.
Adding more logs in the tests:
I remember when I looked at the
test failures in the release pipeline for the first time, I did not know why
the tests failed because there was no log information. Therefore, I added more in
each test. It is easy to track which step causes the test failure.
Set maximum # of attempts =3.
I asked the Azure Devops
engineer to set maximum # of attempts =3. The idea is that UI Automation tests
are not as robust as API tests. If the test fails 3 times, it means there is
something wrong with this test.(Please see this post: Setmaximum # of attempts for Selenium UI automation tests in Azure releasepipeline)
Add wait time when the page loads to wait for elements to appear
Sometimes UI automation tests run too fast to capture the UI element
name. Therefore, using WebDriverWait to add the wait time is a good idea.
WebDriverWait wait
= new WebDriverWait(driver,
TimeSpan.FromSeconds(seconds));
Selenium Simple Click is not really reliable in React App in
Chrome
When Chrome was upgraded from v71 to v74, a lot of UI automation
tests against React App failed because the element.Click() was not working
properly. There was no error because clicking the element had no response. I
filed the bug in Chromium project , but they closed bug with no repro.
In the latest Chrome version, Element.Click() is not working
properly in React App. Using Actions class to hover over element is not always
working either. The only way I found out was that using IJavaScriptExecutor() class handled the click easily. It took us one week to
fix all failed tests and it was the great achievement!
Next time if you would like to debug the test failures in Selenium
against React App in Azure Devops, the information above may be helpful to you.
