Last Thursday, I noticed that one UI automated test failed in Canary in Azure Pipelines. After looking into this issue, it happened in all environments.
Previous situation: When we hit the portal, marketing landing page appears. Clicked Sign In button. The login page was open in another tab.
Current situation: When we hit the portal, marketing landing page appears. Click Sign In button. It is remaining in the same tab and is redirected to the login page.
This change was made by our marketing team and it looks more user-friendly. But it made one UI automated test failed and it was not the shipping blocker.
I fixed it in Dev. In this case, we do not want to re-deploy test code to PPE-Canary again because it is really time consuming.
I talked to my release engineer that we should exclude this test in the specific release pipeline for reporting purpose since everyone knew it. He totally agreed to what I was saying.
In test filter criteria, we added the following one and re-ran the tests. All tests passed.