Search This Blog

July 2, 2020

How to do automated accessibility tests in Selenium?

Today one person asked me about how to do accessibility tests in Selenium. 

I remember after I joined this team, we used Chisel Web Selenium to scan the page directly in Visual Studio. However, we could not know if this test passed or failed locally.

new ChiselWebSelenium().Scan(webdriver, "Scan name");

Once we deployed to DEV/PPE and ran the tests, some test failures appeared in the Azure release pipeline (in accessibility tab). Resetting the baseline could ignore the failures.

In September 2019, we were told that Chisel Web Selenium was deprecated in August 2019 and we needed to move to Selenium Axe as soon as possible.


 At that time if we used Selenium Axe, all tests failed in Visual Studio directly. Since this is the new model, we need to exclude the known failures.

Here is the good example(TestAccessibilityOfPageExcludingKnownIssues ) to show how we needed to update the code.

         AxeResult axeResult = new AxeBuilder(Driver)

                .WithOptions(

                    new AxeRunOptions()

                    {

                        RunOnly = new RunOnlyOptions()

                        {

                            Type = "tag",

                            Values = new List<string>() {  "wcag21a", "wcag21aa" },

                        },

                        Iframes = false,

                    })

                // Exclude some elements

                .Exclude("#meInitialsButton")

                .Exclude("script#mt_script")

                .Analyze();

            axeResult.Violations.Should().BeEmpty();


June 23, 2020

How to test localization using Selenium?

Today someone asked me about how to test localization of the Portal using Selenium. This is the good question.

In our React Portal (Microsoft Dynamics 365 Customer Service Insights) , we need to support 43 languages.  

For example, if we want to test in Spanish, we can just add lang= ‘es-ES’. 

  ChromeOptions options = new ChromeOptions();

  options.AddArgument($"lang={language}");

When you run the test, the Chrome window will be launched in Spanish. Is that simple?

The following 6 languages are not supported in our React Portal because it still shows English.

Malay = "ms-MY";

Basque = "eu-ES";

SerbianCyrillic = "sr-Cyrl-CS";

SerbianLatin = "sr-Latn-CS";

Kazakh = "kk-KZ";

Galician = "gl-ES";


Last year I opened the bug to bugs.Chromium.org. Finally, they closed it and status was “Won’t fix”.

If you see the same issue above, do not feel surprised.


June 19, 2020

CloudTest: No error message appears in the Azure Pipeline if data-driven tests fail.

Our team at Microsoft decided to use CloudTest framework to run Selenium Tests against React Portal in Azure Devops.

I have developed many localization tests. Today I saw many failed, but there was no error message showing up in the release pipeline and it was hard to debug.

I posted the issue in Stackoverflow.microsoft.com where everyone working at Microsoft can post the questions.

The dev in the CloudTest team replied and said the main issue is that these tests are data-driven tests, and they are not handling the results (especially the way they are reported using InnerResults) correctly while parsing trx files. They have created the work item to track this request.

Hopefully this feature can be finished and deployed shortly..

May 21, 2020

Fixed the issue in Azure Pipeline: session not created: This version of ChromeDriver only supports Chrome version 80


We deployed to Canary on May 19 in the Azure Pipeline. When running all UI automation tests in the pipeline, all failed. 

Here is the following error message in each test.

Initialization method UIAutomationTests.Tests.TopicsTests.Setup threw exception. System.InvalidOperationException: session not created: This version of ChromeDriver only supports Chrome version 80


After looking into this issue, here is my note.

·       Google Chrome or Microsoft Edge Chromium version 82 is skipped due to COVID-19 pandemic.  The latest version is 83.


·       The Cloud test framework we use in the Azure pipeline is always installing the latest version of Chrome. Therefore, the Chrome driver 80 or 81 was not supported in Chrome 83. There is no Chrome driver 82 and it is required to use Chrome driver 83 to run the tests.

·       On May 19, I did not see the Selenium.WebDriver.ChromeDriver V83 in NuGet Package.There is no way to update it. 


In order to know the test result as soon as possible, I need to manually download the Chrome driver 83 and updated the code in Visual Studio 2019. 

After updating it and running all tests in my machine, all passed.