The Matrix: A Quick Guide to Cross-Browser Compatibililty Testing
By: Perry Miranda


The ever growing list of legacy and new browser versions can be overwhelming to any developer and tester trying to ensure their web UI behaves adequately across platforms, OS versions, devices, screen resolutions, and several other permutations.  Testing a website or web app across all available browsers on multiple platforms is virtually impossible and impractical even if you have unlimited resources.  Below are some simple guidelines to help you narrow down the browsers to include in your cross-browser compatibility testing efforts, hereinafter referred to as “The Matrix”.  This article highlights some of the tactics we use at WeddingWire to determine our cross-browser compatibility test methodology along with some of the tools and services we use.

1.  Use site traffic metrics to determine your most important browsers.  We use Google Analytics data for the last 6 months to determine the browsers with the highest and lowest traffic.

a)  Typically any traffic below 10% of the total site's traffic is a good candidate for not supporting.

b)  Prioritize the platforms and their respective versions based on the top traffic for each, i.e., Windows, Mac, iOS, and Android.

c)  Prioritize the browsers and their respective versions based on the top traffic for each supported platform, i.e., Internet Explorer, Firefox, Chrome, and Safari.

2.  For newer sites that may not yet have substantial site traffic metrics, understand and identify the target users of your web application, and the browsers they use to make them the top priority in the matrix.  For new commercial web sites, try to support the latest versions of the most popular browsers. The W3Schools site offers a comprehensive monthly view of each browser’s usage trend.  This information is crucial in determining the browser versions to include in the matrix.

3.  For any unsupported browser versions, determine if your web site detects them and offers any options to users.  Include this test scenario in the matrix if applicable.

4.  Investigate, evaluate, and select tools that allow you to test your web application across multiple platforms and browser versions.  Some of them even offer options to integrate with your Continuous Integration system and test automation scripts.

a)  SauceLabs - Provides a virtual machine for testing various browsers and OS platforms.  The service provides great diagnosis for website and mobile applications.

b) BrowserStack - Web-based service that offers the ability to test your site across a massive collection of mobile/desktop OS and browser combinations.  It also provides instant access to a browser inside of a virtual machine.  This is a great tool for both desktop and mobile testing.

c) CrossBrowserTesting - Another great service with an impressive layout comparison feature that lets you choose a "base" browser for comparisons and get a summary of rendering differences, along with a screenshot of side-by-side images to catch and debug layout issues effectively.

d) Browserling - An affordable tool that provides unlimited browser sessions and SSH tunnels for multiple versions of Chrome, Firefox, Safari, Internet Explorer and Opera.

5.  If you're serious about responsive, there's no way around interacting with real mobile and tablet devices.  Be prepared to spend some money on buying the most popular and newer devices.  Fortunately there are some services that offer access to many physical devices from your browser:

a) Perfecto Mobile - Offers a cloud-based solution with thousands of REAL devices located securely around the world.

b) Keynote Mobile Testing - Powered by the DeviceAnywhere Cloud with a huge amount of mobile and tablet devices available with touch and gesture interaction.

6.  Geolocation - Our websites include functionality to geolocate the user within specific regions.  Most mobile cloud-based services provide the ability to set the test device’s location.  For a quick desktop geolocation test, the Geopeeker website provides screenshots of a specific page at various locations.

There is a large amount of value in all of these cross-browser testing tools and services.  But before you dive headfirst into these services, consider building your own "Matrix" - a list of browser versions on specific OS platforms that you want to support, in a prioritized list.


Meet Perry:
Perry is a Senior QA Software Engineer at WeddingWire. Perry is from Ponce, Puerto Rico and has a Computer Engineering degree from the University of Puerto Rico at Mayagüez. When he doesn't have his hands dirty testing software, Perry likes driving his Vespa scooter around DC, jogging around Rock Creek Park, and enjoying his eclectic music collection from all over the world.