How to Choose What to Test When Creating a Responsive Site.

“We have a responsive site now, which means it works on all devices, and we should test all the things!” How many times have you heard something along these lines before? As testers, we know that this is not possible. We have to make the best recommendations we can to cover the most ground and test in a way that eliminates the most amount of risk. But with so many devices and so many different sizes, how do we choose what makes the most sense to test?

First, I want to make it clear what I am referring to when talking about a site that is “responsive.” I will also explain how it works at a high level, as that is important to understanding what to test for on a responsive website.

What is Responsive Design?

The core concept of a responsive site is building one site that displays properly and changes its layout depending on the width of the browser on the device that is being used. The main advantage of this over having a separate “mobile” (usually m.) or “tablet” site is that there is one code base and one content database. When there are separate sites that are maintained independently, often they have different content, and are less functional than the main “desktop version” of the site, which is a negative experience for both the site editors and users. On a responsive site, the same features exist regardless of what device it is viewed on, and when a change is made it is made for all versions.

How is the Layout Determined?

A common misconception about responsive design is that many think the layout format is determined by device detection. This is not the case and would be very difficult to maintain with the deluge of new devices and browsers released on a regular basis. The layout is determined only by the viewport width of the browser. This way the designers/developers of the website only need to worry about how wide the screen is in determining how it is displayed, and it will work universally regardless of the specific device.

Once the viewport width is detected, the site uses CSS media queries to serve up different CSS layouts depending on what breakpoint the browser is between. Breakpoints are the specific viewport widths at which the layout of the page changes. Many sites have only two breakpoints, which equates to three different layouts. An example may be a break point of 600 pixels(px) and a break point of 1200px (Note: the px in viewport widths are referring to CSS pixels and not physical ones. This will be discussed further below). In this example, there are three layouts: small, medium, and large. The small layout will display in any browser that is between 0px – 600px. The medium layout will be served up from 601px – 1199px. The large layout will be shown at 1200+. Since these are not device specific you can actually hit all of the breakpoints in a desktop browser by changing the width of the browser window. If you want to see it in action check out one example at Think with Google. On Think With Google, the breakpoints are 480px and 1000px. There are many different plug-ins that will visualize your viewport size as you change the size of your window. The one I use is the chrome extension “Window Size”. While the above examples only have two breakpoints, there can be as many breakpoints as the developers/designers choose to have.

Language is Important.

Note the language used above when referring to the different layout sizes. I call them small, medium, and large. Many designers, project managers, developers, clients, etc. can often be found referring to the different sizes as mobile, tablet, and desktop. This is both incorrect and confusing. A phone with a large viewport width often displays the medium view, a tablet with a small viewport width can display the small view, many tablets display the large view, a desktop browser may show the medium view if the user has multiple windows open, etc. How it displays has nothing to do with the classification of device. When the language refers to specific device types and the layouts do not meet those expectations, it causes people to be confused and think that there is a bug. As long as the layout that is displayed is within the defined breakpoint,it is working as intended regardless of what device it is being viewed on.

Understanding Viewports.

Another important concept to understand when testing responsive sites is viewport width. Viewport width is often confused with resolution and screen size but they are not the same thing. Resolution and screen size are only part of what determines the viewport width. The viewport width is how wide the CSS sees the page. The viewport width for a website is actually affected by a mix of resolution, screen size, pixel density, dppx (dots per pixel unit), on-screen UI elements, and the rendering software of the device. One of the most recognizable examples of this in action is Apple’s iOS line of devices when they switched over to retina displays. The original non-retina iPhone line had a resolution of 320x480, a relatively low pixel density of 163ppi (pixels per inch), and dppx (dots per pixel unit) of 1. The dppx is the key factor here. In the above example, the dppx of 1 means that there is 1 dot for each css pixel unit. In this case, the viewport width does actually match the resolution of 320 since it is a 1:1 ratio. When the iPhone 4 came out with its new “retina” display it had a much higher resolution of 640x960 and almost double the pixel density of 336. Since the physical size of the screen was the same, if the resolution of the display determined how the page rendered then logically everything on the iPhone 4 would have been much smaller than on the iPhone 3g, like it would have been on any standard computer monitor of the time. However, comparing them side-by-side shows the relative size of everything on the screen is exactly the same, but the “retina” display is just shaper. This is because the retina display has a dppx of 2. In other words, for every CSS “pixel unit” there were actually 2 physical dots on the retina screen, resulting in a ratio of 2:1. So in this case, even with the resolution width of 640px the viewport size was still 320px, and the page rendered with exactly the same proportions.

The low resolution iPhone 3g and the high resolution iPhone 4 have the same viewport width so they render the pages the same.

The above only covers viewport width. Viewport height gets more complicated as most browsers have UI elements on the top and bottom which would shrink the viewport height. That is why when you rotate a device to landscape, the landscape viewport width does not always match the portrait viewport height. While it is possible to have vertical break points, most responsive design sites only work with the width.

The UI elements that are present in portrait make the viewport height different than the viewport width in landscape on the same device.

The iPhone example above is easy to visualize since the devices are physically the same size. That does not always need to be the case though. For example, all of the iPads that have been released as of the writing of this all have the exact same viewport size. This includes the original iPad, iPad mini, and the iPad Air. If you have to test an iPad for a responsive site, you can test it on any retina iPad (iPad 3, iPad 4, iPad Air, iPad Mini 2, iPad Air 2) with the same OS and it should render exactly the same. This could change at some point like it has recently with the iPhone 6 and 6 plus, but still something to keep an eye out for to reduce redundancy in testing.

Another thing to keep in mind when testing on the iPhone is that up until the current line of phones, all iPhones have also had the same viewport sizes. All iPhones used to have a viewport width of 320px, but now the 6 is 375px and the 6 Plus is 414px - this is mostly due to the change in aspect ratios.

I know it seems like a lot of work to figure out what your viewport width is, but luckily other people have already done all of this work for you. There are many sites out there that will give you all of the aforementioned stats on each device including the dppx and viewport size. The easiest way to find a device viewport size is to use viewportsizes.com. Just type in your device and it will give you the viewport sizes. If your device is not included in the list, the site can also provide the viewport width of the device being used to view it.

Deciding What to Test.

So how do you take all of the information above and figure out what to test on? Basically it is a juggling act of covering the most amount of viewport sizes and browser versions while avoiding redundancy.

Before actually picking devices, you should start with a fixed number of devices you will commit to test based on the timeline of the project and bandwidth of your team. This avoids the scope of testing ballooning out of control. Remember you can never test everything - our job is to test the most we can to avoid risk.

One of the common inefficiencies in device selection is redundant testing. I mentioned earlier that all iPads have the same viewport width and as long as they have the same OS, the risk that they will render differently is very low. Therefore testing multiple iPad retina devices is not necessary. Testers, teams, and clients often focus too much on specific devices, especially when it comes to Android. The specific device is not what is important; it’s all about the viewport widths and the browser version. For example, despite the physical size difference in the Note line of phones and the Galaxy S line of phones, most of them have viewport widths of 360. If they have the same OS, testing both of them would, for all intents and purposes, be running the same test twice. It would be like if someone asked you to test IE9 in Windows XP on a Dell computer and an Acer computer. We all know that would be a waste of time as it would render exactly the same.

In order to cover the most ground when making your testing selections, you should try to mix the OS and viewport widths. Instead of testing a Note 3 (viewport width 360px) and a Samsung Galaxy S4 (viewport width 360px) both with Android 4.4, a better choice would be the Galaxy S4 (viewport width 360px) with Android 4.4 and a Nexus 4 (viewport width 384px) with Android 4.2. This way with just two devices you are covering tests for two versions of Android and two different viewport sizes. Basically you went from covering one test scenario on two devices to covering four. Depending on your testing time/bandwidth you may also consider being more extreme in your viewport differences and making sure that you only choose devices with viewports that will actually hit a different breakpoint. In the above scenario if you replace the Nexus 4 with a Nexus 7 (viewport width of 600px) you are now also adding a different layout view to the mix.

Another thing to consider in regard to Android phones is that some devices ship with just the native Android browser as its default browser (eg: the Samsung Galaxy line), while most other Android devices (eg: the Nexus line) ship with mobile chrome as the default browser. Each browser does have the potential to render pages slightly differently so make sure to pay attention to that when mixing up your choices.

Now that you have an understanding of the different variables to pay attention to when making your device selections, the next step is to always have your ear to the screen for new device releases, new OS/browser releases, and changes in market share. I find the best places to keep up are the many tech blogs that are freely available to anyone. I personally frequent Engadget, Gizmodo, and The Verge, to name a few. As far as market share goes, both Apple and Google share up-to-date statistics about the market share of their OS versions. I also suggest checking out some of the third-party usage statistic sites like Net Market Share and StatCounter as they will often paint a less self-serving picture.

Happy testing!