Choosing Devices

Naturally, the first step when building a device lab is gathering devices for testing. This first step will help you sort out the parameters for the rest of your lab – from the overall size of the lab, to what kind of cables and power supplies to purchase, to how big of a footprint you’ll need. There are hundreds of devices out there to choose from; how do you pick from handsets and tablets, from smartwatches to video game systems, from the ever-expanding Internet of Things?

Developers instinctively test on the devices that they have in their pocket, or on their desk. But these are typically the shiniest, newest, fastest devices out there. They’re not often representative of what your site or app’s audience routinely uses to access the Internet. Your personal device probably won’t give you a good picture of what bugs might crop up, how long a page takes to load, or how usable your site’s UI is on a typical screen size. It’s easy to forget that your day-to-day experience is unlikely the same as your site or app’s target audience. A device lab comes in handy, as it’ll open your eyes to the range of user interfaces, screen sizes, heft, and sluggishness that your users experience every day.

That being said, you can’t test for every use case. Having a helpful device lab doesn’t mean you have every device on the market; it means focusing on what your users are actually using to access your site and prioritizing by the amount of traffic and potential for bugs. Just having two or three additional device types, covering a range of operating systems and screen sizes, means that you can get a better sense of how your work performs for others. To figure out what those primary devices should be in your lab, you’ll need your site’s visitor data. We’ll start by analyzing what devices your visitors are using and compare that list to what’s important for you to test on your particular site. Then we’ll walk through technical considerations when purchasing devices to make sure you are getting the most bang for your buck.

Analyze your users

Let’s dig up the data about what visitors are actually using to access your site. In this example, I’ll focus on using Google Analytics, as it’s a fairly common analytics tool, but these tips will apply to any analytics solution’s visitor data.

Once you’re in Google Analytics, open the Audience section, then the Mobile section, and choose Devices. You’ll see a list of users’ devices and various pieces of data about their traffic to your site that will help you gure out what good device coverage looks like.

You can also open up an important secondary dimension: operating system version. You’ll want to see what versions people are using the most when they visit your site, to make sure you have coverage of popular operating system versions in your lab. This is also something you’ll want to check on regularly to see when to upgrade your devices’ operating systems.

Table 1–1 shows an example list of devices used to visit my personal website. I’ve included the top device names, the operating system versions used on those devices, some visit data, and noted my own classification of device size, from handsets through large tablets.

The data in Table 1–1 will help me figure out what operating systems and screen resolutions are important to have coverage of. Now that I have this data, how should I go about deciding what to buy?

Get the biggest bang for your buck

First, focus on roughly the 20% of devices that produce 80% of your traffic. While a large device lab can be beautiful and shiny, we really only need a handful of devices for good coverage of our user base, and it’s important to optimize the total cost of the lab. The devices listed in Table 1–1 make up 74% of total sessions to my site, so this is a great starting list for me. There may be cases where you want to have some unique devices for testing strange bugs – we’ll get to those cases in the next section.

Next, make sure that you have coverage of your visitors’ screen sizes. In Table 1–1, we can see that my site’s visitors mostly use handsets and iPads. It’s hard to tell if the iPads listed are the iPad Mini or the full-size iPad, as these devices send the same user agent. If I invest mostly in handsets with maybe one iPad, I’ll have fairly good coverage of screen sizes.

If you notice that a lot of people are using iPads or other devices with retina screens, make sure that you’re testing on devices with high DPI. All of the newer versions of the devices in Table 1–1 are high DPI or classified as retina devices, so I should purchase devices that allow me to test things like how icons scale up on these screens or which responsive image sizes work best.

Buggy and older devices

In addition to the major device types and versions in your list, there are a few additional considerations for what to initially buy for your device lab. As you’re using the lab for testing your work to troubleshoot bugs and ensure a good user experience, it may be important to purchase some older devices or devices that could generate surprising bugs for your site.

It looks like my users are on recent versions of iOS and Android, so for now, I’ll avoid buying devices running older operating system versions. If on my site I declared supporting older versions of operating systems, then I’d make sure that I had them in the lab so I could test for bugs. At Etsy, for example, we have a handset running Android 2.x, as older devices can be really helpful to reproduce user-reported bugs. That said, if you’re just starting out building a device lab, stick with operating system versions that are representative of the majority of your user base. As you grow it over time, you can become more inclusive of your overall user base.

At Etsy, we found that Kindles can be particularly buggy for our user experience, so we include them in the device lab for testing. If you have a Kindle in your top ten list of devices that visit your site, definitely consider getting one for testing. Similarly, if you have a lot of users on the Windows Surface or Chromebook Pixel, be sure to test on a device with a combination of touch and cursor. We’ve found some significant JavaScript issues that affected people’s ability to log in to our site on these devices.

Target demographic

We’ve been looking very high-level at visitor data, but this may not provide the most accurate picture for what devices you need in your lab. As the last step before purchasing, find out if device usage differs if you segment your analytics data by audience region or target market. It’s possible that when you focus on visitors in your target demographic, you’ll find variances that will impact which devices you should buy. Maybe you know that your primary user is located in the United States. Or maybe you collect age data on your users and it’s important for you to know which devices teenagers are using to visit your site. Verify that the devices you buy cover what your target audience is using. It’s important to make your site accessible to all, but when building a starting list for a device lab, focus on the best coverage of your target demographic.

Iterating over time

The Internet of Things constantly evolves. When we first started Etsy’s lab, we included older Blackberries and a Google Glass. Every quarter or so, it’s important to revisit your visitors’ data to see how things are changing. Is there a new device that’s gaining popularity? A new range of screen sizes to cover? Are you keeping up with the rate at which people are upgrading their operating systems? Be sure to routinely check in and make sure that your device lab stays on par with what your users are truly using to access your site. This way, you can budget to purchase used versions of these devices in the next quarter. Additionally, you’ll want to consider updating a few devices’ operating systems at a time, while continuing to keep some on older versions if necessary to test bugs.

So what would I buy based on the data in Table 1–1? If I purchase one iPhone running iOS8, that’s 52% of my traffic covered. I could also purchase an iPad to cover tablets, and a Nexus 5 to cover Android handsets, bringing me to solid coverage of my top 10 with just three purchased devices! If I had budget for a fourth devices, my next purchase would be an iPad Mini running iOS7, to cover the iOS7 test cases in the list.

Purchasing devices

Device lab costs can quickly escalate without proper planning. Devices alone cost hundreds of dollars, and it’s important to have budget left over for your power chain, replacement parts, and improving the usability of the lab. Let’s focus on optimizing the purchasing process to both lower the costs and increase the efficacy of the lab.

Used devices can give the best representation of what your visitors experience when they access your site. While in our pockets we may have the shiniest, newest devices, these devices don’t reflect what our entire user base has in theirs, and therefore don’t accurately represent their real user experience. Used devices will give you a better depiction of battery life, common bugs, and screen brightness, and they’re generally cheaper than new ones.

When you purchase a device for the lab, you probably won’t need to purchase a data plan. It’s unlikely that you’ll need any carriers or network contracts unless you are aiming to test voice capability or the device’s speed or battery consumption over a cellular network. We recommend skipping the data plan and sticking to connecting your devices to a Wi-Fi network. We’ll cover Wi-Fi setup and troubleshooting in Chapter 4 – Networking.

You can purchase used devices from sites like eBay, Amazon, and local cell phone dealers. Apple sells refurbished devices as well. If you need to estimate pricing for your budget, searching eBay is a good way to get a feel of what people are asking for a particular device. If you’re buying something you won’t be able to test drive in person, make sure the seller has a good return policy. When possible, get something still under warranty, and carefully read the description before purchasing. Does it come with all the necessary accessories and cables? Is the device unlocked? You can usually get a better price on devices with a less-than-mint appearance, but be sure the damage is only external and doesn’t affect the functionality of the device.

In the next chapter, we’ll focus on what lies between your new devices and the wall – your power chain! Few things are less fun than arriving at your lab, ready to get testing, only to find that your devices are drained or dead. After you receive your newly purchased lab devices, head to the next chapter to figure out your power setup.