Pros and cons of the top 4 user testing methods
As UX designers we focus profoundly on designing for real users and the way they interact with our product, service or app.
It’s so easy to make assumptions on what we ‘think’ our users will do, but this is often proved wrong during user testing.
So, why do we perform user testing?
User testing helps us to understand who that person is, in what context they’ll use a product, and what goal they are looking to achieve.
Over the years UX researchers have developed techniques for testing a validating their ideas. The most popular forms of testing are usability testing, focus groups, A/B testing, and surveys.
Usability testing is the best way to learn how real users experience your website or app. During this process, you can collect a range of information about the users and watch/track them while they interact with your product to see if it’s usable. Usability testing is a foundation of UX practice —there are two popular types of tests which both have pros and cons:
a. Moderated Usability Testing
During this process, a moderator will facilitate the user through tasks while they interact with the website or app. This method encourages the user to give live feedback that enables you to get a broader understanding behind their thought process.
Helpful article: 20 great tips for moderated usability testing
When to use
The best time to conduct a moderated test is during the design phase— when a team has a working prototype is in its very early stages. During the trial, you can watch the users behaviours and iterate the designs to support getting the best possible outcome.
Tip: Getting early feedback will save you from spending too much money on design and development of a product that isn’t usable.
- The moderator can help get a deeper understanding of why the user was performing a specific task.
- The moderator can help clarify any confusion which will help you understand their cognitive process.
- Usability testing measures behaviour, not preference.
- Save money on development and redesign efforts.
- Moderators can tell the participants what to do which gives bias results.
- Interferes with the natural progression that participants would make on their own
- The overall session length increases
- Can be more expensive than other usability testing techniques
It is crucial as a UX designer to ensure the moderator has the best balance of keeping the participant on a task without interfering with the user’s natural experience.
b. Unmoderated remote usability testing (URUT)
URUT testing is conducted remotely without a moderator present— it automatically collects user feedback and records their behaviour.
“It provides fast, reliable, and inexpensive user testing results”
When to use
Unmoderated remote usability testing gets conducted when you need to collect a large sample of data to prove key findings from your original moderated research—its perfect to see how users interact with the interface of a simple task.
- The user is often in their environment which allows them to interact with the product more naturally.
- Tasks can be completed in the users own time without the moderator worrying about the schedule.
- The turn-around time for unmoderated tests is often significantly faster than moderated tests.
- The cost is normally considerably low because you don’t need to pay for moderators or an equipment setup.
“You can get the maximum value for minimum cost when tasks are pre-determined”
- The absence of a moderator implies less control, less personal observation, and an increased risk of confusion.
- It provides limited or no qualitative feedback because users won’t often think out loud without a moderator
- It’s difficult to make sure users take your test seriously.
- It doesn’t operate well with complex tasks and prototypes.
- Several users will have technical issues and give up.
To conduct the tests successfully you need to set the user with 3-5 clear tasks that will take no longer than 15-30 minutes.
Focus groups are more of an informal technique that can help you examine user needs and behaviours before interface design and after implementation. In a focus group, you bring together 6-9 users for about 2 hours to discuss issues and concerns about the features of a user interface.
Helpful article: The use and misuse of focus groups
The purpose of focus groups isn’t to evaluate design usability, but to discover what users want from a product—their individual opinions and preferences.
When to use
Focus groups can be conducted before a product’s design and long after its release. They are useless whenever you are looking to explore the user’s needs, thoughts and feelings of a product or a service.
Typically, you should run more than one focus group, because the result of one session may not be representative.
- Great for learning from the banter between people
- Discover what users want from the system
- A group answer can be developed
- Help you assess user needs and feelings
- Focus groups can be very cost effective
- Poor method for evaluating interface usability
- The results can be altered with strong leadership
- Group dynamics sometimes create a personal bias
- Moderators can inadvertently influence the data
I think we are all familiar with surveys and questionnaires. Surveys are a great way to gather large amounts of data with minimal efforts in time and costs. There are several tools online like Survey Monkey and Google Forms that can help you set-up a questionnaire.
“The right questions can help tap into customer needs, wants, and pains”
When to use
Surveys can help you gather valuable information about your users, provide quantitative data about overall user satisfaction and collecting feedback about a new feature.
- Finding out why people visited your site
- Recruiting test participants for moderated user tests with experienced users
- Gathering quantitative data about content quality
- You can ask real visitors during their actual visit
- You can’t study user behaviour with surveys
- Takes a long time to prepare the right questions for the right target audience
- What the usability problems are with your product
Tip: Stakeholders often want to see big numbers. After doing your qualitative research, you can validate your findings with a survey.
A/B testing enables websites to compare variations of the same web page to discover which will produce the best outcomes. This type of test is often carried out when designers can’t decide between two competing elements.
When to use
A/B testing is beneficial when trying to identify small variations in designs or simplifying a whole website redesign to see which elements of the web page have a detrimental effect on conversion.
- The tests give clear, quantitative results
- A good way to measure user response to small tweaks
- Quick to create a modified version of an existing web page
- It tests reality, not the theory —getting real results from real users doing real things.
- A/B testing is that it can often take a long time to see results
- You only get the best option from the available variations
- Tests are often built on internal opinion
- Missing the “why” —no understanding of the reasons for the results
- Need an ample amount of traffic to your web page to run an A/B test
Tip: UX research can help develop more practical hypotheses and distinguish more possibilities for experimentation.
User testing is a vital part of the design process—done in the right way, at the right time with the right user base reduces the risk of building the wrong product.
Different types of user testing suit different types of goals but they all work towards saving time, money and valuable resource. The type of test conducted on your product will depend on what you are looking to learn and how much time you have available.