Moderated and Unmoderated User Testing: The Differences

Usability testing is an essential aspect of the UX design process. This is justifiably so as it has been proven that 85% of the problems in an interface can be found by just 5 users. However, before usability testing comes user testing, which is also an important step in the design process. 

The type of user testing chosen for a design project will determine how effective and useful the test will be. Using the wrong kind of testing would not only be a waste of resources, it can be counterproductive. This is why embracing personalization and choosing the right user testing for your design project is extremely important. If you get it wrong at this stage, chances are the rest of the design process would be a futile effort. 

To make sure you don’t make this dire design mistake, this article goes in-depth into what user testing is about, the two types, when you should use them, their methods, as well as the pros and cons of each. At the end of this article, you will be certain of the type of testing to use for your next design project.

What is user testing?

User testing is the test carried out to find out whether your target audience needs a solution or not.  In other words, this test determines whether a need exists for your product or not. This is usually done at the beginning of the design process. 

The reason for this testing is not far-fetched. No one wants to design a product people will not use. To be sure the problem you want to solve exists, and in what variation, there is the need to test users. For example, knowing the factors that make a compelling infographic is not enough. You must test the product to see if it is usable. Note that this testing can also be used to test an idea or an existing product or feature. 

What is moderated user testing?

A moderated test is a live or real-time interview with a user who tests your product (or its idea, prototype, or design). Traditionally, this user experience research technique is conducted in a UX lab or a corporate context. However, UX methodology has evolved and modern testing methods now enable remote testing, where the participant and moderator can hold the test in different physical locations.

If this testing is done remotely, your engagement with the participants takes place in real time, in the same ‘virtual’ environment. A remote conferencing tool, such as phone, VoIP, chat platform, web conferencing (or a combination of these tools), is used by participants to connect with you.

What is unmoderated testing?

According to experts on UX design, an unmoderated test is one in which participants carry out specific tasks independently, unaided by a moderator. Participants choose when and where to take the test, and they give feedback. The session can be recorded or not. 

The participant thinks aloud as they do the activities, and the researcher reviews the results or recorded sessions (when available) afterward. In contrast to a moderated exam, the researcher does not interact with the participant in real time. Instead, they may add follow-up questions to the test or contact the participant after seeing the session or results. Since changes cannot be made in real time with a participant, unmoderated tests could demand extra work upfront.

When should you use moderated user testing?

How do you know when to use moderated user testing? Here are reasons to use moderated testing.

• When you want to make sure participants are not hindered by a challenging technical task, such as interacting with a complicated or low-fidelity prototype when assessing ideas or concepts
• When you want to observe participants’ behaviors in detail by studying their body language
• When you want to ask individualized follow-up questions in real time

When should you use unmoderated user testing?

The following reasons are why you should consider using unmoderated testing.

• When you want to quickly get feedback
• When you want to finish other tasks while participants are engaged in their activities
• When you want to be sure you’re getting unbiased feedback from participants as the involvement of a moderator can influence participants’ behaviors or responses

Methods of moderated testing

Moderated testing methods include the following:

• Lab study
• Interviews
• Focus groups
• Remote testing
• Card sorting
• Click testing
• Ethnographic study

Methods of unmoderated testing

Unmoderated testing methods include but are not limited to the following:

• Surveys
• A/B testing
• True intent studies
• Diary study
• Tree testing
• Card sorting
• Click testing

Pros of moderated testing

1. Better participant engagement: a higher level of participant engagement results from moderated testing. The moderator can strike up a casual chat with participants, which promotes trust and encourages the completion of the tasks.
2. Possibility of follow-up questions and probing: with moderated testing, UX researchers can ask participants follow-up questions to learn more about any usability problems they may have run into. The moderator might probe further and find out the specific causes of their struggles. For instance, if a participant is having trouble, the moderator might ask questions about their expectations.
3. Real-time feedback: moderated testing allows the moderator to identify subtle behaviors and responses of the participant. For example, if the participant looks confused, the moderator can guess what the problem is. UX researchers can get feedback from participants in real time, which aids in the preparation of a more thorough report.
4. Promotes more empathy: being present during the test can allow moderators to become more empathetic.

Cons of moderated testing

1. Consumes time: it can be time-consuming as moderators can only moderate a limited number of participants at a time. The tester must typically be personally recruited, and as the test must be observed, preparation time is significantly increased. It takes time and effort to prepare all of these details and observe various people while asking each one a specific question. 
2. It can be quite costly: bringing many people together for moderated testing can cost an arm and a leg. In addition, platforms for moderated testing are quite expensive unlike for unmoderated testing.
3. It can be used only for smaller sample sizes: It would be too costly and time-consuming to conduct a moderated test on a large population and this might lead to a lack of statistical significance of the results. 
4. Introduction of bias: the presence of a moderator can introduce bias and affect the credibility of the results. Additionally, participants may behave differently than they would when they are alone because other people are watching them. 

Pros of unmoderated testing

1. Convenience: unmoderated testing is very convenient. Participants can complete the sessions in their own space and time. 
2. Lower possibility of bias: by letting users handle everything on their own, bias is less likely to occur. Participants won’t be led in any particular direction, although they still need to be offered some instructions and asked the proper questions. 
3. More results in a short time: it allows for more results in a limited time. This is a result of being accessible to many people at the same time.
4. Statistically significant data: the amount of data it gives can be statistically significant as it can be administered to a large population.
5. Cost-effective: it is cost-effective as it does not require a dedicated space and equipment to carry out.
6. Improved efficiency: it can improve efficiency as UX researchers can engage in other tasks while the test is ongoing.

Cons of unmoderated testing

1. Difficulties during test: as a result of a researcher not being present during unmoderated testing, participants might become stuck, have technical issues, or have difficulty understanding the questions. This could negatively affect the results.
2.  Lack of real-time observation: the researcher won’t get to see the participants’ nuanced replies to some questions. They would miss subtle signs like hesitation before answering a particular question. 
3. Non-intensive data collection: with unmoderated testing, data gathering is not intensive. The researcher would not be able to go more in-depth with participants or ask any follow-up questions. Only questions that the researcher has already prepared are responded to. 
4. Distortion of test flow:  the absence of real-time interactions during unmoderated testing increases the likelihood of participants skipping the initial activities and going straight to the subsequent ones. Due to the lack of instruction, participants may stray from the testing flow that the researcher had anticipated.

User testing: the key to improved user experience

The UX design process is incomplete without user testing. To carry out this testing, however, it is necessary to be familiar with the two types – moderated and unmoderated. This knowledge will serve as a guide in choosing the right one for your design project. 

Making a choice between the two types of user testing has become easy as you can now tell them apart. You now know what they are, when to use them, their methods as well as their benefits and disadvantages. Ensure you keep all this in mind as you take up your next design project.