The conflict between qualitative and quantitative user experience research methodologies has its origin in a time when computers were only a dream, and with the emergence of the social sciences over a century ago. It lies in a struggle that is much older than the field of Usability, User Testing, and Design Research.
And it is still unresolved. The question arises in almost every company, every UX study, every project. Which path to take, which approach is more preferable for the stakeholders, management, and usability researchers? Which one will enable low hanging fruit and which will bring strategic advances?
“I don’t believe in a quantitative approach”, “We need to quantify this”, “Can we put this in numbers?”, “I only do qualitative interviews”, – are the phrases one hears quite often. But, as we know from everyday experiences, the “Divide and conquer” approach is not helpful in solving problems. And it is not a good way to do research, either. What is the best way to avoid going from one extreme to another?
First of all, lets define the terms and avoid semantics. What do we mean when talking about quantitative or qualitative user experience and usability research methods? Is it all about just adding numbers to reports, proper sampling or opportunities to observe user’ needs? When someone announces their love for a quantitative approach, do they mean proper statistics sampling, accurately measuring time on task and mapping click paths? Do countless interviews with users imply a preference for a qualitative user testing methodology?
The difference between the approaches chosen should reside in the answers each type of usability study, quantitative or qualitative, can help you obtain. Quantitative UX research methods can answer the questions of what, how many, & how much. They will help you compare interfaces, features, navigation paths, or study conditions, as well as differences between user testing Personas.
Qualitative methodologies (observations, interviews, contextual inquiries), on the other hand, answer the question of Why? and can drive insights for further user research. They can help UX researchers analyze the reasons for the paths users choose, the shortcuts they use, and the hacks they leverage while listening to their train of thought.
While quantitative methodologies are meant to provide researchers with objective, measurable results, when applied to real contexts, they are not free from subjective interference. Hundreds of books and articles have been written on statistic sampling and formulating questions for a survey (some companies even conduct meta-studies to find out which questions get a better response rate).
But at a certain point of a user experience testing study we are forced to ask only one question and provide only one set of answer choices from this particular sample. And there is no chance to go back and ask the question again, or to change it. Thus it is very important to conduct pilot testing of every part of a UX study, especially when conducting unmoderated online usability studies.
Conversely, it is possible to enrich qualitative UX research methodologies with quantitative usability metrics, using word counts, clicks, task times, errors, SUS scores, etc.
Quantitative vs Qualitative UX Research
Almost all user researchers are employing both approaches, quantitative and qualitative user research, no matter what their “official” preference or title. Which is a good thing, especially when there are other limitations to the user experience research process.
Online unmoderated usability studies offer a lot of opportunities in terms of saving resources and time: a user experience testing researcher needs to simply define the UX test script, with the instructions, questions, and branching logic, launch the project to as many participants as needed, in as far away locations as necessary, and receive the results within hours.
It is therefore highly scalable, allowing a significant number of tests managed by a single usability researcher. It does not allow for the personalization of a moderated usability study (whether conducted in person or remotely), however, using branching logic it is possible to make an unmoderated UX test study highly personalized to the actions and responses of each individual participant.
Ultimately, the decision on which user testing method to employ (quantitative vs qualitative, moderated vs unmoderated) and how to adjust them for each research project ultimately hinges on the question usability researchers need to always address:
Quantitative vs Qualitative UX Research
The goal of the user experience testing project:
How much it affects our business metrics?
Or
How many users are affected by it?
Or
Why it happens?
