Apple has recently entered the race with its Homepod. Which of the two current leaders, Amazon’s Alexa (Echo) or Google Home, provides the best user experience?
Interactive Voice Response (IVR) systems used by banks and utility companies provide some of the clearest examples of poor usability and user experience. With the appearance of Voice User Interface systems such as Alexa, Google Home, the Homepod etc, it will be increasingly important to apply the online user testing techniques currently used for websites and mobile apps and prototypes to these emerging voice UIs.
Userlytics has recently launched a new capability focused on user testing voice UIs, both IVR systems as well as the emerging AI enabled voice UI systems. We decided to compare the UX and usability of the two current market leaders, Amazon’s Alexa and Google Home.”
Amazon’s Echo has taken the tech world by storm, literally – propelling its parent company to massive profits. Amazon recently boasted it sold “millions” over the 2017 holiday shopping season. But Google isn’t far behind. The Google Home carried the company’s voice assistant out of smart phones and into smart speakers in late 2016.
Both platforms have had more than a year to mature — an eternity in the fast-paced world of technology, but have their user experiences kept up? Let’s find out.
What’s the weather?
One of the most popular uses for voice assistants is getting on-demand weather forecasts without having to tune in to the Weather Channel. Both devices are able to deliver a basic forecast, but a few of the Echo users said they wished Alexa told them more than just the current temperature and conditions, by offering a more detailed forecast about the afternoon and evening, and letting them know if there’s rain on the way
Users who tested with Home said the forecast was very helpful, and one user said he only wished he could say ‘more information’ to get a more detailed forecast.
Winner: Google Home
Who’s Hungry?
Next, our testers asked both assistants for some nearby Chinese Food restaurants, and the telephone number of one of those eateries.
In our tests, both assistants dutifully read of a list of nearby restaurants close to the user’s zip code, but only Alexa went a step further and sent contact information to the users’ phones. That’s a great user experience.
The only complaint from users: neither assistant bothered to tell them the restaurant’s hours, or how far it was from their home. Both assistants answered the question at hand, but users seemed to appreciate Alexa going the extra mile with the additional information in the app.
Winner: Amazon Echo
Geography Quiz
We asked users to put on their thinking caps, and try to use the assistants to determine the largest city in California. You think this would have been an easy win, but in fact, some users said the answers left them wanting more. Listen:
It was nearly unanimous among our testers: most wanted the Echo and Home to tell them the population number, as well. Alexa eluded to the answer, by saying Los Angeles was the “most populous” city, but remember – users asked for the “largest city” and they didn’t tell the assistant what “largest” meant to them.
No assistant provided those demographics, and a couple users seemed to have trouble determining which California city was largest by area. (It’s also Los Angeles)
Winner: Draw
Amazon Alexa vs Google Home
Chili Contest
Next, we had users try on a chef’s hat and have their digital assistants offer up a recipe for Turkey Chili, including a list of ingredients. Both assistants felt they could help, suggesting a single recipe and offering to give the user more information. Google’s Home suggested an option from the New York Times, and asked the user if that sounded good.
Alexa went a step further. When users asked for a recipe, she offered what she said was the most popular recipe she could find, and told users how long it would take to cook.
Alexa also offered to send our users a list of ingredients to their cell phone (“very helpful,” said some.) But one user said her experience with the Google Home was better. Listen:
Both assistants also offer additional “apps” or “skills” to get accurate recipes and additional features, from third party applications. Overall, despite the one outlier in the audio recording above, it seems our testers preferred Alexa for this task.
Winner: Amazon Echo
24K Magic in the Air
For the final test, we pitted our assistants against each other in their knowledge of pop culture. Our users asked both assistants who won Album of the Year at the Grammy’s.
A couple users noted Amazon Echo doesn’t share Google Home’s awareness of your previous question, so this multi-part request required some repetition. That seemed to hurt Alexa’s helpfulness rating on this task.
It’s understandable how Google could come out on top, given the assistants strong connection to Google’s all-encompassing search platform.
Winner: Google Home
Amazon Alexa vs Google Home
Gaining Valuable Insight
Which assistant is better can be a very subjective question, but what’s clear is thoughtful user testing can provide invaluable insight into improving your voice UI. Userlytics offers audio-only recording to quickly test Interactive Voice Response systems or voice-centered applications.
Beyond voice assistant and voice interface user testing, the Userlytics “Audio + Metrics” user testing feature can also be used to test corporate and customer service IVR systems, and even allow you to leverage the continual traffic of visitors to your site or app by having them provide you invaluable user testing feedback in return for a simple promotional code.
Using the Userlytics platform, you can automate the collection of valuable feedback from everyday users and visitors.
Amazon Alexa vs Google Home
Here’s an example:
- You can sign up for the All-You-Can-Eat Userlytics “Enterprise Unlimited” plan and then set up an “Audio + Metrics” study.
- Then, add a redirect to the session that takes users who complete the test to a landing page with a reward for completing the test.
- Set the study to accept hundreds, thousands, or hundreds of thousands of participants.
- Next, add a “Self Recruitment” widget intercept to your site, and offer users an incentive to test a particular part of your site (like a coupon code or discount.)
- Then you can visit your Userlytics Dashboard to review the metrics of the test results and zero in on any quantitative data outliers, such as user experience issues or tasks that are taking users too long.
- You can then listen to those specific audio recordings (“think aloud” commentary) to find out what happened.
If you detect a particularly important issue, you can then launch a full qualitative test, using the “webcam, screen, audio + metrics” recording methodology, to dig deeper into the issues
Userlytics provides you with all the tools you need to answer the question of why your customers and users do what they do..“
