I attended a wonderful webinar about user testing. It was presented by Arthur Bodolec, Co-founder and Designer at Feedly and UserTesting.com. (By the way, I’m a happy user with UserTesting.com)

Arthur mainly talked about:

• Understanding real users and collecting real stories to guide product design
• How to do product discovery and define product features based on user stories
• How to find users
• Tips/resources for user testing

The examples below are from Feedly, but the principles and methods apply to other products as well.

Arthur mentioned 2 kinds of testing, one is about getting to know the problem space, desired features and collecting user stories, typically you need to talk with 10-30 people; the other is about usability testing for your mockups or prototypes, 3-5 people each round are sufficient.

Understanding Your Users

Design for the real people, instead of imagery personas.

 

“Who are they?”

Ask users:

• What is your occupation?
• What is your passion?
• Did you use any other product before [product]?

An example dialog goes like this, the key is to really drill down and get the reason why they use your product:

You: “Why are you using Feedly?”

User: “To have all feeds together.”

You: “Why do you need to gather these feeds?”

User: “To get access to information”

You: “Why is that important?”

 

When user answers “I want to get inspired to write my book”, “I want to learn about my competition” or “I want to learn about the latest HR laws”, then you know you understand the use cases better.

 

“What context are they using [product/service]?”

Ask users:

• Where were you?
• Describe the last time you used [product/service].
• What other services did you use?

 

An example dialog goes like this:

You: “You said you are using Twitter, what is the difference from using Feedly?”

User: “Feedly is where I get information, Twitter is where I talk about it.”

 

Get some usability feedback

Ask users:

• What frustrates you?
• What’s one thing that makes you happy when you use [product/service]?
• If you could add one thing to [product/service], what would it be?

These questions above are to shape the user stories, why they use a product/service, what other products/services they are using, what features they want, etc.

 

Define Features based User Stories

To collect real stories to guide your design, you can talk to 10-30 people, asking questions to know who they are, in what context, why they want to use [product/service]. Tips:

• Do it consistently
• Every time build a new feature, first thing is to gather stories
• Stay organized:
  • Keep a spreadsheet track name, email, features people are interested in
  • Keep separate documents for longer stories from users and each feature

Get to know your users:

• Who are they?
• Why are they using your product?
• Why do they need this feature?

When Feedly was building a new feature called Feedly Profile, they collected stories from user: who they are, what occupations they have, how they use the product/service, what purpose they use it for, what people they collaborate with while using it, what one or more features they need to do in order to accomplish what task, what other similar products they use.
After this, you get make user stories as concise as one or two sentences:

From: the user

(To: the user’s audience, sometimes your product user is not the end user)

Goal: to accomplish something.

With user stories like this, Arthur designed the smallest product — he spent 5 to 10 minutes putting together a mockup for one product feature. Knowing it’s not going to satisfy everyone, he reached out to people to get some quick feedback. He also kept in mind that if he needs to add one or more features, how can it can be improved.

 

What to test?

Testing is useful in product discovery and feature defining phase. Arthur gives us some specifics on how to do testing during these phases.

 

Product Discovery

The goal of product testing in discovery phase is to get to know the first experience a user has with the product. Ask yourself these questions:

• Do they understand the service?
• Can they discover what they are supposed to do?
• Can they find the key features?

Tips for testing:

• Don’t give any instructions, let the participant get lost in your product; if they ask you questions say you don’t know
• If they really get stuck, give some hints, but keep in mind that you want to know how they interact with the product in real life, so do not offer help unless it’s absolutely necessary.

 

Features

The goal of testing features is to understand what value are truly useful for your users. Pay attention to these during testing:

• What features do the users need?
• Can they find the features?

Tips for testing:

• Have a list of the tasks you want your participant to do, such as “save an interesting article”, “delete a feed” and “change your profile picture”.

Testers during user testingKeep in mind that you are testing the product, not the participants. Typically people start blaming themselves if they do something wrong, so the following is what you need to tell participants upfront:

• There is nothing wrong you can do
• Don’t worry about hurting our feelings, we want to learn from this testing
• Think aloud and describe what you are looking and trying to do
• The screen and voice is going to be recorded

 

As the tester, keeping the right attitude is important, too:

• Don’t pitch your product
• Don’t lead the user, let people use your product and get lost
• If they fail to do something, ask them why they did it
• Answer questions with questions
• If their answers are too general, ask for specifics
• Whenever you or the user are in doubt, clarify

 

What to look for during testing:

• Aha moments — you may discover some features that are necessary to add in your prototype
• Confusing and misleading things — why did a user keep clicking on a non-clickable area?

 

How to find users

Know who to talk to, and who NOT to talk to. Once you have a plan, broadcast your upcoming testings to your friends, your social networks(Facebook, Twitter etc), your company support forums, Craigslist or use Usertesting.com.

Keep in mind that:

• Most people welcome the opportunity to provide feedback, so no need to be afraid of asking questions;
• Your testing can be in various formats, such as static images, wireframes and interactive prototypes
• Pick your favorite/most convenient way to communicate with users, such as Skype and Email
• Provide fair compensation for your users’ time and commute, for example, give a $50 gift card for a 1-hour on-site usability testing
• Follow up capture leftover opinions. e.g. follow up with users 2 weeks later, asking “how do you like using it now?”

Tips on posting a Craigslist Ad:

• Make the testing info clear: location, duration, compensation, agreement
• A link to a form to screen people’s qualifications

 

Summary

Test small and test often during the product discovery and design phase, so that we can create the right features for the right problems. For usability testing, start small, ask feedback and edit often before build a big prototype. We want to build products/services that customers love, instead of finishing product implementation and using live customers as test users.

 

Useful Resources for User Testing

• http://www.usertesting.com/ – useful for usability testing. They have micro-testing (2-3 minutes) and focus groups features coming soon.
• http://www.pickfu.com/ – quick UI testing poll
• https://www.uservoice.com/ – get to know how much market interest
• Book: Don’t Make Me Think by Steve Krug (Yingying’s side note: highly recommend this book! This is the book that got me into User Experience Design field)