Skip to main content

5 Things You can Do to Make Usability Test Part of Your Culture

Most of the times, the usability test requests come from the marketing team, shortly before a product or a major release is about to launch. 

It goes like this:

About the time when the product is about the launch, the marketing team comes to the executives, everybody is having these little butterflies in their stomach. We have all too business on everything we have to do, planning our marketing events, developing all the features, that we forget our users.

We did the user tests, well, focus groups actually, and we found a long list of problems users encounters when using our products. And we prioritize them based on the time needed from the development team to fix the issue: if it is the change of a label, priority high; if it is changing of an interaction, priority low etc.

Everybody is frustrated with the process: product team knew the test is too late for the current release, it should have been done much earlier; marketing themselves are upset as they are worried the product will not get user acceptance for all the problems we spotted from the test; development team are stressed as now they have more things to change and fix that they don’t know why (they are not invited to the team of course) on top of the zillion other things they have to do before the launch; 

And the CEO is shucked, s/he can’t believe the number of problems spotted from the test, some of them looks serious. 

Throughout my product manager life, I’ve seen the same story again and again. It is an inefficient and frustrating experience for most of the people involved in the process.

But why don’t we do something different, what stopped us from doing what we should do:

Executives Don’t Support Early Usability Test: cost is one of the main issues, hiring a professional testing company can be expensive, the last time we used TangUX in Shanghai for 4 test sessions on 12 people, it costed us more than half a million RMB (~80K USD). Privacy is another issue I faced, especially for new product, company can be very sensitive in any kind of test involving external parties even through the confidentiality agreement is signed.

Team Don’t Have the Skill to Execute: besides PM, the product team is actually not that busy, the requirements are given and assets produced, but they has very little knowledge on testing, we have brilliant business analysts and graphic designers, but they have no idea how to conduct user tests nor to analyze the findings; 

Those Who Knows Don’t Have Time to Teach Those Who Have Time: as a product managers, I have done all kinds of user test from Focus Group to Usability Benchmarking, but I am so busy trying to clean my to-do list and I have no time to write screener questions, prepare test scripts and do beautiful presentation for the executives. 

After being through the same process 3 times, I came to realize the daunting pattern:

Why is cost high, because we need to hire professionals, they are expensive;
Why can’t we do in-house, because we don’t have the skills;
Why don’t we have the skills, because we didn’t train the people;
Why didn’t you train the people, because I don’t have time;
Why didn’t you do it when you have time, because management don’t approve such a project at that time;
Why executives don’t approve such a project at that time, because they don’t see the need;
Why don’t they see to need then, because we are going to market now;
… …

Don’t we always want to go to market?

… …

I’ve came to realize we will just fall into the routine again and again unless I make an effort to broke the chain. And I’ve been doing 5 simple things which gives me tremendous benefits on making user tests part of the product management cycle:

Create Artifacts at the Best Time: the first 3 things I when I get my hands on a product is to prepare the screener questions, the focus group questions and the test scenarios. In the product envisioning stage, I will do screener questions and focus group questions, these are more generic questions to define the direction of a product; as I start developing each feature, I will write focus group questions and test scenarios, here the scope are limited to the respected feature; timing is very important because, in fact, I’ve done similar thinking and wrote them down in the requirement and PRD, so it just take me extra 10-15 minutes to create a Confluence page or Google Doc and format them for the use of user tests;

Beat the Competitors and We Buy You Lunch: everybody loves to look at competitors, it helps us know our place in the guard scheme of things, know what users recognize us as etc. So I decided the marry this need with my need for competitor research and organize bi-weekly brunch on competitor tests. I need to write script of course, but I need to do it as part of the competitor analysis anyway, and I got to use the office budget for team activities to provide lunch to everyone;

Piggyback on the Giants: look, executives don’t give a damn about products, they only care about revenue, ROI; but what they do care, are what other executives are doing; I subscribe to the search term “usability test” from Google Trends and I forward the interesting articles I found related to executives endorsing usability test to my executives. Be careful not to use the one that end up marketing their product though, you maybe misunderstood as trying to promote their products. Alternatively, you can always find tons of articles from the big consulting firms (McKinsey, Bain etc) that continue to come out with the “food for thought’ article that just so helpful for you to make a point. Don’t forget to write a comment using the key words they are (e.g. meet consumer needs; improve ROI; reduce cost etc). For example: “Great McKinsey article on how banks and large retailers using user tests to meet consumer needs

Outsource, Delegate and Automate: to do the above 3 I have a task list that needs to be done at different times:
  • Write screener questionnaire;
  • Write tester recruitment advertisement;
  • Write focus group questions;
  • Write test script;
  • Find a list of competitor products;
  • Collect and analyze results;
  • … ...

First I see if I can outsource or delegate to somebody else, maybe not now, but in the future; and what is needed to enable it. After that the list becomes sometime like:
  • Write screener questionnaire;
  • Write tester recruitment advertisement template;
  • Write focus group questions template;
  • Write test script template;
  • Review scripts;
  • Collect list of competitor products and scripts;
  • Read Google trend on usability ROI; 
  • … ...

Then I put the list in my calendar and reserve the time for them so I get to them and complete it.

If you can’t, well then read the next point.

The More People that Do it With You, The Easier it is: well, the last point is not really a serpent point, but it is one of the most important one, I consider myself a very disciplined person but even that, I get swapped by work and I just can’t continue, but that’s why we have a team right? My team member feeds me with competitor links, they share me articles that I can pass on to the executives; executives share good product that we test together; I can be a happy librarian centralizing the knowledge and pass them on.

As Kilian, Serrazin and Yeon stated at the end of Building a Design-driven Culture”: Customers increasingly expect products and services that are designed to meet their needs, delight them with unexpectedly great experiences, and address a heightened sense of aesthetics. Companies that meet those needs are rewarded with fierce brand loyalty and higher spending, which translates into fatter profit margins. But that kind of success only happens by design. 

And above all, test my words out, see it works for you. They have worked out for me, 

I would love you can drop a line or two in the comment section to give me some feedbacks. 



Building a Design-driven Culture, McKinsey

Popular posts from this blog

4 Techniques to Make Your UX Review Meetings Successful

As a product manager, I often need to sit down with the executive management team to get their feedback on the new designs. It can be a frustrating process and many times I found that I cannot get things down in the time I am allowed to have.
Nevertheless, not having the sign-off from management is terrible for the team, we face high risk of having to re-work (yes, we always need to re-work, but it feels better if it is an improvement), schedule get delayed etc.
Over my 200+ review meetings, i've came to understand the reasons and learnt skills on how to stir the meeting towards an efficient completion, and I want to share them with you.
There are 4 key reasons of an unsuccessful product review meeting: 
1. Audience Lack the Background Knowledge: especially when introducing a new function, executives don’t know what they are looking at, or how the end users will be using such function, you may have sent the presentation before and again in the meeting invite, it doesn’t matter;

2. T…

How to Do Market Research and Competitor Benchmarking

Congratulation, you have validated your business idea from the people around you and it looks good in their eyes. You want to go ahead and start working on the idea.
No matter if you are a veteran in the industry and have good connections to help you start; or your idea came from your personal experience while you have little experience on the market.The first thing you will end up doing is market research.
Remember, you are doing the research to: Make sure the idea in a correct way;Identify what your end users are comparing toUnderstand what challenges you will face;See how you can make money;
You are not doing the research to: Copy what the other companies are doing Exhaust all potential competitors in the market
What I always to, is a 5+1 market analysis. In the beginning, I would start with a 5-day analysis with the following schedule:
Day 1: Find a list of companies that are targeting the same end users you want to approach 
For most of us, finding the list of companies that are target…

Indie Game: What Developers Should Do and PLayers Should Play

What does indie game stands for? What is the market and market growth for indie games?
Indie games are video games that are created without the support of a publisher (E.g. EA games; Microsoft etc). Indie game developers am to bring innovation to the market, they focus on digital channels (APP stores; online marketplace) to distribute the games.
Perhaps James Swirsky And Lisanne Pajot's movie "Indie Game" best visualize the community working on indie games. This is a group of entrepreneurs seeking to realize their ideal, keep refining the work they have done and hope to find others who believe the same.
I researched on the well recognized forums, bloggers, and platform operators (E.f. Steam) to construct a comprehensive list of 100+ popular indie games to try to see what criteria are important for a good indie game. 

My logic is: indie game mostly face serious gamers (people who spend a significant hours of their week playing games), there are certain special requirements fr…