Skip to main content

5 Things I do to Make User Test Part of the Culture

Most of the times, the usability test requests come from the marketing team, shortly before a product or a major release is about to launch. 

It goes like this:

About the time when the product is about the launch, the marketing team comes to the executives, everybody is having these little butterflies in their stomach. We have all too business on everything we have to do, planning our marketing events, developing all the features, that we forget our users.

We did the user tests, well, focus groups actually, and we found a long list of problems users encounters when using our products. And we prioritize them based on the time needed from the development team to fix the issue: if it is the change of a label, priority high; if it is changing of an interaction, priority low etc.

Everybody is frustrated with the process: product team knew the test is too late for the current release, it should have been done much earlier; marketing themselves are upset as they are worried the product will not get user acceptance for all the problems we spotted from the test; development team are stressed as now they have more things to change and fix that they don’t know why (they are not invited to the team of course) on top of the zillion other things they have to do before the launch; 

And the CEO is shucked, s/he can’t believe the number of problems spotted from the test, some of them looks serious. 


Throughout my product manager life, I’ve seen the same story again and again. It is an inefficient and frustrating experience for most of the people involved in the process.

But why don’t we do something different, what stopped us from doing what we should do:

Executives Don’t Support Early Usability Test: cost is one of the main issues, hiring a professional testing company can be expensive, the last time we used TangUX in Shanghai for 4 test sessions on 12 people, it costed us more than half a million RMB (~80K USD). Privacy is another issue I faced, especially for new product, company can be very sensitive in any kind of test involving external parties even through the confidentiality agreement is signed.

Team Don’t Have the Skill to Execute: besides PM, the product team is actually not that busy, the requirements are given and assets produced, but they has very little knowledge on testing, we have brilliant business analysts and graphic designers, but they have no idea how to conduct user tests nor to analyze the findings; 

Those Who Knows Don’t Have Time to Teach Those Who Have Time: as a product managers, I have done all kinds of user test from Focus Group to Usability Benchmarking, but I am so busy trying to clean my to-do list and I have no time to write screener questions, prepare test scripts and do beautiful presentation for the executives. 

After being through the same process 3 times, I came to realize the daunting pattern:

Why is cost high, because we need to hire professionals, they are expensive;
Why can’t we do in-house, because we don’t have the skills;
Why don’t we have the skills, because we didn’t train the people;
Why didn’t you train the people, because I don’t have time;
Why didn’t you do it when you have time, because management don’t approve such a project at that time;
Why executives don’t approve such a project at that time, because they don’t see the need;
Why don’t they see to need then, because we are going to market now;
… …

Don’t we always want to go to market?

… …

I’ve came to realize we will just fall into the routine again and again unless I make an effort to broke the chain. And I’ve been doing 5 simple things which gives me tremendous benefits on making user tests part of the product management cycle:

Create Artifacts at the Best Time: the first 3 things I when I get my hands on a product is to prepare the screener questions, the focus group questions and the test scenarios. In the product envisioning stage, I will do screener questions and focus group questions, these are more generic questions to define the direction of a product; as I start developing each feature, I will write focus group questions and test scenarios, here the scope are limited to the respected feature; timing is very important because, in fact, I’ve done similar thinking and wrote them down in the requirement and PRD, so it just take me extra 10-15 minutes to create a Confluence page or Google Doc and format them for the use of user tests;

Beat the Competitors and We Buy You Lunch: everybody loves to look at competitors, it helps us know our place in the guard scheme of things, know what users recognize us as etc. So I decided the marry this need with my need for competitor research and organize bi-weekly brunch on competitor tests. I need to write script of course, but I need to do it as part of the competitor analysis anyway, and I got to use the office budget for team activities to provide lunch to everyone;

Piggyback on the Giants: look, executives don’t give a damn about products, they only care about revenue, ROI; but what they do care, are what other executives are doing; I subscribe to the search term “usability test” from Google Trends and I forward the interesting articles I found related to executives endorsing usability test to my executives. Be careful not to use the one that end up marketing their product though, you maybe misunderstood as trying to promote their products. Alternatively, you can always find tons of articles from the big consulting firms (McKinsey, Bain etc) that continue to come out with the “food for thought’ article that just so helpful for you to make a point. Don’t forget to write a comment using the key words they are (e.g. meet consumer needs; improve ROI; reduce cost etc). For example: “Great McKinsey article on how banks and large retailers using user tests to meet consumer needs http://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/building-a-design-driven-culture

Outsource, Delegate and Automate: to do the above 3 I have a task list that needs to be done at different times:
  • Write screener questionnaire;
  • Write tester recruitment advertisement;
  • Write focus group questions;
  • Write test script;
  • Find a list of competitor products;
  • Collect and analyze results;
  • … ...

First I see if I can outsource or delegate to somebody else, maybe not now, but in the future; and what is needed to enable it. After that the list becomes sometime like:
  • Write screener questionnaire;
  • Write tester recruitment advertisement template;
  • Write focus group questions template;
  • Write test script template;
  • Review scripts;
  • Collect list of competitor products and scripts;
  • Read Google trend on usability ROI; 
  • … ...

Then I put the list in my calendar and reserve the time for them so I get to them and complete it.

If you can’t, well then read the next point.

The More People that Do it With You, The Easier it is: well, the last point is not really a serpent point, but it is one of the most important one, I consider myself a very disciplined person but even that, I get swapped by work and I just can’t continue, but that’s why we have a team right? My team member feeds me with competitor links, they share me articles that I can pass on to the executives; executives share good product that we test together; I can be a happy librarian centralizing the knowledge and pass them on.

As Kilian, Serrazin and Yeon stated at the end of Building a Design-driven Culture”: Customers increasingly expect products and services that are designed to meet their needs, delight them with unexpectedly great experiences, and address a heightened sense of aesthetics. Companies that meet those needs are rewarded with fierce brand loyalty and higher spending, which translates into fatter profit margins. But that kind of success only happens by design. 

And above all, test my words out, see it works for you. They have worked out for me, 

I would love you can drop a line or two in the comment section to give me some feedbacks. 


Reference:

TangUX

Building a Design-driven Culture, McKinsey

Comments

Popular posts from this blog

Will AI Believe in God?

I came across an interesting article recently on Gizmono ( Link ) titled “when superintelligent AI arrives, will religions try to convert it”,  The key question seems to be simple at the first glance. The commonality among all different interviewees suggest that there is some kind of criteria the Strong AI has to pass to be qualify as human to be relevant to the religion.  The complication comes in that different party seems to have different criteria, to summarize: If Strong AI has a soul? If Strong AI has a soul an spirit? And for the very fact that it is about AI and religion, let’s refer to Bible for the definition of spirit.  What is the definition of Soul:   the   spiritual   part   of   humans   regarded   in   its   moral  aspect ,   or   as   believed   to   survive   death   and   be subject  to   happiness   or   misery   in   a   life   to   come.  “In Noah’s day . . . a few people, that is, eight souls, were carried safely through the water.” (1 Pe

8 Takeaways on Productive Usability Tests

Please, test it.  The fact you believe in your idea, or that your investor do as well, it means a lot about creating a wonderful team and culture. It does not mean your customer will buy into that.  There is a long way between what the market opportunity (as the market analysis and the founder’s vision is) and what the actual product is. Of the thousands of products in different countries that seems to solve the same problem (you may call them competitors if you want), which one will win the customers’ favor now and in the future? Usability test helps to provides insight on that. Or in other words. usability test tells you, at the current state of mind, your customer will most likely find your product to be value if you do … in the following way … and … in the following way ... As you can see, these findings can be translated to what you can include in your MVP, why a certain design or feature did not meet the expectation (or will ever meet) and what you may decide

2 Myth & 2 Truth About Bio-authentication

Bio-metrics using our physical characteristics to verify our identity. What has been advertised on the market is that it improves security level and make user experience much more simple.  While at the same time, we also hear the opposite voice, from time to time, about the failure of the technology and the risk of using it. In this article, I aim to summarize the research findings to understand the actual benefit and risk of bio-metrics and what what are the key hurdles for implementing biometrics in end-consumer products. TechTarget defined Biometric payment as a point of sale (POS) technology that uses biometric authentication to identify the users and authorize the deduction of funds from a bank account. Fingerprint payment, based on finger-scanning, is the most common biometric payment method.  As the authentication involved physical characteristics (e.g. fingerprint; voice; pupil etc), there certainly need a “point” on which the “Sale” is confirmed. Nevertheless, the POS concept