Most of the times, the usability test requests come from the marketing team, shortly before a product or a major release is about to launch.
It goes like this:
About the time when the product is about the launch, the marketing team comes to the executives, everybody is having these little butterflies in their stomach. We have all too business on everything we have to do, planning our marketing events, developing all the features, that we forget our users.
We did the user tests, well, focus groups actually, and we found a long list of problems users encounters when using our products. And we prioritize them based on the time needed from the development team to fix the issue: if it is the change of a label, priority high; if it is changing of an interaction, priority low etc.
Everybody is frustrated with the process: product team knew the test is too late for the current release, it should have been done much earlier; marketing themselves are upset as they are worried the product will not get user acceptance for all the problems we spotted from the test; development team are stressed as now they have more things to change and fix that they don’t know why (they are not invited to the team of course) on top of the zillion other things they have to do before the launch;
And the CEO is shucked, s/he can’t believe the number of problems spotted from the test, some of them looks serious.
Throughout my product manager life, I’ve seen the same story again and again. It is an inefficient and frustrating experience for most of the people involved in the process.
But why don’t we do something different, what stopped us from doing what we should do:
Executives Don’t Support Early Usability Test: cost is one of the main issues, hiring a professional testing company can be expensive, the last time we used TangUX in Shanghai for 4 test sessions on 12 people, it costed us more than half a million RMB (~80K USD). Privacy is another issue I faced, especially for new product, company can be very sensitive in any kind of test involving external parties even through the confidentiality agreement is signed.
Team Don’t Have the Skill to Execute: besides PM, the product team is actually not that busy, the requirements are given and assets produced, but they has very little knowledge on testing, we have brilliant business analysts and graphic designers, but they have no idea how to conduct user tests nor to analyze the findings;
Those Who Knows Don’t Have Time to Teach Those Who Have Time: as a product managers, I have done all kinds of user test from Focus Group to Usability Benchmarking, but I am so busy trying to clean my to-do list and I have no time to write screener questions, prepare test scripts and do beautiful presentation for the executives.
After being through the same process 3 times, I came to realize the daunting pattern:
Why is cost high, because we need to hire professionals, they are expensive;
Why can’t we do in-house, because we don’t have the skills;
Why don’t we have the skills, because we didn’t train the people;
Why didn’t you train the people, because I don’t have time;
Why didn’t you do it when you have time, because management don’t approve such a project at that time;
Why executives don’t approve such a project at that time, because they don’t see the need;
Why don’t they see to need then, because we are going to market now;
Don’t we always want to go to market?
I’ve came to realize we will just fall into the routine again and again unless I make an effort to broke the chain. And I’ve been doing 5 simple things which gives me tremendous benefits on making user tests part of the product management cycle:
Create Artifacts at the Best Time: the first 3 things I when I get my hands on a product is to prepare the screener questions, the focus group questions and the test scenarios. In the product envisioning stage, I will do screener questions and focus group questions, these are more generic questions to define the direction of a product; as I start developing each feature, I will write focus group questions and test scenarios, here the scope are limited to the respected feature; timing is very important because, in fact, I’ve done similar thinking and wrote them down in the requirement and PRD, so it just take me extra 10-15 minutes to create a Confluence page or Google Doc and format them for the use of user tests;
Beat the Competitors and We Buy You Lunch: everybody loves to look at competitors, it helps us know our place in the guard scheme of things, know what users recognize us as etc. So I decided the marry this need with my need for competitor research and organize bi-weekly brunch on competitor tests. I need to write script of course, but I need to do it as part of the competitor analysis anyway, and I got to use the office budget for team activities to provide lunch to everyone;
Piggyback on the Giants: look, executives don’t give a damn about products, they only care about revenue, ROI; but what they do care, are what other executives are doing; I subscribe to the search term “usability test” from Google Trends and I forward the interesting articles I found related to executives endorsing usability test to my executives. Be careful not to use the one that end up marketing their product though, you maybe misunderstood as trying to promote their products. Alternatively, you can always find tons of articles from the big consulting firms (McKinsey, Bain etc) that continue to come out with the “food for thought’ article that just so helpful for you to make a point. Don’t forget to write a comment using the key words they are (e.g. meet consumer needs; improve ROI; reduce cost etc). For example: “Great McKinsey article on how banks and large retailers using user tests to meet consumer needs http://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/building-a-design-driven-culture
Outsource, Delegate and Automate: to do the above 3 I have a task list that needs to be done at different times:
- Write screener questionnaire;
- Write tester recruitment advertisement;
- Write focus group questions;
- Write test script;
- Find a list of competitor products;
- Collect and analyze results;
- … ...
First I see if I can outsource or delegate to somebody else, maybe not now, but in the future; and what is needed to enable it. After that the list becomes sometime like:
- Write screener questionnaire;
- Write tester recruitment advertisement template;
- Write focus group questions template;
- Write test script template;
- Review scripts;
- Collect list of competitor products and scripts;
- Read Google trend on usability ROI;
- … ...
Then I put the list in my calendar and reserve the time for them so I get to them and complete it.
If you can’t, well then read the next point.
The More People that Do it With You, The Easier it is: well, the last point is not really a serpent point, but it is one of the most important one, I consider myself a very disciplined person but even that, I get swapped by work and I just can’t continue, but that’s why we have a team right? My team member feeds me with competitor links, they share me articles that I can pass on to the executives; executives share good product that we test together; I can be a happy librarian centralizing the knowledge and pass them on.
As Kilian, Serrazin and Yeon stated at the end of “Building a Design-driven Culture”: Customers increasingly expect products and services that are designed to meet their needs, delight them with unexpectedly great experiences, and address a heightened sense of aesthetics. Companies that meet those needs are rewarded with fierce brand loyalty and higher spending, which translates into fatter profit margins. But that kind of success only happens by design.
And above all, test my words out, see it works for you. They have worked out for me,
I would love you can drop a line or two in the comment section to give me some feedbacks.
Building a Design-driven Culture, McKinsey