Co.Design

Usability Is King For Your Product. Here's How We Can Finally Measure It

Businesses live and die by the usability of their services, writes Continuum's Rick McMullen. But how do they know when there's a serious problem?

“Life used to be simpler,” my mom says while making her fourth attempt to update her Windows firmware in order to install Office 2011 on top of Office 2008.

I don’t correct her, but I don’t believe her either. As far as I can tell, life has always been complicated; and certainly as long as my mother has been alive, there has been incredibly sophisticated technology in the world. (When she was my age, people were landing on the moon. Nothing simple about it.) What I do believe, though, is that life used to be more usable. What’s different now is that complex technology has become so freakin’ cheap that it seems free to include “one more” feature in your product. The unforeseen cost, of course, is that those extra features hurt usability.

But we know all this. There is plenty of literature on the subject, and good usability is table stakes for a modern product. If your product isn’t usable, your business is in a dangerous position. Maybe you can get by in the short term by boasting your killer feature set; but the fact is that if people can’t figure out how to use your bells and whistles, you’re going to feel it on your bottom line sooner or later.

It may be old news, or even obvious to some, that poor usability can hurt customer relationships and hold back sales. But what isn’t obvious to business leaders is: How do I tell if it’s happening to me?

Before I get right to answering that, consider customer service for a moment. The adage “the customer is always right” has been uttered many times, in many languages, for many years. Long before the cash register was even invented, businesspeople intuitively knew that cultivating relationships with loyal customers was key to long-term success. Yet still, even in 2012, there is no end of companies who find ways to pull short-term profits out of their customers, at the expense of the longer-term customer relationship. (Example: So-called free three-month magazine subscriptions that you have to remember to opt out of before you get automatically charged.)

As the service economy continues to evolve, more and more companies are working to measure and improve their customer-service performance as a key indicator of success. And to this end, a minor revolution occurred in 2003, when the corporate world was introduced to Net Promoter Scores (NPS). In only a few years, NPS had become far and away the leading measure of customer loyalty.

Here’s the beautiful thing about NPS: Organizations get results from the data, but it’s amazingly simple to collect. You simply ask your customers, on a scale of 1 to 10, how likely they are to recommend your company/product to their friends and colleagues. People who give high marks are “promoters,” (i.e., will recommend you) people who rate low are “detractors,” (i.e., will bad-mouth you), and in the middle are “passives” (i.e., don’t care enough to do either one). The theory, which has been backed up by evidence, is that companies who have more promoters than detractors (i.e., a high Net Promoter Score) will win. They will acquire more customers and make more money.

Critics of the metric say it is a blunt instrument, and maybe it is. But if your NPS is -20, and your biggest competitor’s is 80, you had better do something about it. It could certainly be argued there are better ways to measure customer loyalty; but the success of the Net Promoter Score was closely tied to being the most usable way to measure customer loyalty. This is all by way of saying that even one of the oldest goals in the business world--to keep the customer happy--has only recently been armed with a really useful, usable metric.

Now, back to usability. I suspect that even if you believe wholeheartedly in the power of usability, you probably aren’t measuring it in a useful way and making it a key part of your business strategy. I am willing to bet that one of the major reasons people don’t effectively measure usability is that they don’t know how. One of the other major reasons is that the established usability metrics take a lot of effort and analysis to get anything out of them, so ironically, the usability measurements themselves aren’t terribly usable. So here’s my thought. Let’s apply the broader lessons of the Net Promoter Score to usability. No complicated metrics, no long surveys, just one “ultimate question for usability” that lets us know if we need to invest more in making our products intuitive.

What, then, should that question be? When I was in college, we were taught that the quantitative measures of usability are efficiency (how long a task takes), effectiveness (whether or not a subject can complete a particular task), subjective satisfaction (whether or not the experience is enjoyable), and error rate (how many times the subject makes a mistake, even if they eventually complete the task). All good stuff to know, but too low level for this purpose.

The famed usability expert and evangelist Jakob Nielsen says that quantitative measures of usability are low bang for the buck; he favors qualitative evaluation instead. When it comes to making a product better, I also firmly advocate qual over quant. But qualitative evaluation is just too much work to answer the simple question, “How do I know if I need to invest in usability?” I asked all of my “usability guru” friends, dug into the existing metrics, and came up with some of my own ideas; and the one I came across that most succinctly captured it was “How confident are you using this system/product/service?”

Unfortunately for me, this wasn’t one of my original ideas. It is a question I poached from the System Usability Scale (aka the “SUS”), originally published by John Brooke at Digital Equipment Corporation in 1986. It’s actually a nice survey in and of itself, but it takes several minutes to complete and doesn’t overtly tie to business goals, and I’ve never actually seen it done. Put another way, it’s usable but not usable enough to actually ever get used.

I settled on that notion of self-reported confidence, because it captures two important factors from both a human and business perspective. First, technology is supposed to work for people, and not the other way around. Second, as my uncle Fred once told me, “The worst thing you can do to an adult is make them feel stupid.”

In our modern world of automated interactions, the usability of your product or service is an important part of your relationship with your customers, and it had better not suck. If people knew that the complexity of their products was causing serious brand damage, they would surely make the investment to prevent it. But by no fault of their own, people outside of the design and usability world just don’t necessarily know when there is a problem. So even if this one question is under-nuanced, over-generalized, and does not tell you how to fix your usability issue, it does answer the question, “Do I have a problem I need to address?” with a yes or no.

So I propose that we all ask one simple question of as many potential users of our products as we can. The question I asked above was “On a scale of 1 to 10, how confident do you feel using this system?” But in truth, the question itself is less important than making sure you ask it. It’s a well-understood quirk of human nature that we tend not to take things seriously unless we’re measuring them. Whatever you ask, pay attention to the percentage of scores that are under 7. Those are all the customers you’ll lose, customers you’ll keep without delighting, or customers you’ll perennially frustrate, and for an embarrassingly fixable reason. It’s easy to do, and it might just move us toward a more usable world, which I know my mom would appreciate.

[Images: urfin, Olga Danylenko, and Triff via Shutterstock]

Add New Comment

32 Comments

  • Robert

    Great article. But what a bad reading experience (fonts & article structure)...

  • Jon Innes

    I'm not sure we need a new metric. We just need people less people in the field who claim or imply that usability or UX can't be measured. That just confuses decision makers. NPS is actually quite useful for measuring UX. SUS is pretty darn good for measuring perceived usability. Modern technology makes it easy to gather other metrics based on actual usage or feedback on prototypes.

  • Kayla Block

    I come to usability from a hardcore social science research background. But UX people can also come from visual design or software engineering backgrounds. I suspect those of us who haven't been trained in social science research methodology are unlikely to know and understand usability testing.

  • Ned

    This is a good question for usability.  Thank you.
    NPS is good for overall corporate satisfaction.  If you want the parallel thing for product satisfaction, why not just ask "How likely are you to recommend this product?"  Usability is a subset of product satisfaction which is a subset of supplier satisfaction.

    A simple question can be useful but cannot ever give complete information.  This is a polarity to be managed.

  • Joelm

    Nice article.  No, not an ultimate explanation of usability, but actionable in virtually any setting for virtually any product.  If your product makes the customer feel confident, but the customer wants something more, you probably DON'T need to work on usability (though you might want to work on marketing, to find customers aren't looking for more than you're providing).  That's really useful to know.

  • Pawel Smietanka

    Great article! Thanks for sharing your thougths. In my experience, even if users "say" that product is easy to use, it can be not good in itself because it may not give good results for them. In this case your question could partly measure it, because they could be "not confident" about effects of their actions.

  • Alan J. Salmoni

    Thanks for the article. Your point about confidence reflects some research I did some years ago in medical education: confidence inspired greater participation with the e-learning. It started off as an association but later work showed it to be an encouragement rather than a result.

    For for NPS - I'm unsure if a single question enscapulates the entire issue. It's hard to tell if it's stable over time (I would guess it fluctuates according to experience modulated by most recent experience and severity). It is, however, quite compelling  if convincing evidence could be found for it.

  • Alex Bystrov

    Guys.. what have you done to typography on fascodesign, It is way harder to read now, especially condenced typeface is unreadable. Are you motivating people to use Google Reader for experiencing your content?

  • Rajiv

     the serif fonts make it hard to read, it was ironic to read an article on the greatness of helvetica, univers and other iconic fonts in this irritating type.

  • Aemedd

    This makes perfect sense, I can't tell you how many times I am in applications and am thinking to myself "what on earth was this company thinking, did they not test this themselves". Just because these applications emcompass everything the user could ever want and need to know, does not mean that it has the correct flow. In the end, the users/customers will inevitably be missing steps, gaining frustration and confusion. The user will just give up.

  • Adam Khan

    Confidence does not necessarily translate into delight and satisfaction, which is ultimately what we should be after. Have you ever sat in an "economy" car, confidently breezed through the simple seat and temperature controls, and found yourself wishing that the experience was a bit more sophisticated, engaging or quirky? Sometimes it can be about the kinds of materials you use--the iPhone feels like a piece of glass art, but other touchscreen phones with a similar form factor do not--they twist and creak, feel plasticky, make strange noises when you type on the keyboard. What does this mean for usability? Simply that the human mind can learn new tricks--again, the iPhone is a good example of how people adapted to touchscreen phones, swipe gestures, and more. But convincing the brain to be satisfied takes more than inducing confidence.

  • Adam Khan

    Usability does not necessarily translate into delight and satisfaction, which is ultimately what we should be after. Have you ever sat in an economy car, confidently breezed through the simple seat and temperature controls, and found yourself wishing that the experience was a bit more sophisticated, engaging or quirky? Sometimes it can be about the kinds of materials you use--the iPhone feels like a piece of glass art, but other touchscreen phones with a similar form factor do not--they twist and creak, feel plasticky, make strange noises when you type on the keyboard. What does mean for usability? Simply that the human mind can learn new tricks--again, the iPhone is a good example of how people adapted to touchscreen phones, swipe gestures, and more. But convincing the brain to be satisfied takes more than inducing confidence.

  • David Griffin

    I remember a documentary about the terrible Airbus UI that allegedly resulted in more than one airliner crash.
    They said that from voice recordings they heard two phrases that should have been a dead giveaway:
    "How do you make it do xxx ?"
    "Why did it just do that ?"

    A good UI does exactly what a new user expects it to.  That's pretty much all rteh definition you need.
    It gives the user a mental model of how the system works that actually matches the underlying functionality.

  • Bilalali24434

    Its true to a point where competitors are facing each other over the edges. But these days new technologies are paving their own way to the customer on using their newly introduced product as they want. Now the competiton lies on how softly you bend the customer to stay loyal to it. bilalali24434@gmail.com

  • Noel Lastrella

    Opening with "update her Windows firmware in order to install Office 2011" doesn't lend much to the articles credibility. Windows and Office are software, firmware is typically embedded in chips and is read-only. As for someone who comes from a tech and a design background it was distracting, it bothered me the whole time I read the article.

    You do offer some insight but in the whole, it's fairly obvious that we do have to make the interactions between humans and machines easy and seamless but at the same time we have to push that relationship to train the population on new concepts. 

    For example, the Wii controller or the XBOX Kinnect: these are very simple, and intuitive ways to interface with the machine and software but those concepts had to be communicated first to the audience. You can also add the iPhone into that mix as well. And I believe that the success comes from advertising and marketing geared to engage and train the audience, not just build interest.

    To go back to your topic on how to measure, I feel that in our day and age, the survey is an anachronism. Today we have real time metrics, we can see where the users have trouble, we can see how long it takes them to eventually find a menu item, we can see when they quit applications, where their mouse moves, and even though the technology is still quite new... even where their eyeballs are looking. We can see where people are having trouble in real time, we just have to track it. As far as fixing these problems, my only solution is to get people interested in the product and communicate with them on how to use it and find a way to get them interested in learning (not an easy task I know). Make it fun and interesting like Nintendo, Microsoft and Apple did it. I don't claim to have the answers, just my $ .02.

  • Roderick McMullen

    Noel, you are correct, the firmware is not part of Windows, that was an oversight on my part.  I should have said "...her fourth attempt to update whatever it is that her computer needed her to update in order to then update Windows."  From an end-user perspective, the distinctions between software and firmware (in fact I would argue...most technical distinctions) are lost on all but the most savvy of users. 

    I apologize for the distraction.

    To your second point, I can't disagree that we have more mature metrics, and that surveys are barely state of the art.  If I had my way, everyone would do what you described or something equally elegant.  I hope, though, that those who aren't ready for that do something, no matter how sloppy, in the meantime.

  • Daniel

    At live|work we've been using a measure called the Service Usability Index for a few years now. It's also based on the Net Promoter Score to give a simple "1 number you need to know" approach, but it's derived from users' responses to 4 key criteria of a service: Proposition, Experience, Usability, & Accessibility.

    Ultimately it's a qualitative response measure, and as such, there is still resistance to measuring the subjective (even though it can be scaled up to a quant study).

    So whilst I agree with Roderick McMullen's plea that asking the question is the most important thing, but getting organisations to ask it let alone act on the answer is the real challenge.

    Daniel Letts, Senior Partner at live|work