At the Google Ventures Design Studio, we have a five-day process for taking a product or feature from design through prototyping and testing. We call it a product design sprint.
As day 5 of the product design sprint dawns and the team files into the war room, there’s a certain something in the air. Is it the chemical scent of the whiteboard markers? Maybe. Is it coffee breath? Yeah, almost for sure—in fact, you might want to hand out some gum.
But there’s also something else: anticipation. Today, we test our prototype and learn which ideas worked, which didn’t, and what to do next.
Today, you’re going to be running a user study, showing your prototype to four to six real humans. By "real humans," I mean people who don’t work at your company—more specifically, people who you’d like to have as users. Learn how to recruit great participants for your study here.
Wait, what if I don’t know how to run a user study?
Bad news: This post won’t teach you how to talk to users. This post is for the people watching the study, not for the poor soul conducting the interviews.
Good news: If you’re the interviewer, you can get some great advice from our research guide. Be sure to check out Michael Margolis’ articles on interviewing tips and body-language hacks, and his video about running user studies from the Google Ventures Startup Lab.
And you might as well read this post, too, so you know what’s going on in that observation room. I went to all the trouble of typing the whole thing up, so it’s kind of the least you could do.
Start at the end. When the day is over, you’ll want a concrete list of ways to improve your prototype. Talking about your game plan before the study starts will help you get there.
The interviewer and the observers should make a list of the key questions for the day. Here are a few tips:
• Review your conflicts and assumptions. Those assumptions you flagged as testable with a user study? Now’s the time!
• Are you testing multiple prototypes in a battle royale? If so, be sure the interviewer understands the differences between the two versions, so they can ask the right questions.
• Consider showing participants some real products for comparison—they’re like free prototypes!
• What else do you want to see through your users’ eyes? Today’s study is not a usability test —you have a chance to find out how your users understand your product, what competitors or substitutes they use, and more. Think beyond the prototype and you are guaranteed to get some unexpected insights.
Everybody who participated in the sprint should be in the room. There’s no substitute for watching real humans use your product, and this is a golden opportunity to do it!
I like to reserve two rooms for the day: one for the user interviews, and another where the sprint team can watch live video of the sessions and take notes together. It’s kind of like a Battlestar Galactica marathon, only instead of comparing guesses on who’s a Cylon, you’ll be comparing guesses on which parts of your design are going to go up in flames. It’s actually pretty fun.
Test the A/V ahead of time
Setting up live video for observation shouldn’t be too hard—at the Google Ventures Design Studio, we’ve had success with WebEx, GoToMeeting, Apple Airplay, and Google Hangouts.
Just be sure to test your video before you start the study—ideally the night before and the morning of—because nothing’s worse than putting in all this work and not getting to watch the outcome. OK, there are a few worse things. But you see what I mean.
A few more A/V tips:
• A conferencing mic can provide a huge improvement in sound quality.
• Don’t forget to mute the audio in the observation room!
• Quickly re-test the A/V between each session. Seriously, stuff gets wacky.
Don’t diss the user
If you see people struggle to understand the prototype, keep in mind that we’re testing your design—not the participant. If they don’t get it, it’s not because they’re dumb. It’s because you haven’t nailed the design yet.
Make it clear in the observation room that it’s not OK to diss the participant. It’s tough to wade through a prototype while people are watching. You owe the participants your respect.
Every observer takes notes
Everybody should take notes on things they see during the interviews: good, bad, and other. Insist on paper note-taking—it’s best to keep laptops closed, lest you lose your fellow observers to email.
Designate a court reporter
One person can use a laptop in the observation room, but they have a tough job: the court reporter. They’re responsible for typing a word-for-word transcript of the interview in real time. Assign a different court reporter for each session—it’s cruel and unusual to make the same person do it all day.
It’s kind of a pain, but these notes become an incredibly valuable reference after the study, and a text document is a heck of a lot faster to scan for quotes and reactions than a video recording. Often times, we don’t record the study and rely on these notes instead.
Make a scoreboard
Clear one big whiteboard to collect the group’s notes. Make a column for each participant and a row for each part of the interview (e.g., background, first prototype, second prototype, etc.).
At the end of each session, write down the highlights from everybody’s notes—you can double check any questions against the transcript. I like to color code: green for things that went well, red for problems, and black for everything else. It makes it easier to find patterns at the end of the day.
I’ve never been in mission control when one of those rover things lands on Mars (NASA isn’t returning my calls). But I imagine it’s pretty similar to the atmosphere in the observation room of a user study. There’s tension and excitement and nobody wants to be the one who designed the doohickey that blows up. Here’s what you might expect to feel throughout the day.
First session: "We’re geniuses!" or "We’re idiots!"
The first user may love your prototype. Or they may hate it. Either way, take a deep breath. People are different, and I offer you an ironclad guarantee that not everyone will react in exactly the same way.
Regardless of how that first interview goes, resist the urge to make changes to your prototype. Unless it’s something simple like a typo or broken link, you risk "fixing" something that the next four users would have actually liked.
Sessions 2–4: "Oh, this is complicated…"
It’s not uncommon for the second and third participants to have dramatically different feedback, which means you’re going to feel a little confused. Just sit tight and keep taking notes. It’s still too soon to tell.
Actually, that’s not entirely true. Sometimes you know for sure that part of your prototype is rotten after two or three interviews, and watching more people suffer through it is punishment for all involved. If everyone agrees that something is way off, talk to the interviewer in between sessions and ask them to guide people around that part of the prototype.
Studies 5-6: "There’s a pattern!"
After the final interviews it’s easy to see the big patterns, but it’s worth double checking the notes. Go to the scoreboard and look for things that showed up two or more times. Mark good stuff with a big green dot, and bad stuff with red.
Now make two lists on the whiteboard: "things that work" and "problems to solve." These are your top-line findings. The CEO or decider for the project should bless that list before you leave the room.
The sprint is complete. Take a deep breath.
The vast majority of these studies ends with mixed results. Some of your solutions work, and some don’t. The outcome usually falls in one of three buckets:
A. Most stuff worked
This is pretty uncommon for the first sprint on a project, but if it happens to you, everyone on the team is probably on the same page about the fixes and tweaks you need to make.
What to do next: Tune your existing prototype and keep going. Try starting your next sprint at step 3, when you decided which prototypes to make.
B. Some big questions
The most common outcome after a user study is a mixed bag: a few hits, a few tweaks, and a couple of real head-scratchers. Fortunately Keynote prototypes are easy to change, and as long as some parts of your design are solid, you can probably build on what you have.
What to do next: You can move fast on the tweaks, but you’ll want to come up with multiple solutions for the bigger problems. Start your next sprint at step 2, creating a storyboard.
C. Everything exploded
I’ve seen a lot of of my designs go up in flames. It’s OK. You learned that something didn’t work, and it only took you a few hours to build it in Keynote. This is great progress and—relative to building and launching for real—very cheap progress at that. Think what would have happened if you’d spent weeks or months implementing this solution!
What to do next: Start your next sprint back at the drawing board with step 1 (developing a team understanding). (Hint: the results of this study are perfect fodder for reviewing and building understanding as a group.)
You may feel tired. Actually, if you don’t feel tired, you probably didn’t do the sprint properly. It’s an intense week. But you’ve built up a tremendous head of steam—even if everything exploded, you now have a much better understanding of the problem and possible solutions.
At the end of a sprint, CEOs often tell us they appreciate that they have a clear list of what to do next. Now that you’ve built up all that knowledge and momentum—not to mention a prototype—you’ll find you can do the next sprint more quickly, or with less effort, as long as you do it right away. Don’t let more than another workday go by before you jump back into it. If this problem is important, you’ve got to bear down and finish it off before people get distracted.