How Usability Studies are Like a French Meal [Comic]

It occurred to me the other day that usability studies were like a fine, french meal when I decided to insert a mini-task within a longer user study. This mini-task reminded me of the “amuse-bouche” in french meals, like the bite of sorbet or other spoonful of citrus goodiness to cleanse the palate. After that, the rest of the story just fell into place ;)

All my comics can be found on my scribd account.

How Usability Studies are Like a French Meal

Using remote research to inform social interaction design (SxD)

This was originally posted on the Bolt|Peters blog on February 2, 2010, as a guest author.

What is social interaction design?

Social interaction design (SxD) is the practice of designing for person-to-person interactions mediated by a computer interface, going beyond pure usability and human-computer interaction. Even fairly solitary experiences like editing a Wikipedia page occur in a social context in which other users’ past interactions influence what new editors contribute.

“It’s the the interactions among users that informs design” (Adrian Chan).

sxd sketch

[Sketch and original photo by Kai Chan Vong]

What’s a good example of an SxD problem?

conversation threadVark.com is a question-answering service that routes users’ questions to people in their extended networks who may have relevant knowledge of the topic. The original service operates through IM, Twitter, and email; more recently an iPhone app has been developed.

Let’s consider the difference between the mobile and desktop experiences of Vark.com. Both asking and answering activities work rather well in desktop email and IM. In contrast, responding on-the-go is awkward—more often than not, we’re distracted, hurried, or unable to type a coherent answer without bumping into a fire hydrant.

There’s also an assumption that the answer resides solely in our heads, when in reality, providing an answer often requires sharing links or performing a quick search—that is, we may not have the answer immediately on hand, but we know where to look.

Furthermore, successful answers often manifest as conversations on the desktop (example above), in which messages are exchanged in a back and forth manner so that the questioner can clarify her question and the answerer can refine her response. This type of sustained interaction is much harder to establish with on-the-go users.

mobile vark Finally, iPhone prompts (below) often lack enough information about the nature of the question or your relationship to the questioner. One reason for Vark’s success is that it seeks out answers from people within an extended, personal network, naturally building trust and accountability into the system. But without knowing how you know the questioner, the iPhone app experience feels instead intrusive and disruptive, and lacks any strong social motivator to respond.

Why is remote research useful for SxD?

Traditionally, user-centered designers conducted field studies or shadowed someone to learn more about their practices. The digital space complicates matters—not only is it difficult to shadow someone, but people’s actions are so fluid and varied that it’s hard to isolate specific behaviors in order to study them.

Remote research has emerged as a great way to do needs-finding for SxD, for three reasons:

First, it’s hard to recreate interactions between two or more people in a lab setting. Last year when I was studying user interactions during social search tasks, I realized that I needed to talk to multiple people: both the user who posed the question as well as the people who provided replies. I started by observing the questioning process: how the question was phrased, which communities or individuals were questioned, the historical relationships between the parties. Then I explored the answering process: answerers’ perception of the request, why they chose to reply, if they had a history of interacting like this.

What’s interesting is that answers provided over social networking sites (like Twitter and Facebook) were mostly jokes or “nudges” to attract the user’s attention (“Hey, remember me?”). But answerers in private channels (email, IM, phone) were more serious and thoughtful because people were contacted directly and had longstanding relationships with the user (“She asked me personally, and she’s helped me in the past”).

Second, social interactions unfold over time, and their repercussions aren’t always apparent in a hour-long lab study. I recall one user in my social search study who asked a question on ping.fm. He received a prompt reply which “seemed right”, so he reported it as his “final answer”. I followed up two days later to see if he had received any other replies. In fact, the conversation thread on ping.fm had progressed, and the community had collectively concluded that the earlier reply was incorrect. This observation was only made possible by the passage of time.

Third, social interactions are best understood within the context where they occured. Not just physical location, but also past history (between the people interacting) and reasons for having the interaction. For example, my sister tweets about her new startup, but I’m not familiar with her a field and don’t have a professional relationship with her, so I seldom reply to her tweets. However, when she emails, calls, or writes on my Facebook wall, I reply instantly—even on an unfamiliar topic. If you were only studying my Twitter use, you might wrongly conclude that I’m an ingrateful sister, but this interpretation would be taken out of the full context of my relationship with her.

Thus, whether you’re designing for healthcare, fitness, games, dating, or online privacy, it’s critical to gain insight into where, when, and why people to act the way they do. Community engagement through social media will differ substantially depending on people’s personalities, reputation, location, local culture and rules, nature of their relationships, and history of the community. Remote research methods—like experience sampling, remote observations, and critical incident surveys—are great tools for understanding the many facets of social behavior, and suggest productive avenues for pursuing SxD.

Additional resources:
Social interaction design salon (group blog)
Digital ethnography for social interaction design (slide show)

[Guest author: Brynn Evans is a digital anthropologist, design researcher, and author who studies social interaction design and social search. She extends a thousand thanks and a bear hug to Tony Tulathimutte for help in editing this post!]

Putting the craft in design thinking

This was originally posted on Unstructure on January 30 2010, as a guest author.

Is design thinking really that hard? There is obviously a growing acceptance of the notion behind design thinking as the previous essays and comments pointed out. But it remains that there is no formula for design thinking, and because of that, design thinking may alienate business leaders, managers, or even UX practitioners.

Consider the following quotes:

  • “Design thinking is not about solving design problems, it’s about solving problems with design.”Paula Thornton
  • “It’s not just thinking. It’s a structured approach to organizing design.”Gayle Curtis
  • “Good design is at the intersection of business and human goals. It’s not just about users, and it’s not just about business—it’s about balancing both.”Jess McMullin

A natural reaction to this is: Great! Sign me up! P.S. I have no idea where to begin.

Even as the essays on this panel have hit the nail on the head in how design thinking can be used for innovation in businesses, it still feels like an elusive process that faces many barriers in actual organizations. I’ve been noticing this with one of my clients. After introducing some new user-centered, user-driven design, marketing and sales invariably rework it to echo their time-tested sales pitch, causing it to bloat with extraneous options, text, and check boxes. In the end, we’ve made only an incremental improvement in our design.

Yet, I’ve also noticed a theme emerge across the many articles on design thinking recently. Bruce MacGregor talks about the importance of gaining insights early. Venessa Miemis mentions Tim Brown’s book which outlines an “inspiration phase” (disclosure: I have not read the book myself). And Peter Merholz continually reminds us that users are the central to the design process.

At the same time, I saw this surprising graphic last week: that “science” only makes up a sliver of the design thinking process. Really, I thought? What about the aforementioned importance of understanding users—isn’t that like a “science”? Maybe this is partly explains the uncertainty and confusion around design thinking.

[via http://www.kaplusa.com/blog/2009/12/the-role-of-intuition-in-design/ ]

I prefer the way that David Gillis describes the tradeoff between science and art as more of a continuum. Even still, where does design thinking fit in? A notch closer to the art, or to the science?

[via http://www.teehanlax.com/blog/2010/01/20/the-art-science-of-evidence-based-design/ ]

Taken together, I wanted to write a piece on the “science” in the design thinking process, to reiterate the importance of user-centered design and try to illustrate how this is not just a black box. Hopefully there’s some stuff in here that will help companies grasp exactly what we mean when we talk about “innovation” and “design thinking.”

One way to think of the innovation process is as a funneling of ideas across various stages—stages that span needs-finding, synthesis, ideation, prototyping, and iterating. Of course, this is a cyclical and dynamic process so it’s somewhat misleading to represent it as a sequential progression.

[via http://www.slideshare.net/mikeyk/intro-to-design-thinking ]

Gillis represents this process slightly differently, but still captures the same basic design phases:

[via http://www.teehanlax.com/blog/2010/01/20/the-art-science-of-evidence-based-design/ ]

Now, I prefer to think in terms of craft and creativity rather than science and art. By craft, I’m referring to the well-defined and established process of user-centered design. Creativity is the art, shiny design-y, intuitive part—the window dressing if you will. And to an extent, all these phases can be said to involve both craft and creativity.

It’s the craft part of design thinking that I want to elaborate on in the rest of this post, since the craft can be taught to a greater extent than the intuitive, experiential, creative part. Afterall, there are books and workshops out there teaching contextual and user-centered design. One great resource is IDEO’s Human Centered Design Toolkit.

Investigate / Observe:

Remember that the point of user-centered design is to gain clues about unmet user needs—needs that users themselves may be unable to articulate. The only way to gain this insight is to embed yourself in the community and practice of the people you’re designing for.

To do this, you must first scope your project and define various goals and hypotheses (what IDEO calls a brief).

Next, you talk to end users directly to learn about how they think, act, and engage, either with your product or in the space where you want to design a new product. This involves conducting contextual interviews, field studies, or otherwise observing users in their natural environments (not in the lab). If you’re designing an e-commerce checkout application, watch a user’s entire purchasing process from start to finish. If you’re redesigning your own site’s checkout flow, watch your users’ purchasing process. This is not a usability study. You’re not looking for feedback on specific features of your checkout process; you’re looking holistically at what your users’ goals are, what they’re doing to address those goals, where breakdowns occur, where confusion arises because expectations were violated, and importantly, how they feel (emotions! emotions! emotions!)

Synthesize:

There are number of established ways for documenting and synthesizing your insights. For example, interpretation sessions should be run as soon as possible after gathering user data, and they should always be done with other people—both people who were at your observation sessions as well as people who weren’t. Although it sounds counterintuitive, people who weren’t “in the field” with you often see the problem space from a different perspective, which causes important questions to be raised that might otherwise have been overlooked.

Some of the methods to use in interpretation sessions include building affinity diagrams, modeling workflow and cultural influences, and generating personas. Not all of these activities will be performed in the first interpretation session, but they are all part of the craft of synthesis in the design thinking process.

What you should be left with after this is a set of design principles in which you can begin to think about how to innovate on your product.

Ideate / Brainstorm:

Ideation and brainstorming is as critical to design thinking as is the collection of user data. After you’ve gathered your design principles, the goal is to generate ideas about how to create a product, service, or experience based on those principles. Gayle Curtis has an excellent talk on how to run such brainstorming sessions. Again, it’s a craft to structure the session— although what it generates is intended to be very creative, exploratory, and experiential. The more ideas the better.

Another excellent way to ideate is through what Dennis Schleicher calls a Issue Board. Issue Boards are, in fact, quite structured while still being visually evocative. They are generally built by one or two individuals, but subsequently used in larger brainstorming sessions to generate ideas.

Prototype / Evaluate / Validate:

I’m specifically lumping the prototyping and evaluative phases of design thinking together, to emphasize the point that prototypes are intended to solicit feedback. Yes, prototyping is an activity that can involve lots of creativity and visual aesthetics. But prototypes are not simply a beautifully-packaged, first generation version of your product—prototypes come in all shapes and sizes (from conceptual mockups to paper prototypes to high-fidelity interactive products).

In order to use prototypes to generate feedback, you must be open to testing your ideas early and often. Explore some conceptual mockups, and get feedback from a few users; then move onto paper prototypes and get quick feedback again; etc. This can also involve something like participatory design, whereby users are directly involved in the development of your prototype. There’s a good example of this in the Design Thinking for Social Innovation article: IDEO worked with children directly to develop a comprehensive vision care system for VisionSpring (the local provider).

As most will agree, design thinking is no panacea, even when combined with business thinking. Perhaps organizations become fearful of its outcome; or the numerous stakeholders and deep-seated traditions make it difficult to use design thinking in practice. One way to deal with this problem is to create an emotional connection with business leaders and UX professionals, in much the way we want to create an emotional connection with our users.

To do this, we must continue sharing examples of design thinking across a range of problem areas (which I have not succeeded in doing, but which unstructure has placed a call for). Sooner or later there will be a compelling example that resonates with every industry! We must also providing the necessary resources for others to embark on the design thinking process themselves. The goal for this essay was to do just that: illustrate how to put the craft in design thinking.

Vision board

Last week I made a vision board for 2010, which I have to recommend as an exercise to others! I already had a “themeword” for the year (see my previous post), but I wanted something in addition as a reminder of my goals, hopes, or dreams. When I set out to actually create the vision board, I wasn’t quite sure how it would turn out — and I engaged the Overlap SF group to do it together as an activity for our January meetup.

I really hope other Overlappers will share the output of their vision boarding sessions! For me, mine became a reminder of what social interaction design is — what factors influence social dynamics in a community, and what kinds of questions you have to ask when studying or designing for a community. What appears to be a crack or crevice in the middle of the conversation (in the middle of the board) is supposed to represent this design opportunity. But there is no “one size fits all.” The quote beneath reminds us of that: “I don’t really know what ‘community’ means.” Is that like Facebook? Question mark?

That’s exactly the point. What works in Community A might not work in Community B. (As an example: Until recently, you haven’t been able to reply to Facebook messages via email, although this didn’t stop people from logging into Facebook.com and continuing their experience there. This fluid cross-platform interaction hasn’t worked for LinkedIn, however. Receiving a email notification from a LinkedIn group feels more like RSS than social interaction, and when I receive a message, I haven’t been motivated to log onto the site and reply or interact with people.)

And so, as social interaction designers, we have to carefully consider the social dynamics we want to enable and how to go about doing that. Consider this:

“Social” can’t be solved by an algorithm

I was contacted by a dutch journalist who’s writing an article on the merits of social interaction versus search engines. She read a paper of mine and emailed me with two questions. I thought it’d be useful to post my reply publicly:

First, do you think search engines making use of social networks will improve search results and thus make our daily life a bit easier?

Yes! A lot of fact finding and information discovery already comes from friends, colleagues, and even acquaintances. Online social networks organize our personal relationships in a way that reduces the barriers to information exchange. At the moment, social networking sites aren’t set up for search per se—Facebook and Twitter want to get into search, but with Facebook, they can only reveal what your friends have OK’d to be public, and with Twitter, there’s enough noise, spam, and unknown people making claims that the results can be hard to trust. But, there’s already evidence that people are turning to these online social networking sites to ask their friends questions. On Twitter, these have been dubbed “lazy tweets”:

“Wondering where Bowtie saves it’s themes… Anyone? #LazyTweet” –@iphone360

“Looking for social media trends in the healthcare industry. Anyone out there have resources they can share? #lazytweet #healthcare” –@chrismevans

“Anyone know if there are Brocade SAN and Cisco MDS simulators? #lazytweet #healthcare” –@sloane

The real value for social search is making the search experience more personalized. If search engines can make use of existing ties, relationships, and data coming from social networks, they can use that data to bubble up results that come from a trusted friend network that may be as relevant (if not more relevant due to the trust factor) as traditional search results. The risk, however, is that our personal networks are narrow and not every search we perform may have counterpart results from social networks. This is why search algorithms will continue to play a large role in search, regardless of how “social” it gets.

Another benefit of using social network information in conjunction with search is that a services can begin to “learn” which of your friends have expertise or knowledge about certain topics. Then when you search for those topics, people from your network who may have relevant knowledge could be made available to you. It’s still unknown how visible searchers want other people to be in the search interface. People may only appear as a search result listing, linking to their profile or email address; or they could appear as a direct contact, like through an instant messaging window on the same page as the search results. Either way, the point is that direct person-to-person conversations can greatly supplement an information discovery process (as we pointed out in the “Do your friends make you smarter?” paper).

Another interesting area for social networking support during search is for searches that present difficulties. Anytime you can’t find what you’re looking for on the first try, or you rework your query over and over again — these are use cases that could benefit from asking a friend a question, pinging your social network, or finding a colleague/acquaintance who may have experience with this particular problem.

And second, do you expect these social search engines to become available in any near future?

Yes and no. Yes, in that there are a number of services that provide human answering — but these mostly do not include a search algorithm component, meaning that results are only driven by a direct human contribution. Such services include Aardvark, Cha Cha, Hunch, Quora, and Mahalo

Another popular class of social search services only makes use of aggregated social data from large networks. I call this “collective social search” since it’s like the wisdom of crowds effect, in that you can see trends from the collective that might be useful in guiding your search. Google Search Suggest is an example of this — it shows you the common search phrases for a given few words. Twitter’s Trending Topics and OneRiot are similar. I think the popularity of this approach is that it’s algorithmic, so you can throw more programmers at it and hopefully improve the results. But it’s quite limiting in its utility since searchers will trust people they know or people who can be vouched for, whereas trends across an entire network have no intrinsic relationship to the searcher. Such results may help in the early stages of search when you’re still trying to formulate an ill-formed query, but won’t be as useful when you want to narrow down to a specific answer to your question.

Thus, the kind of “social” component I want to see in search will require combining both of the approaches I mentioned above. This is not trivial, and there are a lot of unknowns about how people will respond to a service that does this. How will people react if their search results are shared with their social network? Will it be different if we see how valuable it is when our networks’ search results are shared with us (the reverse case)? How will reputation and obligation come into play? How will reactions differ by personalities? By location? By past history? By the political climate?

If Facebook’s Beacon experiment taught us anything, it’s that “social” can’t be solved by an algorithm. We’re still a ways off from really solving social search.