Brynn Marie Evans

I’ve been reading Bill Tancer’s book Click. He works for a company called HitWise that has access to millions of search terms from across the web — by finding trends in these queries, he claims that we can learn about people’s actual motivations, interests, behaviors, and, even, fears.

A word before I go on: I generally think it’s dangerous to assume a searcher’s intent based on their search query terms alone. A premise in Click is that at scale, we can pull out things like intentions and motivations from web-wide trends. While Tancer could make a stronger statement about being cautious around intent (he does this in a very subtle way), he also illustrates how huge sets of internet search data do reveal things about us that we may not offer up willingly (if asked about it).

He makes this point about porn and also about fears. Expanding on this notion, I want to further illustrate how what we say is not necessarily what we do.

Tancer notes that the results from a nation-wide survey asking people about their phobias was substantially different from the list of fears he uncovered in his web-wide search query data. The survey, callled the National Comorbidity Survey, found that people’s top nine fears were:

1. Bugs, mice, snakes, and bats
2. Heights
3. Water
4. Public Transportation
5. Storms
6. Closed spaces
7. Tunnels and bridges
8. Crowds
9. Speaking in public

In contrast, the top ten fears from queries containing “fear of” (after filtering for non-phobia fears) were:

1. Flying
2. Heights
3. Clowns
4. Intimacy
5. Death
6. Rejection
7. People
8. Snakes
9. Success
10. Driving

What’s interestingly different from these two sets is that the non-survey results (2nd list) includes more social fears: “fear of intimacy,” “fear of rejection,” “fear of people,” and “fear of success.” Why might these differences be present? Perhaps because people don’t want to admit their social phobias to the person administering the survey? Perhaps because, when asked about a “phobia,” people will think of things (heights, water, tunnels, snakes) and rather than social scenarios.

Either way, what people say and what people do (think or mean) can be seriously different; and this is the greatest danger of relying (exclusively) on survey data.

This is always at the forefront of my mind as I’ve designed, administered, and analyzed many types of surveys. However, surveys can be well-positioned in the research and design process provided a few tips and guidelines are observed:

1) Avoid asking for generalizations!
People are terrible at estimating how long, how much, or how often they do something. There are some exceptions to this: if you truly do something everyday (like shower first thing in the morning), then you’ll be better at discussing your behaviors (in the abstract). Infrequent behaviors are hard to estimate and should generally be avoided in surveys.

But Brynn, you might ask, you just ran a survey asking people about their coffee drinking behaviors. Is this reliable data? That’s an excellent question, but you have to also take into consideration our intended use of that data for the betacup project. We intentionally planned to collect data from a few hundred people to identify apparent use cases and patterns, which we will subsequently investigate more directly through interviews and observations in coffee shops. Therefore, my second tip is:

2) Surveys are well-suited for establishing basic trends and underlying patterns.
I would not design a product or service around survey data alone! They can be suitable as the first step in a research program, but don’t confuse the power of numbers with the power of direct observations. Having 250 or 1000 data points in a survey provides a completely different perspective on a problem than watching 5 people move through, order from, and drink coffee in a coffee shop, for example.

However, I have learned that if you get people to talk concretely about a specific time in their past, they do a reasonable job recounting it.

3) Getting people to talk in specifics will improve the reliability of survey data.
I’ve been using a modified critical incident reporting model for collecting information from people online. Typically critical incident reporting was a technique used in face-to-face interviews: A researcher would ask a user to describe a time when something specific happened (a salient episode from the past). These are called “critical incidents” — but I’ve found that this general method can be applied to everyday episodes if you ask people to describe the last time something happened. For example, can you tell me about the last time you read a book or cooked dinner at home?

One caveat on this approach: actions that people take very frequently or very infrequently are harder to recall. Do you remember the last time you hand-washed your car? (This is presumably an infrequent behavior). Or the exact steps you took to get prepared for work this morning? (This is may be done on auto-pilot if you have a regular routine.)

If you have other tips or guidelines for studying user behaviors from written, remote surveys, please share them here!

In the modern digital age, online surveys are cheap and quick to administer. Yet they are not trivial to design and caution should be taken to make sure every question serves a purpose relative to your overall project goals. As designers we must be cautious about the reliability and generalizability of survey data alone. What people self-report (I have a fear of bugs and mice) may be very different from what people actually mean (I have a fear of people, intimacy, and rejection). Yikes.