Brynn Marie Evans

Peter Pirolli and I are designing a study where we want to do a cognitive task analysis of people’s sensemaking processes. Verbal protocol analysis (most commonly introspective, retrospective, and think-aloud) may help us understand the process people engage in while they find, synthesize, and assimilate new information as part of a broader sensemaking task.

Modeling cognitive activities may be useful for several reasons. First, people constantly handle large amounts of knowledge. When new knowledge is introduced, we must understand how it integrates with and modifies previous cognitive schemas. The output of one’s thinking or sensemaking process may be a material artifact or verbalization of sorts, but does not include the process one went through to achieve the product. Furthermore, information, knowledge, and cognition is obviously distributed over several domains, including at least the specific category domain (e.g., programming techniques, energy policy, mathematics), the information domain (what types of information are required), the technology domain (what tools are used, what knowledge of tools are required), the social domain (who is present—individuals or communities, access to resources through those people), and the cultural domain (how the rules or taboos in an organization or workspace affect your actions). The cognitive process people engage in when problem solving will inevitably be influenced by these factors (very much in line with distributed cognition theory). Our hope is that a verbal protocol analysis will shed light on the critical aspects of these domains that influence people’s sensemaking abilities.

The goal of (verbal) protocol analysis is to understand a person’s thought process while he/she performs a task. In theory, verbalizing your thought process is effective because people temporarily store the steps of their process in working memory (Ericcson & Simon 1980, 1993). This method could be seen as a criticism in my department, however, because it focuses on “verbal reports of thought sequences,” not necessarily “cognition” as it is naturally distributed and influenced by many aspects of our environments. (We take this into account, below.)

There are also some concerns with this method: that people can’t report processes they are not aware of; that people are not even aware of all the factors that go into their thinking; and that talking outloud interrupts their natural flow. (Conrad, Blair & Tracy, 1999). Ericsson & Simon (1993) reported that talking outloud may not actually disrupt your flow—people seem to have just as accurate task performance (albeit with longer completion times), as long as they are not required to reflect on or evaluate their thinking process. However, others have noted there may be interference if people are mapping information from one domain to another: for example, if someone needs to verbally report on how they interpret a visual map, there could be interference with talk-aloud protocols. Russo, Johnson and Stephens (1989) also found that accuracy on gambling tasks increased but accuracy on mental arithmetic decreased when people used talk-aloud protocols.

There are valid concerns of using this method for the present study, as well. We expect that people will be browsing for information online or engaging in real-world interactions with friends or colleagues while they try to solve problems. Their accuracy when mapping these potentially non-verbal activities into a verbal protocol should be considered. On the other hand, Pirolli and others at PARC have successfully used verbal protocol analysis for much of their work on information foraging on the web. Therefore, I am inclined to believe that this protocol could be used successfully in the present study if we are careful about what we demand of our subjects and if we include some additional factors of interest (a la distributed cognition).

I was glad to learn that it has become somewhat standard to combine techniques (Willis, DeMaio and Harris-Kojetin, 1999), such as talk-aloud protocol + “verbal probes” (probing for additional information during a talk-aloud protocol to get people to report on something that they may or may not have been aware of). Of course, there are tradeoffs with probing for information like there are with any technique, especially because experimenters could probe for reflections on people’s thinking process, which could interfere with task performance (as suggested above). However, we hope that probing for details of the people, technologies, and material objects that subjects interact with will only delay their completion time, not actually not change their process. Additionally, retrospective probing (after a task or sub-task has been completed) may still produce valuable information without (severely) interfering with the process.

For our purposes, it is important to consider how the people, technologies, and material artifacts affect a person’s thinking or sensemaking process. Cognition does not exist purely within the head of a single individual, nor is cognition unaffected by environmental factors. We hope that by videotaping subjects’ entire workspace (desk, computer, etc.) and by probing for critical events that appear in their process (among the people, technology, and artifacts they interact with), we will minimize interference while still gaining a reasonable account of an ecologically-situated, cognitive sensemaking process.