Brynn Marie Evans

I attended a great workshop at CHI ’09 this weekend on sensemaking—a term used to describe the process of trying to make sense of information, data, experiences, etc. Through the process of making sense of stuff, we solve problems, make new insights, or develop an understanding (or new framework) that can be applied in other situations (for example).

The workshop was composed of talks on the theory and practice of sensemaking (SM) behaviors on Day 1; and working groups on Day 2 where we explored unresolved questions that arose on Day 1. At the end of the day, we gave short presentations on our sensemaking of sensemaking issues…which I try to sketch out here from my notes.

Part 1. The Essence of Sensemaking
In other words, how does sensemaking compare to problem solving?

• sensemaking is a cyclic iteration between these two stages of activity
• SM may begin with routine problem solving, but a trigger may come along which alerts the user to some issue (a change in the world, a gap, a dilemma, an obstacle)…this may thrust someone into a larger sensemaking process
• thus, SM results in a new understanding of goals or new frameworks (some type of learning that can be applied to other areas).
• with problem solving, one may never develop an understanding of the problem or a framework that can be applied in future situations
• SM may result in just temporary, provisional conclusion that leads to the next point in the sensemaking process; it could be an “Ah ha” moment, but it doesn’t have to (could be pretty mundane).

Therefore, sensemaking is distinct from ‘just understanding’ or ‘just learning about stuff’ or ‘problem solving’.

Sensemaking is a situated activity and must be understood as such.

• it is intrinsically human (it can’t be automated!)
• it can occur in the process of doing (dance, music, parenting, etc.)—it isn’t necessarily an explicit process
• SM is done by the person; it’s is part of us, not part of the problem we’re trying to solve/understand

What is the difference between success and failure in sensemaking?

• success and failure in sensemaking are not absolutes; “success” can be seen as a continuum.
• also, success is socially situated: it’s subjective—what’s successful for me may not be successful for you
• successful SM can involve discovering, defining, or framing a problem; it can involve getting a “good enough” answer to move on to the next issue. It doesn’t have to necessarily involve producing a measurable product, outcome or artifact.***
• but failed sensemaking can mean giving up in frustration, getting stuck in the wrong place, or having a negative outcome
• however, a failed product (outcome) might not be a failed process (the person may still have learned something that can be applied elsewhere)
• perhaps sensemaking should judged in terms of understanding gained; whereas problem solving is judged in terms of products/outcomes

***However, David Kirsh pointed out that a sensemaking process probably can’t be considered to be successful if there is no way the sensemaker can externalize anything about his process or outcome: through an utterance, a gesture, a behavior, an artifact, etc.

Part 2. The Process of Sensemaking
The process of sensemaking can be thought of as a trajectory through a (knowledge) state space: moving from a state of non-sense to a state of sense.

• it may be equally important to understand that artifact/outcome of sensemaking as well as the process you went through to get there.
• there are actually more kinds of knowledge than just know-what and know-how. other dimensions of knowledge that are important to consider: the process to accomplish goals, agreement within group, confidence, risk in outcome, etc.
• people could engage in top-down, bottom-up, or a mixed methods (“middle out”) approach

How do phase transitions work? (moving between two points: S₁ to S₂):

• includes what you know, what you need to know, and what you don’t know you need to know
• there may be such a thing as an optimal gap between what you know and what you need to know: if you don’t have that gap, you may not engage in the sensemaking process at all!
• there may be different reasons for stopping the sensemaking process (criteria for exiting)
• sensemakers need to continually evaluate where they are in the process, in case they identify triggers that may cause a transition between states (gap identification, imbalance, perceived needs, surprise element, etc.)

Transition Operators

• operators can be applied to move between states (you begin with suspicions, develop skepticism, and later synthesize that data)

In talking about an integrated process model, we should consider:

1. Entry points/triggers
2. There may be state spaces (n dimensions, movement operators between phases, trajectories between state spaces or “phases”)
3. Evaluation
4. Exit Points/Triggers
5. Externalization/handoffs (not only current state, but history and process of getting there too)

Part 3: The Methods of Sensemaking
What kind of metrics should we use to evaluate the sensemaking process? What is the purpose of metrics?

• could be to describe the sensemaking process or for an evaluation of a process (for scoring someone, or building a tool to accomplish a thing).
• metrics may help us test different models or hypotheses of sensemaking.
• but there may be differences between individual and group measures
• we can talk about measuring the process or the product (outcome) of sensemaking

David Woods has proposed a model in which there are 3 phases: gathering, interpreting, and synthesizing information, and people transition through these phases due to suspicions, skepticism, and re-formulations (being open to re-planning).

But there is one issue with this model: how to enter and exit this process? So the group has proposed a modified model—the “Woods+” model—where entry/exit points can occur at any time, and you can transition between states fluidly, not linearly.

Measuring the process of sensemaking:

• which phase/stage of sensemaking do we measure?
• or could we measure something like openness (suspicion), adaptability (skepticism), completeness, engagement, communication, exhaustiveness, or efficiency, etc.

Measuring the product (outcome) of sensemaking:

• how can we measure the user’s internal representations (frame)?
• or should we only measure the externalized products? (actions, decisions, diagrams produced, etc.—see next point for more)
• should we measure the entire working process or just the final output?

What kinds of products/outcomes could we measure?

• a decision
• a representation
• an effective action
• efficient performance
• moving from chaos to order
• discovery, clarity, revising, relevance
• redefining goals
• renegotiating in groups

There are some methodological challenges for studying sensemaking:

• there are lots of non-observable phenomena in a SM process
• observing it may disrupt the process
• metrics have various costs
• there are also challenges of mapping different SM states to the metrics you want to measure

Part 4: The Tools of Sensemaking
There are a number of issues facing toolmakers and designers of sensemaking technologies.

can be dealt with in four large pieces:
-phases, context, scale, and requirements

The context of sensemaking is really important to understand.

• it affects the kind of tool you’d want to build
• it helps define what your “data” source is: text, people, etc.
• the environment in which sensemaking occurs can affect the outcome/process
• it may illuminate what the handoff requirements are
• individual versus group SM situations will differ

Another issue is whether we should design for the different sensemaking phases?

• should tools focus on the foraging/searching loop (collecting sources/evidence) or the sensemaking loop (storytelling, organization, presentation, synthesis of material), or be one integrated/monolithic tool?
• the component approach to tool building is probably more appropriate, despite some disagreement on the specific model/process of sensemaking
• if we do this, there is the issue of needing to have common representations of concepts if/when you later to transition the other tools

Scale makes certain types of sensemaking difficult:

• the size of the data set can affect your confidence (maybe you’ve only covered 10% of the possible data out there)
• the scope of the domain is important (in some situations, you won’t even know how much more data there is to cover)
• need to manage multiple points of view with collaborative SM efforts, so we’ll need good communication methods (more process management will be necessary)

Top ten tool requirements: (Think design principles)

1. keep track of the SM progress; keep track of alternatives at each stage. (this will help with in generating hypotheses, and in refining and shifting between representations)
2. help people construct narratives and deal with confirmation bias
3. carry rationale/provenance behind decisions/data; provide integrated communication
4. data and project management are critical
5. enable building up of synergy between individual and group sensemaking
6. keep track of of confidence in the analysis: provide trust measures and evidence of consensus (throughout process)
7. analyze/reduce risks of handoffs
8. prepare for revisiting old information: have undo/history features, readwear and editwear for the state of sensemaking
9. transparency of the state built into the model (tool needs to reveal what it’s doing)
10. stimulate diversity—manage conflict and multiple points of view (aim for balance)

Specifically, what kinds of data structures and operators need to be supported by sensemaking tools:

• different representations of information such as: lists, trees, maps, graphs, spreadsheets, etc.
• different operators to navigate those representations: clustering of information, extracting entities, descriptive statistics, summarization/overviews, anomaly detection

Reflection on our own processes:
We engage in sensemaking all the time—whether it’s explicit or implicit. Think about the tools we use (e.g., spreadsheets, TurboTax, pen and paper, etc.). What specific features could improve the experience in using these tools? How would these features facilitate your process?

In our workshop, we mostly used pen and paper, digital note-taking programs (e.g., Notepad, TextMate), and powerpoint. They were limiting in some ways: paper notes are terrible for refactoring data or reanalyzing our process. In other ways, they were useful: paper was great for recording history, sketching, drawing… Perhaps we could use post-it notes next time to provide optimal functionality.