Evaluating TouchPad Gesture Usability (part 2)

This is the second part of a rough draft article describing our experiences at Synaptics doing usability testing on touchpad gestures. Part 1 is here. This part describes a typical usability test session in detail.

I'm posting this hoping to get some idea of how interesting this topic is to other practitioners so I welcome any comments.

Session Plan

The
following four steps are performed for each gesture and on each device tested.
In most cases we've found it best to study one gesture completely before
proceeding to the next.

1. Introduction and
   practice


2. Familiarization
   Task


3. Accuracy Task


4. Satisfaction
   Questionnaire

The
following sections describe these phases in more detail.

Variations
of this might work better in some cases, depending on timing and number of
gestures and devices -- e.g. If there are few gestures being tested it might be
more efficient to administer one questionnaire at the end rather than one for
each gesture.

Introduction and practice

Introduce
the user to the gesture using documentation or by demonstration/explanation,
depending on what is available and the context for use. Have the user practice
the task on their own until they believe they understand how to perform it.

Familiarization tasks

The
purpose of the familiarization tasks is to give the participant additional
guided practice in the context of a realistic scenario. This ensures that each
person has had a base level of practice with the task before rating it, and has
exposure to the gestures in a typical situation.

Participants
should be instructed to think aloud and describe any problems or issues they
notice. During this activity, the moderator should note any problems seen and
any comments the participant made.

All
of the gestures we're concerned with here can be used in a photo gallery
application, so that's the platform we chose for our tasks. Specifically, we
used the Windows Live Photo Gallery and set up several photo albums.

Sample
familiarization task: Pinch

To
familiarize people with the Pinch Zoom gesture, we asked participants to
perform the following tasks on four large photos. The idea was to simulate a
real-world zooming task on maps and similar large images. All images measured
3000 or more pixels in x and y dimensions, and took at least five Pinch Zoom
gestures to zoom all the way in on or out of.

For
these tasks the moderator speaks the instructions. Each instruction has two
parts: first it gives the general area of the target, then it names the target.
Giving it in two steps prevents people from having to do a lot of searching,
which is not what we want to study here.

1. “Going Under” news
   graphic (on New Orleans flooding): Zoom into the small map on the left side and find the city named Hopedale. Zoom all the way in on Hopedale
   and then zoom all the way back out of the image.


2. Stanford campus
   map: Zoom in on the South Residences (near the top of the map) and find
   the building called The Knoll. Zoom all the way in on The Knoll and then
   zoom all the way back out of the image.


3. New York State
   Theater seating chart: Zoom in on the Orchestra Right section and find row
   H seat 140. Zoom all the way in on this seat
   and then zoom all the way back out of this image.


4. World map: Zoom
   all the way in on Rome, Italy and then zoom all the way back out of the image.

To
reposition the map (if needed), participants were shown how to click-and-drag
the map. They could also use the scrollbars if they preferred.

Sample
familiarization task: Swipe/flick and rotate

To
practice the Pivot Rotate and the Three-Finger Flick gestures, we devised a
compound task that asks participants to re-orient and add captions to a
sequence of images within Windows Photo Gallery. It's similiar to the way one might sort through vacation
photos after a trip. Each picture was a flashcard with a picture and a title,
as shown below.

The
images were pre-configured with random orientations (in the screenshot above, the picture needs to be rotated counter-clockwise by 90 degrees). The participant was asked
to do the following:

1. Rotate the image
   to be upright using the Pivot Rotate gesture.


2. Type the image’s
   title into the caption field and press the Enter key.


3. Three-Finger Flick
   right to go to the next image.

Entering
text was included as part of the task so that participants would need to move
their hands back and forth between the TouchPad and the keyboard, rather than
resting their hands in the same place throughout the task. Each participant performed this task for
eight to ten images.

Familiarization
for other gestures

The
other gestures we have tested have included swipe up or down to enter or exit
slideshow mode, and a "three finger press" gesture to launch an application.
Since these are simple actions we felt it was sufficient just to practice the
task a number of times instead of devising a more complicated scenario.

Scrolling
gestures are a special case that we haven't mentioned here. For scrolling we use a more complex document scrolling experiment.

Accuracy Tasks

To
obtain accuracy measures, the moderator asked the participant to perform 10
repetitions of each gesture in each direction (e.g. Swipe left, Swipe right,
Pinch in, Pinch out, etc.). For each attempt, the moderator recorded the system response as either
"correct," "no response," or "incorrect." In the
"incorrect" case, meaning the system responded with a different
gesture than intended, the moderator also notes what unintended gesture happened
(e.g. a rotate happened when the user made a pinch gesture).

The
accuracy tasks were performed in Windows Live Photo Gallery. To allow for multiple
repeated zooms, we used a very large photo. To make note taking easier, we found it was best
to ask the participant to do only five or fewer gesture attempts at a time.

The
resulting measures were averaged participants and presented as percentages. In
a stacked bar chart you can show percent correct, incorrect and no response.
Additional charts can be used to break out by gesture direction and to show
details of incorrect responses, which can be useful information for developers
working to optimize the gestures.

Additional measures/procedures

• Measures of task
  performance. You could time the flick-rotate photo tagging task on each device
  for comparison. In our experience so far it has been more profitable to
  use the accuracy and user observations than to directly measure task
  performance.

• Videotaping -- we
  have found case-specific video taping to be useful, when a particular
  person is having much difficulty with a gesture in a formative study.
  Videotaping otherwise did not seem as beneficial. We have also on occasion
  used logging tools to capture individual gesture attempts for developers.

Measuring user satisfaction

With
some exceptions, noted below, we have used the following rating questions for
all gesture and device tests. The questions are answered on a 7-point Likert
scale from Strongly Disagree to Strongly Agree and the set of questions applies
to one gesture on one device. Our questionnaires also include comment fields.

1. This gesture was
   easy to perform.


2. This gesture was
   fast.


3. This gesture was
   accurate.


4. This gesture was
   tiring. (note reverse scaling)


5. This gesture felt
   comfortable.


6. The help
   information for this gesture was accurate.


7. I would use this
   gesture if it were available.

Question
2 can be ambiguous for some gestures, such as pinch, so we have used two
alternative questions instead:

1. The amount of
   motion required to activate this gesture was: (7-point scale from Too little -> Too much).


2. The action
   produced by this gesture was: (7-point scale from Too slow -> Too fast).

For
device comparisons we have organized the questions by gesture then by device.
I.e. the user answers the questions for gesture A for all devices, then gesture B for all devices, etc. This can be too large a questionnaire, so if three or
more devices are being tested we have instead had users answer the questions
only once for each device, applying to all the gestures. We then ask users to
comment on any gesture-specific differences that they find. An alternative is
to reduce the number of gesture questions. We have used this approach as well,
asking users for a single rating for a device per gesture.

Discussion/lessons learned

We
have found that this combination of practice, familiarization, and accuracy
tasks yields reliable measures and we have used the method to track progress at
two stages of development and to compare performance between different devices.

The sorts of problems we have identified using this test are: speed/distance differences for triggering gestures, speed of zoom,
difficulty initiating gestures, difficulty learning gestures, gesture
misrecognition problems.

What
this method leaves out:

• For consistency,
  we relied primarily on Windows Photo Gallery (and also Apple's Preview
  application photo gallery), but have also tested inside Adobe Reader, and,
  for scrolling tests, inside the Firefox web browser. It's important to
  consider application-specific differences in implementation and usability
  but for our purposes we have left this to separate software QA testing
  instead of usability testing.


• We have so far only tested a few familiar gestures (pinch, swipe, rotate) and haven't tested
  any more abstract or user-defined gestures. New issues may arise when testing those types of gestures.

Comments

[Barbara Ballard]

This is really good stuff, thanks for posting it. Would you be willing to add it to the Design For Mobile wiki? http://design4mobile.mobi/

[Kevin Arthur]

Barbara - thanks. I'll look at the wiki and try to add something.

[James Field]

This is off-topic but I think it's important. My Sony Vaio laptop of perhaps 18 months of age is becoming increasingly sensitive. I believe the plastic alumanized coating is wearing thin thus altering the electrical properties of the pad. The left mouse key is already worn through to the next layer.

[Kevin Arthur]

James -- I would contact Sony support about this.

[Christopher McManus]

Thanks for posting this!

I have just started work on a project where I am prototyping multitouch gestures on mobile devices. I have found it really difficult drafting a good testing method - this has helped no end.

Thanks again,

Chris