Evaluating TouchPad Gesture Usability (part 2)

This is the second part of a rough draft article describing our experiences at Synaptics doing usability testing on touchpad gestures. Part 1 is here. This part describes a typical usability test session in detail.

I'm posting this hoping to get some idea of how interesting this topic is to other practitioners so I welcome any comments.

Session Plan

The following four steps are performed for each gesture and on each device tested. In most cases we've found it best to study one gesture completely before proceeding to the next.

1. Introduction and practice
2. Familiarization Task
3. Accuracy Task
4. Satisfaction Questionnaire

The following sections describe these phases in more detail.

Variations of this might work better in some cases, depending on timing and number of gestures and devices -- e.g. If there are few gestures being tested it might be more efficient to administer one questionnaire at the end rather than one for each gesture.

Introduction and practice

Introduce the user to the gesture using documentation or by demonstration/explanation, depending on what is available and the context for use. Have the user practice the task on their own until they believe they understand how to perform it.

Familiarization tasks

The purpose of the familiarization tasks is to give the participant additional guided practice in the context of a realistic scenario. This ensures that each person has had a base level of practice with the task before rating it, and has exposure to the gestures in a typical situation.

Participants should be instructed to think aloud and describe any problems or issues they notice. During this activity, the moderator should note any problems seen and any comments the participant made.

All of the gestures we're concerned with here can be used in a photo gallery application, so that's the platform we chose for our tasks. Specifically, we used the Windows Live Photo Gallery and set up several photo albums.

Sample familiarization task: Pinch

To familiarize people with the Pinch Zoom gesture, we asked participants to perform the following tasks on four large photos. The idea was to simulate a real-world zooming task on maps and similar large images. All images measured 3000 or more pixels in x and y dimensions, and took at least five Pinch Zoom gestures to zoom all the way in on or out of.

For these tasks the moderator speaks the instructions. Each instruction has two parts: first it gives the general area of the target, then it names the target. Giving it in two steps prevents people from having to do a lot of searching, which is not what we want to study here.

1. “Going Under” news graphic (on New Orleans flooding): Zoom into the small map on the left side and find the city named Hopedale. Zoom all the way in on Hopedale and then zoom all the way back out of the image.
2. Stanford campus map: Zoom in on the South Residences (near the top of the map) and find the building called The Knoll. Zoom all the way in on The Knoll and then zoom all the way back out of the image.
3. New York State Theater seating chart: Zoom in on the Orchestra Right section and find row H seat 140. Zoom all the way in on this seat and then zoom all the way back out of this image.
4. World map: Zoom all the way in on Rome, Italy and then zoom all the way back out of the image.

To reposition the map (if needed), participants were shown how to click-and-drag the map. They could also use the scrollbars if they preferred.

Sample familiarization task: Swipe/flick and rotate

To practice the Pivot Rotate and the Three-Finger Flick gestures, we devised a compound task that asks participants to re-orient and add captions to a sequence of images within Windows Photo Gallery. It's similiar to the way one might sort through vacation photos after a trip. Each picture was a flashcard with a picture and a title, as shown below.

The images were pre-configured with random orientations (in the screenshot above, the picture needs to be rotated counter-clockwise by 90 degrees). The participant was asked to do the following:

1. Rotate the image to be upright using the Pivot Rotate gesture.
2. Type the image’s title into the caption field and press the Enter key.
3. Three-Finger Flick right to go to the next image.

Entering text was included as part of the task so that participants would need to move their hands back and forth between the TouchPad and the keyboard, rather than resting their hands in the same place throughout the task. Each participant performed this task for eight to ten images.

Familiarization for other gestures

The other gestures we have tested have included swipe up or down to enter or exit slideshow mode, and a "three finger press" gesture to launch an application. Since these are simple actions we felt it was sufficient just to practice the task a number of times instead of devising a more complicated scenario.

Scrolling gestures are a special case that we haven't mentioned here. For scrolling we use a more complex document scrolling experiment.

Accuracy Tasks

To obtain accuracy measures, the moderator asked the participant to perform 10 repetitions of each gesture in each direction (e.g. Swipe left, Swipe right, Pinch in, Pinch out, etc.). For each attempt, the moderator recorded the system response as either "correct," "no response," or "incorrect." In the "incorrect" case, meaning the system responded with a different gesture than intended, the moderator also notes what unintended gesture happened (e.g. a rotate happened when the user made a pinch gesture).

The accuracy tasks were performed in Windows Live Photo Gallery. To allow for multiple repeated zooms, we used a very large photo. To make note taking easier, we found it was best to ask the participant to do only five or fewer gesture attempts at a time.

The resulting measures were averaged participants and presented as percentages. In a stacked bar chart you can show percent correct, incorrect and no response. Additional charts can be used to break out by gesture direction and to show details of incorrect responses, which can be useful information for developers working to optimize the gestures.

Additional measures/procedures

• Measures of task performance. You could time the flick-rotate photo tagging task on each device for comparison. In our experience so far it has been more profitable to use the accuracy and user observations than to directly measure task performance.

• Videotaping -- we have found case-specific video taping to be useful, when a particular person is having much difficulty with a gesture in a formative study. Videotaping otherwise did not seem as beneficial. We have also on occasion used logging tools to capture individual gesture attempts for developers.

Measuring user satisfaction

With some exceptions, noted below, we have used the following rating questions for all gesture and device tests. The questions are answered on a 7-point Likert scale from Strongly Disagree to Strongly Agree and the set of questions applies to one gesture on one device. Our questionnaires also include comment fields.

1. This gesture was easy to perform.
2. This gesture was fast.
3. This gesture was accurate.
4. This gesture was tiring. (note reverse scaling)
5. This gesture felt comfortable.
6. The help information for this gesture was accurate.
7. I would use this gesture if it were available.

Question 2 can be ambiguous for some gestures, such as pinch, so we have used two alternative questions instead:

1. The amount of motion required to activate this gesture was: (7-point scale from Too little -> Too much).
2. The action produced by this gesture was: (7-point scale from Too slow -> Too fast).

For device comparisons we have organized the questions by gesture then by device. I.e. the user answers the questions for gesture A for all devices, then gesture B for all devices, etc. This can be too large a questionnaire, so if three or more devices are being tested we have instead had users answer the questions only once for each device, applying to all the gestures. We then ask users to comment on any gesture-specific differences that they find. An alternative is to reduce the number of gesture questions. We have used this approach as well, asking users for a single rating for a device per gesture.

Discussion/lessons learned

We have found that this combination of practice, familiarization, and accuracy tasks yields reliable measures and we have used the method to track progress at two stages of development and to compare performance between different devices.

The sorts of problems we have identified using this test are: speed/distance differences for triggering gestures, speed of zoom, difficulty initiating gestures, difficulty learning gestures, gesture misrecognition problems.

What this method leaves out:

• For consistency, we relied primarily on Windows Photo Gallery (and also Apple's Preview application photo gallery), but have also tested inside Adobe Reader, and, for scrolling tests, inside the Firefox web browser. It's important to consider application-specific differences in implementation and usability but for our purposes we have left this to separate software QA testing instead of usability testing.
• We have so far only tested a few familiar gestures (pinch, swipe, rotate) and haven't tested any more abstract or user-defined gestures. New issues may arise when testing those types of gestures.

Comments

[Barbara Ballard]

This is really good stuff, thanks for posting it. Would you be willing to add it to the Design For Mobile wiki? http://design4mobile.mobi/

[Kevin Arthur]

Barbara - thanks. I'll look at the wiki and try to add something.

[James Field]

This is off-topic but I think it's important. My Sony Vaio laptop of perhaps 18 months of age is becoming increasingly sensitive. I believe the plastic alumanized coating is wearing thin thus altering the electrical properties of the pad. The left mouse key is already worn through to the next layer.

[Kevin Arthur]

James -- I would contact Sony support about this.

[Christopher McManus]

Thanks for posting this!

I have just started work on a project where I am prototyping multitouch gestures on mobile devices. I have found it really difficult drafting a good testing method - this has helped no end.

Thanks again,

Chris