Touch Usability

Some pluses and minuses of gestural interfaces

Nicholas Nova has a write-up of a recent Lift Lab seminar on gestural interfaces: Lift Seminar @ Imaginove about gestural interfaces (from which I grabbed the above slide). The talks were about free-form gestures in video games.

What would you like to see in a presentation on gesture usability?

In May I'll be giving a 30-minute presentation at UPA 2010 about evaluating (touch) gesture usability. It'll be based somewhat on material that I posted here previously -- part 1, part 2. If you're interested in this topic, I'd be grateful if you'd take a minute and send me your thoughts on what you'd like to see in such a presentation. What would be most useful? What questions do you have about evaluating gestures? I'll do my best to incorporate any suggestions. Thanks.

Jitouch gestures for macbook and magic mouse

Jitouch is an impressive looking suite of gestures to add to your MacBook or Apple Magic Mouse. I haven't tested it out, but I'm impressed with the attention to detail evident from the web site, particularly in how they've documented the gestures.

Link: Jitouch. (Via ignore the code.)

Motorola Backflip with back-of-device interaction

It looks like Motorola is the first to put "back-of-device interaction" into production. That's an idea that HCI researchers including Patrick Baudisch have demonstrated in the past few years (see his nano touch page for details). Putting a touchpad on the back of a device gets your fingers out of the way of the screen and gives potential for more precise interaction as well as new types of gestures.

More at Slashgear and Engadget. (Images from Slashgear.)

p.s. I should have mentioned that the Fuse concept phone also implements this idea (albeit in a prototype). Fuse is a collaboration between Synaptics, TheAlloy, TAT, Immersion, and Texas Instruments: press release, Engadget's coverage.

Are dual touchpads the way of the future?

Probably not, but it's an idea worth exploring, and a couple of dual-touchpad designs have been in the news recently. As I wrote in a previous post, I'm interested in dual touchpads as a way to support two-handed interaction (by enabling separate, parallel actions or combined actions using both hands together).

Asus has shown the NX90Jq, designed by Bang & Olufson, with two "turntable-like" touchpads at either side of the keyboard (see Engadget, second post with video). It sounds like they haven't added any new interaction techniques, but presumably you can do two things in parallel, like point with one hand while scrolling with the other.

The touchpads are at the sides rather than on the palmrest, which is one way to avoid the problem of accidental contact that comes with large touchpads.

Wired's gadget lab also reports on this Asus machine, and asks "Who knows why we would want two touch pads, though." I'll bite! Like I said, there is the possibility of doing things in parallel, or maybe not literally in parallel but more efficiently. You might not have the coordination to scroll and point at the same time, but the switching time between the two tasks might be shorter than if you are using only one hand for both. This classic paper by Buxton and Myers talks about such benefits: A study in two-handed input (from CHI 86).

A couple of weeks ago, also via Engadget, I spotted this Fujitsu MH380 laptop with two touchpads. In this case the second touchpad only does circular scrolling (I think).

I wonder why they put the scroll circle on the right rather than the left. Most users are right-handed and would probably get more of an efficiency boost out of this design if they could scroll with their left hand. My guess is that two-handed use wasn't really part of the thinking here and that the scrolling is on the right because that's where the scroll strip normally is.

In any case, these are neat designs, and I'm guessing we'll see more experiments like this with multiple touch surfaces on laptops.

International Research Study about Touch Gestures

Several usability companies under the banner of International Usability Partners have conducted a major user research study about touchscreen gestures in different cultures. The research was presented at the recent Human Factors and Ergonomics Society conference among other places. Dan Mauney of HumanCentric is blogging about the study at HumanCentric's blog. So far the posts have presented the background and test protocols. Future posts will report on the results. From the first post:

Very few studies appear in the literature on the identification and understandability of gestures on touchscreens.  It is truly an emerging convention. Even fewer, and perhaps even none, have studied the identification and understandability of gestures across cultures.  Yet, many of the products that enable gestures will be intended for a global audience.  Designers have a definite need to know how understandable gestures are, and how universal they are, while the field is still young.

The International Usability Partners (IUP) has run a global study aimed at identifying the most common user-generated gestures for control of a touchscreen within a given country and across cultures.  The (IUP) is an established network of 12 independent usability companies based in 12 different countries (including Belgium, China, Chile, Finland, France, Germany, India, Italy, Spain, Sweden, UK, and USA) who have joined  to provide user experience services worldwide. This research involved collecting user-defined gestures for 28 common commands (like scroll, open, rotate, zoom in, cut, copy, paste) across 9 different countries and analyzing the resultant data for consistency of responses and cultural influence. 40 participants in each country were presented with a “before” screen image and an “after” screen image and requested to make a gesture that will result in that “after” screen image. The gesture made was recorded and catalogued.

Link to all the posts: Gesture Research.

It's very interesting work and I'm looking forward to reading all the results.

IUP also has a new group blog on this theme: TouchThinking.

Report from Workshop on Touch Affordances

The organizers have posted a report summarizing the one-day Workshop on Touch Affordances that was held with Interact 2009: Workshop overview report.

Article about iPhone's new accessibility features

A great article about the little-advertised but very impressive accessibility features Apple has added to the new iPhone: My first experience using an accessible touch screen device (Marco's Accessibility Blog).

Marko writes mostly about the new voice over features. There is also a zoom feature and white-on-black color mode.

Kudos to Apple for putting serious effort into accessibility and (again) being way ahead of the pack.

Mobile Gesture Design at Nokia

Article and video at the Nokia Conversations blog: Mobile Gesture Design at Nokia - developing a new dialect of interaction.

Evaluating TouchPad Gesture Usability (part 2)

This is the second part of a rough draft article describing our experiences at Synaptics doing usability testing on touchpad gestures. Part 1 is here. This part describes a typical usability test session in detail.

I'm posting this hoping to get some idea of how interesting this topic is to other practitioners so I welcome any comments.

Session Plan

The following four steps are performed for each gesture and on each device tested. In most cases we've found it best to study one gesture completely before proceeding to the next.

1. Introduction and practice
2. Familiarization Task
3. Accuracy Task
4. Satisfaction Questionnaire

The following sections describe these phases in more detail.

Variations of this might work better in some cases, depending on timing and number of gestures and devices -- e.g. If there are few gestures being tested it might be more efficient to administer one questionnaire at the end rather than one for each gesture.

Introduction and practice

Introduce the user to the gesture using documentation or by demonstration/explanation, depending on what is available and the context for use. Have the user practice the task on their own until they believe they understand how to perform it.

Familiarization tasks

The purpose of the familiarization tasks is to give the participant additional guided practice in the context of a realistic scenario. This ensures that each person has had a base level of practice with the task before rating it, and has exposure to the gestures in a typical situation.

Participants should be instructed to think aloud and describe any problems or issues they notice. During this activity, the moderator should note any problems seen and any comments the participant made.

All of the gestures we're concerned with here can be used in a photo gallery application, so that's the platform we chose for our tasks. Specifically, we used the Windows Live Photo Gallery and set up several photo albums.

Sample familiarization task: Pinch

To familiarize people with the Pinch Zoom gesture, we asked participants to perform the following tasks on four large photos. The idea was to simulate a real-world zooming task on maps and similar large images. All images measured 3000 or more pixels in x and y dimensions, and took at least five Pinch Zoom gestures to zoom all the way in on or out of.

For these tasks the moderator speaks the instructions. Each instruction has two parts: first it gives the general area of the target, then it names the target. Giving it in two steps prevents people from having to do a lot of searching, which is not what we want to study here.

1. “Going Under” news graphic (on New Orleans flooding): Zoom into the small map on the left side and find the city named Hopedale. Zoom all the way in on Hopedale and then zoom all the way back out of the image.
2. Stanford campus map: Zoom in on the South Residences (near the top of the map) and find the building called The Knoll. Zoom all the way in on The Knoll and then zoom all the way back out of the image.
3. New York State Theater seating chart: Zoom in on the Orchestra Right section and find row H seat 140. Zoom all the way in on this seat and then zoom all the way back out of this image.
4. World map: Zoom all the way in on Rome, Italy and then zoom all the way back out of the image.

To reposition the map (if needed), participants were shown how to click-and-drag the map. They could also use the scrollbars if they preferred.

Sample familiarization task: Swipe/flick and rotate

To practice the Pivot Rotate and the Three-Finger Flick gestures, we devised a compound task that asks participants to re-orient and add captions to a sequence of images within Windows Photo Gallery. It's similiar to the way one might sort through vacation photos after a trip. Each picture was a flashcard with a picture and a title, as shown below.

The images were pre-configured with random orientations (in the screenshot above, the picture needs to be rotated counter-clockwise by 90 degrees). The participant was asked to do the following:

1. Rotate the image to be upright using the Pivot Rotate gesture.
2. Type the image’s title into the caption field and press the Enter key.
3. Three-Finger Flick right to go to the next image.

Entering text was included as part of the task so that participants would need to move their hands back and forth between the TouchPad and the keyboard, rather than resting their hands in the same place throughout the task. Each participant performed this task for eight to ten images.

Familiarization for other gestures

The other gestures we have tested have included swipe up or down to enter or exit slideshow mode, and a "three finger press" gesture to launch an application. Since these are simple actions we felt it was sufficient just to practice the task a number of times instead of devising a more complicated scenario.

Scrolling gestures are a special case that we haven't mentioned here. For scrolling we use a more complex document scrolling experiment.

Accuracy Tasks

To obtain accuracy measures, the moderator asked the participant to perform 10 repetitions of each gesture in each direction (e.g. Swipe left, Swipe right, Pinch in, Pinch out, etc.). For each attempt, the moderator recorded the system response as either "correct," "no response," or "incorrect." In the "incorrect" case, meaning the system responded with a different gesture than intended, the moderator also notes what unintended gesture happened (e.g. a rotate happened when the user made a pinch gesture).

The accuracy tasks were performed in Windows Live Photo Gallery. To allow for multiple repeated zooms, we used a very large photo. To make note taking easier, we found it was best to ask the participant to do only five or fewer gesture attempts at a time.

The resulting measures were averaged participants and presented as percentages. In a stacked bar chart you can show percent correct, incorrect and no response. Additional charts can be used to break out by gesture direction and to show details of incorrect responses, which can be useful information for developers working to optimize the gestures.

Additional measures/procedures

• Measures of task performance. You could time the flick-rotate photo tagging task on each device for comparison. In our experience so far it has been more profitable to use the accuracy and user observations than to directly measure task performance.

• Videotaping -- we have found case-specific video taping to be useful, when a particular person is having much difficulty with a gesture in a formative study. Videotaping otherwise did not seem as beneficial. We have also on occasion used logging tools to capture individual gesture attempts for developers.

Measuring user satisfaction

With some exceptions, noted below, we have used the following rating questions for all gesture and device tests. The questions are answered on a 7-point Likert scale from Strongly Disagree to Strongly Agree and the set of questions applies to one gesture on one device. Our questionnaires also include comment fields.

1. This gesture was easy to perform.
2. This gesture was fast.
3. This gesture was accurate.
4. This gesture was tiring. (note reverse scaling)
5. This gesture felt comfortable.
6. The help information for this gesture was accurate.
7. I would use this gesture if it were available.

Question 2 can be ambiguous for some gestures, such as pinch, so we have used two alternative questions instead:

1. The amount of motion required to activate this gesture was: (7-point scale from Too little -> Too much).
2. The action produced by this gesture was: (7-point scale from Too slow -> Too fast).

For device comparisons we have organized the questions by gesture then by device. I.e. the user answers the questions for gesture A for all devices, then gesture B for all devices, etc. This can be too large a questionnaire, so if three or more devices are being tested we have instead had users answer the questions only once for each device, applying to all the gestures. We then ask users to comment on any gesture-specific differences that they find. An alternative is to reduce the number of gesture questions. We have used this approach as well, asking users for a single rating for a device per gesture.

Discussion/lessons learned

We have found that this combination of practice, familiarization, and accuracy tasks yields reliable measures and we have used the method to track progress at two stages of development and to compare performance between different devices.

The sorts of problems we have identified using this test are: speed/distance differences for triggering gestures, speed of zoom, difficulty initiating gestures, difficulty learning gestures, gesture misrecognition problems.

What this method leaves out:

• For consistency, we relied primarily on Windows Photo Gallery (and also Apple's Preview application photo gallery), but have also tested inside Adobe Reader, and, for scrolling tests, inside the Firefox web browser. It's important to consider application-specific differences in implementation and usability but for our purposes we have left this to separate software QA testing instead of usability testing.
• We have so far only tested a few familiar gestures (pinch, swipe, rotate) and haven't tested any more abstract or user-defined gestures. New issues may arise when testing those types of gestures.