Don Norman: Gesture Wars

Don Norman's latest column at Core 77 is about scrolling direction and gesture consistency: Gesture Wars.

Hinckley and Wigdor: Input Technologies and Techniques (2012)

Ken Hinckley has posted a book chapter written by him and Daniel Wigdor: Book Chapter: Input Technologies and Techniques, 2012 Edition.

This is a revised chapter for the 3rd edition of the HCI Handbook and includes more coverage of direct-touch input than in the 2nd edition (2007).

Lion Trackpad Gestures

A while back I posted some observations about the (then) forthcoming OS X Lion release with lots of changes to the trackpad behavior and options. I've now been using Lion for a few days so here is a follow-up on those observations.

Reverse Scrolling Direction

One of the first things you see after installing Lion is a little tutorial screen alerting you to the fact that your scrolling world has been turned upside down. This is a change to the scrolling metaphor -- you're now manipulating the content rather than the scroll bars (which are now hidden much of the time in Lion, though you can still click and grab them if you try). This is now called the "Natural" scrolling direction by Apple.

I wondered whether users would get used to this. So far, judging by reviews and Twitter comments, it looks like most people are okay with it, but some find it irritating, especially if they have to switch between computers.

One factor that forced this change, I suspect, is the new and more fluid navigation left and right between screens (with either 2, 3, or 4 fingers, depending on how you configure it). This feels right when the screen moves with your finger direction (the "natural" direction). If you go back to the unnatural scrolling direction then sliding left and right feels odd. (But this depends on the particular scroll or swipe navigation gesture -- more on this below.)

More fluid gestures

In addition to the more fluid swiping/scrolling between screens, scrolling now has the edge bounce physics as in iOS. This depends on the application, though; so far it's only supported in Apple applications like Safari and iWork. I don't know if Apple will be opening up this feature to other developers.

Two-finger double-tap to zoom ("Smart zoom")

This works well, but again only in Apple programs for now. I had questioned how this would coexist with the existing two-finger tap option (for secondary click). It looks like my prediction for how they would handle this was right -- there is now a noticeable delay after two-finger tap responds, if you have Smart zoom enabled. This allows the system to unambiguously recognize the two gestures. (If you didn't have this delay then you would get secondary clicks happening before smart zoom happens.)

Different navigation gestures

In the previous post I noted that Apple was introducing two-finger swipes in addition to three-finger swipes for navigation, and that this might confuse users. The reality is even more complicated than that. For several of the navigation gestures on the "More Gestures" tab (pictured above), the user can select between

• scroll with two fingers,
• swipe with three fingers,
• swipe with two or three fingers, and
• swipe with four fingers.

And changing the setting for any one gesture may change the settings for the others so that they don't conflict.

They've dealt with "Three finger drag" conflicting with three finger navigation by linking that setting to the navigation settings; if you enable Three finger drag then you have to use two or four finger gestures for navigation.

There's some inconsistency in the terms that Apple is using for these gestures: there is both "scroll" and "swipe" for the options for Swipe between pages. In Safari, "scroll" left or right seems to mean you get the continuous sliding back and forth for navigation. If you choose "swipe" it's a discrete action and you don't get a preview as with "scroll." For the next setting, however, (Swipe between full-screen apps) the options are called "swipe" yet they give you the smooth version, so I would have thought they'd be called "scroll."

I wondered how two-finger scrolling and two-finger navigation would get along. For example, if you are zoomed in on a page in Safari, will two-finger scroll pan around the page or navigate forward and back? Apple's solution seems to work well: it pans/scrolls until you get to the edge of the page, then stops. If you start another gesture, it then does navigation.

More on "Scroll Direction: Natural" and navigation gestures

The scrolling direction applies not only to two-finger scrolling/panning, but also to the continuous swiping (or scrolling) for navigation between pages or full-screen apps. It does not, however, change the direction of the discrete swiping gestures for navigation. This can lead to some odd configurations, such as with the Swipe between pages settings (pictured above). Scroll direction applies to two-finger scrolling/swiping but not the three-finger swiping. So if you select "Swipe with two or three fingers," then

• if Scroll direction: natural is checked, swiping two fingers right will go back, but swiping three fingers right will go forward,
• if scroll direction: natural is unchecked, swiping two fingers right will go back, and swiping three fingers right will also go back.

That this mismatch between the direction of continuous vs. discrete navigation gestures is allowed seems odd, but I don't know if it will bother users, or just those who go digging, like me. Users may never master all these gesture combinations and instead just stick with one or two that they're familiar with.

I suspect that the reason there is an option for both two and three fingers (and thus the mismatch) is because the two finger scroll/swipe is only supported in Apple programs for now, so if you use Firefox, for example, you need that option to continue using three-finger swipes.

So many choices

If the above has confused you, then I suspect it's not just my confusing writing -- Apple has given users a surprising number of choices for configuring their trackpad in Lion. The settings page itself is much bigger, with three tabs instead of the previous single-page layout. And because of the dependencies I noted above, changing one setting might change other settings automatically, even settings on another tab.

What's missing?

Despite the more complicated settings panel, there are a few controls missing that were present in Snow Leopard (pictured below).

At first I thought that double-click speed, scrolling speed, tap-to-drag, drag lock, and the option to turn off scrolling inertia were all gone, but it turns out they're now hidden away in the Universal Access settings. You have to select the "Mouse & Trackpad" tab and then select Trackpad Options (pictured below). That's a fairly non-obvious place for them to be, but I suppose it's good to have them somewhere. The one that I'm most surprised is hidden is the tap-to-drag option, and it appears that it defaults to off when you install Lion.

The only feature that really appears to be gone for good is the old-style screen zoom, which worked by holding the Control key and scrolling. That's now replaced with Smart zoom using two-finger double taps.

Other observations

I've written mostly about the changes to features, but there are of course a couple of new ones: Launchpad is done with a pinch with thumb and three fingers, Show desktop is done with a spread with thumb and three fingers. These are a bit awkward to perform but seem to work well enough.

As far as I can tell there are no other changes to the recognition or responsiveness of other gestures like pinch and rotate. Likewise, "resting finger" support on the trackpad appears to work the same as before. (You can rest your thumb at the bottom of the pad, for example, and point or scroll with your fingers -- the system ignores your thumb in this case. Resting finger support is one of the most technically challenging aspects of clickpads.)

If you have a Magic Mouse, you'll notice that Apple has increased the choices there as well, but that's a different post.

eBook quoting and highlighting

This may be old news, but I was surprised to discover when reading an ebook yesterday that there was no simple way to copy a snippet (and, say, paste it into a twitter app). Selecting the text in either the Kindle or iBooks app on an iPhone or iPad doesn't give you the usual option to copy it (see image below).

I assume this is due to a concern over copyright violation, but that seems a bit extreme as it makes it difficult to copy even small snippets for note-taking or other writing tasks (and I'm all in favor of reasonable copyright protection). It's absurd that it's easier for me to tweet a sentence from a paper book to an iDevice than it is to tweet from an ebook on that same device.

The second thing I noticed was that highlighting a passage -- a feature all the e-readers advertise -- has a huge usability issue. You can't highlight a passage that goes over a page boundary. In the situation pictured above I tried all the gestures I could think of to move the marker onto the next page while still highlighting, but nothing worked. I then tried decreasing the text size to make everything fit on one page. That still left a page break, but after a number of increase/decrease text operations the pagination magically shifted so that I got the passage I wanted onto one pagein order to highlight.

I'm surprised that this problem wasn't uncovered and fixed before release -- any usability test of the highlighting feature would likely have found it. I was using the Kindle app but I found the same problem in iBooks, on both the iPhone and iPad.

(The book is Adrenaline Junkies and Template Zombies: Understanding Patterns of Project Behavior, by Tom DeMarco and others, and it's a good read.)

iPad Nudge Gestures

An interesting post at the Omni blog about some new gestures added to their OmniGraphSketcher app: A Nudge Forward.

UPA 2011

I'm attending UPA 2011 this week and looking forward to learning stuff and meeting other usability folks. I'm especially interested in chatting about touch and hardware usability with anyone else so inclined -- please look me up!

Trackpad Gestures in Mac OS X Lion

The forthcoming OS X Lion has a few trackpad gesture changes. Here is a brief rundown of what looks interesting to me from a UX point of view. This is based on the WWDC presentation and Apple's web site (this page in particular). I'll update this after I've tried Lion myself. Update: Please see this follow-up post.

Reverse scrolling direction

Apple has reversed the vertical scrolling direction so that moving your fingers down on the trackpad will move the screen content down, rather than up, as currently happens. This is a change in the metaphor from "grabbing the scrollbar" to "grabbing the content." The latter metaphor is what Apple has used on the iPad and iPhone, which are direct-touch devices where manipulating the content directly is much more intuitive. Overall, Apple is making the notebook experience closer to the touchscreen experience so this change makes sense (and the scrollbar now hides so this aligns with that change too).

Apple is highlighting this new metaphor on their website:

"Multi-Touch gestures in OS X Lion make it feel as though you’re controlling your content more directly than ever before. So when you scroll down on your trackpad or Magic Mouse, your document scrolls down. When you scroll up, your web page scrolls up. When you swipe left, your photos move left."

Question: Will people get used to this? In the Lion previews there was a checkbox to toggle the direction -- is it still there?

Related: a post at the Human Factors blog about this topic: Scroll direction, touch screens, trackpads.

More fluid responses and animations -- "Gestures that feel real"

Again a carry-over of features like rubber-banding from iOS. Apple has done a better job than Windows PC makers to make their gestures respond smoothly and to use animations. This is no doubt partly a result of Apple's tighter control over applications and OS features. The touchpad gesture experience on Windows is influenced by at least three parties: Microsoft, the OEM, and the touchpad maker. As a result it's more difficult to refine the user experience of features like this.

Two-finger double-tap to zoom

This is another feature that will be familiar to iOS users. I'm curious whether this replaces or supplements the Control-scroll feature for screen zooming. And will the "intelligent" zooming feel right on a big screen? Zooming in to a tiny column of text could be a visually jarring.

In Snow Leopard, two-finger tap could be configured to do a secondary (right) click to bring up a context menu. Is this option still available in Lion? If so I'm curious how they got the two to co-exist well. One option would be to delay the single-tap response in order to determine if the user will do a double-tap; the double-tap action (zoom) would then cancel the single-tap action (context menu).

Different navigation gestures and their implications

Before Lion, three-finger swipe left or right meant navigate backwards or forward. In Lion there are two navigation gestures:

• Three-finger swipe left or right to move between full-screen apps.
• Two-finger swipe left or right to move between "web pages, documents, and more."

So two-finger swipe now seems like the main navigation gesture, and three-finger swipe is for the special case of full-screen applications. This raises a few questions about the usability:

• Will users keep this straight or will they end up swiping two fingers when they meant three and vice-versa? What will happen if they make this mistake? Hopefully nothing too disruptive.
• Would a modifier key have been more intuitive? For example -- swipe two fingers to navigate within the application; hold Control and swipe two fingers to navigate between applications.
• In Snow Leopard, Apple introduced the option of using three fingers for dragging instead of swiping. Is this option still there? I thought this was quite an interesting change because it promised to make dragging simpler and perhaps faster than the old, awkward "tap and a half" gesture. I tried it for a while and liked it, but I missed the three-finger navigation too much to stick with it.
• How well do two-finger swipe and two-finger horizontal scroll get along? Does the gesture engine do a good job of disambiguating between the two? Or is horizontal scrolling now limited or gone altogether? (Limited in the sense of perhaps having to start a scroll gesture vertically before going horizontally.)
• What about two-finger swipe and two-finger vertical scroll? One of the great things about Apple's current two-finger scroll is that if you're in a situation where only vertical and not horizontal scrolling is possible, you can be very relaxed about the angle at which you move your fingers to scroll. And people do often scroll vertically with their fingers moving at an angle of about 30 degrees above horizontal. Will some of these strokes now get mistaken for swipes? (And will some swipes be mistaken for scrolls?) That's the cost of introducing a new gesture into the same parameter space (in this case the space of two-finger strokes) -- the recognizer has to be more sophisticated about discriminating between gestures.

With all these questions I don't mean to doubt the usability of what Apple has done -- as I said I haven't even tried Lion yet. I assume they've done their usual great job. My intention here was just to highlight some of the usability questions I'd look to answer when implementing and testing these features.

iPad Usability Report from NN Group

Raluca Budui and Jakob Nielsen of the Nielsen Norman Group have published a report from a very large usability study of iPad applications. This is a sequel to a report from a year ago.

The summary is here, with a link to the full report at the bottom: iPad Usability: Year One.

Brave NUI World

Brave NUI World: Designing Natural Interfaces for Touch and Gesture is a new book out later this month by Daniel Wigdor and Dennis Wixon. I've read it and recommend the book for anyone working in the area of gestures and touch. The book contains perhaps more detail than any other so far about how to design touch and gesture interactions that work well.

(And, yes, I find the title a little cringe-worthy.)

Activity Zones for Touchscreens

Dan Saffer has made some nice illustrations of "activity zones" on touchscreen phones and tablets, based on which areas are easiest to reach given normal ways of holding the devices. Put frequent actions in the "easy" zones and infrequent or dangerous ones in the "reach" zones.

See his post for the rest of the drawings: Activity Zones for Touchscreen Tablets and Phones (Kicker Studio).

For some research that validates this idea of zones, I'd recommend the publications from Amy Karlson of Microsoft Research. She has done a lot of research on one-handed thumb use of mobile phones in particular.