Touch Usability

The Hand by Frank Wilson

David Birnbaum of Immersion has posted a good review of Frank Wilson's 1999 book The Hand: How its use shapes the brain, language, and human culture, a book that's going on my to-read list... Excerpt:

The Hand by Frank Wilson is a rare treat. It runs the gamut from anthropology (both the cultural and evolutionary varieties), to psychology, to biography. Wilson interviews an auto mechanic, a pupeteer, a surgeon, a physical therapist, a rock climber, a magician, and others—all with the goal of understanding the extent to which the human hand defines humanness.
Wilson is a neurologist who works with musicians who have been afflicted with debilitating chronic hand pain. As he writes about his many interviews, a few themes emerge that are especially relevant to my interests.

Link: The Hand (tactilicio.us)

Limitations of touch, and touch input vs. output

There's a good post and video by Steven Hoober over at the Little Springs Design blog about how current touch interfaces are very impoverished compared to our tactile experience in the real world. Link: Humans are tool users.

I left a comment over there but I don't think it's up yet. I agree with Steven's points for the most part, but I'm not quite as pessimistic about touch input without haptics -- I think it works pretty well if it's designed well and there is good visual feedback.

Touchscreen Pop-Up Buttons

Research from CMU's Human-Computer Interaction Institute featured in Technology Review:

Touch-screen technology has become wildly popular, thanks to smart phones designed for nimble fingers. But most touch screens have a major drawback: you need to keep a close eye on the screen as you tap, to make sure that you hit the right virtual buttons. As touch screens become more popular in other contexts, such as in-car navigation and entertainment systems, this lack of sensory feedback could become a dangerous distraction.

Now researchers at Carnegie Mellon University have developed buttons that pop out from a touch-screen surface. The design retains the dynamic display capabilities of a normal touch screen but can also produce tactile buttons for certain functions.

Graduate student Chris Harrison and computer-science professor Scott Hudson have built a handful of proof-of-concept displays with the morphing buttons. The screens are covered in semitransparent latex, which sits on top of an acrylic plate with shaped holes and an air chamber connected to a pump. When the pump is off, the screen is flat; when it's switched on, the latex forms concave or convex features around the cutouts, depending on negative or positive pressure.

Link: Touch Screens with Pop-Up Buttons (via Engadget)

The research was presented earlier this month at CHI 2009. One of the authors, Chris Harrison, has set up a project page with more details and the CHI paper for download: Providing Dynamically Changeable Physical Buttons on a Visual Display.

Here's a video about the work that they put on YouTube:

It'll be a while before this makes it to your mobile phone, but it's still pretty cool stuff.

Tactile Motion Aftereffect

This doesn't have anything to do with user interfaces, but it's kind of an interesting finding about tactile perception:

In the classic waterfall illusion, if you stare at the downward motion of a waterfall for some period of time, stationary objects -- such as rocks -- appear to drift upward. MIT neuroscientists have found that this phenomenon, called motion aftereffect, occurs not only in our visual perception but also in our tactile perception, and that these senses actually influence one another. Put another way, how you feel the world can actually change how you see it -- and vice versa.

In a paper published in the April 9 online issue of Current Biology, researchers found that people who were exposed to visual motion in a given direction perceived tactile motion in the opposite direction. Conversely, tactile motion in one direction gave rise to the illusion of visual motion in the opposite direction.

"Our discovery suggests that the sensory processing of visual and tactile motion use overlapping neural circuits," explained Christopher Moore of the McGovern Institute for Brain Research at MIT and senior author of the paper. "The way something looks or feels can be influenced by a stimulus in the other sensory modality."

[...]

To test the effect of tactile motion on visual perception, adjacent rows of pins vibrated in rapid succession, creating the sensation of a tactile object sweeping up or down the subjects' fingertips. After 10 seconds of this stimulus, the monitor displayed a static pattern of horizontal stripes. Contrary to the prevailing assumption that vision always trumps touch, subjects perceived the stripes as moving in the opposite direction to the moving tactile stimulus.

Demos of the motion stimuli used in this study can be seen at http://web.mit.edu/~tkonkle/www/CrossmodalMAE.html.

Link: MIT News: How you feel the world impacts how you see it.

Interactive Displays Conference

Interactive Displays is a technology conference happening April 21-23 here in San Jose, CA. It's featuring some big name speakers like Perceptive Pixel's Jeff Han and Microsoft's Steven Bathiche as well as lots of exhibitors. From the site:

Interactive Displays 2009 will bring together the interactivity community to discover advances in technologies from multi-touch to holography, and haptics to gestures, offering a collaborative environment in which to tackle the challenges and hurdles of development and integration.

Interactive Displays 2009 is the must attend event for developers and end-users of interactive display technologies, along with the manufacturers that are utilizing, seeking or integrating those emerging technologies.

Link: Interactive Displays.

Lynn Marentette has gathered some additional links to related content: Interactive Displays 2009.

Future directions for tactile feedback technology

Peter Odum at the blog Idlemode has a good round-up of various existing technologies and future concepts for adding tactile feedback to touchscreens. His list: physical overlays, air or liquid filled bladders, piezoelectric mechanisms, pressable mechanical touchscreen, the band-aid display, nanotech and ultrasound. From his introduction:

A variety of devices currently on the market provide simple forms of touch feedback, but none is an unqualified success - they all lack some aspect of physical experience, a correspondence with the way we actually interact with the world. Current tactile solutions fall short either in reconfigurability or in pre-interaction feedback. This pre-interaction feedback would provide the physical feeling of a button which the user can press or not, rather than just a tactile confirmation that they have just pressed that button. Reconfigurability would allow physically felt controls to change with the content of the display. In short, it’s easy to make static physical buttons, but not to make them disappear when not needed. And it’s easy to provide a physical sensation after the user interacts, but not to provide buttons that can be physically felt *before* the interaction is committed.

Link: Future Directions for Tactile Feedback.

Three-state touch -- Now Apple has it too!

Apple's new MacBooks and MacBook Pros implement the same idea RIM implemented in the Storm (see previous post). They made the entire touch surface a button. From Apple:

It all just clicks.
The first thing you might notice — or not notice — is the missing click button. Now the entire trackpad is the button, so you can click anywhere. Without a separate button, your hands have 39 percent more room to move on the large, silky glass surface.

The BlackBerry Storm's Three-State Touchscreen

RIM has done something very interesting with the touchscreen on their new BlackBerry Storm. As Phone Scoop describes it (and everything I write here is based on media reports -- I have no insider knowledge):

The big news about the display is the touch technology. First, it's capacitive, like the iPhone and the HTC G1. This alone is great news, but the real innovation is the true tactile feedback. The whole screen is essentially one giant physical button that you can press down for a gratifyingly real "click" action. There are no haptics or funny tricks here; just good ol' tactile feedback, and it works very well.

As you would expect, the click action lends a great deal to the experience of pressing virtual buttons and text entry. However, it also enables a whole new dimension (literally) in touch interaction. Capacitive touch technology requires no pressure; it responds to the very lightest touch. This means the Storm has two distinct ways to press the screen. A light touch is just a "touch", while a more forceful press results in a "click". This makes the Storm the first phone we know of with what could be considered a "3D" touch screen.

Link: Hands-On: BlackBerry Storm (see also RIM's official site and Gizmodo.)

The part above that I highlighted in italics is what I find most interesting. From an HCI-theory perspective, what RIM has done is transform the touchscreen from a two-state to a three-state input device, according to Bill Buxton's three-state model of interaction. (I am assuming that you can drag your finger while pressing on the Storm; if you can't do that then I'm wrong and it's not a three-state device.) That model is described in Buxton's 1990 paper from the Interact conference: A three-state model of graphical input (pdf). (He also describes it in chapter 4 of his more recent book-in-progress on input devices.)

The three states in Buxton's model can be described roughly as "out of range," "tracking," and "dragging," and he shows how this state transition model is very useful in understanding the differences between all sorts of input devices, including mice, joysticks, tablets, and touchscreens. The picture below (copied from his paper) shows the three states as they apply to a graphics tablet with a stylus, which is a three-state device.

 

In comparison, a basic mouse provides only states 1 and 2, a touchpad provides states 0 and 1, and a touchscreen provides states 0 and 2 (for the reasons why please read the Buxton paper). For touch devices you need to add more complexity to get the extra state transitions needed for more complex actions. This is done typically by adding external buttons or recognizing actions like double taps or tap-and-hold, and usually these additions aren't very satisfying or intuitive.

RIM isn't the first to extend touch interfaces this way. Scott Mackenzie and Aleks Oniszczak published a paper at CHI 1998 describing a very similar technique applied to a touchpad (see A Comparison of Three Selection Techniques for Touchpads). They basically put a button underneath a touchpad so that the device could transition between the three states of Buxton's model in an intuitive way. Pressing harder allows you to drag or rubber-band in a simpler way than pressing an external button or doing a "lift and tap." Their study showed that their "tactile touchpad" gave the best performance of the three methods on a standard Fitts pointing test and people also liked it better. There were some quirks about their prototype that made it less than perfect, but in general the idea was a winner. So why didn't it take off? Why, 10 years later, don't we all have tactile touchpads on our laptops? PC companies and touchpad makers (like, ahem, Synaptics) seem to have missed an opportunity there.

There are similar ideas in the iPod's clickwheel and in touchscreens with selective haptic feedback, but none of those extend the device as much as these two examples do.

Bloggers reporting from MobileHCI

Belen Barros of iQ Blog is at MobileHCI in Amsterdam and has posted some notes from the first day: Tutorial Day at MobileHCI.

Full sets of slides from the tutorials are also available at Albrecht Schmidt's site (he is chair of the tutorial sessions): MobileHCI 2008 Tutorial.  Patrick Baudisch's slides, in particular, have quite a bit of touch-related material.

Nokia Haptikos Tactile Touchscreen Details

A newly revealed patent application from Nokia describes their "haptikos" technology for tactile touchscreens.  According to Engadget:

Basically, the tech uses layers under the touchscreen to allow control of surface roughness and friction. The patent describes a "plurality of closely space voltage controllable protruberances" that can raise or lower based on where the user is touching the screen, resulting in what feels like resistance and tactile feedback. Filled with fluid, the protruberances increase fiction and help users feel like they're actually touching something rather than poking at a plane of glass.

Link: Nokia's Haptikos tactile feedback tech revealed in patent application (Engadget)