Touch Usability

Article about iPhone's new accessibility features

A great article about the little-advertised but very impressive accessibility features Apple has added to the new iPhone: My first experience using an accessible touch screen device (Marco's Accessibility Blog).

Marko writes mostly about the new voice over features. There is also a zoom feature and white-on-black color mode.

Kudos to Apple for putting serious effort into accessibility and (again) being way ahead of the pack.

ThinkPad W700ds with dual touch and related thoughts

Darren Rowse at the site Digital Photography School posted a review of Lenovo's ThinkPad W700ds notebook. You may not recognize the model number but you've probably seen a photo of it -- it's the beast with two screens and a built-in Wacom digitizing tablet.

Link: Lenovo ThinkPad W700ds review.

I haven't tried this machine but I've been curious about what users think of it so I enjoyed reading Darren's review. It prompted a few thoughts about touchpad size and the palm-rest area:

"the touchpad really is small"

Yesterday's touchpads are small. The one on the W700ds looks like a standard pre-2008 (or so) touchpad that measures a little under 60 mm wide by 40 mm high. MacBooks now have trackpads that are about 100 mm x 75 mm and even Windows machines are getting up there, like Lenovo's Ideapad Y650 which has a 104 mm x 64 mm touchpad. (Shown below with home keys and touchpad highlighted -- ignore the orange lines please. The images are from Lenovo's site)

  

Bigger is better for any kind of multifinger gestures and people seem to like the extra space for pointing as well. But it has its disadvantages. The biggest problem is that you're much more likely to accidentally touch it when you're typing. This is especially likely if the touchpad is not centered to the keyboard. Unfortunately the Y650 is centered to the palmrest. That may be more aesthetically pleasing, but it means your right hand very often brushes the touchpad. When it does the cursor moves and scrolling gets activated if you brush the scroll-zone on the right side. (Touchpad drivers generally have some algorithms built in to filter out accidental touches but they're not perfect.)

Back to the W700ds, which has a Wacom tablet right where you might rest your hand. It isn't sensitive to finger input so there's no risk of accidental input, but users may still not like it. Darren writes:

"at times it felt ‘wrong’ to have my right wrist leaning against it as I typed as it is placed directly in front of the keyboard area."

That observation makes me wonder whether we shouldn't just leave the palmrests for the palms. Nevertheless, I'm very interested in what you could do with huge touchpads covering the palmrests. (And lots of other people have thought about this previously too.) Here's a mockup:



To my thinking there is a benefit in dividing the touchpad into two regions, one for each hand. The dividing line could either be real or imaginary -- there could just be two touchpads side-by-side or there could be one wide touchpad with a virtual invisible dividing line.

Two-handed input has been the subject of a lot of HCI research that argues that we'd have better user interfaces if we did a better job of balancing the interactions across both hands (see for example Mackenzie and Guiard, 2001).

Much of that research builds on Yves Guiard's kinematic chain model that says (approximately) that people often use their two hands in concert like two links in a chain. The dominant hand is like the bottom link and does finer manipulations, while the non-dominant hand is like the top link and frames what the other hand does. A common illustration of this is handwriting: people tend to use both hands when writing on paper -- one hand to write and the other to continually reposition the paper.

There are some possible dual-touchpad interaction techniques that are analogous to this. For example, you might have a mode where the left touchpad allows scrolling and panning while the right allows pointing (for right-handed users). Bill Buxton and others have shown that even though users are still not likely to scroll and point in parallel, the interface is more efficient because of a cognitive benefit in splitting the tasks across the hands (see Leganchuk et al., 1998).

Another asymmetric technique that you could implement on dual touchpads is the Toolglass technique (see Bier et al., 1993) where the left hand positions a toolbox and the right hand selects tools from it.

Some gestures also might work better with two hands instead of one (see Moscovich 2008 for an experiment on this). Scaling an image is likely to be done more precisely with two index fingers than with the thumb and finger on one hand.

I'm hoping to set up a dual-touchpad prototype and test some of these ideas. I welcome any comments... Would you buy a laptop with two touchpads?

Mind-Mapping on a Touch Surface

Mindflow is a student project by Toni Da Luz that supports collaborative creation of mind-maps.

Toni's site has a couple of nice video demonstrations, including this one that shows the idea in animated paper prototype form. The narration is in French but it's still fairly easy to follow if you're like me and don't remember any French from school.

mindflow [work in progress] from Toni Da Luz on Vimeo.

Evaluating Gesture Usability

[This is a rough draft/work in progress. Comments/feedback are welcome!]

There is some great work being done on how to design gestural interfaces but we don't see much work focused on how to evaluate their usability. Having reliable and repeatable evaluation techniques for gestures is important. Without them we can't tell developers if their system is getting better or compare two alternative implementations or products.

The following is a discussion of general issues to do with testing gestures for usability, followed by a description of specific test procedures and my experiences with them.

Gesture platforms
I've been working mostly with gestures on touchpads, so that's my focus here. Many of the issues are the same for touchscreens and some are the same for free-form (in the air) gestures.

On a touchpad there's not necessarily a direct one-to-one mapping between your finger motions and the motion of things on the screen (in contrast to gestures on a large touchscreen, for example). I'm talking about gestures like pinch, swipe, and rotate, but also more abstract ones like drawing an infinity symbol to launch your favorite application.

Automated evaluation and learning algorithms
First off, I should say that I'm not talking about automated gesture testing with a dataset. Gestures are often implemented using machine learning algorithms. The developer can train and also evaluate the system using a very large dataset of gesture input that has either been recorded or simulated. The evaluation will produce percentages of correct and incorrect gesture recognition. This kind of automated testing should ideally be done as part of algorithm development and I'm assuming it's already been done.

You don't always need to implement gestures using learning algorithms. For simpler and smaller sets of gestures it might be easier to use heuristics in hand-written software. In that case it may still be useful to have prerecorded gesture input for evaluating the algorithms in a consistent way over time.

But I said I wasn't going to talk about automated evaluation... Assume the developers have done their job and you're a usability person asked to evaluate the gestures and give feedback. Maybe the developers have a dataset of gesture input data but they're not sharing it with you.

Evaluation scenarios
Here are some of the scenarios for when a usability test of gestures is useful:

• Formative usability tests to help developers know if and how their algorithms are improving or not. These can also evaluate the gestures within a realistic context -- evaluate whether the gestures work well in specified applications, whether the feedback and actions performed by the gestures are working appropriately.
• Summative and competitive tests. Document the performance of the gestures for future reference. Compare the gestures on system A and system B, where the systems might have different hardware, firmware, or software, may be running different end-user applications, may have touchpads of different sizes or shapes, have different surface textures or different bezels. There are many variations you could compare.

What's different about gestures
Some aspects of gestures make them different from testing other user interfaces:

• Gestures are inter-related. If your system has a single two-finger gesture, say pinch, then it's easier to make it work better than if it has two two-finger gestures, like pinch and rotate, or more. The power to discriminate between gestures gets smaller the more gestures you have in the same effective parameter space. This means that you can't test gestures in isolation.
• Gesture interfaces typically don't have affordances. This means they must be taught. The user has to be prompted in some way to know to perform the gesture. Perhaps they looked at some documentation, watched a demo video, learned gestures on another system, or were just told that the machine had a certain gesture. The way people are instructed on gestures will have a major impact on how effectively they can use them.
• Gestures don't just need to work -- they need to not work when they're not supposed to. It should be difficult for a user to accidentally trigger a gesture.
• Variation among users. Testing a variety of users isn't unusual, but for touch gestures things like finger size and fingernails can make a very big difference so it's important that the test participants are representative.

Research questions

The main questions that gesture usability tests might try to answer include the following:

1. Are gestures better than alternative (and existing) interaction techniques?
   
2. How accurate is each gesture (on each platform)? What are the rates of correct and incorrect recognition? What are the failure cases like?
3. How satisfying are the gestures to use? (and other aspects of subjective responses -- how comfortable, tiring, responsive do they seem to users?)
4. How effectively do people learn the gestures? How much, if any, training or documentation is needed?
5. What problems do people encounter with using the gestures? (development feedback)

Introducing gestures and testing the documentation

The documentation for gestures (or lack of it) may have a big impact on how well users can perform them. The documentation should itself be usability tested. Bring in users unfamiliar with the gestures and simulate the introduction that a typical customer might get to the gestures. Specifically, have them read the help text, watch the help video, or do neither. Then have them try each gesture. Record whether users can perform the gestures, or if not, how many need more instruction.

Part 2 is here.

Workshop on Touch Affordances at Interact 2009

The Workshop on Touch Affordances is being held as part of the Interact 2009 conference on human-computer interaction in Uppsala, Sweden on August 25. Here is the call for participation:

Interfaces involving touch-based interactions (e.g. (multi)touch screens, NFC/RFID, ubiquitous interfaces) require very simple user actions. However, such interfaces often are unfamiliar to users and therefore do not trigger correct interactions. This leads to the question of how to design simple, intuitive touch-based interfaces. In most cases the interaction is made clear by adding (graphic) signage. However, in this workshop, the concept of touch affordances will be approached from a broader perspective.

The goals of this workshop are to:
•    start up a community around this topic
•    create a common understanding of touch affordances
•    generate new research ideas

The creative, generative workshop will have four sessions:
1.    Keynote presentation and wrap-up of position papers
2.    Introduction to creative part of the workshop and a brainstorm exercise.
3.    Group exercise: developing low-fi touch affordance prototypes
4.    Prototype testing

Anyone interested in touch-based interfaces and interaction is invited to participate. Participants are asked to submit a position paper (about 4 pages) about their work or vision related to the field of touch affordances. Examples of relevant issues are:
•    Methods to communicate touch-based interaction in an intuitive, implicit way
•    Success stories of intuitive touch interfaces
•    The value of “touch” with regard to user perception
•    The impact of a touch-related interface on user experience

Link: Touch Affordances.

The deadline for submitting position papers is June 15.

Scalable Multi-Touch Prototype from Moto Labs

Cnet has a story on a new large "moderate cost" capacitive touchscreen prototype from Motorola: Moto Labs screens interactive display concept (via @kickerstudio). Video embedded below...

There's more about it on the Moto Labs blog.

Touchscreen Pop-Up Buttons

Research from CMU's Human-Computer Interaction Institute featured in Technology Review:

Touch-screen technology has become wildly popular, thanks to smart phones designed for nimble fingers. But most touch screens have a major drawback: you need to keep a close eye on the screen as you tap, to make sure that you hit the right virtual buttons. As touch screens become more popular in other contexts, such as in-car navigation and entertainment systems, this lack of sensory feedback could become a dangerous distraction.

Now researchers at Carnegie Mellon University have developed buttons that pop out from a touch-screen surface. The design retains the dynamic display capabilities of a normal touch screen but can also produce tactile buttons for certain functions.

Graduate student Chris Harrison and computer-science professor Scott Hudson have built a handful of proof-of-concept displays with the morphing buttons. The screens are covered in semitransparent latex, which sits on top of an acrylic plate with shaped holes and an air chamber connected to a pump. When the pump is off, the screen is flat; when it's switched on, the latex forms concave or convex features around the cutouts, depending on negative or positive pressure.

Link: Touch Screens with Pop-Up Buttons (via Engadget)

The research was presented earlier this month at CHI 2009. One of the authors, Chris Harrison, has set up a project page with more details and the CHI paper for download: Providing Dynamically Changeable Physical Buttons on a Visual Display.

Here's a video about the work that they put on YouTube:

It'll be a while before this makes it to your mobile phone, but it's still pretty cool stuff.

Microsoft at IDC 2009

Over on the Microsoft Surface blog they've posted several of the videos that Steven Bathiche and Andy Wilson showed during their keynote presentation at the Interactive Displays Conference last week: Interactive Displays Conference.

They showed several Surface applications as well as pre-Surface research and other interesting bits. They also showed that 2019 Office Labs concept video that was floating around a while back but I never managed to watch. It's worth watching if you haven't yet.

Design for Mobile Conference Summaries

Little Springs Design has posted several excellent detailed summaries from the D4M conference that they hosted this week: see their blog or the shortcuts at the top of the conference site: Design for Mobile.

Interactive Displays Conference Highlights

Over the next week or so I'm going to be posting some highlights from the IDC 2009 conference that's happening today and tomorrow in San Jose. The conference has brought together a slew of big-name exhibitors and speakers on display and touch technology.

Today's highlight without a doubt was the keynote talk by Jeff Han of Perceptive Pixel. He talked about some of the graphics and HCI research he did before he hit multitouch demo fame, as well as some of his inspirations. He showed a great clip from an early 80s TV show called Bits and Bytes that featured a young Bill Buxton demonstrating some of his tablet work at the University of Toronto. Buxton was showing touch as a way to control music, and the clip included free-form interaction on a tablet as well as touching through a template that set out some virtual sliders and buttons. Unfortunately I can't find that clip online, but I did find this 1982 clip on Buxton's web site in which he demonstrates what looks like the same system: Templates on Touch Tablets to Support Virtual Devices (Flash video, see also this page which has several other videos from the U of T Input Research Group).

Jeff Han's point in showing this clip was not just to share what inspired him as a kid to pursue computer science as a career. He also wanted to make the point that none of this stuff is really new. He urged the audience not to jump on whatever tech is cool this week, but to be aware of the history and to do the research. Be thoughtful and careful about what you're making. He said one of his fears with the multitouch craze is that the waters will be poisoned by bad and poorly conceived implementations. He said "don't add noise" to the ecosystem by using terms sloppily -- like "multitouch".

The importance of being more thoughtful and mindful of prior work are not exactly new to most of us with design, HCI, or CS backgrounds, but the audience here is largely made up of marketing or other business types, I believe, who sometimes tend to get a bit carried away, you might say. I mean no disrespect to my friends in marketing...

(Note also that this is all just my paraphrasing based on some notes, and I seem to be making it sound he was scolding the audience... it wasn't like that. It's possible I've recalled some things incorrectly so please keep that in mind.)

Han also showed some impressive work that Perceptive Pixel has done to create applications for medical visualization and other things in collaboration with academic and industry colleagues. Of course he also showed one of those whizbang dizzy-fast multitouch demo reels (you know where to find them). It was nice to see real, useful applications on big multitouch screens, though.

By the way, Bits and Bytes was a wonderful, goofy show that I too watched as a kid. You can find some clips on YouTube if you search, and some screenshots and background about it here: Bits and Bytes.

I'll post more conference recaps and observations in the days ahead.

Minority Report quote of the day: "It's sort of like Minority Report, I'm afraid." - Brad Gleeson of TargetPath Global, acknowledging the downside of new digital signs that record information about the people who look at them.