OCR: IBM, Carnegie Mellon and Oregon State     


OCR: IBM, Carnegie Mellon and Oregon State - overview

IBM, CMU, Oregon State research predictive usability tool

An IBM Open Collaborative Research project that began by predicting how long an ideal simulated expert user takes to perform a given user interface task is now exploring new usability projects.

cogtoolPeople are good at telling you if the streusel-topped blueberry muffin recipe you followed to the very last teaspoon of salt actually performs the way you thought it would.

Maybe they're not so good at helping you figure out if the code you wrote for a widget does what you want it to do in the shortest time possible. The right testers, for example, might be hard to round up. You might have to pay them for their time. And when they finally point out a flaw in the widget's usability, you realize you've got to go back to the drawing board. Rethinking your code is going to be expensive, time-consuming and really frustrating.

Enter Rachel Bellamy, an IBM researcher who is working on an IBM Open Collaborative Research (OCR) project with computer scientists from Carnegie Mellon University and Oregon State University to further develop CogTool, an open-source usability-testing tool. CogTool is the brainchild of CMU researcher Bonnie John, who joined the OCR team to help IBM researchers predict how long an ideal simulated expert user would take to perform a task on any given user interface (UI). Now OCR participants are extending the tool to include programming and screen reader use.

The OCR team, for example, is testing the usability of parallel programming Eclipse tools, including the parallel debugger and the X10 development tools. Team members are also working with Rational and Lotus design partners to demonstrate the value of automated usability assessment during a customer engagement.

Research directions for CogTool

CogTool already has proved useful in mocking up a variety of user interfaces. Now IBM researchers and their university collaborators are looking into other potential uses. Among the possibilities:

Aviation. Researchers could use CogTool to create several iterations of a model of an airplane cockpit task. Each iteration intentionally would produce errors that point to additional theory or device knowledge. This theory and knowledge then would be incorporated into the next iteration by modifying the mock-up of the device — not by changing the implementation of the underlying cognitive engine. Using CogTool to employ this "rapid theory prototyping" technique would give researchers a platform to rapidly explore theory and knowledge requirements in the aviation domain. Read more

Customer Relationship Management. University researchers could conduct studies to help them better understand how users interact with a set of online customer relationship management products. One study might explore a methodology that combines expert and novice performance data in an effort to measure intuitiveness. Another might create a methodology that combines verbal and nonverbal cues to gain an understanding of the emotional effect these products have on users. CogTool could make predictions about expert performance, which the CRM company could use as a new measure of intuitiveness. Read more

Mobile phones. A university-industry collaboration could research a "cognitive crash test dummy" that perceives, learns, feels, moves and makes mistakes like a human being so that consumer product companies can test design ideas as they emerge in the engineering lab. Such a cognitive model could be valuable in the consumer product domain to predict human behavior in using mobile phones, for example. CogTool could make predictions of skilled execution time and novice exploration behavior and compare these predictions against the way people actually perform mobile phone tasks. After verifying the trustworthiness of these predictions, researchers could determine how mobile phone designers can use CogTool most effectively to save money and time during the design process. Read more

"The advantages of using something like CogTool should be obvious," says OCR participant Rachel Bellamy. "And yet it remains a challenge to get people to think through the details of the tasks their application will support from the very start. We'd like people to understand that beginning with a detailed description of a UI task can end up saving developers time and money."

As with all OCR projects, the usability testing collaboration with CMU and Oregon State will yield open source code.

Download Cogtool

Last updated on June 14, 2010

The theory behind CogTool

The psychological theory that underlies CogTool's predictions of skilled execution time is called the Keystroke-Level Model, created by Stu Card, Tom Moran and Allen Newell in the 1980s. It has been validated through decades of research and documented in over 100 research publications.

The predictions can be trusted to be within 20% of what you could measure if you trained users in your new designs, practiced them until they were skilled, and ran controlled time-on-task studies.

CogTool has increased the accuracy of the KLM because it applies the theory more consistently through its "modeling by demonstration" approach, and has been reported to be within about 10% of empirical data. Use Cogtool today