Next Billions - IBM Picture Discussions

Engaging citizens in developing regions in information creation and information sharing leverages people's existing social networks to facilitate transmission of critical information, exchange of ideas, and distributed problem solving. In this project, we are exploring how multisensory modes of human interaction using mobile phones can support these activities. For example, photos, drawings, and graphics can provide context for and serve to organize aural information and conversely, spoken comments can guide users to appropriate parts of complex visual information.

IBM Picture Discussions is a mobile computing application framework that supports asynchronous conversation anchored by pictures. We are exploring several issues in the design of mobile social computing interfaces, including how to create visual representations of digital speech that afford effective interactions between mobile phone users, how to coordinate peer production of content, and how to organize question and answer exchanges.

The use of pictures and sound affords the possibility of communicating without text, and thus reaching users who for a variety of reasons — lack of access to computers, insufficient literacy, or disabilities — have been unable to benefit from information technology. IBM Picture Discussions augments speech with three types of visual component: comment proxies, pictorial contexts and visual controls. Comment proxies are visual representations of digital speech that depict various types of meta-information, such as the identity of the speaker and the length of the comment. Pictorial contexts are diagrams or photographs that provide a background for a particular conversation; pictorial contexts serve both to represent the conversation as a whole, and allow comment proxies to take on additional meaning by virtue of their location with respect to the pictorial background. Visual controls are visual user interface components for controlling the system, for example, a message play button.

We have implemented IBM Picture Discussions prototypes on several devices: Web browsers, Android phones, and camera phones connected to an interactive voice response system. Each client connects to a centralized Web service that stores photos, audio clips, and links between them.

  • The Android phone implementation of IBM Picture Discussions leverages the phone's touch screen, high resolution display, and built-in microphone and speaker.
  • The camera phone implementation of IBM Picture Discussions accepts picture messages sent over the telecommunications network using the Multimedia Messaging Service (MMS) and supports audio posting through the voice channel. Users can be notified of new discussion postings by Short Message Service (SMS).
  • The Web implementation uses Flash Media Server to create a rich internet application able to display multiple discussions at once.

IBM Picture Discussions's Web service transcodes photos and audio recordings to formats appropriate for each device, enabling cross-device discussion. For example, one user can take a picture on a camera phone and call the IBM Picture Discussions phone number to provide a spoken introduction to this picture, then another user might discover this photo on their Android phone's high resolution display and tap on the picture to respond to the first user's posting. The limitations of using mobile devices to access rich structured multimedia content by users with limited literacy skills exposes human-computer interaction challenges that are key to enabling broad access to information by the coming billions of information technology users.

Associated Teams

Social Computing Group