Position Paper for the
CHI 97 Basic Research Symposium
(March 22-23, 1997, Atlanta, GA)

Building on the GUI Toolkit Paradigm:
Creating Multimedia System Toolkits

Joseph A. Konstan
University of Minnesota
Department of Computer Science
200 Union Street SE -- Rm 4-192
Minneapolis, MN 55455
E-mail: konstan@cs.umn.edu
URL: http://www.cs.umn.edu/Research/GIMME


Index

Abstract || Keywords || Position Statement || References

Abstract

Project GIMME attempts to apply the lessons of graphical user interface toolkits to the challenge of supporting authors of flexible, interactive multimedia presentations. As we design and implement a toolkit for multimedia creation, we have focused on three areas that we see as interesting research problems. To support creative presentation synchronization, we have developed new models for specifying and implementing inter-stream synchronization. To support creation of new media and media controls, we have invented a stream of commands that supports a full range of playback options. To create interfaces on demand in response to available media streams, we have designed an "embeddable application" interface modelled on user interface widgets.

Keywords

multimedia systems, multimedia toolkits, synchronization, flexible multimedia presentations, command media, multimedia user interfaces.

Position Statement

Until recently, interactive graphical user interfaces were difficult to implement and lacked consistency. The development of GUI toolkits both added consistency to the interface and greatly simplified the implementation process freeing creators of graphical user interfaces to focus on design and usability rather than technical details. Multimedia systems today seem to be ripe for toolkit development. While there are several commercial-quality systems for creating static multimedia presentations, creating interactive flexible presentations is still a technical challenge. The multimedia systems research community would benefit greatly from the presence of standardized toolkits to relieve some of the technical burden and allow authors to focus on creative and usable interaction.

Project GIMME

Project GIMME (Graphical Interfaces for Multi-Media Environments) at the University of Minnesota was founded in an effort to apply lessons learned developing GUI toolkits to the problem of creating multimedia toolkits. There are three key research thrusts in the project: flexible multimedia synchronization, interactive scripting within multimedia applications, and defining the multimedia equivalent of widgets. This position paper discusses our general approach and the tools we are using, and then discusses each of thse thrusts individually.

Approach and Tools

Our approach is heavily application-centered. We have been working on a range of applications, most of them education-focused, where simple media playout is not enough to support the complicated interactive tasks. These applications range from a low-media on-line course in Medical Pharmacology (http://www.med.umn.edu/ISAP) to computer-based physics laboratories to distance learning appliations. In each case, we work with content-area experts (who are also generally pedagogy experts) to explore the range of the possible. As an example, the physicist with whom we worked was initially expecting little more than computer-based movies with some limited simulation capabilities, but was very excited to find that we could create both on-line laboratories and multimedia on-line assessments. We are then responsible for enhancing our multimedia toolkit prototype to meet the new demands. While this is a slow process, we expect it to lead us to a usable toolkit motivated by real applications, rather than an arbitrary set of tools based on a particular research agenda.

To free us from many of the technical details of delivering multimedia content, we use the Berkeley Continuous Media Toolkit ( http://bmrc.berkeley.edu/projects/cmt) which provides media objects to stream, transmit, play, and synchronize audio and video on a basic timeline. CMT is a flexible framework built using the Tcl language and Tk toolkit. It allows us to build interface objects, create new media, and experiment with different synchronization models while using existing infrastructure.

Flexible Multimedia Synchronization

We have found that currently available commercial multimedia systems provide only very limited facilities for composing flexible presentations. Timelines, which define fine-grained synchronization, are prevalent in a variety of systems (including CMT), and they are extremely good for defining a presentation that has only a single sensible temporal playout. More flexible presentations, such as interactive educational applications, often can benefit from more semantically-designed synchronization. Schnepf [below] explores the example of an architecture lesson where narrative and background music are tied only to groups of slides from architectural genres, allowing the student to navigate within a single genre without interrupting the narrative description.

Our first effort towards supporting flexible synchronization was the FLIPS synchronization model, which uses the abstractions of barriers and enablers to provide greater expressiveness and flexibility. FLIPS was implemented on both Unix and PC/Windows platforms along with the slideshow application.

Our current effort is n-Synch (formerly SynchMe) a synchronization toolkit that uses data propagation constraints and temporal constraints to implement course-grained synchronization primitives on top of the fine-grained CMT timelines. For example, a skew constraint that prevents one stream from getting "too far ahead" of another would be expressed as a temporal constraint: "when {clock1.value > clock2.value + 2} corrective action." The temporal constraint manager ensures that the corrective action is executed when the condition becomes true.

We have implemented the n-Synch system core, and have built several demonstration applications. Other researchers are also experimenting with n-Synch as a foundation for synchronization research. We have some key questions to address in our future work.

Research Questions

Interactive Scripting: The Command Stream

As part of defining flexible presentations, we had two content goals: the ability to easily create new media types, and the ability to stream control alongside content. The first goal was motivated by theatrical presentations where lighting, curtains, and other effects need to be synchronized with traditional content. We also found new media types to be valuable for creating interactive laboratories such as the physic application. The second goal came from the observation that the interactive part of presentations often involved overlaying user interfaces and changing the playout of the presentation based on the results (trips to both the Smithsonian's Information Age exhibit and the multimedia museum Tech 2000 provided us with ample examples).

The command stream is a new medium designed to solve both problems. The "frames" of this medium are executable command chunks (written in Tcl) that include code fragments indicating how to undo the command if jumping backwards and how to rush through the command to execute only the essentials when jumping forwards. This stream of commands can include interface and presentation controls as well as code to implement new media types. We have implemented two full presentations using the command stream: a karaoke application that includes a dancing stick figure and time-sensitive controls, and a mary poppins stage scene where the user controls lighting and stage cues as mary flies onto the stage and back off. We are also using the command stream in our physics application.

Technically, we are happy with the command stream, but there are several critical usability concerns that we know must be addressed. The first is that command streams are extremely difficult to write. Lacking good editing tools, authors must write quintuples including four segments of Tcl code and a time at which the chunk should execute. While the sheer volume of code is staggering (it required 48 code fragments to control a button that counts down from 10 to 0 and then disappears), the larger problem is proper definition of the inverse and rush through code fragments. We know that writing command streams will require much more effective authoring tools.

Research Questions

Multimedia Widgets: Embeddable Applications

Our educational application led us to think more heavily about distance learning and the nature of the multimedia browsers that would be needed to view flexible presentations. We started pursuing a model based on heavyweight widgets that we call embeddable applications. These objects correspond to media streams and include both author controls and user controls. The user controls, such as forward, reverse, and pause for a video stream, can be selected from a set of different interfaces or hidden all together to allow the author to substitute application-specific controls.

We have created a number of interesting controls to accompany players from CMT, but have not packaged embeddable applications into a usable toolkit yet.

Research Questions

Goals for BRS

We are interested in presenting and discussing this work at the Basic Research Symposium for two key reasons: 1) We think it is important to continue thinking about interfaces beyond synthetic graphics and WIMP interfaces. Multimedia has the opportunity to embrace greater artistic creativity and a greater sense of the temporal, which nicely builds upon some of the work presented at the 1995 and 1996 symposia. 2) We are very interested in learning how other parts of HCI view the challenge of authoring flexible multimedia documents, presentations, environments, and experiences. We hope to learn a great deal about other metaphors and abstractions that may be helpful to us as toolkit builders.

References

J. Schnepf, J. Konstan, and D. Du. "Doing FLIPS: Flexible Interactive Presentation Synchronization." IEEE Journal on Selected Areas in Communication, January 1996.

J. Herlocker and J. Konstan. "Commands as Media: Design and Implementation of a Command Stream." Proceedings of ACM Multimedia '95.

J. Herlocker and J. Konstan. "Tcl Commands as Media in a Distributed Multimedia Toolkit." Proceedings of the Tcl/Tk Workshop 95, Usenix Association, 1995.

S. Iyengar and J. Konstan. "TclProp: A Data-Propagation Formula Manager for Tcl and Tk." Proceedings of the Tcl/Tk Workshop 95, Usenix Association, 1995.

B. Bailey and J. Konstan. "SynchMe: A Toolkit for Multimedia Synchronization Research," University of Minnesota Department of Computer Science Technical Report.

Return to Top of Page || Index.