Everywhere Displays Project

Everywhere Displays at SIGGRAPH'01

The exhibition
The Emerging Technologies Exhibition at SIGGRAPH is a juried, 6-day exhibition of the most inovative and advanced technologies in computer graphics and interaction technology. The Everywhere Displays project was selected to be presented at the ET exhibition of SIGGRAPH'01, held in Los Angeles, California, August 12-17, 2001

The concept
Among the various applications envisioned for the ED-projector, we are particularly interested in augmented reality (AR) scenarios. Unlike traditional AR, the use of ED-projectors enable user visualization of information without encumbering goggles and interaction without contact with input devices. A typical AR application is the augmentation of an assembly task. In our demo at SIGGRAPH'01, assembly instructions are superimposed on the objects to be assembled. Each individual user contributes to the assembly of an object by executing specific parts of the assembly process, without knowledge of the complete structure of the assembled object.
Based on these premises, we conceived and implemented an augmented reality assembly environment to be experienced by the visitors of SIGGRAPH'01. Given the characteristics and interests of our audience, we chose the object to be assembled to be a picture made of M&M's (multi-colored sugar-coated chocolates) where each M&M is regarded as a "pixel" of the picture. Our experiment built pictures of 60x50 M&M "pixels" based on portraits painted by Van Gogh. Each picture used a pallete of 18 colors of the 21 colors available from M&M ColorWorks. Since each visitor placed between 1 to 15 M&M's, in average 200 people were needed to complete a picture.

The setup
In the front of the space is a table; on the table lies a flat transparent Plexiglas board coated with double-sided transparent stick tape. This board is used as a mounting "canvas" for the M&M's . On the back walls of the space we shelved unlabeled buckets, each containing one of the 18 different colors of M&M's used. Two other surfaces are also used for projection: a painter's palette mounted on an easel and a wood board first covered with 1 inch-thick foam and then topped with white fabric.The ED-projector is positioned directly above the Plexiglas board. No sensors of any kind are embedded on any surface of the environment. The interaction is always detected only by the camera.

The experience


As the visitor arrives in the space, she sees the image of an M&M projected on the painter's palette with the instruction "Touch to begin". The goal of this image is simply to familiarize the user with the idea of touching a surface in order to interact with the system.				The ED-projector is then steered towards the foam-covered-board where images of different colored M&M's are projected along with the invitation "Pick a color". Here, the user can select which color to place in this iteration by simply making a "clicking" gesture on the M&M of the desired color.

The ED-projector then rotates the mirror so it highlights exactly the bucket that contains the M&M's of that color. It displays the message "Get n", (where n is the number of M&M's to be picked up from the bucket) on the bucket's frontal face. In addition, a button with the word "done" is also displayed on the can. After retrieving the appropriate number of M&M's from inside the bucket, the visitor has to touch this "done" button to communicate that the picking has been completed. Immediately afterward an instruction to "Go to the table" is projected on the bucket.

The ED-projector then indicates the precise location where the M&M's should be placed by projecting an image of the target locations on the canvas.

At this point, the user is invited to "Finger paint", that is, to wave her hand over the board to reveal the completed picture. The camera system tracks the position of the user's hand and fills in circles of the appropriate color, interactively completing the picture. At any time the user can stop this activity by touching the "yes" button to the right of the image.

The video
Click HERE or on the picture to watch a video (19Mbytes, AVI), showing a run of the system, from the point of view of the ED-projector camera. This is exactly the image fed to the computer vision system used to detect the interaction. Copyright IBM Corporation, 2001.

The results


Publications
A publication describing this experience and the basic technology involved is available.

For general information, contact Claudio Pinhanez M&M and ColorWorks are registered trademarks of Mars, Inc. All pictures by Claudio Pinhanez, Copyright IBM Corporation 2001.

Last update: 10/10/02