Hotaru: Intuitive Manipulation Techniques
for Projected Displays of Mobile Devices

Kosuke Miyahara, Hiroshi Inoue, Yuji Tsunesada, Masanori Sugimoto
Interaction Technology Lab., University of Tokyo
{miyahara, inoue, tsunesada, sugi}

A near future technology is presented: a mobile device with a projector and intuitive manipulation techniques by using a video camera mounted on the device. It is difficult to realize a mobile device with a small and light projector that still retains the feature of mobility, so we have developed a system to project displays of mobile devices by tracking their three-dimensional positions and orientations. The proposed system called Hotaru (a firefly, in English) allows users to annotate, rotate or transfer files between multiple devices by touching their projected displays with fingers.

Figure 1. Hotaru in Use

2.1 Requirement

Figure 2. System Overview

2.2 Location & Orientation Identification
A method with infrared LEDs and a stereo camera is used for identifying locations and orientations of mobile devices. Three LEDs are mounted on a circuit board designed to be attached to each PDA. By capturing the shape of the triangle formed by the LEDs through the stereo camera above users, locations and orientations of the PDA are estimated. The position and orientation errors with this method are less than six centimeters and ten degrees, respectively. Different blinking patterns of LEDs are assigned to individual PDAs for their identification.

Figure 3 PDA with IR LEDs

3.1 Recognition of Projected Displays
The system first performs the distortion correction of an image caused due to the wide-angle lens, and converts it into a binary format image. When four vertices of a projected display cannot be determined due to occlusion by human hands, the system applies the Hough transform and recognizes the projected display. Finally, by using the four vertices of the projected display in the camera coordinate and those of the PDA screen in the world coordinate, a transformation matrix between the two coordinates is calculated.

3.2 Recognition of Operations by Fingers


Figure 4. Annotation (left), Rotation (center) and file transfer with a finger(right)

Figure 5. File transfer by overlapping projected displays of their devices

Informal user studies proved that Hotaru could effectively support collaborative tasks in co-located situations. We will further investigate the design and implementation issues through intensive user studies.

PUBLICATION (selected)