VR Lab HITLabNZPOSTECH Univ. of Canterbury

Occlusion based Interaction Methodsfor

Tangible Augmented Reality Environments

Gun A. Leeα Mark Billinghurstβ Gerard J. Kimα

α Virtual Reality Laboratory, Pohang University of Science and Technology

β Human Interface Technology Laboratory New Zealand,University of Canterbury

endovert@postech.ac.kr

VRCAI 2004

VR LabHITLabNZ

2VRCAI 2004

Introduction

■ Tangible Augmented Reality[Kato 2001]

■ Each virtual object registered to physical object

■ Users manipulate virtual objects by manipulating corresponding physical objects

VR LabHITLabNZ

3VRCAI 2004

Introduction

■ Tangible AR Applications

▲MagicBook

ARGroove ►

▼ VOMAR

◄ Tiles

VR LabHITLabNZ

4VRCAI 2004

Introduction

■ Tangible Augmented Reality■ Easy natural 3D interactions

■ Full 6DOF 3D Direct manipulation

■ 1/2D interactions?■ Buttons?■ Sliders?■ Menus?■ Keypads?

VR LabHITLabNZ

5VRCAI 2004

Related Works

■ 2D Input methods in AR■ Screen stabilized fashion

■ Tinmith [Piekarski 2002]

■ Magic Ring [Dias 2003]

■ World stabilized fashion■ Windows on World■ ARGUI [Geiger 2003]

■ Hand gesture mouse [McDonald 2003]

■ Used indirect input methods (mouse, getsture, etc) rather than direct touch

VR LabHITLabNZ

6VRCAI 2004

Related Works

■ Large scale touch sensor■ SmartSkin [Rekimoto 2002]

■ Touch implies occlusion■ Commercial products using

occlusions as inputs■ Virtual Keyboard, Canesta■ Parapara Paradise, KONAMI

VR LabHITLabNZ

7VRCAI 2004

Occlusion based Interaction

■ 2D Interaction using pointers■ Pointer centered view

■ Tracking the pointers■ Cost depends on number of pointers + interaction

points■ Interaction point centered view

■ Sensing on interaction points■ Cost depends on number of interaction points

■ In TAR environments, interaction points (props) are trackedrather than pointers (hands).

VR LabHITLabNZ

8VRCAI 2004

Occlusion based Interaction

■ Occlusion Detection in TAR■ Tracking a set of visual markers with

predefined spatial relationship■ Provides robustness■ Invisible marker positions are inferred from

the visible ones

■ Occlusion = Invisible + Within the view■ Two methods for detecting occlusions

■ Boundary Marker Method■ Estimated Marker Projection Method

VR LabHITLabNZ

9VRCAI 2004

Occlusion based Interaction

■ Boundary Marker Method

VR LabHITLabNZ

10VRCAI 2004

Occlusion based Interaction

■ Estimated Marker Projection Method

VR LabHITLabNZ

11VRCAI 2004

Occlusion based Interaction

■ Boundary Marker Method■ Easy and straight forward, reliable

implementation■ Lesser computation

■ Estimated Marker Projection Method■ No need of boundary markers■ More computation required■ Estimation errors due to lens distortion

■ Over estimating the size of markers helps

VR LabHITLabNZ

12VRCAI 2004

Occlusion based Interaction

■ Interaction Design■ Time out constraints

■ Point & hold -> Explicit command■ Preventing accidental commands■ Repeating the command (common in

keyboards)

VR LabHITLabNZ

13VRCAI 2004

Occlusion based Interaction

■ Interaction Design (cont’d)

■ Sub-marker level measurement■ Two consecutive markers occluded

-> The pointer is in-between them■ Relatively lesser markers needed

VR LabHITLabNZ

14VRCAI 2004

Occlusion based Interaction

■ Interaction Design (cont’d)

■ Tip point marker detection■ 2D grid of markers■ Top-left one from the

occluded marker blob■ Other heuristics applicable

– e.g. select top-right one when the blob is skewed to the right

VR LabHITLabNZ

15VRCAI 2004

Implementation

■ Desktop■ Custom built PC with Windows XP■ Athlon 1.5GHz / 512MB■ NVIDIA GeForce4 MX

■ Mobile■ Apple Macintosh iBook with Mac OS X■ G3 900MHz / 640MB■ ATI Radeon 7500

VR LabHITLabNZ

16VRCAI 2004

Implementation

■ ARToolKit (2.65DS / 2.61Mac)■ Logitech Notebook PRO USB Camera

■ 320x240 @ 30Hz■ i-Visor HMD

■ Video image stretched to fit 800x600■ OpenGL drawn in full resolution

■ Performance: 19~30fps■ 5fps with 35 markers on mobile

configuration (low computation power)

VR LabHITLabNZ

17VRCAI 2004

Applications

■ Buttons

VR LabHITLabNZ

18VRCAI 2004

Applications

■ Sliders

VR LabHITLabNZ

19VRCAI 2004

Applications

■ Menu bars

VR LabHITLabNZ

20VRCAI 2004

Applications

■ Ball-pushing Game

VR LabHITLabNZ

21VRCAI 2004

Applications

■ Calculator

VR LabHITLabNZ

22VRCAI 2004

Applications

■ Board games

VR LabHITLabNZ

23VRCAI 2004

Discussion

■ Informal user study■ 6 subjects■ 7 scale questionnaire

■ Easy to learn: 5.6 (stddev=0.55)■ Easy to use: 5 (stddev=0.7)

■ Two 2D game applications demonstrated on public■ Over 100 peoples tried and gave

positive feedbacks

VR LabHITLabNZ

24VRCAI 2004

Discussion

■ The interaction appeared natural■ Using bare hands■ Direct manipulation (pointing)

■ Occlusion of users’ hands by virtual objects were reported unnatural ■ Partially solved by making the interface

semitransparent when occluded■ Easily applicable to mobile and/or

wearable systems

VR LabHITLabNZ

25VRCAI 2004

Conclusion & Future Works

■ Occlusion based Interaction for Tangible AR Environments■ Simple, easy to use, and natural 2D interaction■ Low development and computational cost

■ Future Works■ Try other type of visual markers or visual

tracking methods■ Add passive tactile feedbacks