Finger Walking in Place
The effect of transformed natural user interaction for spatial learning in 3D immersive virtual environments
- Project Lead: Dr. Ji-Sun Kim Graduated PhD Student
- Dr. Denis Gracanin Associate Professor
- Dr. Francis Quek Professor Center for Human Computer Interaction
Note: This is a recently completed project that is currently dormant
Action-based interaction technique design in virtual environments(VEs): The premise of the research is that our ability to operate in a virtual space is grounded in our experience (action-based) in the real world, and that this action-based experience can be activated by using other physical body frames as well as our full-body frame. This prediction is based on Motor Equivalence Theory that states that physical action can map to different body parts while activating the neural machinery relating to the original physical action. This has tremendous implication, for example, for the design of prosthetics. The utility of this theory for our Finger Walking in Place (FWIP) project is that if the well-learned and innate capacity for bi-pedal natural activity can be mapped to one's fingers walking on a smart-phone or any multi-touchable surface, one could have a comfortable way to interact with any virtual reality system (CAVE, desktop, 3DTV or other).
More importantly, all of virtual reality is based on the coupling of human perception with action. This coupling along with the synchronized digital production of the immersive sensory stream transports the user into the virtual place. A key question for FWIP is whether one's allied sensory and perceptual system is primed regarding spatial learning because the similarity of the finger walking to real walking activates the equivalent motor system. If so, one might expect to see perceptual and spatial understanding gains when using this mode of operation in VEs.
Fig. 1 - 3 illustrates this concept. Fig. 1. illustrates the neural activation associated with Bipedal Natural Walking in yellow. The allied perceptual machinery associated with active vision and proprioception that activates with bipedal natural walking is shown in blue. Fig. 2. shows the Motor Equivalence hypothesis. FWIP activates most of the same areas related to bipedal natural walking. Fig. 3. illustrates how FWIP may activate similar perceptual/proprioception machinery if indeed the Motor Equivalence hypothesis is correct. This means that there should be perception gains when using FWIP over some locomotion technique that is less likely to be motorically equivalent (e.g., using a joystick).
The implications of having such a light-weight interaction system as FWIP that yields strong 3D perception gains is likely to enable a more ubiquitous application of VR technology. To date, we have shown gains of spatial cognition in wayfinding and navigation tasks. We are proceeding with studies to investigate the perceptual gains per se.
- Provisional patent application: Ji-Sun Kim and Denis Gracanin, Finger Walking in Place (FWIP). U.S. Patent Application No.: 61/114,555. Filed date: Nov. 11, 2008.
- iPhone, iPad or other multi-touchable surfaces are applicable to implement FWIP technique that is usable to navigate in virtual environments.
- In open maze space, route knowledge acquisition performance was better with FWIP technique than Joystick-based flying technique.
- In closed maze space, route knowledge and survey knowledge acquisition performance was better with FWIP technique than Joystick-based flying technique.
- In closed maze space, route knowledge and survey knowledge acquisition performance was better with Sensor-Fusion Walking in Place technique than Joystick-based flying technique (not published yet). -- Sensor-Fusion Walking in Place technique is directly mapped to the same body parts as natural bipedal walking.
Thus, these results are supporting the concept described above toward interaction technique design methodology to improve spatial learning performance.
- Ji-Sun Kim, Denis Gracanin, and Francis Quek, "Sensor-Fusion Walking-in-Place Interaction Technique using Mobile Devices". in IEEE Virtual Reality 2012, March 4-8 2012
- Ji-Sun Kim, and Denis Gracanin, "An Approach to Comparative Studies in CAVE using a Virtual Black Wall". in 'IEEE Virtual Reality 2012, March 4-8 2012
- Ju-Sun Kim. "Walking-in-Place technique based on mobile devices". Presented in ACM Student Research Competition (1st Round), Grace Hopper Celebration 2011
- Ji-Sun Kim, D. Gracanin, K. Matkovic and F. Quek. The Effects of Finger-Walking in Place (FWIP) for Spatial Knowledge Acquisition in Virtual Environments. In Proc. of 10th International Symposium on SMART GRAPHICS 2010, LNCS 6133, pp. 56-67, Springer-Verlag Berlin Heidelberg 2010
- Ji-Sun Kim, Denis Gracanin, Kresimir Matkovic and Francis Quek, iPhone/iPod Touch as Input Devices for Navigationi n Immersive Virtual Environments (Poster), in the Proc. of IEEE VR, 2009.
- Ji-Sun Kim, Denis Gracanin, Kresimir Matkovic and Francis Quek. Finger Walking in Place (FWIP): a Traveling Technique in Virtual Environments. In Proc. of 8th International Symposium on SMART GRAPHICS 2008, LNCS 5166, pp. 58–69, Springer-Verlag Berlin Heidelberg 2008.
- For more information about Ji-Sun's publications, please refer to Ji-Sun Kim's homepage