Author: Zhenchao Xia
Mentor: Hao Tang
Institution: BMCC
Abstract: The Mixed Reality (MR) Cane utilizes simple auditory and haptic feedback on an iPhone to help the cane’s users interact with virtual objects and environments as they are walking in real world situations. The visually impaired can use the app to learn the physical layout of unknown environments using virtual reality (VR), which encourages them to travel independently. However, we found that users sometimes had difficulty understanding the direction of the object of interest in the facility, because the iPhone simulated the long cane and no additional sensor was used to track user’s head. When the head pose of the visually impaired isn’t correctly tracked, the 3D audio feedback can’t be played correctly so the blind users have difficulty to learn the correct spatial layout.
In this study, we try to improve the estimation of the user’s head pose by using iPhone’s front camera. We develop an algorithm to process the video of user’s head movement captured by iPhone’s front camera and compute the orientation of user’s head in real time. The algorithm analyzes the video to recognize the user’s face in each video frame and computes the orientation of user’s head. The algorithm keeps track user’s head in the video sequences and ensures the head is correctly estimated even the iPhone is swept to simulate the long cane. In the experiment, we found the 3D audio feedback is greatly improved due to the proposed head tracking algorithm.
Improve Head Tracking for MR Cane – A Virtual Reality App to Learn Spatial Layout for the Visually Impaired
