Math for Blind
Mathematics Instruction for the Blind
- Project Lead: Francisco Oliveira PhD - Completed 2011
- Dr. Francis Quek Professor, Center for Human Computer Interaction
- Dr. David McNeill "Professor, Psychology, U. of Chicago"
- Dr. Mary Ellen Bargerhyuff, Assoc. Prof. Special Education, Wright State U.
- Mr. Jeffrey Vernooy, Director, Disability Services, Wright State U.
- Bing Fang PhD - Completed 2011
- Yannick Verdie MS - Completed
Note: This is a recently completed project that is currently dormant
We are studying the impacts on academic performance of those who are blind when they become able to understand the teacher's deictic references when she is at the board.
This project is focused on enhancing the ability of learning of the visually impaired. It is a multidisciplinary project involving Phisiology, Psychology, Electric Engineering and computer Science. We are investigating ways of keeping teacher/student communication situated. Blind students do not have access to the teacher’s gestures. We have developed haptic gloves that help blind student to navigate through instructional material. This material is a raised-line version of the class notes. The student’s reading finger must be on the point the teacher is referring to during the lecture. We call this point PIF – Point of instructional focus. We use computer vision techniques to track both teacher’s and student’s hand and calculate the direction the student need to move his/her hand to reach the PIF. This direction is then, continuously sent through the glove.
The research described above raises many investigation questions about how humans use their affordances to engage collaborative work in different environments, how technology modifies these environments providing novel ways for humans interaction and how humans react to this new augmented world.
Related Research Areas
Embodied Interaction, Haptic Interaction, Accessibility, Education
The Haptic Glove
The glove contains a set of eight vibrating motors arranged in such way that each motor corresponds to a direction: North, Northeast, East, etc. The glove is connected to a box with a PIC18F452-I/P microcontroller via serial cable. The controller box is in turn connected to a computer also via serial cable. The program on the microcontroller controls the timing and intensity of the vibration of each individual motors on the glove. The motors are assembled into copper tubes, which are put into the glove’s small pockets. The spinning within the pocket produces the sensation of vibration.
- The glove conveys the sense of direction. An experiment was designed to answer the question: Can the gloves convey directional information? Eight direction signals were sent randomly to the glove. Before each trial, the experimenter asked the participant whether she was ready to receive the signal. Upon an affirmative answer, the experimenter clicked a button to start sending the signals. The participant was asked to quickly and accurately tap on the arrow corresponding to the perceived direction on a touch-sensitive pad.The response times averaged around half a second and no significant difference was found among the glove models (Anova 95%). Participants who reported that they could clearly perceive the directions also had higher hit rates. More...
- The Glove vibration does not interfere with fingertip reading. We wanted to make sure that the vibration produced by the motors on the palm of the hand wouldn’t interfere with fingertip reading. For that, we assign two targets on a board with dots forming numbers. The program helped on the blindfolded participant’s navigation to the targets. Once they found the targets, we asked them what number they were reading. The glove never stopped vibrating. The hit percentage for all participants were way above guessing. More ...
- One can navigate with the help of the system while listening to a story. Furthermore, It is possible to fuse the information received from hearing and tactile modes. These two findings are very important pre-requisites because without them, we could not advance to actual instruction experiments. We told five different one-paragraph-long stories to each participant. Some numerical story content (e.g. how many bedrooms were in a house) was conveyed on the tactile board (like that used in experiment 2) and not by speech. Subjects were guided to the numbers co-temporally with the speech using the glove. This simulates the simultaneous presentation of graphical material with deictic references while speaking. After each story, we asked five questions about the story. For each of them, five possible answers were asked and the participant had to pick one. To correctly answer the questions, the participant had to recall information obtained from either the board or the speech. The participants who reported that they could clearly perceive the directions while listening to the stories also answered correctly a higher number of questions and were able to navigate faster. More ...
A Game for the Blind
Individuals who are disabled are motivated to learn the assistive technologies devices and tend to use them over a long period of time. To address this we developed a computer game which is placed permanently at Office for Disabilities Services at Wright State University - Dayton, Ohio.
This demonstrates the use of games as means to motivation and engagement for training.
- F. Quek and F. Oliveira, “Enabling the Blind to See Gestures”, to appear in Transactions of Computer-Human Interaction, ToCHI.
- F. Oliveira, F. Quek, H. Cowan and B. Fang. "The Haptic Deictic System - HDS: Bringing blind students to mainstream classrooms", to appear, IEEE Transactions on Haptics.
- M.E. Bargerhuff, H. Cowan, F. Oliveira, F. Quek and B. Fang. Developing Technology to Support Learning of mathematics by Students Who Are Blind: The Gamers Have It!. In Journal of Visual Impairment and Blindness, November 2010, Volume 104, Number 11. 2010.
- F. Oliveira, H. Cowan,B. Fang and F. Quek. Fun to Develop Embodied Skill: How games help the Blind to Understand Pointing. In ICMI '10. International Conference on Multimodal Interfaces and Workshop on Machine Learning for Multi-modal Interaction. 2010.
- F. Oliveira, H. Cowan,B. Fang and F. Quek. Fun to Develop Embodied Skill: How games help the Blind to Understand Pointing. In PETRA '10. 3rd International Conference on Pervasive Technologies Related to Assistive Environments. 2010.
- B. Fang, F. Oliveira and F. Quek. Using a Vision Based Tracking to Support Real-Time Graphical Instruction for Student Who Have Visual Impairments. In CVPRW '10. 3rd Workshop on Computer Vision Applications for the Visually Impaired, pages 9-14. 2010.
- F. Oliveira and F. Quek. A Multimodal Communication with A Haptic Glove: On the Fusion of Speech and Deictic Over a Raised Line Drawing. In PETRA 2008: International Conference on Pervasive Technologies Related to Assistive Environments. 2008.
- F. Quek, D. McNeill, and F. Oliveira. Enabling multimodal communications for enhancing the ability of learning for the visually impaired. In ICMI ’06: Proceedings of the 8th international conference on Multimodal interfaces, pages 326–332. 2006.
This research has been supported by NSF grant: Embodiment Awareness, Mathematics Discourse and the Blind, IIS-0451843, and CRI: Interfaces for the embodied mind, IIS-0551610.