Embodied Interaction Introduction
Written by Dr. Francis Quek (N.B . A large part of the material for this page comes from: Quek, F., "Embodiment: We're Just Human," in Workshop on Embodied Interaction: Theory and Practice in HCI (at CHI 2011).
A student in my Embodied Interaction class once asked if a keyboard and mouse are embodied because obviously the body is used in operating such interfaces. Curiously, the answer must be ‘yes’. They engage our ability to offload work into automatic motor activity with a familiar keyboard layout, and our Heideggerian capacity to couple action through the mouse as tool. Without such capacity for ‘what our body knows’, mouse and keyboard activity would be impossible. The challenge with this perspective is that the concept of embodied action may become so trivially true as to become scientifically uninteresting. Furthermore, for HCI, on what criteria may one judge one interface as being more ‘embodied’ than another?
A major thrust of our research is to explore a model of embodiment that focuses on mind. Put succinctly, our proposition is that the embodied mind is one that supports activity in a 1. physically spatial, 2. temporally dynamic, 3. social and cultural, and 4. emotional world. Embodied interaction then is the design of interactive systems that engages this mind. Further, what does this model say about about the systems by which we interact with the computational world, and through whose mediation we interact with one another? The diagram below illustrates our model of embodiment, which guides most of the HCI research projects in VISlab. For convenience of presentation, the figure combines the Social and Emotional aspects of the model. We shall discuss our model, and situate the various research in the VISLab within the framework.
Examples of recent and current projects that fall into the major research strands of the lab are given below:
A. Physical-Spatial Aspect of Embodiment
The mind is 'designed' for us to function in a physical world that is laid out in space. This implies that both space and physical materiality are implicated under this dimension of embodiment. Our work in multimodal human discourse construction indicates how our tremendous capacity for spatial memory, reasoning, sensing and perception can be brought into service of a broad range of activities. This has implications in how we develop systems that employ physical space to support reasoning, information access, and general interaction.
It is not just space that is important, but that this space is physical, and not just graphical, virtual, or abstract. This space has physical extent, and is populated by physical objects. We appeal to Lev Vygotsky's theories to give us a handle to understand the relationship between physicality, space, and thinking. Here, Vygotsky’s concept of the material carrier (MC) helps us to frame interaction. MCs work with another Vygotskian concept – that of signs as the mental objectification of thought, and allows us to bring our capacity to manipulate that which is abstract or distal from our physical presence.
Under the heading of the spatial-physical aspect of embodiment, we list the following projects:
- Finger Walking In Place: We investigate how the innate and well-learned behavior of walking may be transferred to the fingers on a touch surface. Motor equivalence theory suggests this possibility. The key question is whether the theory also means that the allied active perception capacity will transfer as well to FWIP. An associated theoretical construction is the common coding theory.
- TanTab: We investigate how smooth transitions among physical and virtual manipulatives on a tabletop may support both the development of geometric problem-solving skills as well the basic geometric operations in Pre-K to Primary 3 children. We manipulate both the nature of collaboration among children and the kinds of interaction with the physical and virtual tangrams.
- Technology Ecologies for Learning: We study how smartphone, slate-type devices, tabletops and ambient large displays functioning together in ecologies may support learning through the objectification of information objects. Our study includes how such ecologies may be appropriated by the learner, and what interactions across the technology are necessary for such appropriated.
- Spatial and Tactile Audio Annotator and Reader for the blind, STAAR: This research aims to enable Individuals with Blindness or Severe Visual Impairment (IBSVI) to engage their broad spatial cognition and memory resources in reading and annotating textual material.
- Beyond the Talking Head and Animated Icon: Behaviorally Situated Avatars for Tutoring reconstructs full-body avatars with behaviorally correct co-speech gestures from gestures performed on a pen-tablet.
- Embodiment, Mathematics Discourse, and the Blind employs computer vision and a haptic glove to give blind students access to the deictic behavior of teachers to fuse situated spatial and temporal foci in language. We demonstrate how the the technology furnishes blind students with access to the salient information in instruction.
B. Temporal-Dynamic Aspect of Embodiment
Another aspect of our embodiment is the temporal and dynamic nature of our world. One challenge to HCI, however, is to understand how one may harness this dynamism in real-world interactive systems. Here, we connect our model of embodiment with an allied concept – that of situated cognition and context, and with two further concepts: attentional focus and familiarity. I suggest that the structure of the world leads to predictability within context. This permits the development of experience, and experience essentially allows us to focus our processing so that we attend to those things that are essential, and ignore things that are unimportant. An experienced driver does not think faster, she knows where on the road to direct her focus, and where dangers lurk. An experienced chess player may not examine more alternatives, but she knows what alternatives need to be examined. Context allows the human mind to optimize for function within situations – with a finite set of resources. It is part of the strategy of mind to deal with finite bodily action within a complex time-pressured world.
This does not mean that we need to build interfaces entirely based on familiarity. This would be counterproductive, locking us into the past, and ignoring possibilities presented by new technologies like new slate-type computing. Fortunately, human embodiment is not marked by familiarity, but the capacity to become familiar. Drop an American into the chaotic traffic situation in Malaysia where they drive on the other side of the street, and she would become disoriented. With experience, she would adapt and know that she needs to look right, then left, then right again before crossing a street (as opposed to the other way around in the US). The seeming chaos eventually yields to a frame-work of predictability. A deer, on the other hand, would never learn the order to look before crossing a street in either country.
This intuition about familiarity to function in a dynamic world contributes to the following projects:
- Grounded Creativity: See description in the Social-Cultural Aspect of Embodiment section.
- Narratives for Elderly: We develop a gaming system grounded in narrative as opposed to performance for older adults. The rationale is that as one ages, a trade-off takes place between functional capabilities (e.g. mental processing speed, short-term memory capacity) for experience and knowledge. We showed a preference among older adults for familiar narrative game scenarios, and that this motivates greater engagement in game play that supports mental exercise.
C. Social-Cultural Aspect of Embodiment
Society is not an accident. Humans are social beings, and our mind and body are designed for us to function within community. It has, for example, been posited that humans have distinctive white sclera to support nuances of social communication, and that the capacity to recognize human facial displays has both innate and learned components. This suggests that we attach tremendous amount of meaning to our social interactions.
Human society evolves over time through cultural advancement. This brings the study of creativity support and studies directly into purview of embodied interaction. Human innovation, and the social mechanisms by which these innovations are admitted to alter the culture are the means by which the individual contributes to culture. To operationaize these concepts, we return to the theories of Lev Vygotsky that describe creativity as having two components: process and substance. For Vygotsky, the creative process is one of recombination of prior knowledge that has been properly decomposited and internalized as usable signs. Fluency (familiarity) with knowledge or skill in the domain of activity is the substance to be recombined. This distinction may explain the apparent torrent of (embodied) creative activity in young children before they achieve sufficient knowledge to truly contribute to culture. They are practicing the process of creative recombination. Innovative activity is as much a part of being human as hunting is to tigers. Consequently as tiger cubs practice the kill in play, so human young exercise the creative process ahead of acquiring sufficient knowledge to create. This may also partially explain the ‘at risk’ period in children of 9 to 13 years where creativity wanes (known as the 4th grade slump), as they become aware that they do not have sufficient knowledge or skill to really contribute.
Projects associated with the social-cultural aspect of embodiment are:
- SocialOrb: This project investigates the use of location-awareness technology for socially-based tagging to help users to organize and find information. The key intuition is that we consider our information in terms of the people who were physically present, and with whom we work.
- Grounded Creativity: What is the role of the technological medium in the creative process of children? This projects investigates the design of affordances in technological systems (e.g. enactment-based systems) in the domain of creative storytelling. The goal is to support and nurture children's motivation to create as they become more socially aware during the pre-adolescence period, and develop a desire for their cultural contributions to be recognized by others .
- Socially-based Crowd Simulation: This project uses Herbert Clark's theory of 'common ground' to develop a crowd model that exhibits complex behavior driven by individual decisions of agents while the simulation achieves an efficient performance
- Embodiment, Mathematics Discourse, and the Blind: This project cross-listed in the Physical-Spatial Aspect of Embodiment section has a strong social component in mediating situated social communication between instructors, students who are blind, and students with vision in an inclusive classroom.
- TanTab: This project cross-listed in the Physical-Spatial Aspect of Embodiment section also investigates how manipulating social interaction (in our case norms of who 'owns' the tangrams) affects learning.
D. Emotional Aspect of Embodiment
The fourth aspect of our model is that emotions are part of the situated embodiment of the mind. Emotion and affect are critical to our functioning within society, and play a critical role in our decision-making and sense of well-being. Hence, emotion is part of the aspect of mind that helps us to function in a dynamic and social world.
- Remote-Social Touch: Our work on remote affective touch explores how affect may be conveyed remotely through a haptic armband device. Our hypothesis is that touch is an immediate conveyer of affect in that it does not require encoding of a symbolic message by the originator and subsequent decoding into a symbolic message by the recipient. A child hugs her parent not to pass a message, but simply because she wants and needs to. We posit that for touch be an immediate conveyer of affect, only if it is accompanied by another contextualizing communicative channel. Our studies have shown that remote touch can impact emotional experience and enhance a sense of connectedness.
E. Projects Motivated By Embodied Interaction Research
In the course of our research in Embodied Interaction outlined above, we encountered related research or research requirements that needed to be addressed. We highlight two of these research directions that have become significant research or education projects in their own right.
First, many of our research projects begin with a scientific insight (of embodied interaction in our case) that informs the design of a system that operationalizes the insight. This often puts us in an area of the space of interactions that is far from existing user experience. The challenge is that the system we build needs aspects of design that may have nothing to do with the original insight, and design and usability issues can compromise the study of the scientific insight using the system. Knowing how difficult it is to realize any good design further highlights this problem. We found that many of our own design approaches over the years have been rather ad hoc, relying as much on our own design intuition as on HCI methods such as participatory and user-centered design. We also realized that many of the known and practiced HCI design methods are well-suited to incremental design, but do not address problems in radical design. This led us to a research trajectory to understand processes that can lead to radical designs.
Second, we have found the need to think beyond the computing platforms and devices that can be purchased whole. In a number of our projects, we have had to develop our own custom devices (e.g., see Mathematics Instruction for the Blind and Remote Social Touch). More generally, in the world of the Nintendo Wii, Microsoft Kinect, and various physically mobile platforms like smartphones and slate-type computing, HCI researchers needs to embrace a more physical model of computing. The day when computing means rows of flat screens with mice and keyboards will one day be as quaint as the typing pools of the previous generation. This vision will require new research in mobile computing, always-on high-speed networks, and the construction of new physical computing prototypes.
- Radical Design in HCI: In this project, we identify published projects over the last decade that may be considered to result in radical designs. We are interviewing the principals of these projects to find patterns of design approaches. We also interview leaders in the field of HCI to glean their insights on how one may arrive at a testable radical design.
- Physical Computing: The need for physical computing requires research infrastructure for electronics and physical prototyping. We secured an NSF grant to allow us to develop such infrastructure. This project involves the development of such an infrastructure, training students to use engage in physical prototyping as part of their research/design repertoire, and the fostering of a research culture where HCI researchers engage in physical prototyping as part of their research.