Temporal calibration between the visual, auditory, and tactile senses: A psychophysical approach

Abstract

Human observers acquire information about physical properties of the environment through different sensory modalities. For natural events, these sensory signals show a specific temporal, spatial and contextual configuration that aids the integration into a coherent multisensory percept. For multimodal virtual environments, however, signals have to be created and displayed separately for different modalities, which may result in a miscalibration of these signals. This, in turn, can greatly reduce the observer’s sense of immersion and presence. Using psychophysical methods, we investigate fundamental questions regarding how the temporal alignment of signals from the visual, auditory and tactile modalities is achieved. A first project examines the perception of subjective simultaneity of signals. Simultaneity detection poses a non-trivial matching problem to the human brain: physical and neural transmission times differ greatly between the senses. As there is only partial compensation for these differential delays, subjective simultaneity may result from presenting stimuli with a physical delay. Here, we are interested in whether this phenomenon reflects an amodal timing mechanism that works across all modalities in a uniform fashion. Further, we look at the sensitivity for asynchrony detection for different modality pairs as well as at interindividual differences. In a second project, we examine the ability of the human cognitive system to adapt to asynchronous information in different modalities. Adaptation may be used to reduce the disruptive effects of temporal miscalibration between signals in different modalities. We are interested in the strength of adaptation as well as the mechanism underlying this effect. Future projects aim at the investigation of - the precise relationship between the perception of synchrony and multimodal integration, - the influence of prior knowledge about a common origin of signals on the perception of synchrony - the influence of timing on the perception of cause and effect - the neural basis of the detection of synchrony In conclusion, our research seeks to understand the mechanisms underlying temporal calibration between different sensory modalities with the goal to identify factors that foster multimodal integration and, in turn, the sense of presence.