Evidence for an Event-Integration Window: A Cognitive Temporal Window Supports Flexible Integration of Multimodal Events

Just as the perception of simple events such as clapping hands requires a linkage of sound with movements that produce the sound, the integration of more complex events such as describing how to give an injection requires a linkage between the instructor's utterances and their actions. However, the mechanism for integrating these complex multimodal events is unclear. For example, it is possible that predictive temporal relationships are important for multimodal event understanding, but it is also possible that this form of understanding arises more from meaningful causal between-event links that are temporally unspecified. This latter approach might be supported by a cognitive temporal window within which multimodal event information integrates flexibly with few default commitments about specific temporal relationships. To test this hypothesis, we assessed the consequences of disrupting temporal relationships between instructors' actions and their speech in both narrated screen-capture instructional videos (Experiment 1) and live-action instructional videos (Experiment 2) by displacing the audio channel forward or backward relative to the video by 0, 1, 3, or 7 s. We assessed learning, event segmentation, disruption awareness, segmentation uncertainty, and perceived workload. Across two experiments, 7-s temporal disruptions consistently increased uncertainty and workload and decreased learning in Experiment 2. None of these effects appeared for 3-s disruptions, which were barely detectable. One-second disruptions produced no effects and were undetectable, even though much intraevent information falls within this range. Our results suggest the presence of an event-integration window that supports the integration of events independent of constraining temporal relationships between subevents.