A Temporal Hierarchy of Semantic Integration in Human Cortex Derived from Audio Stories and Their Corresponding Word Vectors

Abstract

Previous research has shown that fMRI responses correlate with the semantic meaning of naturalistic stimuli. However, these methods assumed that the brain response is only determined by the current stimulus. While this may be the case for some brain areas, other research has shown that other regions of the brain integrate information over longer time scales. Thus, we may use a trailing window of time step information to represent stimuli over multiple timescales to better predict fMRI response. The optimal size of the trailing window in a brain region would correspond to its actual timescale of integration. Then we can compare a map of timescales to previous, similar maps derived from other methods to test if a trailing window is a proper representation of integrating current and past time steps of information in the brain. fMRI data sets were collected from subjects listening to different audio narratives. For each audio data set, the audio stories\textsc{\char13} transcripts were converted into a series of word vectors over time. To represent how the brain might integrate past information with current information, for each time step, the number of previous vectors concatenated onto a current time step was manipulated. ROI and whole brain searchlight analyses were conducted for each data set, where for each ROI or searchlight area, the fMRI data were aggregated for all subjects using a Shared Response Model and a linear map from the word vectors to the fMRI response was created to predict the fMRI response given new word vector data. The number of previous time steps that led to the best prediction in the ROI or searchlight area was recorded. The searchlight analysis was used to create a temporal hierarchical map of semantic integration of the brain. The hierarchical maps created from the stimulus information corroborate previous maps derived from fMRI response alone. This research is a starting point in understanding the different timescales at which the brain integrates information, and can be used to improve fMRI decoding or to build computational models of semantic integration in the brain.