Perceptual and Cognitive Neuroscience Lab aims to understand how the brain represents and processes a range of perceptual and cognitive information – from low-level sensory features to high-level contexts and predictions. Contrary to our rich and seamless experiences of the external world, raw sensory data are only partial. To investigate how the brain creates enriched internal models of our environment from impoverished sensory data, we examine the human brain at the network level, using functional magnetic resonance imaging (fMRI) and computational approaches (model-based encoding/decoding using machine learning, functional connectivity, graph theory). Functional brain imaging enables us to identify the brain circuits and networks underlying perception, attention, and memory, and to examine the interaction between regions and the functions of populations of neurons at each stage of the information processing stream. We also link psychophysical measures to fMRI responses to elucidate how the neural information relates to our cognition and behavior.
Neural representations in the visual processing hierarchy
Kim, I., Hong, S.W., Shevell, S.K., & Shim, W.M. (2019). Neural representations of perceptual color experience in the human visual pathway. PNAS. Link
Perceptual color experience vs. Physical light
We aim to understand how subjective perceptual experiences (color percept) that are distinct from retinal stimulation are represented across the visual hierarchy. By linking the population-level, color-selective responses in a series of hierarchically organized cortical regions and moment-to-moment color experiences in each subject under interocular switch rivalry, this research provides insights into how the neural representation that corresponds to conscious perception evolves across the visual processing hierarchy.
Neural representations of color in LGN
Perception of color emerges from hierarchical neural processing from the retina via lateral geniculate nucleus (LGN) to visual cortex. Despite its commonly known functional role as a relay station, neural representation of color in human LGN remains elusive. Using fMRI in conjunction with encoding methods, we investigate the characteristics of color representation in human LGN. Our results suggest that color representation in LGN is more complicated than previously considered, which inherits representative characteristics of color processing from both the retina and primary visual cortex elucidating LGN’s functional role as an intermediate transition step in a multi-stage color processing stream.
Dynamics of attention and working memory
Yu, Q., & Shim. W. M. (2017). Occipital, parietal, and frontal cortices selectively maintain task-relevant features of multi-feature objects in visual working memory. Neuroimage, 157, 97-107. Link
Visual working memory
Our visual world changes constantly: Objects come in and out of view, people move across the visual field, and even static scenes change as we move through them. We can hold a limited amount of information in working memory for brief periods of time when visual objects are no longer in view, which is essential for maintaining spatio-temporal continuity in this ever-changing environment. Using fMRI and machine learning approaches, we study the cognitive and neural mechanisms underlying visual working memory. Our recent work demonstrates two important characteristics of a highly selective mechanism of visual working memory: 1) only task-relevant features are maintained at feature-selective responses in the early visual and higher-order frontoparietal regions and 2) this mnemonic information is modulated by temporal-order-based attentional signals in the frontoparietal regions.
Neural representations during different attentional states
Sustained attention intrinsically fluctuates, moment to moment, between optimal and less optimal states. Prior research has introduced two attentional states based on reaction time (RT) variability: a behaviorally less variable attentional state, defined as optimal or “in the zone”, and a more variable attentional state, described as sub-optimal or “out of the zone”. Using fMRI, we aim to examine how such time-varying attentional states modulate the neural representation of sensory inputs at the cortical level.
Construction of narrative structures during natural vision
Song, H., Park, B. Y., Park, H., & Shim, W. M. (2021). Cognitive and Neural State Dynamics of Story Comprehension. Journal of Neuroscience, 41, 8972-8990. Link
Understanding stories
Story understanding represents a real-world process of information accumulation and integration, where higher levels of story understanding are achieved by integrating the accumulated narrative information into a coherent structure. As subjects watch movies inside a scanner, we investigate how changes in one’s level of understanding are represented as dynamic reconfiguration of whole brain functional networks. Using supervised and unsupervised machine learning techniques, we seek to predict evolving levels of understanding using time-resolved neural signatures. In particular, we investigate how the memory of causally-related previous events are spontaneously reinstated when processing the events of the present moment.
Reconstruction of stories in memory
Even when everyone encounters the same story, the interpretations may differ across individuals. Likewise, an ability to understand complex stories may intrinsically differ across individuals. Using fMRI and natural language processing, we seek to find the neural signatures that underlie such individual differences in story understanding. We are decoding trajectories of real-world thoughts as we continuously update situational contexts of the story.
Social prediction within naturalistic social context
As apt social agents, we constantly predict others’ states and actions for successful interactions with them by tracking numerous types of social information. Using fMRI and computational models, we investigate social information processing in the human brain while watching movies with rich social information. The narrative structures in movies provide predictive power over social events and enable us to track social content.
Context transition during free speaking
While humans think spontaneously, their thoughts can sometimes leap to new topics, indicating a change in context. As participants freely spoke their though inside a MRI scanner, we investigate how context transitions in continuous internal thoughts are represented as neural dynamics.
High-level cognition in action under naturalistic settings
In real-world contexts, we do not only passively receive information but also actively explore the environment by interacting with biological entities and social agents and make hierarchical strategic planning to achieve behavioral goals and adaptively update them online. To examine such high-level cognitive processes linked to ecological brain functions, we are currently developing a range of interactive Minecraft-based 3D video game tasks in which subjects engage in active social interactions, strategic planning through hierarchically organized subgoals, and adaptive online strategy updates while navigating the virtual world. By creating interactive tasks in naturalistic environment, we aim to examine cognition and behaviors that occur beyond the laboratory settings and deepen our understanding of complex high-level cognitive functions and adaptive behaviors in a complex, multidimensional world.
Spatial foraging
Foraging, a successive action of seeking reward that yields the greatest long-term benefit, involves a complex cognitive process that can be found in creatures as diverse as insects and primates. Building an efficient foraging strategy inside a naturalistic environment requires navigation, memory and successive economic decision making. Yet, it is not well known how humans explore and exploit spatial regularity of rewards distributed inside a naturalistic 3D world for their optimal foraging. Using fMRI and Minecraft-based interactive games, we aim to seek how human transform external value into a foraging action under given environmental states and behavioral goals.
Hierarchical action planning
How do humans plan and perform hierarchically organized strategic actions to solve complex tasks? In a multi-dimensional environment, a single action may not directly lead to the final goal and reward. To achieve the final goal, it often needs to be decomposed into sub-goals and further into primitive actions. It is not well known how humans can substitute complex goals and actions with a less-complex structure and link them to an intrinsic reward system. Using fMRI and Minecraft-based interactive games, we aim to study high-level cognitive processes that are required to plan and perform multi-stage, hierarchical actions to achieve intended goals in a complex environment.
Social cognition
In everyday life, humans continuously learn complex relationships between objects (e.g., object network) or agents (e.g., social network). People can represent such relational knowledge as a form of “cognitive map” and infer the meaning of novel items using this map structure. Cognitive maps have been studied in many knowledge domains, such as spatial location, object identity, abstract knowledge, and social relationship. Using fMRI and Minecraft-based interactive games, we focus on how people learn and construct complex relational knowledge structures and what commonalities and differences exist in neural representations of social and non-social information.
Ultra high-field (7T) layer-specific imaging
Ultra high-field laminar fMRI
Recent developments of ultra-high field fMRI provided an unprecedented opportunity to investigate mesoscale neuroanatomical and functional information across cortical layers and columns in the human brain. In particular, the layer-specific imaging enables us to address questions on the directionality of information flow (feedforward vs. feedback) by comparing the relative contributions of each layer. Using ultra high-field laminar fMRI, we aim to resolve neural responses with laminar specificity and investigate how distinct laminar responses to feedforward and feedback inputs arise as the integration of information progresses.
High-resolution imaging and large-scale dataset
Laminar fMRI
Recent developments of ultra-high- field fMRI provided an unprecedented opportunity to investigate mesoscale neuroanatomical and functional information across cortical layers and columns in the human brain. In particular, layer-specific imaging enables addressing questions on the directionality of information flow through comparing the relative contributions of each layer. Using ultra-high-field laminar fMRI, we aim to resolve neural responses with laminar specificity and investigate how distinct laminar responses to feedforward and feedback inputs arise as the integration of information progresses. Specifically, we target the primary visual cortex to compare the layer-specific representation of physical stimulus and that of illusory representation.
7T NatPAC (Naturalistic Perception, Action, and Cognition) dataset
Large-scale human neuroimaging data sets have played a pivotal role in mapping brain and cognitive functions at a macroscopic level across large populations. Such data sets provide unprecedented opportunities to examine the healthy and diseased human brain. In this project, we aim to go beyond these big data projects and utilize the state-of-the-art high-resolution functional imaging technique at 7T in order to comprehensively characterize the cognitive landscape of individuals by acquiring large human brain anatomy and functional 7T data sets. The unique strengths of our data sets can be found 1) in its extensive coverage of a breadth of sensory, cognitive, and affective processes within individual subjects using a wide range of tasks, from sophisticated functional localizers to naturalistic cognitive tasks, 2) large scale, high-spatial resolution (1.5 mm3 isotropic voxels) whole-brain images obtained at 7T that can provide new insights into human cognitive processes when combined with computational neuroscience and machine learning, and 3) newly developed naturalistic tasks designed to probe diverse higher-order cognitive functions, including audio-video narratives, free speech, and interactive video games on a Minecraft platform. We will make use of the data for examining low-level sensory to high-level cognitive processes in an all-encompassing fashion, developing novel data analysis methods and computational models, benchmarking cutting-edge artificial intelligence methods, and providing high-quality anatomical and functional data as normative databases combined with behavioral and epidemiological measures.