Prediction of cognitive scores by joint use of movie-watching fMRI connectivity and eye tracking via Attention-CensNet  

作　　者：Jiaxing Gao Lin Zhao Tianyang Zhong Changhe Li Zhibin He Yaonei Wei Shu Zhang Lei Guo Tianming Liu Junwei Han Xi Jiang Tuo Zhang 

机构地区：[1]School of Automation,Northwestern Polytechnical University,XIi'an 710072,China [2]Cortical Architecture Imaging and Discovery Laboratory,Department of Computer Science and Bioimaging Research Center,The University of Georgia,Athens,GA 30602,USA [3]School of Computer Science,Northwestern Polytechnical University,Xi'an 710072,China [4]School of Life Science and Technology,MOE Key Laboratory for Neuroinformation,University of Electronic Science and Technology of China,Chengdu 611731,China

出　　处：《Psychoradiology》2023年第1期193-202,共10页精神影像学（英文）

基　　金：supported by the National Natural Science Foundation of China (31971288,U1801265,61936007,62276050,61976045,U20B2065,U1801265 and 61936007);the National Key R&D Pro-gram of China under Grant 2020AAA0105701;High-level researcher start-up projects (Grant No.06100-22GH0202178);Innovation Foundation for Doctor Dissertation of Northwestern Poly-technical University CX2022052.

摘　　要：Background:Brain functional connectivity under the naturalistic paradigm has been shown to be better at predicting individual behaviors than other brain states,such as rest and doing tasks.Nevertheless,the state-of-the-art methods have found it difcult to achieve desirable results from movie watching paradigm functional magnetic resonance imaging(mfMRI)-induced brain functional connectivity,especially when there are fewer datasets.Incorporating other physical measurements into the prediction method may enhance accuracy.Eye tracking,becoming popular due to its portability and lower expense,can provide abundant behavioral features related to the output of human's cognition,and thus might supplement the mfMRI in observing participants'subconscious behaviors.However,there are very few studies on how to effectively integrate the multimodal information to strengthen the performance by a unified framework.objective:A fusion approach with mfMRI and eye tracking,based on convolution with edge-node switching in graph neural networks(CensNet),is proposed in this article.Methods:In this graph model,participants are designated as nodes,mfMRI derived functional connectivity as node features,and different eye-tracking features are used to compute similarity between participants to construct heterogeneous graph edges.By taking multiple graphs as different channels,we introduce squeeze-and-excitation attention module to CensNet(A-CensNet)to integrate graph embeddings from multiple channels into one.Results:The proposed model outperforms those using a single modality and single channel,and state-of-the-art methods.Conclusions:The results indicate that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes.

关 键 词：functional connectivity naturalistic stimulus eye movement CensNet ATTENTION 

分 类 号：R749[医药卫生—神经病学与精神病学]