该文系统地研究了中国手语模态对句子加工的影响，使用事件相关电位（ERP）技术来探索句子加工过程中手势模态的语义和语音信息对词汇通达的影响和时间过程。设计实验材料时，对句尾变化条件下目标手势的语义和音韵形式进行了系统的人为操纵。通过18个被试的实验，观察到将初级视觉特征映射到手势音韵特征的手势感知证据（时长100~200ms），紧接着是将目标手势音韵特征与语境信息匹配的负成分（时长200~400ms），以及表征语篇语境对语义制约的动态建构的负成分（时长400~650ms），它促进了手势的音义信息与句子信息的整合。音近和义近手势在时间进程和效应方向上存在着相似性，表明手语模态的词汇通达与有声语言类似，并消耗类似的在线处理成本。同时发现了手语模态独有的表征手语音韵特征加工已经完成的N300效应，它代表了手势音韵元素识别和手势识别之间的直接联系。实验结果表明：在进行句子加工时，被试首要考虑的是语境和目标手势匹配，并不是与手势的音韵特征匹配。这种结果可能是词汇选择时音近和义近相关信息相互作用的标志。
The effects of Chinese sign language (CSL) modality on processing sentences are investigated by using the event-related potential method to focus on the influence of both semantic and phonological information on the lexical accessment during sentence processing and its timing. The semantic expectancy and phonetic form of the target signs with the final inflections are manipulated during stimuli selection in the test design. The electroencephalogram data of 18 native CSL signers were recorded and analyzed. The sign perception evidence (100-200 ms) reflects the process of mapping the primary visual features into the phonological features of the signs. This was followed by two negativities associated with matching the phonological features of the target sign with contextual information (200-400 ms) and the dynamic construction of the semantic restriction of the discourse context (400-650 ms) that promoted integration of the lexicon and sentence context. The phonological and semantic-related signs shared a similar timing pattern and the effect direction verified the similarity of the semantic and phonological properties access and the similar online processing cost of CSL and ordinary Chinese sentence comprehension. An N300 effect was found that uniquely indicates the successful processing of phonetic features in the sign language modality. This effect represents the direct link between the phonological feature recognition and the sign recognition. The results show that the subject considers the match between context and target sign as a priority, rather than the phonological features of the signs. This result indicates the interaction between the phonological and semantic-related information at the moment of lexical selection.
[1] Kutas M, Hillyard S A. Reading senseless sentences: Brain potentials reflect semantic incongruity [J]. Science, 1980, 207(4427): 203-205.
[2] Friederici A D. Towards a neural basis of auditory sentence processing [J]. Trends in Cognitive Sciences, 2002, 6(2): 78-84.
[3] Kolk H H J, Chwilla D J, Van Herten M, et al. Structure and limited capacity in verbal working memory: A study with event-related potentials [J]. Brain and language, 2003, 85(1): 1-36.
[4] Kuperberg G R. Neural mechanisms of language comprehension: Challenges to syntax [J]. Brain Research, 2007, 1146: 23-49.
[5] Bornkessel I, Schlesewsky M. The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages [J]. Psychological Review, 2006, 113(4): 787.
[6] Brouwer H, Fitz H, Hoeks J. Getting real about semantic illusions: rethinking the functional role of the P600 in language comprehension [J]. Brain Research, 2012, 1446: 127-143.
[7] Stevens K N, Blumstein S E. The search for invariant acoustic correlates of phonetic features [J]. Perspectives on the Study of Speech, 1981: 1-38.
[8] Stokoe W C. Sign language structure: An outline of the visual communication systems of the American deaf [J]. Journal of deaf studies and deaf education, 2005, 10(1): 3-37.
[9] Morgan G, Barrett-Jones S, Stoneham H. The first signs of language: Phonological development in British Sign Language [J]. Applied Psycholinguistics, 2007, 28(01): 3-22.
[10] Corina D P, Hildebrandt U C. Psycholinguistic investigations of phonological structure in ASL [J]. Modality and Structure in Signed and Spoken Languages, 2002: 88-111.
[11] Carreiras M, Gutiérrez-Sigut E, Baquero S, et al. Lexical processing in Spanish sign language (LSE) [J]. Journal of Memory and Language, 2008, 58(1): 100-122.
[12] Dufour S, Peereman R. Inhibitory priming effects in auditory word recognition: When the target's competitors conflict with the prime word [J]. Cognition, 2003, 88(3): B33-B44.
[13] Gutierrez E, Williams D, Grosvald M, et al. Lexical access in American Sign Language: An ERP investigation of effects of semantics and phonology [J]. Brain Research, 2012, 1468: 63-83.
[14] Neville H J, Coffey S A, Lawson D S, et al. Neural systems mediating American Sign Language: Effects of sensory experience and age of acquisition [J]. Brain and Language, 1997, 57(3): 285-308.
[15] Eimer M. Does the face-specific N170 component reflect the activity of a specialized eye processor? [J]. Neuroreport, 1998, 9(13): 2945-2948.
[16] Eimer M. The face-specific N170 component reflects late stages in the structural encoding of faces [J]. Neuroreport, 2000, 11(10): 2319-2324. -2324.
[17] Chauncey K, Holcomb P J, Grainger J. Effects of stimulus font and size on masked repetition priming: An event-related potentials (ERP) investigation [J]. Language and Cognitive Processes, 2008, 23(1): 183-200.
[18] Emmorey K, Corina D. Lexical recognition in sign language: Effects of phonetic structure and morphology [J]. Perceptual and Motor Skills, 1990, 71(3f): 1227-1252.
[19] Dambacher M, Rolfs M, Giillner K, et al. Event-related potentials reveal rapid verification of predicted visual input [J]. PLoS ONE, 2009, 4(3): e5047.
[20] Kim A, Lai V. Rapid interactions between lexical semantic and word form analysis during word recognition in context: Evidence from ERPs [J]. Journal of Cognitive Neuroscience, 2012, 24(5): 1104-1112.
[21] Emmorey K, Damasio H, McCullough S, et al. Neural systems underlying spatial language in American Sign Language [J]. Neuroimage, 2002, 17(2): 812-824.
[22] Kutas M, Neville H J, Holcomb P J. A preliminary comparison of the N400 response to semantic anomalies during reading, listening and signing [J]. Electroencephalography and Clinical Neurophysiology Supplement, 1987, 39: 325-330.
[23] Colin C, Zuinen T, Bayard C, et al. Phonological processing of rhyme in spoken language and location in sign language by deaf and hearing participants: A neurophysiological study [J]. Neurophysiologie Clinique/Clinical Neurophysiology, 2013, 43(3): 151-160.
[24] Barrett S E, Rugg M D. Event-related potentials and the semantic matching of pictures [J]. Brain and cognition, 1990, 14(2): 201-212.
[25] McPherson W B, Holcomb P J. An electrophysiological investigation of semantic priming with pictures of real objects [J]. Psychophysiology, 1999, 36(01): 53-65.
[26] Luck S J. An introduction to the event-related potential technique [M]. MIT press, 2014.
[27] Holcomb P J, Grainger J, O'rourke T. An electrophysiological study of the effects of orthographic neighborhood size on printed word perception [J]. Journal of Cognitive Neuroscience, 2002, 14(6): 938-950.
[28] Massol S, Grainger J, Dufau S, et al. Masked priming from orthographic neighbors: An ERP investigation [J]. Journal of Experimental Psychology: Human Perception and Performance, 2010, 36(1): 162-174.
[29] Diependaele K, Ziegler J C, Grainger J. Fast phonology and the bimodal interactive activation model [J]. European Journal of Cognitive Psychology, 2010, 22(5): 764-778.
[30] Niznikiewicz M, Squires N K. Phonological processing and the role of strategy in silent reading: behavioral and electrophysiological evidence [J]. Brain and Language, 1996, 52(2): 342-364.
[31] McClelland J L, Rumelhart D E. An interactive activation model of context effects in letter perception: I. An account of basic findings [J]. Psychological Review, 1981, 88(5): 375-407.
