Probabilistic Linguistic Expectations, Uncertain Input, and Implications for Eye Movements in Reading

Abstract

Abstract: One nearly ubiquitous assumption in models of linguistic comprehension and of eye movement control in reading alike is of partial modularization between word-level and sentence-level processing: that the outcome of word recognition, and thus the input to sentence-level comprehension, is a categorial representation. Yet such a partial modularization throws away residual uncertainty regarding word identity that might potentially be of value to the comprehender further downstream in the sentence. Here I describe a line of research combining computational modeling with experimental eye-tracking work to explore the consequences of removing this partial modularity assumption.

摘要： One nearly ubiquitous assumption in models of linguistic comprehension and of eye movement control in reading alike is of partial modularization between word-level and sentence-level processing: that the outcome of word recognition, and thus the input to sentence-level comprehension, is a categorial representation. Yet such a partial modularization throws away residual uncertainty regarding word identity that might potentially be of value to the comprehender further downstream in the sentence. Here I describe a line of research combining computational modeling with experimental eye-tracking work to explore the consequences of removing this partial modularity assumption.

Roger Levy. Probabilistic Linguistic Expectations, Uncertain Input, and Implications for Eye Movements in Reading[J]. Studies of Psychology and Behavior, 2011, 9(1): 53-64.

Roger Levy. Probabilistic Linguistic Expectations, Uncertain Input, and Implications for Eye Movements in Reading[J]. 心理与行为研究, 2011, 9(1): 53-64.

References

[1] Altmann, G. T., & Kamide, Y.(1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73 (3), 247-264.
[2] Bicknell, K., & Levy, R.(2009). A model of local coherence effects in human sentence processing as consequences of updates from bottom-up prior to posterior beliefs. In Proceedings of the north american chapter of the association for computational linguistics-human language technologies(NAACL-HLT)conference.
[3] Bicknell, K., & Levy, R.(2010). A rational model of eye movement control in reading. In Proceedings of the annual meeting of the association for computational linguistics (pp.1168-1178).
[4] Booth, T. L.(1969). Probabilistic representation of formal languages. In Ieee conference record of the 1969 tenth annual symposium on switching and automata theory (pp.74-81).
[5] Chen, S., & Goodman, J.(1998). An empirical study of smoothing techniques for language modeling (Tech. Rep.). Computer Science Group, Harvard University.
[6] Chomsky, N.(1956). Three models for the description of language. IRE Transactions on Information Theory, 2 (3), 113-124.
[7] Church, K., & Patil, R.(1982). Coping with syntactic ambiguity, or how to put the block in the box on the table. Computational Linguistics, 8 (3-4), 139-149.
[8] Cover, T., & Thomas, J.(1991). Elements of information theory . John Wiley.
[9] Crocker, M., & Brants, T.(2000). Wide - coverage probabilistic sentence processing. Journal of Psycholinguistic Research, 29 (6), 647-669.
[10] Earley, J.(1970). An efficient context - free parsing algorithm. Communications of the ACM, 13 (2), 94-102.
[11] Ehrlich, S. F., & Rayner, K.(1981). Contextual effects on word perception and eye movements during reading. Journal of Verbal Learning and Verbal Behavior, 20 , 641-655.
[12] Ford, M., Bresnan, J., & Kaplan, R. M.(1982). A competence - based theory of syntactic closure. In J. Bresnan(Ed.), The mental representation of grammatical relations (pp. 727-796). The MIT Press.
[13] Frazier, L., & Rayner, K.(1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14 , 178-210.
[14] Gibson, E.(2006). The interaction of top-down and bottom-up statistics in the resolution of syntactic category ambiguity. Journal of Memory and Language, 54 , 363-388.
[15] Ginsburg, S.(1966). The mathematical theory of context - free languages . McGraw - Hill.
[16] Hale, J.(2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the north american chapter of the association for computational linguistics (pp. 159-166).
[17] Hale, J.(2003). The information conveyed by words in sentences. Journal of Psycholinguistic Research, 32 (2), 101-123.
[18] Hale, J.(2006). Uncertainty about the rest of the sentence. Cognitive Science, 30 (4), 609-642.
[19] Hopcroft, J. E., & Ullman, J. D.(1979). Introduction to automata theory, languages, and computation . Addison - Wesley.
[20] Jelinek, F., & Lafferty, J. D.(1991). Computation of the probability of initial substring 24 generation by stochastic context free grammars. Computational Linguistics, 17 (3), 315-323.
[21] Jurafsky, D.(1996). A probabilistic model of lexical and syntactic access and disambiguation. Cognitive Science, 20 (2), 137-194.
[22] Jurafsky, D., & Martin, J. H.(2008). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (Second ed.). Prentice - Hall.
[23] Kamide, Y., Altmann, G., & Haywood, S.(2003). The time - course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language, 49 (1), 133-156.
[24] Kliegl, R., Nuthmann, A., & Engbert, R.(2006). Tracking the mind during reading: The influence of past, present, and future words on fixation durations. Journal of Experimental Psychology: General, 135 (1), 12-35.
1., H., & Francis, W. N.(1967). Computational analysis of present - day American English. Providence, RI: Brown University Press.
[25] Levenshtein, V. I.(1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10 , 707-710.
[26] Levy, R.(2008a). Expectation - based syntactic comprehension. Cognition, 106 , 1126-1177.
[27] Levy, R.(2008b). A noisy - channel model of rational human sentence comprehension under uncertain input. In Proceedings of the 13th conference on empirical methods in natural language processing (pp. 234-243).
[28] Levy, R.(2010, March). On hallucinated garden paths . Poster presented at the CUNY sentence processing conference.
[29] Levy, R., Bicknell, K., Slattery, T., & Rayner, K.(2009). Eye movement evidence that readers maintain and act on uncertainty about past linguistic input. Proceedings of the national academy of sciences, 106 (50), 21086-21090.
[30] Levy, R., Reali, F., & Griffiths, T. L.(2009). Modeling the effects of memory on human online sentence processing with particle filters. In Proceedings of the 22nd conference on neural information processing systems(nips) .
[31] Manning, C. D., & Schütze, H.(1999). Foundations of statistical Kuera natural language processing . MIT Press.
[32] Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A.(1994). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19 (2), 313-330.
[33] Morgan, E., Keller, F., & Steedman, M.(2010). A bottom - up parsing model of local coherence effects. In Proceedings of the cognitive science society conference .
[34] Narayanan, S., & Jurafsky, D.(1998). Bayesian models of human sentence processing. In Proceedings of the twelfth annual meeting of the cognitive science society .
[35] Narayanan, S., & Jurafsky, D.(2002). A Bayesian model predicts human parse preference and reading time in sentence processing. In Advances in neural information processing systems (Vol.14, pp. 59-65).
[36] Norris, D.(2006). The Bayesian Reader: Explaining word recognition as an optimal Bayesian decision process. Psychological Review, 113 (2), 327-357.
[37] Rayner, K.(1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124 (3), 372-422.
[38] Rayner, K.(2009). The thirty-fifth Sir Frederick Bartlett Lecture: Eye movements and attention during reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62 , 1457-1506.
[39] Rayner, K., & Well, A. D.(1996). Effects of contextual constraint on eye movements in reading: A further examination. Psychonomic Bulletin & Review, 3 (4), 504-509.
[40] Roark, B.(2001). Probabilistic top-down parsing and language modeling. Computational Linguistics, 27 (2), 249-276.
[41] Russell, S., & Norvig, P.(2003). Artificial intelligence: A modern approach (Second ed.). Prentice Hall.
[42] Smith, N. J., & Levy, R.(2008). Optimal processing times in reading: A formal model and empirical investigation. In Proceedings of the 30th annual meeting of the cognitive science society.
[43] Smith, N. J., & Levy, R.(2010). Fixation durations in first-pass reading reflect uncertainty about word identity. In Proceedings of the 32nd annual meeting of the cognitive science society.
[44] Smith, N. J., & Levy, R.(Submitted). A rational model of predictability effects on cognitive processing times .(Manuscript, UC San Diego).
[45] Stolcke, A.(1995). An efficient probabilistic context-free parsing algorithm that computes prefix probabilities. Computational Linguistics, 21 (2), 165-201.
[46] Sutton, R. S., & Barto, A. G.(1998). Reinforcement learning. MIT Press.
[47] Tabor, W., Galantucci, B., & Richardson, D.(2004). Effects of merely local syntactic coherence on sentence processing. Journal of Memory and Language, 50 (4), 355-370.
[48] Tabor, W., & Hutchins, S.(2004). Evidence for self - organized sentence processing: Digging in effects. Journal of Experimental Psychology: Learning, Memory, & Cognition, 30 (2), 431-450.
[49] Tanenhaus, M. K., Spivey - Knowlton, M. J., Eberhard, K., & Sedivy, J. C.(1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268 , 1632-1634.
[50] Taylor, W. L.(1953). A new tool for measuring readability. Journalism Quarterly, 30 , 415.