Swets et al. (2008) present evidence that the so-called ambiguity advantage (Traxler et al., 1998), which has been explained in terms of the Unrestricted Race Model, can equally well be explained by assuming underspecification in ambiguous conditions driven by task-demands. Specifically, if comprehension questions require that ambiguities be resolved, the parser tends to make an attachment: when questions are about superficial aspects of the target sentence, readers tend to pursue an underspecification strategy. It is reasonable to assume that individual differences in strategy will play a significant role in the application of such strategies, so that studying average behavior may not be informative. In order to study the predictions of the good-enough processing theory, we implemented two versions of underspecification: the partial specification model (PSM), which is an implementation of the Swets et al. proposal, and a more parsimonious version, the non-specification model (NSM). We evaluate the relative fit of these two kinds of underspecification to Swets et al.’s data; as a baseline, we also fit three models that assume no underspecification. We find that a model without unspecification pro- vides a somewhat better fit than both underspecification models, while the NSM model provides a better fit than the PSM. We interpret the results as lack of unambiguous evidence in favor of underspecification; however, given that there is considerable existing evidence for good-enough processing in the literature, it is reasonable to assume that some underspecification might occur. Under this assumption, the results can be interpreted as tentative ev- idence for NSM over PSM. More generally, our work provides a method for choosing between models of real-time processes in sentence comprehension that make qualitative predictions about the relationship between several de- pendent variables. We believe that sentence processing research will greatly benefit from a wider use of such methods.
Tuesday, November 10, 2015
Understanding underspecification: A comparison of two computational implementations (Logacev et al) accepted in: Quarterly Journal of Experimental Psychology
Thursday, October 15, 2015
Tuesday, October 13, 2015
Saturday, October 3, 2015
Some thoughts after attending a conference in Copenhagen
I just got done with a very nice conference in Copenhagen on grammar vs lexicon.
One thing that struck me afresh about several of the people I spoke to there and the talks I heard there is that scientists feel compelled to hold or stand for a theoretical position. People often design their careers around a position that they hold, and then they proceed to defend it no matter what data comes their way. Doing science is very much like a forecasting problem. Your job is to come up with a prediction of what will happen if a particular experiment is run.
The way we do science, however, is as follows. We first find out what the experiment showed. Then we make the "prediction" based on our favorite theory. Researchers routinely use the word prediction even when they already know the outcome of an experiment. If this was a weather forecasting problem, it would be like publishing the probability of rain yesterday. Of course you would get everything right! It is this unfortunate tendency to predict after the fact that people are so confident about their theories and positions. After the fact prediction gives an illusion of being right all the time.
I just read a great review of a book on forecasting by the greatest reviewer I have ever encountered on the web: RK, of RK's musings fame.
He discusses a book, Superforecasters, in which the author lays out the qualities of a good forecaster. I quote from the blog almost verbatim:
And here is another quote from the blog, which itself is a quote from the book:
Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others-and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.
And finally, RK also excerpts a composite portrait of a good forecaster from the book:
Scientists in psycholinguistics tend to be the exact opposite of a good forecaster.
They hunker down and defend to death one position, never never never back down in the face of counterevidence, never entertain multiple alternative theories simultaneously, never express any self-doubt (at least not publicly) that their favorite position might be wrong. Whenever we write papers, we end up converging on what we claim is the most plausible explanation for the result we have found. We never end on an equivocation, because that would mean rejection from the top journal we have submitted our paper to.
If anyone other than me is reading this blog, maybe you should read RK's original review of the book, Superforecasters, and maybe also read the book (I know I will), and then think about what's wrong with the way you are doing science, because it is bass-ackwards. We are terrible forecasters, and there's a damn good reason for it!
One thing that struck me afresh about several of the people I spoke to there and the talks I heard there is that scientists feel compelled to hold or stand for a theoretical position. People often design their careers around a position that they hold, and then they proceed to defend it no matter what data comes their way. Doing science is very much like a forecasting problem. Your job is to come up with a prediction of what will happen if a particular experiment is run.
The way we do science, however, is as follows. We first find out what the experiment showed. Then we make the "prediction" based on our favorite theory. Researchers routinely use the word prediction even when they already know the outcome of an experiment. If this was a weather forecasting problem, it would be like publishing the probability of rain yesterday. Of course you would get everything right! It is this unfortunate tendency to predict after the fact that people are so confident about their theories and positions. After the fact prediction gives an illusion of being right all the time.
I just read a great review of a book on forecasting by the greatest reviewer I have ever encountered on the web: RK, of RK's musings fame.
He discusses a book, Superforecasters, in which the author lays out the qualities of a good forecaster. I quote from the blog almost verbatim:
- Good back of the envelope calculations
- Starting with outside view that reduces anchoring bias
- Subsequent to outside view, get a grip on the inside view
- Look out for various perspectives about the problem
- Think three/four times, think deeply to root out confirmation bias
- It's not the raw crunching power you have that matters most. It's what you do with it.
And here is another quote from the blog, which itself is a quote from the book:
Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others-and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.
And finally, RK also excerpts a composite portrait of a good forecaster from the book:
Scientists in psycholinguistics tend to be the exact opposite of a good forecaster.
They hunker down and defend to death one position, never never never back down in the face of counterevidence, never entertain multiple alternative theories simultaneously, never express any self-doubt (at least not publicly) that their favorite position might be wrong. Whenever we write papers, we end up converging on what we claim is the most plausible explanation for the result we have found. We never end on an equivocation, because that would mean rejection from the top journal we have submitted our paper to.
If anyone other than me is reading this blog, maybe you should read RK's original review of the book, Superforecasters, and maybe also read the book (I know I will), and then think about what's wrong with the way you are doing science, because it is bass-ackwards. We are terrible forecasters, and there's a damn good reason for it!
Saturday, September 26, 2015
ESSLLI 2016 course: Sentence Comprehension as a Cognitive Process: A Computational Approach
Felix Engelmann and I will teach a one-week course at ESSLLI 2016 in Bolzano. I made a preliminary web page for the course, available here.
This is our first step towards writing our book (with the same title as the course), on contract with Cambridge University Press.
This is our first step towards writing our book (with the same title as the course), on contract with Cambridge University Press.
Thursday, September 3, 2015
New paper (Paape and Vasishth, Lang. and Speech): Local coherence and preemptive digging-in effects in German
Dario Paape's new paper has been accepted for publication by Language and Speech:
Title: Local coherence and preemptive digging-in effects in German
Abstract: SOPARSE (Tabor & Hutchins, 2004) predicts so-called local coherence effects: locally plausible but globally impossible parses of substrings can exert a distracting influence during sentence processing. Additionally, it predicts digging-in effects: the longer the parser stays committed to a particular analysis, the harder it becomes to inhibit that analysis. We investigated the interaction of these two predictions using German sentences. Results from a self-paced reading study show that the processing difficulty caused by a local coherence can be reduced by first allowing the globally correct parse to become entrenched, which supports SOPARSE’s assumptions.
pdf: http://www.ling.uni-potsdam.de/~paape/LCpaper.pdf
Title: Local coherence and preemptive digging-in effects in German
Abstract: SOPARSE (Tabor & Hutchins, 2004) predicts so-called local coherence effects: locally plausible but globally impossible parses of substrings can exert a distracting influence during sentence processing. Additionally, it predicts digging-in effects: the longer the parser stays committed to a particular analysis, the harder it becomes to inhibit that analysis. We investigated the interaction of these two predictions using German sentences. Results from a self-paced reading study show that the processing difficulty caused by a local coherence can be reduced by first allowing the globally correct parse to become entrenched, which supports SOPARSE’s assumptions.
pdf: http://www.ling.uni-potsdam.de/~paape/LCpaper.pdf
Monday, August 31, 2015
New paper: Locality and expectation in separable Persian complex predicates
Here's a new paper by Molood Sadat Safavi, Samar Husain, and Shravan Vasishth, which shows evidence from two self-paced reading studies in Persian against one of the key predictions of the expectation accounts (Hale 2001, Levy 2008).
Title:
Locality and expectation in Persian complex predicates
Abstract:
In sentence comprehension, it is well-known that processing cost increases with dependency distance (Gibson 2000, Lewis and Vasishth 2005); this often referred to as the locality effect. However, the expectation-based account (Hale 2001, Levy 2008) predicts that delaying the appearance of a verb renders it more predictable and therefore easier to process. Following up on previous work (Husain et al 2014), we investigated whether strengthening the expectation can increase facilitation at the verb even further. We operationalize strong expectation as prediction of the lexical entry for the verb; by contrast, weak expectation refers to the prediction of some upcoming verb phrase (these are the cases discussed by Levy 2008). We used Persian for this investigation. This language has a special construction called complex predicates, which are separable Noun-Verb configurations in which the verb (the precise lexical item) is highly predictable given the noun. In two self-paced reading experiments, we delayed the appearance of the verb by interposing a relative clause (Expt 1, 42 subjects) or a long PP (Expt 2, 40 subjects). As a control, we included a simple predicate (Noun-Verb) configuration; the same distance manipulation was applied here as for complex predicates, but here, the exact lexical entry for the verb is not predicted but rather a verb phrase is predicted. Thus, we had a 2x 2 design, with Expectation Strength (Strong/Weak) and Distance (Short/Long). Based on the Husain et al study, which had a similar design using Hindi complex predicates, we expected a slowdown in the weak expectation condition (i.e., locality effects), but a facilitation in the strong expectation conditions (i.e., expectation effects). Surprisingly, both experiments showed clear effects of locality in both the strong and weak expectation conditions. We also find evidence that could be consistent with expectation effects: the high-predictable verbs are read faster than the low-predictable verbs. However, this result is difficult to interpret because the verbs used in the strong and weak expectation conditions are different. In sum, these studies show strong and unequivocal evidence in favor of argument-verb dependency distance influencing integration processes at the verb, falsifying a key prediction of the expectation based account of Levy 2008.
Title:
Locality and expectation in Persian complex predicates
Abstract:
In sentence comprehension, it is well-known that processing cost increases with dependency distance (Gibson 2000, Lewis and Vasishth 2005); this often referred to as the locality effect. However, the expectation-based account (Hale 2001, Levy 2008) predicts that delaying the appearance of a verb renders it more predictable and therefore easier to process. Following up on previous work (Husain et al 2014), we investigated whether strengthening the expectation can increase facilitation at the verb even further. We operationalize strong expectation as prediction of the lexical entry for the verb; by contrast, weak expectation refers to the prediction of some upcoming verb phrase (these are the cases discussed by Levy 2008). We used Persian for this investigation. This language has a special construction called complex predicates, which are separable Noun-Verb configurations in which the verb (the precise lexical item) is highly predictable given the noun. In two self-paced reading experiments, we delayed the appearance of the verb by interposing a relative clause (Expt 1, 42 subjects) or a long PP (Expt 2, 40 subjects). As a control, we included a simple predicate (Noun-Verb) configuration; the same distance manipulation was applied here as for complex predicates, but here, the exact lexical entry for the verb is not predicted but rather a verb phrase is predicted. Thus, we had a 2x 2 design, with Expectation Strength (Strong/Weak) and Distance (Short/Long). Based on the Husain et al study, which had a similar design using Hindi complex predicates, we expected a slowdown in the weak expectation condition (i.e., locality effects), but a facilitation in the strong expectation conditions (i.e., expectation effects). Surprisingly, both experiments showed clear effects of locality in both the strong and weak expectation conditions. We also find evidence that could be consistent with expectation effects: the high-predictable verbs are read faster than the low-predictable verbs. However, this result is difficult to interpret because the verbs used in the strong and weak expectation conditions are different. In sum, these studies show strong and unequivocal evidence in favor of argument-verb dependency distance influencing integration processes at the verb, falsifying a key prediction of the expectation based account of Levy 2008.
Subscribe to:
Posts (Atom)