Thursday, October 15, 2015
Tuesday, October 13, 2015
Saturday, October 3, 2015
Some thoughts after attending a conference in Copenhagen
I just got done with a very nice conference in Copenhagen on grammar vs lexicon.
One thing that struck me afresh about several of the people I spoke to there and the talks I heard there is that scientists feel compelled to hold or stand for a theoretical position. People often design their careers around a position that they hold, and then they proceed to defend it no matter what data comes their way. Doing science is very much like a forecasting problem. Your job is to come up with a prediction of what will happen if a particular experiment is run.
The way we do science, however, is as follows. We first find out what the experiment showed. Then we make the "prediction" based on our favorite theory. Researchers routinely use the word prediction even when they already know the outcome of an experiment. If this was a weather forecasting problem, it would be like publishing the probability of rain yesterday. Of course you would get everything right! It is this unfortunate tendency to predict after the fact that people are so confident about their theories and positions. After the fact prediction gives an illusion of being right all the time.
I just read a great review of a book on forecasting by the greatest reviewer I have ever encountered on the web: RK, of RK's musings fame.
He discusses a book, Superforecasters, in which the author lays out the qualities of a good forecaster. I quote from the blog almost verbatim:
And here is another quote from the blog, which itself is a quote from the book:
Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others-and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.
And finally, RK also excerpts a composite portrait of a good forecaster from the book:
Scientists in psycholinguistics tend to be the exact opposite of a good forecaster.
They hunker down and defend to death one position, never never never back down in the face of counterevidence, never entertain multiple alternative theories simultaneously, never express any self-doubt (at least not publicly) that their favorite position might be wrong. Whenever we write papers, we end up converging on what we claim is the most plausible explanation for the result we have found. We never end on an equivocation, because that would mean rejection from the top journal we have submitted our paper to.
If anyone other than me is reading this blog, maybe you should read RK's original review of the book, Superforecasters, and maybe also read the book (I know I will), and then think about what's wrong with the way you are doing science, because it is bass-ackwards. We are terrible forecasters, and there's a damn good reason for it!
One thing that struck me afresh about several of the people I spoke to there and the talks I heard there is that scientists feel compelled to hold or stand for a theoretical position. People often design their careers around a position that they hold, and then they proceed to defend it no matter what data comes their way. Doing science is very much like a forecasting problem. Your job is to come up with a prediction of what will happen if a particular experiment is run.
The way we do science, however, is as follows. We first find out what the experiment showed. Then we make the "prediction" based on our favorite theory. Researchers routinely use the word prediction even when they already know the outcome of an experiment. If this was a weather forecasting problem, it would be like publishing the probability of rain yesterday. Of course you would get everything right! It is this unfortunate tendency to predict after the fact that people are so confident about their theories and positions. After the fact prediction gives an illusion of being right all the time.
I just read a great review of a book on forecasting by the greatest reviewer I have ever encountered on the web: RK, of RK's musings fame.
He discusses a book, Superforecasters, in which the author lays out the qualities of a good forecaster. I quote from the blog almost verbatim:
- Good back of the envelope calculations
- Starting with outside view that reduces anchoring bias
- Subsequent to outside view, get a grip on the inside view
- Look out for various perspectives about the problem
- Think three/four times, think deeply to root out confirmation bias
- It's not the raw crunching power you have that matters most. It's what you do with it.
And here is another quote from the blog, which itself is a quote from the book:
Unpack the question into components. Distinguish as sharply as you can between the known and unknown and leave no assumptions unscrutinized. Adopt the outside view and put the problem into a comparative perspective that downplays its uniqueness and treats it as a special case of a wider class of phenomena. Then adopt the inside view that plays up the uniqueness of the problem. Also explore the similarities and differences between your views and those of others-and pay special attention to prediction markets and other methods of extracting wisdom from crowds. Synthesize all these different views into a single vision as acute as that of a dragonfly. Finally, express your judgment as precisely as you can, using a finely grained scale of probability.
And finally, RK also excerpts a composite portrait of a good forecaster from the book:
Scientists in psycholinguistics tend to be the exact opposite of a good forecaster.
They hunker down and defend to death one position, never never never back down in the face of counterevidence, never entertain multiple alternative theories simultaneously, never express any self-doubt (at least not publicly) that their favorite position might be wrong. Whenever we write papers, we end up converging on what we claim is the most plausible explanation for the result we have found. We never end on an equivocation, because that would mean rejection from the top journal we have submitted our paper to.
If anyone other than me is reading this blog, maybe you should read RK's original review of the book, Superforecasters, and maybe also read the book (I know I will), and then think about what's wrong with the way you are doing science, because it is bass-ackwards. We are terrible forecasters, and there's a damn good reason for it!
Saturday, September 26, 2015
ESSLLI 2016 course: Sentence Comprehension as a Cognitive Process: A Computational Approach
Felix Engelmann and I will teach a one-week course at ESSLLI 2016 in Bolzano. I made a preliminary web page for the course, available here.
This is our first step towards writing our book (with the same title as the course), on contract with Cambridge University Press.
This is our first step towards writing our book (with the same title as the course), on contract with Cambridge University Press.
Thursday, September 3, 2015
New paper (Paape and Vasishth, Lang. and Speech): Local coherence and preemptive digging-in effects in German
Dario Paape's new paper has been accepted for publication by Language and Speech:
Title: Local coherence and preemptive digging-in effects in German
Abstract: SOPARSE (Tabor & Hutchins, 2004) predicts so-called local coherence effects: locally plausible but globally impossible parses of substrings can exert a distracting influence during sentence processing. Additionally, it predicts digging-in effects: the longer the parser stays committed to a particular analysis, the harder it becomes to inhibit that analysis. We investigated the interaction of these two predictions using German sentences. Results from a self-paced reading study show that the processing difficulty caused by a local coherence can be reduced by first allowing the globally correct parse to become entrenched, which supports SOPARSE’s assumptions.
pdf: http://www.ling.uni-potsdam.de/~paape/LCpaper.pdf
Title: Local coherence and preemptive digging-in effects in German
Abstract: SOPARSE (Tabor & Hutchins, 2004) predicts so-called local coherence effects: locally plausible but globally impossible parses of substrings can exert a distracting influence during sentence processing. Additionally, it predicts digging-in effects: the longer the parser stays committed to a particular analysis, the harder it becomes to inhibit that analysis. We investigated the interaction of these two predictions using German sentences. Results from a self-paced reading study show that the processing difficulty caused by a local coherence can be reduced by first allowing the globally correct parse to become entrenched, which supports SOPARSE’s assumptions.
pdf: http://www.ling.uni-potsdam.de/~paape/LCpaper.pdf
Monday, August 31, 2015
New paper: Locality and expectation in separable Persian complex predicates
Here's a new paper by Molood Sadat Safavi, Samar Husain, and Shravan Vasishth, which shows evidence from two self-paced reading studies in Persian against one of the key predictions of the expectation accounts (Hale 2001, Levy 2008).
Title:
Locality and expectation in Persian complex predicates
Abstract:
In sentence comprehension, it is well-known that processing cost increases with dependency distance (Gibson 2000, Lewis and Vasishth 2005); this often referred to as the locality effect. However, the expectation-based account (Hale 2001, Levy 2008) predicts that delaying the appearance of a verb renders it more predictable and therefore easier to process. Following up on previous work (Husain et al 2014), we investigated whether strengthening the expectation can increase facilitation at the verb even further. We operationalize strong expectation as prediction of the lexical entry for the verb; by contrast, weak expectation refers to the prediction of some upcoming verb phrase (these are the cases discussed by Levy 2008). We used Persian for this investigation. This language has a special construction called complex predicates, which are separable Noun-Verb configurations in which the verb (the precise lexical item) is highly predictable given the noun. In two self-paced reading experiments, we delayed the appearance of the verb by interposing a relative clause (Expt 1, 42 subjects) or a long PP (Expt 2, 40 subjects). As a control, we included a simple predicate (Noun-Verb) configuration; the same distance manipulation was applied here as for complex predicates, but here, the exact lexical entry for the verb is not predicted but rather a verb phrase is predicted. Thus, we had a 2x 2 design, with Expectation Strength (Strong/Weak) and Distance (Short/Long). Based on the Husain et al study, which had a similar design using Hindi complex predicates, we expected a slowdown in the weak expectation condition (i.e., locality effects), but a facilitation in the strong expectation conditions (i.e., expectation effects). Surprisingly, both experiments showed clear effects of locality in both the strong and weak expectation conditions. We also find evidence that could be consistent with expectation effects: the high-predictable verbs are read faster than the low-predictable verbs. However, this result is difficult to interpret because the verbs used in the strong and weak expectation conditions are different. In sum, these studies show strong and unequivocal evidence in favor of argument-verb dependency distance influencing integration processes at the verb, falsifying a key prediction of the expectation based account of Levy 2008.
Title:
Locality and expectation in Persian complex predicates
Abstract:
In sentence comprehension, it is well-known that processing cost increases with dependency distance (Gibson 2000, Lewis and Vasishth 2005); this often referred to as the locality effect. However, the expectation-based account (Hale 2001, Levy 2008) predicts that delaying the appearance of a verb renders it more predictable and therefore easier to process. Following up on previous work (Husain et al 2014), we investigated whether strengthening the expectation can increase facilitation at the verb even further. We operationalize strong expectation as prediction of the lexical entry for the verb; by contrast, weak expectation refers to the prediction of some upcoming verb phrase (these are the cases discussed by Levy 2008). We used Persian for this investigation. This language has a special construction called complex predicates, which are separable Noun-Verb configurations in which the verb (the precise lexical item) is highly predictable given the noun. In two self-paced reading experiments, we delayed the appearance of the verb by interposing a relative clause (Expt 1, 42 subjects) or a long PP (Expt 2, 40 subjects). As a control, we included a simple predicate (Noun-Verb) configuration; the same distance manipulation was applied here as for complex predicates, but here, the exact lexical entry for the verb is not predicted but rather a verb phrase is predicted. Thus, we had a 2x 2 design, with Expectation Strength (Strong/Weak) and Distance (Short/Long). Based on the Husain et al study, which had a similar design using Hindi complex predicates, we expected a slowdown in the weak expectation condition (i.e., locality effects), but a facilitation in the strong expectation conditions (i.e., expectation effects). Surprisingly, both experiments showed clear effects of locality in both the strong and weak expectation conditions. We also find evidence that could be consistent with expectation effects: the high-predictable verbs are read faster than the low-predictable verbs. However, this result is difficult to interpret because the verbs used in the strong and weak expectation conditions are different. In sum, these studies show strong and unequivocal evidence in favor of argument-verb dependency distance influencing integration processes at the verb, falsifying a key prediction of the expectation based account of Levy 2008.
Tuesday, August 25, 2015
New paper (Engelmann, Jäger, Vasishth)
Here is a new paper by Felix Engelmann and Lena Jäger and myself that people interested in sentence comprehension processes may be interested in.
Title:
The determinants of retrieval interference in dependency resolution: Review and computational modeling
Abstract:
We report a comprehensive literature review of retrieval interference in reflexive-antecedent dependencies, number agreement, and non-agreement subject-verb dependencies, and computationally evaluate the predictions of cue-based retrieval theory with reference to published results. A novel finding from the review and modeling is that, contrary to claims in the literature, results on number agreement are not entirely compatible with cue-based retrieval theory. We also show that the cue-based retrieval account in its current form cannot explain several reported interference effects, such as (i) speed-ups observed in presence of a syntactically unlicensed distractor when the correct dependent is a full match to the retrieval cues and (ii) slow-downs when the correct dependent only partially matches the retrieval cues. We demonstrate that these effects can be explained by two theoretical and independently motivated constructs: distractor prominence and cue confusion. The cue-based retrieval model is therefore extended to incorporate distractor prominence and cue confusion, and quantitative predictions are derived from this extended model. We show that the extended cue-based retrieval model provides a better explanation of published results than the classical retrieval account.
The pdf is here:
http://www.ling.uni-potsdam.de/~engelmann/publications/EngelmannEtAl_JML_subm_150825.doc.pdf
Title:
The determinants of retrieval interference in dependency resolution: Review and computational modeling
Abstract:
We report a comprehensive literature review of retrieval interference in reflexive-antecedent dependencies, number agreement, and non-agreement subject-verb dependencies, and computationally evaluate the predictions of cue-based retrieval theory with reference to published results. A novel finding from the review and modeling is that, contrary to claims in the literature, results on number agreement are not entirely compatible with cue-based retrieval theory. We also show that the cue-based retrieval account in its current form cannot explain several reported interference effects, such as (i) speed-ups observed in presence of a syntactically unlicensed distractor when the correct dependent is a full match to the retrieval cues and (ii) slow-downs when the correct dependent only partially matches the retrieval cues. We demonstrate that these effects can be explained by two theoretical and independently motivated constructs: distractor prominence and cue confusion. The cue-based retrieval model is therefore extended to incorporate distractor prominence and cue confusion, and quantitative predictions are derived from this extended model. We show that the extended cue-based retrieval model provides a better explanation of published results than the classical retrieval account.
The pdf is here:
http://www.ling.uni-potsdam.de/~engelmann/publications/EngelmannEtAl_JML_subm_150825.doc.pdf
Subscribe to:
Posts (Atom)