von der Malsburg, T. and Angele, B., 2015. False positive rates in standard analyses of eye movements in reading arXiv:1504.06896 [stat.AP]. Working Paper. arXiv.
Full text available as:
PDF ((arXiv preprint) arXiv:1504.06896v1 [stat.AP])
1504.06896v1.pdf - Submitted Version
Official URL: http://arxiv.org/abs/1504.06896
In research on eye movements in reading, it is common to analyze a number of canonical dependent measures in order to study how the effects of a manipulation unfold over time. Although this gives rise to the well-known multiple comparisons problem, i.e. an inflated probability that the null hypothesis is incorrectly rejected (Type I error), it is accepted standard practice not to apply any correction procedures. Instead, there is a widespread belief that corrections are not necessary because the increase in false positives is too small to matter. To our knowledge, no formal argument has ever been made to justify this assumption. Here, we report an investigation of this issue using Monte Carlo simulations. Our results show that false positives are in fact increased to unacceptable levels when no correction is applied, which casts doubt on the assumptions that Type I error rate increases are too small to matter in practice. We also tested two stricter alternative criteria for determining the reliability of an effect and found that the Bonferroni correction controls false positives effectively while only moderately reducing power. Thus, there is little reason why the Bonferroni correction should not be made a standard requirement for analyses of eye movement data in reading.
|Item Type:||Monograph (Working Paper)|
|Group:||Faculty of Science and Technology|
|Deposited By:||Unnamed user with email symplectic@symplectic|
|Deposited On:||22 Jun 2015 12:07|
|Last Modified:||22 Jun 2015 12:07|
Downloads per month over past year
|Repository Staff Only -|