False positive rates in standard analyses of eye movements in reading arXiv:1504.06896 [stat.AP].

Abstract

In research on eye movements in reading, it is common to analyze a number of canonical dependent measures in order to study how the effects of a manipulation unfold over time. Although this gives rise to the well-known multiple comparisons problem, i.e. an inflated probability that the null hypothesis is incorrectly rejected (Type I error), it is accepted standard practice not to apply any correction procedures. Instead, there is a widespread belief that corrections are not necessary because the increase in false positives is too small to matter. To our knowledge, no formal argument has ever been made to justify this assumption. Here, we report an investigation of this issue using Monte Carlo simulations. Our results show that false positives are in fact increased to unacceptable levels when no correction is applied, which casts doubt on the assumptions that Type I error rate increases are too small to matter in practice. We also tested two stricter alternative criteria for determining the reliability of an effect and found that the Bonferroni correction controls false positives effectively while only moderately reducing power. Thus, there is little reason why the Bonferroni correction should not be made a standard requirement for analyses of eye movement data in reading.