As for the response confidence scores, the opinion judgments are reported here as proportions rather than percentages. There was a positive relationship between standard question confidence and standard question opinion judgments for incorrect reasoners, r(225) = .480, p< .001, such that incorrect reasoners who were less confident in their response were also less likely to think other reasoners could answer the standard question correctly. This strong relationship lends support to the notion that opinion judgments and response confidence scores are reflecting similar cognitive processes. De Neys W, & Bonnefon J-F (2013). The ‘whys’ and ‘whens’ of individual differences in thinking biases. Trends in Cognitive Sciences, 17, 172–178. doi: 10.1016/j.tics.2013.02.001 [ PubMed] [ CrossRef] [ Google Scholar] Shaw JS III (1996). Increases in eyewitness confidence resulting from postevent questioning. Journal of Experimental Psychology: Applied, 2, 126–146. doi: 10.1037/1076-898X.2.2.126 [ CrossRef] [ Google Scholar] De Neys W, & Glumicic T (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106, 1248–1299. doi: 10.1016/j.cognition.2007.06.002 [ PubMed] [ CrossRef] [ Google Scholar] Hoover JD, & Healy AF (2017). Algebraic reasoning and bat-and-ball problem variants: Solving isomorphic algebra first facilitates problem solving later. Psychonomic Bulletin & Review, 24, 1922–1928. doi: 10.3758/s13423-017-1241-8 [ PubMed] [ CrossRef] [ Google Scholar]

Campitelli G, & Gerrans P (2014). Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach. Memory & Cognition, 42, 434–447. doi: 10.3758/s13421-013-0367-9 [ PubMed] [ CrossRef] [ Google Scholar] For the second and third samples, nine simple math problems were included between the experimental questions and the memory questions to serve as interference to limit recall and recognition based on working memory (see the Supplemental Materials). Neither math problems nor any other activity occurred between the De Neys et al. (2013) experimental and the memory questions for the first sample. Specifically, we started by testing 126 MTurk participants and then examined their data. We did not have a precise stopping rule for the sample size, but we decided from the outset to pause data collection after examining the data from an initial sample of MTurk participants. We noted poor memory performance by these initial participants. On the basis of our observations, we tested two additional samples of participants (one from MTurk and one from UCB), each approximately the same size as the initial sample (128 participants), and gave them the math problems to create interference.B. T., & Stanovich KE (2013). Dual-process theories of higher cognition: Advancing the debate. Perspectives on Psychological Science, 8, 223–241. doi: 10.1177/1745691612460685 [ PubMed] [ CrossRef] [ Google Scholar] Mata A, Ferreira MB, Voss A, & Kollei T (2017). Seeing the conflict: An attentional account of reasoning errors. Psychonomic Bulletin & Review, 24, 1980–1986. doi: 10.3758/s13423-017-1234-7 [ PubMed] [ CrossRef] [ Google Scholar] Bago B, & De Neys W (2017). Fast logic? Examining the time course assumption of dual process theory. Cognition, 158, 90–109. doi: 10.1016/j.cognition.2016.10.014 [ PubMed] [ CrossRef] [ Google Scholar]

Lichtenstein S, Fischhoff B, & Phillips LD (1982). Calibration of probabilities: The state of the art to 1980. In Kahneman D, Slovic P, & Tversky A (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 306–334). Cambridge, UK: Cambridge University Press. [ Google Scholar] Gangemi A, Bourgeois-Gironde S, & Mancini F (2015). Feelings of error in reasoning—in search of a phenomenon. Thinking & Reasoning, 21, 383–396. doi: 10.1080/13546783.2014.980755 [ CrossRef] [ Google Scholar] Bourgeois-Gironde S, & Vanderhenst J-B (2009). How to open the door to System 2: Debiasing the Bat and Ball problem. In Watanabe S, Bloisdell AP, Huber L, & Young A (Eds.), Rational animals, irrational humans (pp. 235–252). Tokyo: Keio University Press. [ Google Scholar]De Neys W (2014). Conflict detection, dual processes, and logical intuitions: Some clarifications. Thinking & Reasoning, 20, 169–187. doi: 10.1080/13546783.2013.854725 [ CrossRef] [ Google Scholar] Keren G (1988). On the ability of monitoring non-veridical perceptions and uncertain knowledge: Some calibration studies. Acta Psychologica, 67, 95–119. doi: 10.1016/0001-6918(88)90007-8 [ PubMed] [ CrossRef] [ Google Scholar] The body of work on conflict detection in decision-making provides the theoretical framework for explaining the findings outlined above (e.g., Aczel, Szollosi, & Bago, 2016; De Neys, 2012; Pennycook, Fugelsang, & Koehler, 2012). In this line of theorizing, reasoning errors arise from an inability to inhibit prepotent intuitive responses that, importantly, cannot be explained solely due to miserly cognition because incorrect reasoners demonstrate that they are unsure their reasoning was accurate. Indeed, there is a rich body of research in support of error sensitivity in particular and logical intuitions more generally (e.g., Bago & De Neys, 2017; De Neys, 2012, 2014; De Neys & Bonnefon, 2013; De Neys & Glumicic, 2008; De Neys et al., 2013; Gangemi, Bourgeois-Gironde, & Mancini, 2015; Mata, Schubert, & Ferreira, 2014; but see Singmann, Klauer, & Kellen, 2014, for some caveats regarding this literature). Furthermore, sensitivity to error processing has been demonstrated though converging evidence in latency response investigations ( De Neys & Glumicic, 2008; Frey, Johnson, & De Neys, 2017; Johnson, Tubau, & De Neys, 2016) neuroimaging ( De Neys, Vartanian, & Goel, 2008), and alternative measurements of confidence ( De Neys, Cromheeke, & Osman, 2011). However, sensitivity effects were not obtained by indexing reasoner’s mouse-movements ( Travers, Rolison, & Feeney, 2016) nor their eye-movements ( Mata, Ferreira, Voss, & Kollei, 2017; see the subsequent debate concerning the studies by Mata et al. and Frey et al. discussed by Mata & Ferreira, 2018). On a more general level, unconscious processes have not been found to have much (if any) explanatory power (for a review, see Newell & Shanks, 2014). B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278. doi: 10.1146/annurev.psych.59.103006.093629 [ PubMed] [ CrossRef] [ Google Scholar]

The analysis of variance was conducted on proportions to enable comparison with the findings of De Neys et al. (2013), who used an analysis of variance on percentages to compare standard and control problems. However, because the dependent variable is dichotomous (0 or 1), a mixed effects logistic regression (with subject as the random variable) is more appropriate and yielded comparable results. Specifically, participants were significantly more likely to respond accurately to isomorphic control questions than to standard variants, b = −3.73, odds ratio ( OR) = 41.57, χ 2 = 153.54, p< .001, 95% confidence interval ( CI) [0.01, 0.04]. That is, participants’ odds of answering the isomorphic control questions correctly was about 42 times more likely than answering the standard variants correctly. Travers E, Rolison JJ, & Feeney A (2016). The time course of conflict on the Cognitive Reflection Test. Cognition, 150, 109–118. doi: 10.1016/j.cognition.2016.01.015 [ PubMed] [ CrossRef] [ Google Scholar]For recognition, all incorrect reasoners’ responses were included in the analyses because their answers came in the form of a multiple-choice forced response. A mixed effects logistic regression (with subject as the random variable) was again conducted due to the dichotomous dependent variable (with or without “more than”). Once again, incorrect reasoners usually recognized the standard problem, but not the control, as containing “more than” (see Table 1), with this effect of condition significant, b = 3.00, odds ratio ( OR) = 20.07, χ 2 = 50.34, p< .001, 95% confidence interval ( CI) [8.76, 45.94]. De Neys W, Cromheeke S, & Osman M (2011). Biased but in doubt: Conflict and decision confidence. PloS one, 6( 1), e15954. [ PMC free article] [ PubMed] [ Google Scholar] Keren G (1991). Calibration and probability judgments: Conceptual and methodological issues. Acta Psychologica, 77, 217–273. doi: 10.1016/0001-6918(91)90036-Y [ CrossRef] [ Google Scholar]