reflectR - Automatic Scoring of the Cognitive Reflection Test
A tool for researchers and psychologists to automatically
code open-ended responses to the Cognitive Reflection Test
(CRT), a widely used class of tests in cognitive science and
psychology for assessing an individual's propensity to override
an incorrect gut response and engage in further reflection to
find a correct answer. This package facilitates the
standardization of Cognitive Reflection Test responses analysis
across large datasets in cognitive psychology, decision-making,
and related fields. By automating the coding process, it not
only reduces manual effort but also aims to reduce the
variability introduced by subjective interpretation of
open-ended responses, contributing to a more consistent and
reliable analysis. 'reflectR' supports automatic coding and
machine scoring for the original English-language version of
CRT (Frederick, 2005) <doi:10.1257/089533005775196732>, as well
as for CRT4 and CRT7, 4- and 7-item versions, respectively
(Toplak et al., 2014) <doi:10.1080/13546783.2013.844729>, for
the CRT-long version built via Item Response Theory by Primi
and colleagues (2016) <doi:10.1002/bdm.1883>, and for CRT-2 by
Thomson & Oppenheimer (2016) <doi:10.1017/s1930297500007622>.
Note: While 'reflectR' draws inspiration from the principles
and scientific literature underlying the different versions of
the Cognitive Reflection Test, it has been independently
developed and does not hold any affiliation with any of the
original authors. The development of this package benefited
significantly from the kind insight and suggestion provided by
Dr. Keela Thomson, whose contribution is gratefully
acknowledged. Additional gratitude is extended to Dr. Paolo
Giovanni Cicirelli, Prof. Marinella Paciello, Dr. Carmela
Sportelli, and Prof. Francesca D'Errico, who not only
contributed to the manual multi-rater coding of CRT-2 items but
also profoundly influenced the understanding of the importance
and practical relevance of cognitive reflection within
personality, social, and cognitive psychology research.
Acknowledgment is also due to the European project STERHEOTYPES
(STudying European Racial Hoaxes and sterEOTYPES) for funding
the data collection that produced the datasets initially used
for manual multi-rater coding of CRT-2 items.
