| Title: | Automatic Scoring of the Cognitive Reflection Test |
|---|---|
| Description: | A tool for researchers and psychologists to automatically code open-ended responses to the Cognitive Reflection Test (CRT), a widely used class of tests in cognitive science and psychology for assessing an individual's propensity to override an incorrect gut response and engage in further reflection to find a correct answer. This package facilitates the standardization of Cognitive Reflection Test responses analysis across large datasets in cognitive psychology, decision-making, and related fields. By automating the coding process, it not only reduces manual effort but also aims to reduce the variability introduced by subjective interpretation of open-ended responses, contributing to a more consistent and reliable analysis. 'reflectR' supports automatic coding and machine scoring for the original English-language version of CRT (Frederick, 2005) <doi:10.1257/089533005775196732>, as well as for CRT4 and CRT7, 4- and 7-item versions, respectively (Toplak et al., 2014) <doi:10.1080/13546783.2013.844729>, for the CRT-long version built via Item Response Theory by Primi and colleagues (2016) <doi:10.1002/bdm.1883>, and for CRT-2 by Thomson & Oppenheimer (2016) <doi:10.1017/s1930297500007622>. Note: While 'reflectR' draws inspiration from the principles and scientific literature underlying the different versions of the Cognitive Reflection Test, it has been independently developed and does not hold any affiliation with any of the original authors. The development of this package benefited significantly from the kind insight and suggestion provided by Dr. Keela Thomson, whose contribution is gratefully acknowledged. Additional gratitude is extended to Dr. Paolo Giovanni Cicirelli, Prof. Marinella Paciello, Dr. Carmela Sportelli, and Prof. Francesca D'Errico, who not only contributed to the manual multi-rater coding of CRT-2 items but also profoundly influenced the understanding of the importance and practical relevance of cognitive reflection within personality, social, and cognitive psychology research. Acknowledgment is also due to the European project STERHEOTYPES (STudying European Racial Hoaxes and sterEOTYPES) for funding the data collection that produced the datasets initially used for manual multi-rater coding of CRT-2 items. |
| Authors: | Giuseppe Corbelli [aut, cre] (ORCID: 0000-0002-2864-3548) |
| Maintainer: | Giuseppe Corbelli <[email protected]> |
| License: | GPL-3 |
| Version: | 2.1.3 |
| Built: | 2024-10-25 05:17:04 UTC |
| Source: | https://github.com/g-corbelli/reflectr |
Applies coding logic to any number of provided CRT question responses and supports multiple coding schemes. This function can output original coded responses, binary-coded responses, and aggregate scores based on these binary codings.
CRT( item1 = NULL, item2 = NULL, item3 = NULL, codingscheme = "categ", na.rm = TRUE )CRT( item1 = NULL, item2 = NULL, item3 = NULL, codingscheme = "categ", na.rm = TRUE )
item1 |
Vector of responses to the first CRT question, or NULL if not provided. |
item2 |
Vector of responses to the second CRT question, or NULL if not provided. |
item3 |
Vector of responses to the third CRT question, or NULL if not provided. |
codingscheme |
A character string indicating the desired coding scheme. Options are "categ" for the original 1, 2, 3 coding, "sum" for a sum of binary-coded correct answers, or "mean" for an average of binary-coded correct answers. The default is "categ". |
na.rm |
Logical, indicating whether to treat missing values as empty responses or preserve them as missing information. When TRUE, NAs are coded as "other" incorrect responses; when FALSE, NAs are preserved. Default is TRUE. |
A list containing the coded and, if applicable, binary-coded responses for each provided CRT question. For "sum" or "mean" coding schemes, additional vectors representing these aggregate scores are included.
Developed by Giuseppe Corbelli, email: [email protected], [email protected]
# Automated scoring for CRT responses using the categorical coding scheme: reflectR::CRT( item1 = c("meh", "five", "10", "ten"), item2 = c("5", "one hundred", "100", "five"), item3 = c("47", "24", "forty-sevenn", "who knows"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRT responses based on binary-coded correctness: reflectR::CRT( item1 = c("meh", "five", "10", "ten"), item2 = c("5", "one hundred", "100", "five"), item3 = c("47", "24", "forty-sevenn", "booo"), codingscheme = "sum", na.rm = FALSE)$crt_sum# Automated scoring for CRT responses using the categorical coding scheme: reflectR::CRT( item1 = c("meh", "five", "10", "ten"), item2 = c("5", "one hundred", "100", "five"), item3 = c("47", "24", "forty-sevenn", "who knows"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRT responses based on binary-coded correctness: reflectR::CRT( item1 = c("meh", "five", "10", "ten"), item2 = c("5", "one hundred", "100", "five"), item3 = c("47", "24", "forty-sevenn", "booo"), codingscheme = "sum", na.rm = FALSE)$crt_sum
Applies coding logic to any number of provided CRT question responses and supports multiple coding schemes. This function can output original coded responses, binary-coded responses, and aggregate scores based on these binary codings.
CRT4( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, codingscheme = "categ", na.rm = TRUE )CRT4( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, codingscheme = "categ", na.rm = TRUE )
item1 |
Vector of responses to the first CRT question, or NULL if not provided. |
item2 |
Vector of responses to the second CRT question, or NULL if not provided. |
item3 |
Vector of responses to the third CRT question, or NULL if not provided. |
item4 |
Vector of responses to the fourth CRT question, or NULL if not provided. |
codingscheme |
A character string indicating the desired coding scheme. Options are "categ" for the original 1, 2, 3 coding, "sum" for a sum of binary-coded correct answers, or "mean" for an average of binary-coded correct answers. The default is "categ". |
na.rm |
Logical, indicating whether to treat missing values as empty responses or preserve them as missing information. When TRUE, NAs are coded as "other" incorrect responses; when FALSE, NAs are preserved. Default is TRUE. |
A list containing the coded and, if applicable, binary-coded responses for each provided CRT question. For "sum" or "mean" coding schemes, additional vectors representing these aggregate scores are included.
Developed by Giuseppe Corbelli, email: [email protected], [email protected]
# Automated scoring for CRT4 responses using the categorical coding scheme: reflectR::CRT4( item1 = c("four", "nineeee", "maybe 4?"), item2 = c("29", "thirty", "30"), item3 = c("twentyyyy", "ten I think", "dunno"), item4 = c("your behind", "poorer", "richer"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRT4 responses based on binary-coded correctness: reflectR::CRT4( item1 = c("four", "nineeee", "maybe 4?"), item2 = c("29", "thirty", "30"), item3 = c("twentyyyy", "ten I think", "dunno"), item4 = c("your behind", "poorer", "richer"), codingscheme = "sum", na.rm = FALSE)$crt_sum# Automated scoring for CRT4 responses using the categorical coding scheme: reflectR::CRT4( item1 = c("four", "nineeee", "maybe 4?"), item2 = c("29", "thirty", "30"), item3 = c("twentyyyy", "ten I think", "dunno"), item4 = c("your behind", "poorer", "richer"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRT4 responses based on binary-coded correctness: reflectR::CRT4( item1 = c("four", "nineeee", "maybe 4?"), item2 = c("29", "thirty", "30"), item3 = c("twentyyyy", "ten I think", "dunno"), item4 = c("your behind", "poorer", "richer"), codingscheme = "sum", na.rm = FALSE)$crt_sum
Applies coding logic to any number of provided CRT question responses and supports multiple coding schemes. This function can output original coded responses, binary-coded responses, and aggregate scores based on these binary codings.
CRT7( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, item5 = NULL, item6 = NULL, item7 = NULL, codingscheme = "categ", na.rm = TRUE )CRT7( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, item5 = NULL, item6 = NULL, item7 = NULL, codingscheme = "categ", na.rm = TRUE )
item1 |
Vector of responses to the first CRT question, or NULL if not provided. |
item2 |
Vector of responses to the second CRT question, or NULL if not provided. |
item3 |
Vector of responses to the third CRT question, or NULL if not provided. |
item4 |
Vector of responses to the fourth CRT question, or NULL if not provided. |
item5 |
Vector of responses to the fifth CRT question, or NULL if not provided. |
item6 |
Vector of responses to the sixth CRT question, or NULL if not provided. |
item7 |
Vector of responses to the seventh CRT question, or NULL if not provided. |
codingscheme |
A character string indicating the desired coding scheme. Options are "categ" for the original 1, 2, 3 coding, "sum" for a sum of binary-coded correct answers, or "mean" for an average of binary-coded correct answers. The default is "categ". |
na.rm |
Logical, indicating whether to treat missing values as empty responses or preserve them as missing information. When TRUE, NAs are coded as "other" incorrect responses; when FALSE, NAs are preserved. Default is TRUE. |
A list containing the coded and, if applicable, binary-coded responses for each provided CRT question. For "sum" or "mean" coding schemes, additional vectors representing these aggregate scores are included.
Developed by Giuseppe Corbelli, email: [email protected], [email protected]
# Automated scoring for CRT7 responses using the categorical coding scheme: reflectR::CRT7( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("four", "nineeee", "maybe 4?"), item5 = c("29", "thirty", "30"), item6 = c("twentyyyy", "ten I think", NA), item7 = c("your behind", "poorer", "richer"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRT7 responses based on binary-coded correctness: reflectR::CRT7( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("four", "nineeee", "maybe 4?"), item5 = c("29", "thirty", "30"), item6 = c("twentyyyy", "ten I think", NA), item7 = c("your behind", "poorer", "richer"), codingscheme = "sum", na.rm = FALSE)$crt_sum# Automated scoring for CRT7 responses using the categorical coding scheme: reflectR::CRT7( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("four", "nineeee", "maybe 4?"), item5 = c("29", "thirty", "30"), item6 = c("twentyyyy", "ten I think", NA), item7 = c("your behind", "poorer", "richer"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRT7 responses based on binary-coded correctness: reflectR::CRT7( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("four", "nineeee", "maybe 4?"), item5 = c("29", "thirty", "30"), item6 = c("twentyyyy", "ten I think", NA), item7 = c("your behind", "poorer", "richer"), codingscheme = "sum", na.rm = FALSE)$crt_sum
Applies coding logic to any number of provided CRT question responses and supports multiple coding schemes. This function can output original coded responses, binary-coded responses, and aggregate scores based on these binary codings.
CRTlong( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, item5 = NULL, item6 = NULL, codingscheme = "categ", na.rm = TRUE )CRTlong( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, item5 = NULL, item6 = NULL, codingscheme = "categ", na.rm = TRUE )
item1 |
Vector of responses to the first CRT question, or NULL if not provided. |
item2 |
Vector of responses to the second CRT question, or NULL if not provided. |
item3 |
Vector of responses to the third CRT question, or NULL if not provided. |
item4 |
Vector of responses to the fourth CRT question, or NULL if not provided. |
item5 |
Vector of responses to the fifth CRT question, or NULL if not provided. |
item6 |
Vector of responses to the sixth CRT question, or NULL if not provided. |
codingscheme |
A character string indicating the desired coding scheme. Options are "categ" for the original 1, 2, 3 coding, "sum" for a sum of binary-coded correct answers, or "mean" for an average of binary-coded correct answers. The default is "categ". |
na.rm |
Logical, indicating whether to treat missing values as empty responses or preserve them as missing information. When TRUE, NAs are coded as "other" incorrect responses; when FALSE, NAs are preserved. Default is TRUE. |
A list containing the coded and, if applicable, binary-coded responses for each provided CRT question. For "sum" or "mean" coding schemes, additional vectors representing these aggregate scores are included.
Developed by Giuseppe Corbelli, email: [email protected], [email protected]
# Automated scoring for CRTlong responses using the categorical coding scheme: reflectR::CRTlong( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("3 elves", "dunno", "six"), item5 = c("29", "thirty", "30"), item6 = c("15", "fifteen", "0"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRTlong responses based on binary-coded correctness: reflectR::CRTlong( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("3 elves", "dunno", "six"), item5 = c("29", "thirty", "30"), item6 = c("15", "fifteen", "0"), codingscheme = "sum", na.rm = FALSE)$crt_sum# Automated scoring for CRTlong responses using the categorical coding scheme: reflectR::CRTlong( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("3 elves", "dunno", "six"), item5 = c("29", "thirty", "30"), item6 = c("15", "fifteen", "0"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRTlong responses based on binary-coded correctness: reflectR::CRTlong( item1 = c("five", "5 cents", "10"), item2 = c("5", "one hundred", "100"), item3 = c("47", "24", "forty seven"), item4 = c("3 elves", "dunno", "six"), item5 = c("29", "thirty", "30"), item6 = c("15", "fifteen", "0"), codingscheme = "sum", na.rm = FALSE)$crt_sum
Applies coding logic to any number of provided CRT question responses and supports multiple coding schemes. This function can output original coded responses, binary-coded responses, and aggregate scores based on these binary codings.
CRTtwo( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, codingscheme = "categ", na.rm = TRUE )CRTtwo( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, codingscheme = "categ", na.rm = TRUE )
item1 |
Vector of responses to the first CRT question, or NULL if not provided. |
item2 |
Vector of responses to the second CRT question, or NULL if not provided. |
item3 |
Vector of responses to the third CRT question, or NULL if not provided. |
item4 |
Vector of responses to the fourth CRT question, or NULL if not provided. |
codingscheme |
A character string indicating the desired coding scheme. Options are "categ" for the original 1, 2, 3 coding, "sum" for a sum of binary-coded correct answers, or "mean" for an average of binary-coded correct answers. The default is "categ". |
na.rm |
Logical, indicating whether to treat missing values as empty responses or preserve them as missing information. When TRUE, NAs are coded as "other" incorrect responses; when FALSE, NAs are preserved. Default is TRUE. |
A list containing the coded and, if applicable, binary-coded responses for each provided CRT question. For "sum" or "mean" coding schemes, additional vectors representing these aggregate scores are included.
Developed by Giuseppe Corbelli, email: [email protected], [email protected]
# Automated scoring for CRTtwo responses using the categorical coding scheme: reflectR::CRTtwo( item1 = c("first place", "second place", "1", "seconddd", "meh"), item2 = c("7", "eightt", "seven", NA, "8"), item3 = c("emily", "i think emily", "JUNEE", "maybe june", "the name is emily"), item4 = c("nothing", "27 sqmt", "0", "it's empty", "i suck at math"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRTtwo responses based on binary-coded correctness: reflectR::CRTtwo( item1 = c("first place", "second place", "1", "seconddd", "meh"), item2 = c("7", "eightt", "seven", NA, "8"), item3 = c("emily", "i think emily", "JUNEE", "maybe june", "the name is emily"), item4 = c("nothing", "27 sqmt", "0", "it's empty", "i suck at math"), codingscheme = "sum", na.rm = FALSE)$crt_sum# Automated scoring for CRTtwo responses using the categorical coding scheme: reflectR::CRTtwo( item1 = c("first place", "second place", "1", "seconddd", "meh"), item2 = c("7", "eightt", "seven", NA, "8"), item3 = c("emily", "i think emily", "JUNEE", "maybe june", "the name is emily"), item4 = c("nothing", "27 sqmt", "0", "it's empty", "i suck at math"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for CRTtwo responses based on binary-coded correctness: reflectR::CRTtwo( item1 = c("first place", "second place", "1", "seconddd", "meh"), item2 = c("7", "eightt", "seven", NA, "8"), item3 = c("emily", "i think emily", "JUNEE", "maybe june", "the name is emily"), item4 = c("nothing", "27 sqmt", "0", "it's empty", "i suck at math"), codingscheme = "sum", na.rm = FALSE)$crt_sum
Applies coding logic to any number of provided CRT question responses and supports multiple coding schemes. This function can output original coded responses, binary-coded responses, and aggregate scores based on these binary codings.
itaCRTtwo( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, codingscheme = "categ", na.rm = TRUE )itaCRTtwo( item1 = NULL, item2 = NULL, item3 = NULL, item4 = NULL, codingscheme = "categ", na.rm = TRUE )
item1 |
Vector of responses to the first CRT question, or NULL if not provided. |
item2 |
Vector of responses to the second CRT question, or NULL if not provided. |
item3 |
Vector of responses to the third CRT question, or NULL if not provided. |
item4 |
Vector of responses to the fourth CRT question, or NULL if not provided. |
codingscheme |
A character string indicating the desired coding scheme. Options are "categ" for the original 1, 2, 3 coding, "sum" for a sum of binary-coded correct answers, or "mean" for an average of binary-coded correct answers. The default is "categ". |
na.rm |
Logical, indicating whether to treat missing values as empty responses or preserve them as missing information. When TRUE, NAs are coded as "other" incorrect responses; when FALSE, NAs are preserved. Default is TRUE. |
A list containing the coded and, if applicable, binary-coded responses for each provided CRT question. For "sum" or "mean" coding schemes, additional vectors representing these aggregate scores are included.
Developed by Giuseppe Corbelli, email: [email protected], [email protected]
# Automated scoring for itaCRTtwo responses using the categorical coding scheme: reflectR::itaCRTtwo( item1 = c("al primo", "secondo", "1", NA), item2 = c("7", "non so", "sette", "otto"), item3 = c("pprimo", "carlo", "CARLOO", "si chiama boh"), item4 = c("nulla", "27 metri cubi", "mille", "zero"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for itaCRTtwo responses based on binary-coded correctness: reflectR::itaCRTtwo( item1 = c("al primo", "secondo", "1", NA), item2 = c("7", "non so", "sette", "otto"), item3 = c("primo", "carlo", "CARLOO", "si chiama boh"), item4 = c("nulla", "27 metri cubi", "mille", "zero"), codingscheme = "sum", na.rm = FALSE)$crt_sum# Automated scoring for itaCRTtwo responses using the categorical coding scheme: reflectR::itaCRTtwo( item1 = c("al primo", "secondo", "1", NA), item2 = c("7", "non so", "sette", "otto"), item3 = c("pprimo", "carlo", "CARLOO", "si chiama boh"), item4 = c("nulla", "27 metri cubi", "mille", "zero"), codingscheme = "categ", na.rm = TRUE) # Compute the sum score for itaCRTtwo responses based on binary-coded correctness: reflectR::itaCRTtwo( item1 = c("al primo", "secondo", "1", NA), item2 = c("7", "non so", "sette", "otto"), item3 = c("primo", "carlo", "CARLOO", "si chiama boh"), item4 = c("nulla", "27 metri cubi", "mille", "zero"), codingscheme = "sum", na.rm = FALSE)$crt_sum
The reflectR package provides automated coding tools for open-ended responses to various forms of the Cognitive Reflection Test (CRT). This package facilitates the standardization of Cognitive Reflection Test responses analysis across large datasets in cognitive psychology, decision-making, and related fields. By automating the coding process, it not only reduces manual effort but also aims to reduce the variability introduced by subjective interpretation of open-ended responses, contributing to a more consistent and reliable analysis.
The package includes functions to code responses from:
CRT: Original CRT version by Frederick (2005)
CRT4: 4-item CRT version by Toplak et al. (2014)
CRT7: 7-item expanded CRT version by Toplak et al. (2014)
CRTlong: CRT LONG version by Primi et al. (2016)
CRTtwo: CRT version 2 by Thomson & Oppenheimer (2016)
itaCRTtwo: Italian version of CRT version 2 by Thomson & Oppenheimer (2016)
While reflectR draws inspiration from the scientific literature on the CRT, it has been independently developed and does not hold any affiliation with the original authors of these tests.
The development of this package benefited significantly from the kind insight and suggestion provided by Dr. Keela Thomson, whose contribution is gratefully acknowledged. Special thanks are also due to Dr. Paolo Giovanni Cicirelli, Prof. Marinella Paciello, Dr. Carmela Sportelli, and Prof. Francesca D'Errico. Acknowledgment is also due to the European project STERHEOTYPES for funding the data collection that produced the datasets initially used for manual multi-rater coding of CRT-2 items.
Maintainer: Giuseppe Corbelli [email protected] (ORCID: 0000-0002-2864-3548)
Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4), 25-42. doi:10.1257/089533005775196732
Toplak, M. E., West, R. F., & Stanovich, K. E. (2014). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147-168. doi:10.1080/13546783.2013.844729
Primi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J. (2016). The development and testing of a new version of the Cognitive Reflection Test applying item response theory (IRT). Journal of Behavioral Decision Making, 29(5), 453-469. doi:10.1002/bdm.1883
Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99-113. doi:10.1017/s1930297500007622
Useful links: