Cognitive Reflection Test

The Cognitive Reflection Test (CRT) is described by some to be the world’s shortest intelligence test, consisting of only 3 questions. Due to its popularity and some people’s familiarity with the questions, other lesser-known items have been added to revised versions of the test.

Take the test here:

Who the test is designed for

• Adults (age 16+)

What it tests

• The CRT was not actually created as a measure of intelligence, but as a simple measure of one type of cognitive ability—the ability or disposition to reflect on a question and resist reporting the first response that comes to mind.^[1]Cognitive Reflection and Decision Making (Frederick, 2005)
• The CRT thus seems to correlate with response inhibition, one of the executive functions.
• CRT scores are predictive of the types of choices that play a large role in tests of decision-making theories, such as expected utility theory and prospect theory.^[2]Cognitive Reflection and Decision Making (Frederick, 2005)
• But the CRT does have a moderate correlation with measures of intelligence, such as the IQ test.
• The CRT also correlates highly with various measures of mental heuristics,^[3]Cognitive abilities and behavioral biases (Oechssler, Roider & Schmitz, 2009) which are strategies make decisions, quickly form judgments, and determine solutions to complicated problems.
• Test scores are also significantly related to subjects’ time and risk preferences.^[4]Cognitive abilities and behavioral biases (Oechssler, Roider & Schmitz, 2009)

Versions & translations

• The CRT has been translated into different languages.

Taking the test

The original CRT consists of just 3 questions, which require logic and calculus to answer. Answers will consist of numbers. Take the test below!

(The correct answers to the questions can be found in the Answers section)

Cognitive Reflection Test

1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.
   How much does the ball cost?
   _____ cents.
2. If it takes 5 machines 5 minutes to make 5 widgets,
   how long would it take 100 machines to make 100 widgets?
   _____ minutes.
3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake,
   how long would it take for the patch to cover half of the lake?
   _____ days.

Scoring

• Scoring range: 0–3
• Threshold score: 2
• 83% of people missed at least one of the questions.
• NB: If you score 0, but you have an average to high IQ, this test could indicate dyscalculia.^[5]Cognitive reflection vs. calculation in decision making (Sinayev & Peters, 2015)

The CRT is only available as a self-scoring test.

In his book David and Goliath, Malcolm Gladwell says the following about the average scores on the CRT:

Frederick gave the CRT to students at nine American colleges, and the results track pretty closely with how students from those colleges would rank on more traditional intelligence tests. Students from the Massachusetts Institute of Technology—perhaps the brainiest college in the world—averaged 2.18 correct answers out of three. Students at Carnegie Mellon University in Pittsburgh, another extraordinarily elite institution, averaged 1.51 right answers out of three. Harvard students scored 1.43; the University of Michigan, Ann Arbor, 1.18; and the University of Toledo 0.57.

Validity

How reliable, accurate, valid, and up to date is the test?

• The original CRT is an attractive test as it involved only the first three items. Despite requiring only a few minutes to administer, the CRT’s predictive validity equals or exceeds other cognitive tests.
• The CRT is the world’s shortest scientifically validated test. However, an estimated 44–51% of the people who take the test are already familiar with the questions.^[6]Has the Standard Cognitive Reflection Test Become a Victim of Its Own Success? (Haigh, 2016)^[7]A limitation of the Cognitive Reflection Test: familiarity (Stieger & Reips, 2016) As such, researchers have developed revised versions, such as the CRT-Long,^[8]The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT) (Primi et al., 2015) the CRT-7,^[9]Assessing miserly information processing: An expansion of the Cognitive Reflection Test (Toplak, West & Stanovich, 2013) and the CRT-2,^[10]Investigating an alternate form of the cognitive reflection test (Thomson & Oppenheimer, 2016) which feature different questions you may not yet be familiar with.
• People who have seen the CRT tend to outscore those who have not. And yet research shows that the CRT remains robust despite multiple exposures.^[11]The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT) (Primi et al., 2015)
• There is, however, a strong male bias in the test. Shane Frederick already showed that men perform better in the CRT,^[12]Cognitive Reflection and Decision Making (Frederick, 2005) and many other studies have shown the same. A study based on 44,558 people (52.76% of whom were female) shows that:^[13]Cognitive reflection test: Whom, how, when (Brañas-Garzaa, Kujal & Lenkei, 2015)
  • Men perform better in every single question;
  • Women are more likely to answer none of the questions correctly; and
  • Men are more likely to answer all three questions correctly.
• A study on the gender differences (based on the longer version of the test, the CRT-L) shows that the difference in performance between men and women is due to differences in math anxiety (tension and anxiety around working with numbers) and mathematical reasoning. So cognitive reflection is influenced by numerical skills and related feelings like anxiety.^[14]Are there gender differences in cognitive reflection? Invariance and differences related to mathematics (Primi et al., 2017) Since the CRT-2 relies less on numeracy,^[15]Investigating an alternate form of the cognitive reflection test (Thomson & Oppenheimer, 2016) for women that is probably a more accurate measure of the test than the CRT.

Discussion

Natalie:

The things to keep in mind are:

1. Has the person seen the test before (the ball/bat question is often shared on social media).
2. Men score higher than women.
3. A person with dyscalculia will be at a disadvantage with this test, even if they are bright.
4. The CRT was thought to measure the ability to recognize and override errors that are made on quick and effortless intuition and instead use a system that is more controlled and slow. However, CRT items require math ability to be answered correctly, and research has shown that mathematical ability is more important than cognitive reflection.

Kendall:

On seeing three statements, each involving numbers, I knew the CRT would be a problem. A score of 0 would be disheartening if I were unaware of my dyscalculia. Thus it is highly relevant to understand this when you are considering the results.

Eva:

I imagine knowing what the CRT actually measures can significantly influence your performance on the test. I remember that when I first heard the bat-and-ball question, I got the answer wrong. So when I took the CRT, I was careful to think it through rather than offering an intuitive response. If you take the test thinking it’s an intelligence test, you will be more likely to give the intuitive and therefore incorrect answer.

On the other hand, not knowing what the test measures may give a more accurate measure of your propensity to override an intuitive—but incorrect—response with a more analytically correct response. I think it will give a fair indication of your intelligence irrespective of whether you understand what the test measures and correlates with though. But if you want to take this test as a measure of intelligence and you have challenges with numbers, do take the CRT-2 instead.

Answers

1. Correct answer = 5 cents; intuitive answer = 10 cents.
2. Correct answer = 5 minutes; intuitive answer = 100 minutes.
3. Correct answer = 47 days; intuitive answer = 24 days.

Explanations

1. A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? (Answer: 5 cents.)

Note that the bat costs $1.00 MORE than the ball—whatever the price of the ball may be. So the bat doesn’t cost $1.00, and the ball $1, as that would mean the bat costs $0.90 more than the ball. The ball should cost $0.05 in order for the bat to cost $1.00 more (= $1.05), for a total of $1.10.

If you are more mathematically inclined, let’s say the ball costs X. We know that the bat costs $1 more than X, meaning that its price is X + 1. The total price of the bat and ball, then, is X + 1 + X. That’s a total of 2X + 1, which we already know is 1.1 ($1.10). In that case, we know that 2X = 0.1, which means that X = 0.05.

2. If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? (Answer: 5 minutes.)

Each machine makes a widget in 5 minutes. The number of machines does not affect how long it takes, because it will always take 5 minutes. So 100 machines working together will produce 100 widgets in 5 minutes.

Note also that the question can be interpreted such that at least 5 machines are required to work in unison to produce 5 widgets, but this also has no bearing on the answer.

3. In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? (Answer: 47 days)

Every day the patch doubles in size, so every PREVIOUS day the patch was half the size. So if the patch takes 48 days to cover the lake, then the before-last day, on day 47, the patch covers half the lake.

If you are looking for an autism assessment,
Dr. Natalie Engelbrecht can offer help!
You can find more information here:

Online autism assessments