A Blog by Ed Yong

IQ scores reflect motivation as well as ‘intelligence’

Ever since there have been IQ tests, people have debated what they actually measure. Is it “intelligence”, is it an abstract combination of mental abilities, or is it, as Edwin Boring said, “the capacity to do well in an intelligence test”? Regardless of the answer, studies have repeatedly shown that people who achieve higher scores in IQ tests are more likely to do well in school, perform well in their jobs, earn more money, avoid criminal convictions, and even live longer. Say what you like about the tests, but they have predictive power.

However, Angela Lee Duckworth from the University of Pennsylvania has found that this power is overrated. The link between our IQs and our fates becomes muddier when we consider motivation – an aspect of test-taking that is often ignored. Simply put, some people try harder in IQ tests than others. If you take this into account, the association between your IQ and your success in life becomes considerably weaker. The tests are not measuring intelligence alone, but also the desire to prove it.

Many standardised tests assume that the people who take them are alert and motivated. As such, their scores reflect the height of their abilities. IQ tests are no different. The questions are ordered by difficulty to keep people’s morale up. Edward Thorndike, a pioneer of intelligence testing, wrote that “all our measurements assume that the individual in question tries as hard as he can to make as high a score as possible”, although he admitted that no one knew if that was the case.

To look at how motivation affects IQ scores, Duckworth reviewed 25 previous studies, which included a total of 2,008 people. She found that people achieved higher IQ scores on average if they were given material incentives to take the tests, such as money or sweets, particularly if they had above-average IQs anyway. This alone suggests that motivation can skew the results of the tests.

Next, Duckworth looked at the scores of 508 young boys who had taken an IQ test in 1987. The boys were part of the Pittsburgh Youth Study, and researchers kept in touch with them into adulthood, for at least 12 years after the original test. As usual, their scores predicted their eventual academic performance, the number of years they spent in education, their odds of being employed as adults, and their number of criminal convictions.

But there was more. The original tests were all delivered verbally and the sessions were filmed. Duckworth recruited three independent researchers to review the footage for signs of low motivation, such as refusing to take part, or wanting the session to end. The team found that boys with lower IQ scores were also less motivated when they took the test, and their degree of motivation also predicted the course of their lives. Accounting for motivation weakened the link between IQ and life-success, especially for employment and criminal convictions.

Duckworth says, “It is important not to overstate our conclusions.” IQ tests can still predict other aspects of our lives. Motivation reduces that predictive power, but it doesn’t destroy it altogether. The point is that people with above-average IQs also tend to try harder on IQ tests, and they do so more consistently. Duckworth writes, “These findings imply that earning a high IQ score requires high intelligence in addition to high motivation. Lower IQ scores, however, might result from either lower intelligence or lack of motivation.”

The problem, as ever, lies in making too much of the test results, in seeing them as a sign of natural ability and future potential. After all, motivation is itself affected by a person’s background, and their beliefs in their future options and their chances of success. It could partially explain differences in test scores between people of different genders, social backgrounds and nationalities.

If you think it’s obvious that motivation would confound the results of IQ tests, then Robert Stenberg, who studies intelligence at Oklahoma State University, agrees with you. “D’uh!”, he says. Sternberg thinks that Duckworth has produced a “great research study” but adds, “To almost anyone except some subset of psychologists who study IQ testing, it will come as little surprise that motivation is an extremely powerful determinant of performance in school and in life. Most employers, for example, are at least as eager to know about job applicants’ motivation as they are to know about their cognitive skills.  Teachers also know that ability without high motivation typically results in little success in a challenging curriculum.”

Duckworth herself recognises that people who actually administer the tests will be well aware of the issue of motivation. She says, “Where the problem lies, in our view, is in the interpretation of IQ scores by economists, sociologists, and research psychologists who have not witnessed variation in test motivation firsthand. [They] might erringly assume that a low IQ score invariably indicates low intelligence.”

Is this view common? Sternberg thinks so, pointing to the fact that Duckworth’s study was newsworthy enough to be published in PNAS, one of the world’s most prestigious scientific journals. “[This shows] how off-track our society has gone in its acceptance of commercial persuasive appeals to buy into standardized tests as some kind of panacea for predicting almost any outcome in life that we value.

“The irony of the study is that it shows the tests indeed can be useful, but as joint measures of cognitive skills and motivation.  The tests also indirectly measure many other variables, such as quality of schooling, type of socialization in the home, and parents’ ability to provide their children with a home environment that fosters the kinds of skills that tests measure.  IQ tests, like all tests, are agglomerate measures of many things.  They are not pure measures of some kind of “intelligence” or anything else.

“Ultimately, it would be desirable if we recognized that many skills are required for success in life and if we grew beyond using contemporary tests, which are minor cosmetic variants of tests used a century ago.  (Imagine where we would be if our medical testing were essentially the same as it was 100 years ago.)  Instead of limiting ourselves to narrow standardized tests, we might seek as well directly to assess motivation as well as creativity, practical skills, wisdom, and even ethics. If we did, we might find our society advancing to levels of economic productivity and, for that matter, well-being that we previously believed to be out of reach.“

Reference: Duckworth, Quinn, Lynam, Loeber & Stouthamer-Loeber. 2011. Role of test motivation in intelligence testing. PNAS http://dx.doi.org/10.1073/pnas.1018601108

Image by Luc Legay

More on intelligence

13 thoughts on “IQ scores reflect motivation as well as ‘intelligence’

  1. I would bet that high motivation also increases likelihood to seek out learning, productive work, and personal measures of success. If anything, these results are likely a lower bound. In the age of the internet, in which facts are there for the taking (and not as important to memorize) as long as one goes looking, I think that motivation may be more important than ever. And I second the call for some type of creativity/problem solving test, which is generally lacking in IQ tests.

  2. Finally, some evidence that IQ is not set in stone. I wonder about strategy as well. Just my personal experience: in college I was not happy my grades in certain classes. At some point when I “completed” an exam, I decided to sit quietly for 5 minutes, then take the rest of the time (i.e. until I was practically thrown out of the classroom) to review my answers. Sure enough, my grades improved significantly.

    Also a girl in my grad school cohort (one of the smartest in the program) who was in a study group for our statistics class, when we were about to leave for the night, would ask us to not tell the building staff that she was still in the computer room. She would hide until they had locked up the building for the night, then continue working on the problems. I don’t know if she was brilliant or just psychotically motivated.

  3. Michael Gelb wrote a great book about IQ: here. It speaks of the 7 different “types” of intelligence, which include more than the two that are tested (mathematical and linguistic). The book is even featured in the film, “The Italian Job,” starring Marky Mark Wahlberg. It’s titled: How to Think Like Leonardo da Vinci: Seven Steps to Genius Every Day

    William Sidis was allegedly the most intelligent person of all time, weighing in at the 275-300 range. So, what did he do with his life?

    I wish to share two thoughts about MENSA, a group of people who fancy themselves “the most intelligent”:

    1) I was invited to join MENSA, and went to see if I liked the people there, so I went and I didn’t. Too many people debating the size of their intellectual penises, and praising Richard Feynman for his expert use of the real one, assuming morals go out the door. I have since invoked Groucho Marx’s attitude re same, that being “I would never join any organization that would have ME as a member.”

    2) Bureaucratic screw-ups, they happen. Here’s my favorite example, which I’m sure you’ve read if you remember reading Steve Martin’s novel “The Pleasure of My Company.” Paraphrasing: “I once took an IQ test and sent it in to MENSA for grading. They were great! I got a really quick response. But they seemed to make one wee little mistake, which is fine because everyone makes them, even MENSA.
    I looked at my score, and the mistake was obvious. Where was the missing digit ‘1’ in front of the 92? “

  4. Y’know, I was a member of Mensa for a long time. You must have gone to a rare event; usually the ones I went to, no-one was talking about how smart they were, they were just busy having a good time having fun with people who were interested in lots of different things. (And arguing about hugging.)

  5. I haven’t read the study (not sufficiently motivated!), but the methodology as described seems shaky. How can you tell merely by visual observation whether someone is not ‘motivated’? They might be bored or messing around because the tests are too difficult for them. Does this count as lack of motivation or lack of intelligence? I would be more impressed if it were shown that monetary or other rewards (which would plausibly increase motivation) had a major effect on test results, but as far as I recall the results of such interventions are marginal. Countless very expensive programmes like Head Start have been aimed at improving cognitive and scholastic performance, without much effect. Surely someone along the way would have discovered if it were possible to get better results just by paying the kids!

  6. “Many standardised tests assume that the people who take them are alert and motivated.” I guess they’ve never had to proctor the tests at a high school!

  7. I suppose the question would then be: What effect would this have on the predictive effects of IQ tests if we could conceivably control for things like motivation and alertness?

    If the predictive effects of IQ scores in sociological studies was in a large part due to motivation and alertness effects, would that suggest that we could vastly improve the performance of those with below average scores by somehow making them more motivated or alert? If this is something that merely generates statistical noise or even obscures greater disparities, what does that mean for policy?

    Ultimately, the Duh! response seems the best here. I know that I scored significantly better on a pre-college standardized test when I had an adequate amount of sleep than when during the first time, when I was inordinately tired. This seems like a preliminary study and one that would have to uncover distributional effects to really be interesting. Reading the last paragraph:

    Ultimately, it would be desirable if we recognized that many skills are required for success in life and if we grew beyond using contemporary tests, which are minor cosmetic variants of tests used a century ago. (Imagine where we would be if our medical testing were essentially the same as it was 100 years ago.) Instead of limiting ourselves to narrow standardized tests, we might seek as well directly to assess motivation as well as creativity, practical skills, wisdom, and even ethics. If we did, we might find our society advancing to levels of economic productivity and, for that matter, well-being that we previously believed to be out of reach.

    My main response was to ask for an alternative that was more effective and provided the customers with what they were looking for (in the case of ACT, SAT, LSAT, GRE, MCAT, etc., we’re looking at higher education institutions that are looking to filter their base of applicants). If the authors want to improve on the current set of metrics used, then perhaps they should be focusing their energy on developing and testing the predictive effects of alternative metrics.

  8. I use IQ tests a lot, and I always stress that a single test score is just a measure of performance on a specific occasion – if they are unhappy with the result, they might well be able to do better under different circumstances. On the other hand, if they are pleasantly surprised, they should trust the IQ test – while it is always possible to have an “off” day and for some reason not manage to do your best, it is not possible to do better than your best… Still, it seems like a few things have been overlooked here. For one, motivation is quite likely to actually make people more intelligent – by driving them to learn and exercise their brains, and not give up when something seems difficult. And less intelligent people may well be less motivated when taking an IQ test, simply because they don’t expect to be good at it. Motivation and intelligence are not independent factors here.

  9. OK, I looked at the Abstract (full text apparently requires subscription), and it says that financial incentives increase scores on average by .64 of a standard deviation, which is about 10 IQ points. This is not huge, but it is bigger than I expected – I thought previous studies only showed a few points. Also, they say that the effect is bigger for lower IQs, which makes sense. So I’m impressed, provided the studies used in the meta-analysis are kosher (e.g. they exclude anything involving Rick Heber’s Milwaukee project).

  10. I have always believed in this fact. I have taken a few of IQ tests and my IQ has varied from 105 which is average to 129 which is above average. And each time I took them, my mental state had a lot to do with it. It was not only motivation but also many other factors that were all mentally related.

    It is also very cultural I believe.

  11. An interesting graphic in Joanne Rand Whitmore’s ‘Giftedness, Conflict, and Underachievement,’ (1980) page 186, Figure 6-1:

    A circle of engagement, in which the ‘pupil’ receives ‘increasing rewards for interaction,’ while outside the circle, decreasing rewards for interaction, until the pupil “checks out” to more attractive alternatives, and develops coping mechanisms (defenses);
    1. rationalization
    2. projections …
    3. devaluing …
    4. hostility …
    5. avoidance …
    6. minimum compliance
    7. chronic complaining
    8. focus on alternate source of reward.

    Also described in the book, children who are forced to repeat the first grade, wherein a great deal of rote compliance (over individual expression or systems learning) is required, end up with similar personality traits.

  12. I’m glad someone mentioned this, I’ve always been an under-acheiver. In grade school, when I took the itbs my scores were in the 85th national percentile, certainly if the predictive power of intelligence testing held true, I should be making 6 figures by now(I’m not and I’m 30), my sat was over 1100, which should translate to about 115-120 in iq, but it took me 9 years to finish my b/s with a 2.5, so intelligence at least for me has proven very little

  13. IQ tests show current ability for any situation. if you want to record the smartest man in the world then you’ll need to give him a knowledge test such as, ACT, SAT, or anything else similar. Then, you would need to grade his IQ over about a year sparingly, maybe 4 or 5 times total. After that, who ever has the highest knowledge test score and highest average IQ is the smartest. In general, I don’t understand why IQ test aren’t recommended at ever entry level job. Easiest way to see if they are wiling to learn. All I’ve ever used and nothing but success.

Leave a Reply

Your email address will not be published. Required fields are marked *