Friday, May 23, 2008

Does the Wonderlic test predict QB performance?

The Wonderlic is a kind of intelligence test, used to screen potential employees. Apparently the NFL has been using it as part of scouting the draft.

Does a player's Wonderlic score predict how well he'll perform in the NFL? According to
this study, by Arthur J. Adams and Frank E. Kuzmits, the answer is no. They found no statistically-significant correlations between Wonderlic score and measures of draft position, salary, QB rating, or games played.

However, a subsequent
study by a testing company called "Criteria" *did* find an effect. Criteria looked at the 61 QBs drafted between 2000 and 2004. For performance measures, they looked at total passing yards and TD passes (which you would think would be so highly correlated with each other that they would be measuring almost the same thing, but never mind). They found a mild positive correlation of about .2.

But: when they included only quarterbacks with at least 1,000 yards – effectively eliminating the ones that didn't get any playing time – the correlation jumped to .5. As
Brian Burke points out on his own blog, .5 is a HUGE correlation in a study of this type.

Now, it should be mentioned that Criteria has a vested interest in making aptitude tests look effective. If this is a trial and I'm on the jury, I'm going to assume that they tried a bunch of possible cutoffs and methods and chose the one that makes the test look the most predictive. Brian notes that if you change the cutoff from 1,000 yards to 2,000, and take Tom Brady (high Wonderlic, very high NFL performance) out of the study, the correlation drops to about .26 (still pretty significant).

I'm even more skeptical than Brian. Well, I am and I'm not. First, I would be willing to bet a lot of money that there IS a significant correlation between the Wonderlic and QB effectiveness. There's no doubt that QB requires good intelligence and quick decision-making skills, which is partly what the Wonderlic measures. In fact, I'd bet that
Vince Young's supposed score of 6 out of 50 is a mistake – I don't believe that anyone with Wonderlic skills that low could have been a successful college quarterback, never mind a top-ranked one. (See these sample questions for yourself – supposedly Vince Young would only score 2 out of 15.)

So, yes, of course intelligence (or whatever you want to call what the Wonderlic measures) is important to making a good quarterback. But that's really not the question. The question is, *given what you already know* about the quarterback, in four years of extremely competitive college play, does the Wonderlic give you any *additional* useful information about the player?

On that, I'd bet that it doesn't.

Look at it this way: switch to baseball for a minute, and suppose I ask you to predict what Albert Pujols will do in the remainder of this season. You'd have a pretty good idea, and would make a fairly accurate estimate. Now, suppose I boot on over to St. Louis, give Pujols the Wonderlic test, and tell you his score. Will that affect your estimate? No, of course not. Whatever effect Pujols' Wonderlic skills have on his performance has already shown itself in several seasons of major league play. No matter how much Wonderlic correlates with having a high OPS, if you know the player's skill in achieving OPS, no intelligence test is going to give you a better estimate.

Where the Wonderlic would help you is if you didn't know what real-world skills an applicant has. It would be much more useful to know the Wonderlic scores of presidential candidates than football players, because, for instance, we don't have a good idea of their president skills -- we haven't seen Obama or Hillary play president for four years in college. It would be less useful for George W. Bush, because we have strong, first-hand observations on what he's like as president. Did he achieve his Bushness with a high Wonderlic, a medium Wonderlic, or a low Wonderlic? That might be interesting trivia, but shouldn't affect how we evaluate his potential, because we've actually seen him in action.

And the same is true for drafting quarterbacks. The question we *really* want to answer, in the NFL context, is this: given what you know about the player, and have discovered in the NFL combine and from watching him play college football for four years, does the Wonderlic tell you anything *additional* to that?

That is: you have two identical quarterback candidates, with identical college records playing for identically-skilled college teams against identical opponents. At the combine, they look exactly the same in every respect, except the Wonderlic. One of them has a Wonderlic score of 30, and one has a Wonderlic score of 20 (24 is average for QBs). Should you expect a difference in NFL performance?

It's possible to study that question, but I think you'd have trouble finding enough data, and adjusting for the different environments in which quarterbacks play, to be able to make any significant conclusions.

And intuition suggests that they'd probably be about the same. If A and B had equal performance, despite a being significantly "smarter" than B, it follows that B must have had *other* compensating aptitudes that made up for it. Doesn't it?

As for this study, if you believe the results, all we have is a conclusion that better and faster thinkers make better quarterbacks. Well, duh.

Hat tip:
NFL Stats (Brian Burke)

Labels: , ,


At Friday, May 23, 2008 6:44:00 PM, Anonymous Anonymous said...

I looked at the sample questions. It seems quite possible to me that a player who is reasonably intelligent but poorly educated could fail to perform well on it.

At Wednesday, June 18, 2008 12:55:00 PM, Blogger Unknown said...

I have taken the wonderlic test and there is no correlation between education and scores. the majority of the questions are of elementary or middle school difficulty. how fast you are able to process the answer is where you shine.

At Thursday, July 31, 2008 12:34:00 PM, Anonymous Anonymous said...

The issue here really is that the wonderlic is a test of general intelligence. Although it can give you a decent overall picture of a person's mental capacity, a much more accurate measure would be a test of QB IQ or Football IQ. The general intelligence test score includes a lot of noise because it measures things that may not necessarily help or hurt an NFL QB's performance.

Consider the 40-time and RB performance... The 40 time tells us how fast a RB is, but how often does a RB get to just run along in open space. We know the general speed of a player, but that speed doesn't translate perfectly to the game.

I won't even go into the whole debate on whether general intelligence tests actually test intelligence or just your ability to take that particular test. But realize that the jury is still out on that one.


Post a Comment

<< Home