Capitalizing on Chance

In another post, I spoke of the "sin" of capitalizing on chance. What does this mean?

Let's say you do a simple treatment study with two groups. You aren't really sure whether the treatment being examined is going to be effective in treating anxiety, and the literature is unclear. So you decide to go the conservative route and use a two-tailed t-test.

You gather your anxiety data and run your stats--and much to your disappointment, your result just barely falls short of the value of t needed to be significant at .05 with a two-tailed test. Your value of t is in the right direction, but it just isn't big enough. But you also happen to notice that t would have been significant if you had used a one-tailed test (yes, this does happen sometimes!).

So you begin to think to yourself, "You know, I sort of had a hunch that it would go this way. After all, the treatment is effective with other disorders, why not with anxiety? I was too hasty in making the decision to go with the two-tailed test. I'm just going to say it should have been a one-tailed test in the first place. After all, if I don't, I won't have much to say, and who else would know? It was an honest mistake."

That sounds reasonable, doesn't it? But what's the problem? To begin with, it may be true that the original decision was too hasty, but that isn't in itself a reason to change the decision, unless there is evidence other than the statistical result itself that a change would be warranted. The lesson to learn here is: don't be hasty. Think it through carefully, and make the right decision the first time.

But beyond that, the bigger and related problem has to do with ignoring the implications of the probabilistic nature of hypothesis testing in the first place. Any result, even a significant one, may still be spurious, the random convergence of a variety of factors. The proper order of things is to assert your predictions on the basis of theory and prior research, then design a study to test your prediction, collect the data, and perform the test. If your prediction is right, and the method sound, the result should be significant.

But in the example above, to take hold of an unanticipated and unplanned significant result after the fact is to reverse that logic. Now you are working backwards, essentially establishing the "facts" on the basis of a statistically significant test that you stumbled across. To write it up as if this was the hypothesis you intended all along is dishonest, because it hides from the reader the fact that you have made up the supporting argument after you had the finding.

Such dishonesty aside, in a somewhat more subtle sense, we run the risk of capitalizing on chance anytime we try to make more out of a statistically significant result than its probabilistic nature actually warrants. So-called "fishing expeditions" are guilty of this: we ransack the data, running test after test without a clear plan, until we discover a significant result. Then we write up the results as intrinsically meaningful.

If we're honest, we'll tell the reader that we were exploring, and show what we actually did: we ran 100 correlations, for example, and show that only 5 of them were significant. The reader should rightly argue that if only 5 coefficients out of 100 are statistically significant at .05, we don't have much confidence that these are anything more than spurious findings. But if we're dishonest, we'll write up the project in a way that suggests that we were looking for those 5 significant relationships all along. (It makes us look smarter.)

Bottom line: make a careful study of the literature and make the best predictions you can, and determine your primary plan of analysis before you hit the data. Be honest with your readers about what you did and why. Remember that exploratory analyses have to be interpreted with special care, because the thrill of discovering something you didn't anticipate leads to the temptation of overinterpretation, and thus a greater risk of capitalizing on chance.

And keep in mind that no research project ever gives the last word on a subject. If things didn't work out the way you expected, learn from it--there may be an important contribution or correction to the literature just around the corner. If things did work out the way you hoped, replicate it! The more you do, the more confident you are that you haven't inadvertently capitalized on chance.