In four of the last five calendar years, [Warren Buffett] has underperformed his own benchmark, the S.&P. 500 with dividends, often by significant margins. (In 2011, his return of 4.6 percent beat the benchmark by 2.6 percentage points.) In addition, data provided by Morningstar shows that he underperformed the average stock mutual fund investor in four of the five years.)
By contrast, in the previous decades, he had underperformed the S.&P. only six times. Mr. Mehta said his calculations showed that given such a long period of outperformance, there is only a 3 percent chance that the recent stretch of underperformance was a matter of bad luck. [emphasis added]
You don’t need to know anything about finance, Warren Buffett, or Mr. Mehta’s calculations to be able to sniff this out as total nonsense. How? Because there is absolutely no way that this writer for the NY Times, or any other mortal, could know the a priori probability that some force other than luck has started affecting Warren Buffett’s work roughly 5 yrs ago, in such a way that would cause his performance to change in the way that it has. And you’d have to know this in order to make the quoted claim.
More simply: in many cases you can be almost certain that a statement is bogus when it takes the form “based on this evidence, there is an x% probability that some force other than luck is involved.” The exceptions are cases where, prior to seeing the new evidence, you would have been able to say a) what the probability of some cause other than luck being involved (this is often very difficult) and b) what the probability of observing the new evidence would be, given that cause.