The Difficulty of Backtesting

One of the benefits of momentum based systems is that you don't have to deal with the vagaries of fundamental data. Leaving aside revisions to economic data, when they appear in real-time, and changes to their methodology, earnings data is difficult enough. You would think it would be a simple enough thing to just add up companies earnings to get a P/E or to use operating earnings instead of as reported earnings, but some even criticize those two points. Also, you have to deal with changes to index methodology.

Let me explain. In the late 1990s, MSCI changed the methodology by which they calculate indices. In the past, they used simple market-cap weights and tried to include 60% of the publicly traded firms (by market-cap) in the index. At the time, most other indices were float adjusted, where they only recognized shares available for trading in computing weights. Countries like Japan, which saw massively overvalued multiples in the late 80s and early 90s, also had companies that were not widely held as part of their indices. Since they weren't widely held, they should have had less weight in the index. Less weight in the index would have meant that prices wouldn't have gone as high and stocks wouldn't have been as overvalued in the early 90s.

The other main change was increasing the weight 60% limit to something like 80-85%. For instance, in the U.S. GM had 60% of the U.S. auto market so only GM was included, but Ford (despite being huge) was not.

How does this matter? Well, if the index methodology is changed to correct a massive overvaluation in Japanese stocks, it means that you cannot compare historical price or PE ratios to the current levels for most historical indices. For example, let's say I build a global tactical asset allocation model with a 50% weight on momentum and a 50% weight on value. I determine momentum for a country's equity index by how its momentum compares against the other countries and then I compare it to its history (using a z score). Then I take 5 pieces of valuation data (like P/E, P/Cash flow, P/B, RoE, RoA) for each country and compare each country relative to each other and again to the history for each series. In each case, comparing the multiplies to each other should have no bias. However, if I'm using an index that has changed its methodology and was severely biased upward, all my valuation and momentum data shouldn't mean much when I compare it historically. For instance, the standard deviation of P/Es would be much larger and the average value would be higher (due to Japan being overvalued, for instance). The z-scores could be telling me something that isn't true. So what might appear to be historically cheap now, after correcting the bias, actually may not be cheap. As a cross-check, it would make sense to also look at alternative indices that have not substantially changed their methodology over the period.

If Siegel is right (from WSJ article above), then even the earnings data should be biased.