Incredibly Accurate, Incredibly Simple Multi-year Stock Market Forecast and Why That is Only Modestly Helpful

Forecasting the stock market 3 to 5 years into the future can be straightforward and fairly accurate if three basic concepts are followed:

1.  Long term multi-year forecasts are inherently more accurate than short-term forecasts.
2.  Estimating the collective future of thousands of companies gives more success than forecasting just a few.
3.  Avoid forecasting areas where speculative money is dominant.

In brief, trying to forecast the near term future of just a few companies that are the focus of huge piles of fast-moving hot money is just begging for failure.

This translates to a single rule: Do not attempt short-term or single year forecasts of the Dow Jones 30 Industrial Average, the S&P 500 or the NASDAQ composite, or at least, don't expect great accuracy if you do.  The calculation of each of these indexes is dominated by just a small number of huge companies.  Each is followed closely by millions of nervous investors all trying to beat the market and usually failing.  And speculation in each of these indexes is cheap and simple using exchange traded funds. All the makings of a frustrating forecast. Most market forecasters make their annual predictions using these popular averages, and not surprisingly, most of these annual forecasts end up being shown up as wildly inaccurate.

So what's left?  The rest of the stock market, all the thousands of other companies whose names are not household words and people really don't know or care much what they do.  Prices of these thousands of companies -- much of the rest of the economy -- collectively tend to move in a much more predictable way. They follow a path of constant moderate growth very closely over the span of several years. Not month-by-month, or year-to-year, but over a multi-year time span.

What explains this steady long term appreciation? The Law of large numbers. Millions upon millions of people work for these companies.  They go to work 5 days a week or so and they try pretty hard to do a good job regardless of whether the stock market is up or down.  Even in a severe recession most all of these workers are still plugging away at their jobs. The result is a rather predictable rate of overall corporate growth.

Like the more speculative parts of the stock market, prices of the less-speculative stocks go up and down with each bubble and recession.  Their collective price swings, however, are less dramatic and are more closely tied to economic fundamentals.

How predictable is 'the rest of the market'?  Statisticians use a measure called R-squared which evaluates how far, on average, forecast points differ from what actually occurs.  R-squared of 1.0 is a perfect match.  R-squared equal zero or negative means the model is essentially worthless.

Since 1984 the S&P 500 has followed a somewhat constant rate of growth with R-squared = .84, actually pretty good. Simply put, buy and hold investing makes sense: over the long haul the index grows at a somewhat steady pace -- except for all the speculative crashes and bubbles along the way.

I follow the Value Line Arithmetic Index  (VALUA) which focuses more on 'the rest of the market.' VALUA gives equal importance to each of the 1,700 or so stocks that the Value Line Investment Survey tracks. No company is treated as more important than any other in calculating the index.  VALUA  follows a constant growth rate with an R-squared value of .98.  Not perfect, but very good. An ETF near-equivalent is the Guggenheim S&P 500 Equal Weight ETF (RSP)

`

(Click on image to enlarge.)

Looking at the graph above it is obvious that neither VALUA nor the S&P 500 follow their long term steady growth trends exactly. But, even over just a 3 to 5 year time span, the growth of VALUA has matched the steady growth trend very well: R-squared = .96.

So what? -- stock indexes tend to follow their long term growth paths if you look from a multi-year perspective. Whoopie! Not news to anyone.

A more important and useful observation, however, is that VALUA has a very strong tendency to 'regress to its mean', to move back in line with the long term growth trend.  If the VALUA is above the long term growth trend the likelihood is that the next few years will show sub-par appreciation.  But, if the index is at or below trend, as is the case now, then average or somewhat better than average price appreciation is probable.  When the average is near the long term trend line a tremendous price gain becomes less likely. If prices do shoot up, a severe correction looms in the future.