Fallible Forecasting

July 19, 1965

There has long been a need for someone to keep score on the errors of the business forecasters. This is at last being done, systematically and on a broad scale, by the nonprofit National Bureau of Economic Research.

The task has been subdivided among several staff members. One of them is trying to find out how good have been the predictions of turning points in business. He is finding these “often difficult to interpret or evaluate because of vagueness or hedging.” But another staff member, Victor Zarnowitz, had the easier task of comparing numerical predictions of the nation’s gross national product (GNP) with what the official figures turned out to be.

He presents a tabulation on the results of eight sets of annual forecasts. Four of these are company forecasts, another set groups those of 50 business economists. Altogether, the table records the efforts of 300 or 400 individual forecasters.

The errors in these eight sets of forecasts averaged nearly $10 billion a year, up or down, from the actual figures for the eleven years 1953–1963 inclusive. The errors appear small—about 2 percent—when compared with the average level of actual GNP. But, as Zarnowitz points out, they are big enough to make the difference between a good and a bad business year.

HOW TO COUNT ERROR

The average actual year-to-year change in GNP over the period was $22 billion. Thus, as the study notes, the errors were not quite one-half the size of those that would have occurred by assuming that each year’s GNP would be the same as the year before. Furthermore, the errors were almost as large as those that would have occurred by merely assuming that each year would show the same gain in GNP as the average gain in preceding years. Such an assumption would have gone wrong by an average of less than $12 billion—only $2 billion worse than the forecasts.

This, I think, is the valid way to count the percentage of error in a prediction. The predicted GNP should not be compared merely with the actual GNP, but also with the “automatic” prediction. By the automatic prediction I mean the figure that would result from taking the average percentage increase in GNP for, say, the last five to ten years, giving a little extra weight to later compared with earlier years, and making some allowance for the observed economic trend in the three months preceding the prediction. These are the kinds of figures that could be fed into an automatic computer. Then only if a prediction came nearer to the actual figure than did this automatic extrapolation would it deserve a compliment for insight.

MISSING THE TURNS

As Zarnowitz puts it, though the simplest measure of error is obtained by comparing predicted with actual levels, it is more important to compare predicted with actual changes.

And the forecasts have persistently underestimated changes. They have turned out best in “normal” years, worst in abnormal years. They have also turned out worse for longer periods ahead than for shorter. To cite one example: in a semiannual forecast of GNP for 1955–63 by a large group of business economists, the mean errors of change were, for a six-month span, $6.7 billion; for a twelve-month span, $12.3 billion.

In brief, business forecasting is not yet an exact science. Nor will it ever be. We can’t foresee the future by simply extrapolating from any number of figures or trends in the past. There are too many “outside” factors that cannot be foreseen—droughts, floods, tornadoes, wars, revolutions, political decisions, sudden strikes. Estimates of the effects of other factors—monetary policy, deficits, tax changes, economic controls, labor-union demands—must involve a large element of guesswork. Most important of all, economic forecasts themselves affect the future they predict.

Yet to guide our decisions, we must all try to forecast. By intelligent study we may all hope to reduce the extent by and frequency with which our forecasts go wrong.

The aim of every speculator and businessman, however, is not merely to be right, but to be right sooner than his competitors. And that result no science can provide.