Following up my last post I thought I’d expand on my problem with econometrics. Essentially the core problem with econometrics is that it is heavily dependent on its assumptions and can be easily twisted to say whatever its designer wants it to.
A while ago I read “Confessions Of An Economic Hitman” by John Perkins when I came across a very interesting passage. To put it in context Perkins claims he travelled around the world giving advice to Third World countries. He claims that the advice he gave was deliberatively false in order to trick countries into taking out loans they could never pay back. America could then use this debt in order to gain leverage and power over the countries. But first Perkins had to find a method of convincing governments to believe his overoptimistic projections.
“Bruno came up with an idea for an innovative approach to forecasting: an econometric model based on the writings of a turn-of- the-century Russian mathematician. The model involved assigning subjective probabilities to predictions that certain specific sectors of an economy would grow. It seemed an ideal tool to justify the inflated rates of increase we liked to show in order to obtain large loans, and Bruno asked me to see what I could do with the concept.
I brought a young MIT mathematician, Dr. Nadipuram Prasad, into my department and gave him a budget. Within six months he developed the Markov method for econometric modeling. Together we hammered out a series of technical papers that presented Markov as a revolutionary method for forecasting the impact of infrastructure investment on economic development.
It was exactly what we wanted: a tool that scientifically “proved” we were doing countries a favor by helping them incur debts they would never be able to pay off. In addition, only a highly skilled econometrician with lots of time and money could possibly comprehend the intricacies of Markov or question its conclusions. The papers were published by several prestigious organizations, and we formally presented them at conferences and universities in a number of countries. The papers — and we — became famous throughout the industry.”
Now what you take from this depends on how much you believe Perkins story. His book in general sometimes comes off as too incredible to be believed, but there’s little information on the internet to clarify the issue. However, regardless of the details, I think Perkins makes an excellent point. Firstly, econometrics can be twisted to say whatever the author wants them to say. This is a serious problem as it casts doubt upon on all studies, both good and bad. Secondly, it is incredibly hard to critically examine the claims of an econometric study. You need to be really smart and have a lot of time and money to fully investigate the claims of a study. Most people lack these qualities and therefore take even the most questionable of studies at face value. I don’t know how true Perkins story is, but it definitely could have happened.
A serious problem with econometric models was revealed during the Financial Crisis. You see models are created by taking events and trying to measure the likelihood that it will happen. This assumes that the world is predictable. However, others argue that uncertainty can dominate markets. The difference between the two is that if you flip a coin there is a 50% probability of getting heads. However, the chance that in 20 years time the president of America will be either a Democrat or a Republican is uncertain. We simply lack the information to possibly know. Financial companies based their actions upon models which would supposedly measure the risk they were subject to and allow them to take precautionary measures. To say they were wrong is an understatement.
In 2008, the econometric models were revealed to be disastrously wrong. To give you an idea of how wrong they were, the Chief Financial Officer of Goldman Sachs, David Viniar, said that “We were seeing things that were 25-standard deviation moves, several days in a row.” Now how unlikely is this? Well to put it in context something that is 5 standard deviations (or a 5 sigma event to use the economic jargon) will occur once every 14,000 years. A 6 sigma event will occur once every 4 million years. A 7 sigma event will occur once every 4 billion years. An 8 sigma event will occur once in the lifetime of the universe. There is no possible way to explain how unlikely a 25 sigma event is (most software programs run out of numbers). The easiest way to explain it would be if you won the lottery 22 days in a row. Viniar was saying these incredibly improbable events were happening several days in a row.
How did they get it so wrong? Well this comes back to a core problem of econometrics. A model is only as good as its data and assumptions. If you only use recent data when house prices are rising and the economy is booming then you will have a model that predict that the boom will get boomier. The problem is that the rest of us don’t know that. So the news headline will read “Study predicts economy to boom”, the pundits will ridicule critics and everyone will presume that if the serious economists who should know what’s going on think everything’s fine, then everything must be fine. It is not until after the crash that economists will come out and point out the flaws in the model, at which point it will do little good.
Herein lies the problem with econometrics. Despite what many may hope economics is not a science but rather a deeply political field. However, no one likes to admit they are political, instead they prefer to give the impression that their views are based upon objective facts. As a result econometrics can be misused to say whatever people want in order to push their agenda rather than being a purely evidence based activity. Now almost anything can be made political, but the sheer complexity and abstractness of econometrics means that few can decipher the layers of algebra within. So groups can use econometrics to masquerade as neutral judges while merely acting in their own self-interest. That is the problem with econometrics.