The Problem With Econometrics

Following up my last post I thought I’d expand on my problem with econometrics. Essentially the core problem with econometrics is that it is heavily dependent on its assumptions and can be easily twisted to say whatever its designer wants it to.

A while ago I read “Confessions Of An Economic Hitman” by John Perkins when I came across a very interesting passage. To put it in context Perkins claims he travelled around the world giving advice to Third World countries. He claims that the advice he gave was deliberatively false in order to trick countries into taking out loans they could never pay back. America could then use this debt in order to gain leverage and power over the countries. But first Perkins had to find a method of convincing governments to believe his overoptimistic projections.

“Bruno came up with an idea for an innovative approach to forecasting: an econometric model based on the writings of a turn-of- the-century Russian mathematician. The model involved assigning subjective probabilities to predictions that certain specific sectors of an economy would grow. It seemed an ideal tool to justify the inflated rates of increase we liked to show in order to obtain large loans, and Bruno asked me to see what I could do with the concept.

I brought a young MIT mathematician, Dr. Nadipuram Prasad, into my department and gave him a budget. Within six months he developed the Markov method for econometric modeling. Together we hammered out a series of technical papers that presented Markov as a revolutionary method for forecasting the impact of infrastructure investment on economic development.

It was exactly what we wanted: a tool that scientifically “proved” we were doing countries a favor by helping them incur debts they would never be able to pay off. In addition, only a highly skilled econometrician with lots of time and money could possibly comprehend the intricacies of Markov or question its conclusions. The papers were published by several prestigious organizations, and we formally presented them at conferences and universities in a number of countries. The papers — and we — became famous throughout the industry.”

Now what you take from this depends on how much you believe Perkins story. His book in general sometimes comes off as too incredible to be believed, but there’s little information on the internet to clarify the issue. However, regardless of the details, I think Perkins makes an excellent point. Firstly, econometrics can be twisted to say whatever the author wants them to say. This is a serious problem as it casts doubt upon on all studies, both good and bad. Secondly, it is incredibly hard to critically examine the claims of an econometric study. You need to be really smart and have a lot of time and money to fully investigate the claims of a study. Most people lack these qualities and therefore take even the most questionable of studies at face value. I don’t know how true Perkins story is, but it definitely could have happened.

A serious problem with econometric models was revealed during the Financial Crisis. You see models are created by taking events and trying to measure the likelihood that it will happen. This assumes that the world is predictable. However, others argue that uncertainty can dominate markets. The difference between the two is that if you flip a coin there is a 50% probability of getting heads. However, the chance that in 20 years time the president of America will be either a Democrat or a Republican is uncertain. We simply lack the information to possibly know. Financial companies based their actions upon models which would supposedly measure the risk they were subject to and allow them to take precautionary measures. To say they were wrong is an understatement.

In 2008, the econometric models were revealed to be disastrously wrong. To give you an idea of how wrong they were, the Chief Financial Officer of Goldman Sachs, David Viniar, said that “We were seeing things that were 25-standard deviation moves, several days in a row.” Now how unlikely is this? Well to put it in context something that is 5 standard deviations (or a 5 sigma event to use the economic jargon) will occur once every 14,000 years. A 6 sigma event will occur once every 4 million years. A 7 sigma event will occur once every 4 billion years. An 8 sigma event will occur once in the lifetime of the universe. There is no possible way to explain how unlikely a 25 sigma event is (most software programs run out of numbers). The easiest way to explain it would be if you won the lottery 22 days in a row. Viniar was saying these incredibly improbable events were happening several days in a row.

How did they get it so wrong? Well this comes back to a core problem of econometrics. A model is only as good as its data and assumptions. If you only use recent data when house prices are rising and the economy is booming then you will have a model that predict that the boom will get boomier. The problem is that the rest of us don’t know that. So the news headline will read “Study predicts economy to boom”, the pundits will ridicule critics and everyone will presume that if the serious economists who should know what’s going on think everything’s fine, then everything must be fine. It is not until after the crash that economists will come out and point out the flaws in the model, at which point it will do little good.

Herein lies the problem with econometrics. Despite what many may hope economics is not a science but rather a deeply political field. However, no one likes to admit they are political, instead they prefer to give the impression that their views are based upon objective facts. As a result econometrics can be misused to say whatever people want in order to push their agenda rather than being a purely evidence based activity. Now almost anything can be made political, but the sheer complexity and abstractness of econometrics means that few can decipher the layers of algebra within. So groups can use econometrics to masquerade as neutral judges while merely acting in their own self-interest. That is the problem with econometrics.

6 thoughts on “The Problem With Econometrics”

  1. In physics if we throw an object up into a gravitational field we know we will get something near a parabolic arc. The results are quite predictable. Are any economists studying the way their data “arcs” to see if there are any recognizable patterns in the data? If there are not, then nothing is predictable as the data are chaotic. This analysis is much simpler to do than the complex mathematic models I have seen being used and should be definitive. Econometrics that are chaotic would then have to be replaced by some that were … and if such data cannot be found ..

  2. I think you are conflating econometrics and modeling. Econometrics is about finding relationships between variables (e.g., correlation). Predictions and forecasting are usually handled by modeling. One can certainly try to use regression results to inform a model that makes predictions, but from what I’ve seen in the literature, this is not how econometrics is primarily used for. It’s used to answer questions.

    One of my favorite papers on the financial crisis is “The Consequences of Mortgage Credit Expansion: Evidence from the U.S. Mortgage Default Crisis” by Atif Mian and Amir Sufi (2009).

    It tests the theory that the securitization of mortgages (CDOs, etc.) had a role in causing the financial crisis. So they procured mortgage data by zip code, and noted that the amount of mortgages granted in a neighborhood should normally be correlated with the credit-worthiness of that neighborhood, for which income is a decent proxy.

    Now, if securitization caused credit standards to be loosened (because lenders knew they would no longer have “skin in the game”), you would expect the amount of mortgage credit to be correlated with the amount of securitization in that neighborhood instead. This is a testable hypothesis, and econometrics can be used to factor out effects of different variables in order to simulate a laboratory control. Indeed, they found mortgage credit expansion was correlated with securitization. There are also numerous similar empirical studies using different methods to reach similar conclusions.

    Of course, real-world data is messy, and there is a judgment aspect in regards to how to correct for certain types of errors, but that’s where peer review comes in. Good researchers will explain the decisions made, supplying a rationale for any difficult ones, and supply the data to anybody who wants to see whether they failed to correct for an error, made a mistake, or merely what happens if different decisions are made.

    Note, econometrics is definitely not an area of expertise of mine, but from what little I understand of it, the crisis did not disprove econometrics (if anything, the main area of econ most damaged was DSGE modeling). If anything, I think econometrics has been one of the best tools available to study the crisis.

  3. I think your problem with econometrics is that you do not know any. You are masters student, you have probably just taken a mid term in introductory metrics. How can you critique something you know epsilon about…. Then again, hasn’t stopped you before. As the great KW says “it’s a good idea to learn what something is before dismissing it.”

  4. Econometrics can be used and abused in various ways. First, that most innocent of applications; the correlation; is open to data mining – if you run enough regressions you will eventually get a false positive – econometrics should be used to test theoretical predictions. There is also the problem of cointegration – in simple terms; series with similar trends will appear correlated even if completely unrelated. Next we come to causality. A may appear to cause C but B could cause both or A could cause B and B could in turn cause C. A real can of worms. Finally there is modelling (forgive me father for I have sinned). Everybody wants to now the future but for a salutory lesson in the reliability of models take a simple one and project it forward with the confidence intervals on the parameters – you may be surprised at how soon it is totally meaningless. For all that econometrics is a useful tool when used correctly but like any tool it can hurt the novice.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: