I was at a conference a few days ago and, after a theory speaker had finished his talk, a senior empirical faculty turned to me looking angry and said “you know, now that you’ve got tenure, you should write a paper about how empiricists should read theory.”
The problem is, I don’t know myself how theorists should read theory – so, next best thing to writing a paper about something you don’t know – I’ll jot down a few thoughts in a blog and hope it all comes together at the end.
Thought 1: Read more than the title
Make a prediction that can be tested in a data set: chances are more than one theory could deliver this prediction.
I’ll take a running example: consider the confirmation theory of Dye (1983) and Gigler and Hemmer (1999). The theory predicts that a verifiable message (e.g., earnings) makes otherwise unverifiable communication (e.g., forecasts) credible. But, this can only occur in the very different theory of Einhorn and Ziv (2008), a world where disclosures are credible by assumption, but the manager can pretend to be uninformed and withhold information. Realized earnings, in this model, inform investors about the manager’s information endowment.
Theories are about mechanisms or the means to get to a prediction, and within each paper, there is enough information to test which mechanism is operating. So a good reading of theory should identify the empirical content of the mechanisms (to be tested as well), not just the end prediction.
Returning to our running example, under confirmation theory, credibility is assured by writing a contract that punishes the manager for missing a forecast, controlling for earnings. Do we see this in the sample? Under dynamic disclosure theory, we should see that low earnings, indicative that the manager probably withheld strategically, should cause more disclosure in the next period, a very different time-series prediction.
Thought 2: Normative research provides guidance about what to study empirically, and why.
An old tradition in accounting – which by the way is no longer mainstream even in empirical research – is to view normative research (“how things ought to be”) as suspect and probably wrong. Recall that this came to be, over the 70s (see Demski 1973) because a number of individuals were dishing out accounting knowledge in the form of religious edicts, and this had to stop for serious scientific research to begin.
The majority of theory work has clear normative implications, and some of it does not contain much in terms of testable predictions, so having appreciation for what it is trying to do is important.
To talk about what normative research is, I’ll make a quick parenthesis along STEM, Science-Technology-Engineering-Mathematics. To over-simplify, let’s identify Science as the scientific method and take testability as a core principle. Let’s also set aside Technology and Mathematics since these are tools that are rare. However, Engineering is different. If I want to build a bridge, I make plans to do it based on sound principles validated by science. I’m not building a bridge only for the purpose of testing and, if I were to require testability always, no first bridge would ever be built.
This is what normative research is all about, namely, make plans for improvements that do not yet exist. Normative research requires good assumptions as inputs, hopefully assumptions that we believe have been tested. This is important: after all, without engineering/normative, then all the scientific knowledge we could accumulate would have no means of creating better outcomes.
So, how should empiricists consume normative theory? Normative theory is the natural end-point of the knowledge we create but it makes many assumptions. Knowing whether these assumptions are descriptive or not is the realm of positive theory and empirical research. Therefore, normative theory gives guidance to empiricists over what meaningful assumptions should be tested. It makes the empirical exercise relevant.
Thought 3: Find the selected math that summarizes.
Ten years ago, I remember attending a week-long seminar by an accounting theorist. I was shocked, rather than tell us about all the great things he had found, he started his talk by saying that “math is an unnecessary evil” and, then, looking at one particular faculty in the crowd, said they should shut down one top econ journal this faculty had published in (for, apparently, having published a few papers there, he was now its representative).
Anecdote aside, let’s ask the question: why do theorists use symbols when they do their work? Is it something that’s back-end material to be entirely ignored by empiricists?
Math in argumentation has many purposes, and one of them being to support a tight logical argument. For example, how often is it than one is lost in a wordy hypothesis where everything seems to float in the air, and multiple logics seem to be operating?
But is math only useful as a method of validity, so that empirical readers may safely ignore the math once a referee or journal has verified its correctness? To answer this question, consider absorbing Adam Smith’s Wealth of Nations, and compare it to an undergrad micro textbook treatment of the welfare theorems: which one is easier?
Empiricists can learn from math in a study because a few equations can provide a concise simple summary of otherwise difficult trade-offs. You don’t need to bury yourself in the appendix, or even to follow every main argument to get to these equations. So, to use a theory papers, remember those few equations that summarize the assumptions and main results; this is often much easier than remembering convoluted steps and implications of a wordy logic.
Thought 4: Take a theory seriously, not literally.
Any theory is a simplification of reality, it does not aim to be descriptive of everything and, in social sciences, the most successful theories only get to first-order effects. We can’t take theories literally to be exact representations but we can take a theory seriously enough so that it may explain empirical behavior.
Unfortunately, many empiricists view theory as merely motivational. By motivational, I mean that a theory is here to introduce a topic but nothing more, or that it is part of an enormous bundle of theories that have nothing to do with one another and together deliver predictions – some of which may occur in one of the theory and not the other, in both or in none.
So taking one theory seriously really means the following: let’s shut down for a moment all other theories, and assume that the world that we see in a data set has been generated by this theory (and only this theory). Is what I see in line with what the theory says? Is the theory complete enough to speak to all the empirical facts I want to study?
Doing this requires some specialization: if one is serious about a theory, one needs to know it very well – but the payoff is to be able to test multiple predictions of something parsimonious and clear. Few studies can claim this type of transparency.
What about alternative explanations? The good thing about it is that, once we are binded to multiple deep tests of a theory (being serious about it, that is!), most alternative theories will often fall as being naturally ruled out by these tests without having to design specific extra tests.
Thought 5: Theories tell you about (unobserved) exogenous variation in observational samples.
Some disagree, but I like the idea of theory as a poor man’s substitute to unavailable data. If we lived in a world where we could experiment anything instantly and at no cost, we would not need any theory to make better things – we would simply proceed by an infinite set of free experimentation.
I think that’s the deep problem with advocates of the self-called ‘credibility’ revolution who believe that experimentation is required to solve any problem. Yes, experimentation is better but isn’t free or always feasible – so, wherever it is missing, we need to rely on theory.
The greatest example of this is observational data. I was at a conference at Kellogg law and a statistician complained that most research designs in the social sciences would not meet the standards of medical science – he was quite critical of observational data outside of a controlled experiment and believed that instrumental variable methods (even assuming the exclusion held) had serious statistical flaws. I think his point of view was that only carefully planned experimentation could meet the standard of proof, noting that economists always asked him to “Believe..”
Yes, indeed, theory does require to believe. Believing makes things less credible, but there is often no alternate course. But let’s be more precise now, what does ‘believe’ mean in an observational design? Theory is a statement from a source of exogenous (but often unobservable variation) and how this exogenous variation can cause outcomes. The theoretical exercise accepts we need to make assumptions, but requires that these assumptions be clear and logically used – this is the least we can do.
What does the theory conjecture is the source of exogenous variation? While it does not offer certainties, this can inform the empirical design about what assumptions are being made, and (within the theory) how these assumptions are used in a consistent manner.
The evil word, here, is of course endogeneity. Endogeneity is a fundamental characteristic of any observational study. Theory does not solve the endogeneity, if we mean, by solving, providing the same level of confidence as if there had been an experiment. However, theory does clarify a plausible mechanism for the endogenous relationship between variables, and links them to an exogenous source. Theory clarifies the assumed source of (unobserved) exogenous variation.