How should you judge a theoretical model?

When faced with a model of the world (in physics, neuroscience, economics, ecology), how should you judge that theory? Cyrus Samii suggests 5 ways. Here is number 2:

2. If any result can be engineered then results themselves have no special ontological status.

This is another way of asking whether a model has empirical content, which we typically take as falsifiability. Yet Karl Popper suggested:

The empirical content of a statement increases with its degree of falsifiability: the more a statement forbids, the more it says about the world of experience.

And he suggested “two criteria determine the empirical content of a theory are their level of universality (Allgemeinheit) and their degree of precision (Bestimmtheit).”

I also really like the question at the start of number 4:

How complicated can the problems be that we allow our agents to solve in a model? Is a dynamic program ever admissible as a reasonable assumption on the objective function of an agent?

Charles Krebs (or Judy Myers) says:

Recommendation – no paper on models should be published or talked about unless it makes specific, testable predictions of how the model can be tested.

I actually disagree with this rather strenuously. There are several reasons to make models, only one of which is to make predictions. Another is to confirm hypotheses.

Let’s say that you think that honeybees are dying because of the excessive use of mint toothpaste and you collect data to prove it. The problem is that data is simply a collection of facts (or “facts”) with no organizing structure. A model can give those facts that structure: you put what you know together with some of the data, and see if what you know is sufficient to replicate the observations of the world. Of course, you have to interpret these types of models carefully; they are not predictive models in the sense that they tell you anything about the world. Rather, they tell you about whether you have a consistent and complete story. But it’s still just a story.

3 thoughts on “How should you judge a theoretical model?

  1. Also a model can give us a simple or intuitive understanding of a system, even if it does not make any novel, or even accurate, predictions. The Ising model is not a very realistic model of a ferromagent but it has been very useful.

  2. This is another way of asking whether a model has empirical content, which we typically take as falsifiability.

    I completely disagree with this statement. An example that satisfied Cyrus Samii’s second criterion but not your ‘equivalent’ formulation is the Church-Turing thesis.

    I completely agree with your disagreement with Krebs. For instance, heuristic models are very useful and dominate most of science, yet they are usually devoid of predictive content. There are times when I would advocate for more predictive capacity, and other times I would actually push back against it (for more discussion on prediction in political science, see also Adam Elkus’ G+ post).

    It is important to always recognize when we are using these arbitrary criteria of a ‘good’ model as just a way to have power over people that we are unwilling to engage with in open critical discussion. We should always err on the side of more permissive rather than less, and any studies of ‘good’ models should be descriptive not prescriptive. In other words, I philosopher shouldn’t tell me how to build a good model, but should instead look at the literature and find common trends among models that scientists (or maybe some disembodied ‘history’, to avoid some of the more obvious band-wagon-ing) generally consider ‘good’.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s