How do you do your science?

I spend too much time thinking about what the best way is to do science. How should I structure my experiments if I want to maximize the likelihood that what I discover is both true and useful to other people? And how many different experiments do I need to do? Especially as a theoroexperimentalist.

The background philosophy of science chatter has picked up a bit over the last week, and I was spurred by something said on Noahpinion:

I don’t see why we should insist that any theory be testable. After all, most of the things people are doing in math departments aren’t testable, and no one complains about those, do they? I don’t see why it should matter if people are doing math in a math department, a physics department, or an econ department.

Also (via Vince Buffalo)

As a mathematical discipline travels far from its empirical source, or still more, if it is a second and third generation only indirectly inspired from ideas coming from ‘reality’, it is beset with very grave dangers. It becomes more and more purely aestheticizing, more and more purely l’art pour l’art. This need not be bad, if the field is surrounded by correlated subjects, which still have closer empirical connections, or if the discipline is under the influence of men with an exceptionally well-developed taste. But there is a grave danger that the subject will develop along the line of least resistance, that the stream, so far from its source, will separate into a multitude of insignificant branches, and that the discipline will become a disorganized mass of details and complexities. In other words, at a great distance from its empirical source, or after much ‘abstract’ inbreeding, a mathematical subject is in danger of degeneration. At the inception the style is usually classical; when it shows signs of becoming baroque the danger signal is up. It would be easy to give examples, to trace specific evolutions into the baroque and the very high baroque, but this would be too technical. In any event, whenever this stage is reached, the only remedy seems to me to be the rejuvenating return to the source: the reinjection of more or less directly empirical ideas. I am convinced that this is a necessary condition to conserve the freshness and the vitality of the subject, and that this will remain so in the future. ::: John von Neumann

Right before I left the Salk Institute, I was chatting with an older scientist and he said something along the following lines (paraphrasing):

The best science is done when you can identify a clear mechanism that you can test; [anonymized scientist] was known for having a lot of confirmatory evidence that pointed at some result, but nothing conclusive, no mechanism. Pretty much all of it ended up being wrong.

Basically, he was of the opinion that you can provide evidence of a direct mechanism, or you can provide evidence for a general idea that is consistent and points to that mechanism like so:

Experimental designWhere you should you collect your evidence? Each situation makes one of these arrows easier to collect than others.

But if you want to maximize your likelihood of making a lasting impact on knowledge, where do you want to place your bets? Can theories come before mechanism?

I don’t know.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s