- What is the importance of a single experiment to science? Very little.
- A massive study attempted to replicate 100 psychology experiments. 36% still had ‘significant’ p-values and 47.4% still had a similar effect size.
This is a description of one attempt at replication:
“The study he finally settled on was originally run in Germany. It looked at how ‘global versus local processing influenced the way participants used priming information in their judgment of others.’… Reinhard had to figure out how to translate the social context, bringing a study that ran in Germany to students at the University of Virginia. For example, the original research used maps from Germany. “We decided to use maps of one of the states in the US, so it would be less weird for people in Virginia,” he said…Another factor: Americans’ perceptions of aggressive behavior are different from Germans’, and the study hinged on participants scoring their perceptions of aggression. The German researchers who ran the original study based it on some previous research that was done in America, but they changed the ratings scale because the Germans’ threshold for aggressive behavior was much higher…Now Reinhard had to change them back — just one of a number of variables that had to be manipulated.”
- In the simplest “hypothesis-test” experiment, everything is held constant except one key parameter (word color; monetary endowment; stereotype threat). In reality, this is never true. You arrive at a laboratory tired, bored, stressed, content.
- Experiments are meant to introduce major variations into already noisy data. The hope is that these variations are larger than anything else that occurs during the experiment. Are they?
- Even experiments that replicate quite often can turn out to be false
- Even experiments that fail to be replicated can turn out to have grains of truth
- The important regression is the likelihood of replication given n independent replications already in existence. If 66% of experiments fail to replicate when contained in only one publication, what percent fail when contained in two? Three? Science is a society, not the sum of individuals.
- Given time and resource constraints, what is the optimal number of experiments that you expect to replicate? What if it should be lower?