Publication bias

This is almost common sense but if you can characterize it by value - excellent! The simple statement is that things that are found in data are more likely to be published than things that are not found.

For example, if a scientific paper finds (cosmology hit again) non-gaussianity in the CMB it is more likely to be published than anything that doesn’t find non-gaussianity (if judging by non-gaussianity alone). A rule of thumb is generally that if a (new) effect is found to ~95% (2σ) confidence, don’t believe it: there’s a one in twenty chance of it being wrong. To prove something new you need a certainty of around 5σ (something like 99.99999980%). If you need a real life corollary, and you happen to be/know a D&D enthusiast,  it’s effectively the difference between rolling a 20 on a fair d20 (20-sided) die versus dying of radiation poisoning from eating a single banana.

Markov chain Monte Carlo

Briefly… I want to only touch on Markov chain Monte-Carlo (MCMC) simulations. To explore probabilities of multiple parameters (probability-space) is computationally expensive. If you, say take 10 variations of a single parameter that will cost 10 computations. Two parameters will cost 100 computations. If you, say, compute 8 parameters, it will cost 100,000,000 evaluations. Supercomputers typically push a few million computations.. but that is pushing it. And so MCMC comes into play.

  1. Monte Carlo: any calculation that involves an element of randomness e.g. making a hop of a random size (normally drawn from a gaussian).
  2. Markov chain: where the next hop is taken depends on where it ends up -> only a position of greater likelihood in probability space is taken.


So an MCMC run will gradually converge to a point in probability-space, where the most likely combination of parameter values (i.e. model) is found. To prevent false minima form being found, you can launch several Markov Chains: most will end up at the true maximum.

Computationally, MCMC scales linearly with additional parameters, as opposed to the multiplicative grid effect. Of mention, is the Metropolis-Hastings algorithm, which resembles Markov-Chains but sometimes allow steps to positions of lower likelihood.. allowing exploration of the shape of the maximum.