• Question: How many repeats of an experiment would you have to do in your line work to make your results reliable?

    Asked by astroaaron to Ben, Jony, Katharine, Mark, Peter on 17 Nov 2011.
    • Photo: Mark Basham

      Mark Basham answered on 16 Nov 2011:

      Hi astroaaron,

      In our experiments we always check something called signal to noise, This is basicaly the quality of the result compared to the errors in the collection. This way we always make sure that we collect the right quality of data so we don’t have to repeat the experiments too often.

      This is mainly due to the fact that a scientist may need to wait for a year to come and use our equipment, so if they don’t get the results they need they will have to wait a long time to repeat the experiment.

      Mark

    • Photo: Peter Williams

      Peter Williams answered on 16 Nov 2011:

      mark is right . S/N is the value to use.
      With accelerators it usually either works or doesn’t! But often we use the accelerators to look at subtle effects. For example, in particle physics there are many billions of events. Advanced statistical techniques have been developed to grapple with data on this scale. If you are looking an effect you quote your result in terms of a sigma confidence level. This is based on the Gaussian (or normal) distribution, which occurs pretty ubiquitously in Nature. Think about the distribution of heights of people in your class – most people will be around some average, the number of people who are either very tall, or very small, is small. Statisticians have used this to construct precise ways of describing your confidence in a result. The details however are pretty involved!
      Go and study statistics modules at a-level maths to find out more.

    • Photo: Katharine Schofield

      Katharine Schofield answered on 16 Nov 2011:

      In particle physics the aim of the game is to get what’s known as a ‘five sigma’ result. I found a neat explanation of what this means on the bbc website:
      “Particle physics has an accepted definition for a “discovery”: a five-sigma level of certainty.
      The number of standard deviations, or sigmas, is a measure of how unlikely it is that an experimental result is simply down to chance rather than a real effect.
      Similarly, tossing a coin and getting a number of heads in a row may just be chance, rather than a sign of a “loaded” coin.
      The “three sigma” level represents about the same likelihood of tossing more than eight heads in a row.
      Five sigma, on the other hand, would correspond to tossing more than 20 in a row.
      A five-sigma result is highly unlikely to happen by chance, and thus an experimental result becomes an accepted discovery.”

      Although this doesn’t mean that doing an experiment 20 times will get you a result in particle physics… you need to build up lots of data and understand all your errors in really thorough detail. One thing I found when I was doing my PhD is that you spend a LOT of time thinking about your error bars.

      Nowadays I’m not a researcher, so I don’t need to think about this stuff too deeply anymore. A good grasp of statistics is really useful in life though!

    • Photo: Ben Still

      Ben Still answered on 17 Nov 2011:

      As Katherine said we require a difference in expectation of 5-sigma for a confident measurement of something in a particle physics experiment, but this is often still not enough for the scientific community to take as evidence if coming just from one experiment. In particle physics we require repeats from different experiments to cross-check the systematic (experimental) errors as well as look at methods of measuring the same thing. To do this we usually have a few experiments doing similar things but with slightly different set-ups.

      The Large Hadron Collider will not claim a discovery of the Higgs “God” particle for instance unless it is unanimously seen in at least the two big Atlas and CMS detectors, These two detectors use similar but subtly different techniques for looking for the Higgs. They also have different strengths and weaknesses. But if both see evidence for the Higgs above the 5-sigma level then the scientific community will see this as discovery. If just one experiment saw the Higgs at 5-sigma and the other not at all then acceptance would be more difficult.

      In my research of neutrino physics we are not in the game of discovery but measurement of the properties of the neutrino, but the same rules apply. We use different types, energy and distance over which we fire neutrinos to cross check each experiments result. My experiment uses a beam of muon-type neutrinos made from a particle accelerator and other experiments use electron-type anti-neutrinos from nuclear reactors, but we are both trying to measure the same properties.

Comments