03-Feb-2017 - University of Toronto

Certainty in complex scientific research an unachievable goal

A University of Toronto study on uncertainty in scientific research could shed light on anomalies that arose in early attempts to discover the Higgs boson and even how polls failed to predict the outcome of Donald Trump winning the U.S. presidential election.

The study suggests that research in some of the more complex scientific disciplines, such as medicine or particle physics, often doesn't eliminate uncertainties to the extent we might expect.

"This is due to a tendency to under-estimate the chance of significant abnormalities in results." said study author David Bailey, a professor in U of T's Department of Physics.

Looking at 41,000 measurements of 3,200 quantities - from the mass of an electron to the carbon dating of a sample - Bailey found that anomalous observations happened up to 100,000 times more often than expected.

"The chance of large differences does not fall off exponentially as you'd expect in a normal bell curve," said Bailey.

A long tail of uncertainty

"The study shows that researchers in many fields do a good job of estimating the size of typical errors in their measurements, but usually underestimate the chance of large errors," said Bailey, noting that the larger-than-expected frequency of large differences may be an almost inevitable consequence of the complex nature of scientific research.

"As measurements become more and more accurate, the smallest things matter more and more," Bailey said.

"If two measurements agree, you're happy. If not, you see there's something you need to investigate," he said. "You track down the cause of the variation and report the cause or you say that you don't know the cause and this reduces the trust in your result."

But with finite time and financial resources, researchers often have to make a choice between having a large sample of data, such as tens of thousands of people in a survey, and having a large number of variables you want to understand.

"You start with a very large sample that just lumps everyone together. You then might have to ask if your result is the same for both men and women. Is it the same for different backgrounds, Canadians versus Americans, for example," says Bailey. "At that point, you have to ask if your results hold for the smaller data set. Your sample is getting smaller and more can go wrong."

Impossible not to be a little wrong?

Physics studies did not fare significantly better than the medical and other research observed. However, the highly quantifiable way in which values and uncertainties are reported, may make physics more useful in terms of the degree of reproduce-ability of results that researchers should reasonably expect.

"Scientists will still aim for the most accurate results, but their expectations of how well those aims are met may be tempered in light of this research," said Bailey.

He believes his study can help researchers better analyze their data, motivate more care with novel results, and encourage more realistic expectations by both scientists and the public about the accuracy of scientific research.

"These insights can be beneficial given the inherently complex nature of scientific research," says Bailey. "But the chance of avoiding being wrong in some way on some level is almost impossible."

Facts, background information, dossiers

  • scientific research
  • statistics
  • statistical analyses
  • measurement uncertainty
  • experimental uncertainty

More about University of Toronto

  • News

    Discovery of 'knock-on chemistry' opens new frontier in reaction dynamics

    Research by a team of chemists at the University of Toronto, led by Nobel Prize-winning researcher John Polanyi, is shedding new light on the behaviour of molecules as they collide and exchange atoms during chemical reaction. The discovery casts doubt on a 90-year old theoretical model of t ... more

    'Breathing' enzymes in fast motion

    Researchers from the Department of Atomically Resolved Dynamics of the Max Planck Institute for the Structure and Dynamics of Matter (MPSD) at the Center for Free-Electron Laser Science in Hamburg, the University of Potsdam (both in Germany) and the University of Toronto (Canada) have piece ... more

    Advance ability to control chemical reactions

    Scientists at the University of Toronto have found a way to select the outcome of chemical reaction by employing an elusive and long-sought factor known as the 'impact parameter'. The team of U of T chemists, led by Nobel Prize-winning researcher John Polanyi, have found a means to select t ... more

q&more – the networking platform for quality excellence in lab and process

The q&more concept is to increase the visibility of recent research and innovative solutions, and support the exchange of knowledge. In the broad spectrum of subjects covered, the focus is on achieving maximum quality in highly innovative sectors. As a modern knowledge platform, q&more offers market participants one-of-a-kind networking opportunities. Cutting-edge research is presented by authors of international repute. Attractively presented in a high-quality context, and published in German and English, the original articles introduce new concepts and highlight unconventional solution strategies.

> more about q&more

q&more is supported by: