Introduction

Overview

Questions

• What is gained from good experimental design?

Objectives

• Connect experimental design with data quality and meaningful findings.

Good experimental design generates information-rich data with a clear message. In both scientific research and industry, experimental design plays a critical role in obtaining reliable and meaningful results. Reliable, trustworthy science is not the result of fancy analyses or algorithms. In fact, analysis and design are intrinsically linked and dependent on one another. What you want to find out in your analysis dictates how to design the study, and that design in turn limits the kinds of analysis that you can realistically do. In other words, you would analyze as designed.

graphic of design and analysis

No analysis, no matter how elegant or complex, will be reliable or meaningful if based on a flawed design. Sir Ronald A. Fisher famously said:

“To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.” Presidential Address to the First Indian Statistical Congress, 1938.

Experimental design aims to describe and explain variation in natural systems by intervening to affect that variation directly. In an experiment, the experimenter actively controls the conditions. For example, the Salk polio vaccine trials collected data about variation in polio incidence after injecting more than 600,000 schoolchildren with either vaccine or placebo. In addition, more than a million additional children were vaccinated and served as observed controls. This study hypothesized that the vaccine would reduce the incidence of polio in schoolchildren and the placebo-controlled trials were necessary to demonstrate the effectiveness of the Salk vaccine. The results showed that the vaccine was 80-90% effective in reducing polio incidence. The intervention successfully affected variation in polio incidence directly.
“A calculated risk”: the Salk polio vaccine field trials of 1954 by Marcia Meldrum

Key Points

• Good experimental design generates information-rich data with a clear message.