Experimentation is a key issue in science and engineering. But it is one of software engineering’s stumbling blocks. Quite a lot of experiments are run nowadays, but it is a risky business. Software engineering has some special features, leading to some experimentation issues being conceived of differently than in other disciplines. The aim of this technical briefing is to help participants to avoid common pitfalls when analyzing the results of software engineering experiments. The technical briefing is not intended as a data analysis course, because there is already plenty of literature on this subject. It reviews several key issues that we have identified in published software engineering experiments, and addresses them based on the knowledge acquired after 19 years running experiments. The technical briefing starts by recalling what a SE experiment is, and its distinctive features: control and causality. Then, five topics related to data analysis will be explored. The technical briefing will end with participants’ questions.

The five topics covered are:

One-tailed vs. two-tailed tests. SE experimenters very often opt for a one-tailed hypothesis, but this can be a shortcoming in many experiments.
Choosing the right data analysis technique. The selected data analysis technique should match the experimental design. However, the choice is not straightforward, and several issues have to be taken into consideration.
Analyzing tricky designs. We will discuss two designs which are commonly not properly analyzed: blocked and crossover designs. -Parametric vs. non-parametric tests. We will discuss the different options that can be used when data do not meet the parametric tests assumptions.
The 3 musketeers: statistical significance, effect size and power. We will discuss the meaning and implications of the three parameters, how they relate to each other, and how they should be used to properly interpret the results of an experiment.