5 Summarising quantitative data

Objective: find ways to summarise the distribution of a quantitative variable with a small number of statistics.

Continuing the motivating example

Recall the Economist article on AI and learning from the previous chapter: one of its central questions is whether pupils’ performance changed on average after AI tools became available — and whether the typical pupil’s story is the same as the average pupil’s story. Answering that requires summarising a distribution with numbers, not just a picture. That’s the goal of this chapter.

5.1 Measures of center

Example: Galton’s family heights. We’ll use the childHeight column of Galton’s family heights dataset (934 adult children; see here for a brief description) as a running example throughout this section.

5.1.1 The mean

The mean is the sum of all the values of a variable, divided by the number of data values, \(n\): \[ \bar{x} = \frac{x_1 + x_2 + \dots + x_n}{n} = \frac{1}{n}\sum_{i=1}^n x_i. \]

Mean child height: 66.7 in

5.1.2 The median

The median is the value in the middle of the sorted data: half the values fall below it, half above.

Median child height: 66.5 in

For this roughly symmetric distribution, the mean and median nearly coincide.

5.1.3 Mean or median?

The mean and median don’t always agree. Which one should you report?

The median looks only at the middle of the sorted data, so it is barely affected by outliers or by a skewed tail. This makes it a good choice for skewed distributions — income or company size, for example.
The mean takes every value into account, which makes it sensitive to outliers and skew, but also makes it the right choice when a summary needs to reflect the whole dataset — e.g. an airline computing mean baggage weight to plan fuel, where every bag counts toward the total regardless of how “typical” it is.

Example: Cavendish’s density measurements. In 1798, Henry Cavendish ran 29 experiments to measure the mean density of the Earth relative to water (see here for a brief description). One measurement, 4.88, was almost certainly a recording error — Cavendish himself treated it as 5.88 in his own analysis. Column density keeps the original (uncorrected) value; density2 applies the correction.

column	mean	median
density (uncorrected)	5.4	5.5
density2 (corrected)	5.5	5.5
% change	0.6	0.2

Correcting the single outlier moves the mean noticeably more than it moves the median (in percentage terms, about 0.63% vs. 0.18%).

Warning

The fact that the median barely moved doesn’t mean you’re “off the hook” for outliers. Always investigate them: correct them if they’re errors, and set them aside if they genuinely don’t belong.

It’s good practice to look at both the mean and the median for every quantitative variable you summarise. If they’re close, either is a fine summary. If they differ noticeably, investigate outliers, and ask yourself whether skew is the more likely explanation.

Synthetic examples. The plots below simulate three common situations

Positively skewed (long right tail): the mean is pulled above the median, toward the tail.
Negatively skewed (long left tail): the mean is pulled below the median, toward the tail.
A few low outliers: the mean is pulled toward the outliers; the median barely moves.

Tip

Look at the mean and median for the weight variable of the pokemon data. In which situation are we?

5.2 Measures of spread

Two variables can share the same mean/median while looking completely different. The second piece of information we usually report is how much the data varies around the distribution centre, in other words, the spread.

5.2.1 The interquartile range

Just as we can ask what fraction of the data falls below a given value, we can ask for the value below which a given fraction of the data falls. The first quartile (\(Q_1\), or 25th percentile) is the value with a quarter of the data below it; the third quartile (\(Q_3\), or 75th percentile) has three-quarters of the data below it. (The median is technically the 50th percentile, or “second quartile”.)

The interquartile range (IQR) is the range of the middle half of the data: \[ \text{IQR} = Q_3 - Q_1. \]

A box plot is a compact picture of a distribution constructed entirely from percentiles, making it easy to compare several variables that share the same units side by side:

the box spans \(Q_1\) to \(Q_3\), so its height is exactly the IQR;
a line across the box marks the median;
whiskers extend from the box out to the most extreme value still within \(1.5 \times \text{IQR}\) of the box;
individual points beyond the whiskers flag the more unusual values, without letting them stretch the box or whiskers themselves.

Example: Galton’s family heights. father, mother, and childHeight are all measured in inches:

variable	whisker_low	q1	median	q3	whisker_high
father	64.0	68.0	69.0	71.0	75.5
mother	59.0	63.0	64.0	65.9	69.0
childHeight	56.0	64.0	66.5	69.7	78.0

The IQR (the height of the box) ignores the most extreme values on each side, so the handful of outlying points beyond the whiskers have little effect on it.

5.2.2 The standard deviation

Another measure, the standard deviation, takes every value’s distance from the mean into account.

The differences between each value and the mean, \(x_i - \bar{x}\), are called residuals. Squaring and averaging them (dividing by \(n-1\) rather than \(n\)¹) gives the variance: \[ s^2 = \frac{\sum_{i=1}^n (x_i - \bar{x})^2}{n - 1}. \]

Variance is in squared units (squared inches, squared euros, …), which is awkward to interpret. Taking the square root brings it back to the original units — this is the standard deviation: \[ s = \sqrt{\frac{\sum_{i=1}^n (x_i - \bar{x})^2}{n - 1}}. \]

Variance: 12.81 in^2, standard deviation: 3.58 in

5.2.3 Thinking about variation

Measures of spread quantify what we don’t know: values clustered tightly around the centre give a small IQR and standard deviation; values scattered far away give large ones. If every observation were identical, every measure of spread would be zero, and there’d be nothing left to do statistics about.

Warning

If a distribution has two or more modes, a single measure of spread number mostly reflects how far apart the modes are, not how variable the data are within each mode. Recall the bimodal Old Faithful waiting times from the previous chapter: the standard deviation of all waiting times combined says little about how consistent any single eruption cycle is. It’s usually more informative to summarise the centre and spread separately within each mode.

It’s good practice to report a measure of spread alongside any measure of centre — a centre number on its own doesn’t say how well it represents the data.

5.3 Choosing and reporting summaries

Warning

Don’t compute numerical summaries of a categorical variable. A mean ZIP code or a standard deviation of student ID numbers is meaningless, even though software will happily compute one if you ask. For a categorical variable, report the mode and (relative) frequencies instead (see the previous chapter).

Summarise the centre with the mean and/or the median, and always pair it with a measure of spread: mean with standard deviation, median with IQR.
If the mean and median are close, and the distribution is roughly symmetric with no outliers, the mean/standard deviation pair is usually the more informative summary.
If the distribution is skewed or has outliers, prefer the median/IQR pair — but still investigate any outliers rather than treating the median as an excuse not to.
Some fields have their own conventions regardless (economists, for instance, typically summarise income with the median). When in doubt, report both.

Dividing by \(n\) would make \(s^2\) systematically a little too small, on average, as an estimate of the variability in the population the data came from. Dividing by \(n - 1\) corrects for this.↩︎