7 Correlation

Objective: describe and quantify the relationship between two quantitative variables.

Motivating example: Galton’s family heights

Recall Galton’s family heights dataset (Galton 1886), which contains the heights of children and their parents. The big question is: is a child’s height related to their parents’ height? Galton introduced the concept of correlation while trying to answer this question (Stigler 1989).

7.1 Scatterplots

A scatterplot displays two quantitative variables on a 2D plane: each observation is a point positioned according to its values on each variable. Scatterplots are the starting point for exploring any relationship between two quantitative variables.

Which variable goes on x, and which on y?

Before plotting two variables, ask: which one should be \(x\), and which should be \(y\)?

The two variables play different roles. The response variable is the one we’re interested in explaining or predicting; the explanatory variable (or predictor) is the one we think might account for it. By convention, the explanatory variable goes on the x-axis and the response on the y-axis. Here, it’s more natural to ask whether midparentHeight explains childHeight than the other way around, so midparentHeight is the explanatory variable (x-axis) and childHeight is the response (y-axis).

7.1.1 What to look for in a scatterplot

direction: do you see an upward or downward trend?
form: are the observations arranged along a line, a curve, or something different?
strength: how spread out are the observations?
outliers: are there outliers, or separate clusters of points that stand apart?

In the plot above, the direction is positive: children with taller parents tend to be taller themselves. The form is quite round: for any given midparent height, child heights still vary widely. There’s no single point that stands sharply apart from the rest.

Note

In Galton’s Height example, some of the variability arises from pooling sons and daughters in the same plot, since sons tend to be taller than daughters regardless of their parents’ heights.

Within each colour, observations are closer to an upward line. Conditioning on Gender makes patterns appear more clearly.

7.1.2 Strength, in pictures

To see what “strength” looks like concretely, here are three simulated scatterplots with a known correlation (\(r = 0.95\), \(r = 0.5\), and \(r = 0.05\)):

As the correlation \(r\) moves away from 0, observations are closer to a straight line. A correlation \(r\) close to 0 means the points are scattered with no visible trend.

7.2 Correlation

A scatterplot is only a visualisation; we’d also like a single number to summarise correlation. That number is built from the \(z\)-score of each value, which measures how many standard deviations a value sits from the mean: \[ z_i = \frac{x_i - \bar{x}}{s}. \] A positive \(z\)-score means the value is above the mean; a negative one means it’s below the mean. The size of \(z_i\) says how far, in standard-deviation units (recall the standard deviation from the previous chapter). The correlation coefficient \(r\), also known as Pearson’s \(r\), is given by \[ r = \frac{\sum_{i=1}^n z_{x_i} z_{y_i}}{n - 1}. \]

For each case, \(z_{x_i}\) and \(z_{y_i}\) have the same sign whenever the point sits on the same side of the mean for both variables. When most points behave this way, the products \(z_{x_i} z_{y_i}\) are mostly positive and \(r\) comes out positive; when most points sit on opposite sides for the two variables, \(r\) comes out negative.

Properties of \(r\):

\(r\) is always between \(-1\) and \(1\). Values of exactly \(-1\) or \(1\) mean that every point lies on a straight line.
\(r\) has no units: because it’s built from \(z\)-scores, changing the units of either variable (inches to centimetres, dollars to euros) leaves \(r\) unchanged.
\(r\) treats the two variables symmetrically: the correlation of \(x\) with \(y\) equals the correlation of \(y\) with \(x\).
\(r\) only measures the strength of a linear relationship. A strong curved relationship can have a small \(r\).
\(r\) is sensitive to outliers: a single unusual point can inflate a weak correlation or deflate a strong one.

Warning

Before interpreting a correlation, check:

Variables are quantitative.
Observations are relatively close to a straight line. If the pattern is clearly curved, \(r\) isn’t a meaningful summary.
There are no outliers.

See below what happens when a condition fails.

On the left, \(x\) and \(y\) follow a near-perfect parabola; there is a strong and predictable relationship. Yet \(r\) is close to 0, because it only detects straight trends. On the right, x and y are unrelated for 59 of the 60 points (\(r\) near 0 among just those), but one distant point in the upper right drags \(r\) up to a moderately strong 0.65 all on its own.

Back to the Galton example.

r = 0.32

midparentHeight and childHeight are both quantitative, and the scatterplot is reasonably straight with no glaring outlier, so the conditions are satisfied well enough to compute a correlation. The correlation is \(r = 0.32\): a weak-to-moderate positive association.

female: r = 0.51
male: r = 0.48

Conditioning on Gender confirms what our conclusion from the scatterplots: \(r = 0.51\) for daughters and \(r = 0.48\) for sons. Correlation is higher within each group.

7.2.1 A single point can move r

Many real datasets have one or more outliers, and correlation is sensitive to them. Take the Pokémon dataset, and look at height and weight:

r = 0.66

\(r = 0.66\), moderately strong and positive: taller Pokémon tend to be heavier. But one point stands apart from the rest, in the lower right: a Pokémon that’s very short but extremely heavy. The solid red line is the least-squares fit with that point included; the dashed green line is the fit without it — visibly different, especially at the low end of the range. Removing the outlier changes the correlation:

r without cosmoem = 0.69

Dropping a single point moves \(r\) from 0.66 to 0.69. For a smaller dataset, the change in correlation could be even larger.

7.2.2 Correlation tables

With more than two quantitative variables, it’s common to compute the correlation between every pair and arrange them in a correlation table. Here’s one for four quantitative variables in the Pixar films dataset we saw previously:

	rotten_tomatoes	metacritic	critics_choice	run_time
rotten_tomatoes	1.00	0.80	0.85	-0.22
metacritic	0.80	1.00	0.86	-0.13
critics_choice	0.85	0.86	1.00	-0.13
run_time	-0.22	-0.13	-0.13	1.00

The diagonal is always 1.00 (a variable correlates perfectly with itself). The three rating variables are all strongly and positively correlated with each other (0.80 to 0.87): critics who like a Pixar film on one scale tend to like it on the others as well. run_time is only weakly and negatively correlated with the ratings (around \(-0.13\) to \(-0.22\)): longer films tend to receive slightly lower ratings. We’ll come back to that run_time/rotten_tomatoes correlation shortly.

A correlation table is a quick way to scan many relationships at once. Still, it hides everything a scatterplot would show: whether each relationship is actually linear, and whether an outlier is pulling around any of these correlations. Reading off a correlation table is never a substitute for looking at the plots.

How strong is strong?

No fixed cutoff makes a correlation “weak,” “moderate,” or “strong”: it depends on context. A correlation of 0.7 between two attempts to measure the same underlying skill might be disappointin. In constrast, the same 0.7 between a new economic indicator and stock prices might be an exciting find. Report the value of \(r\) itself, and let the scatterplot and the context support any qualitative conclusion you draw from it.

7.3 Beyond straight lines: Spearman’s \(\rho\) and Kendall’s \(\tau\)

The correlation coefficient \(r\) only measures the strength of a linear relationship, and it can be thrown off by a single outlier or by data that are really ranks or ratings (a 1–5 satisfaction scale, say) rather than measurements on a proper quantitative scale. Two alternative measures of association relax these requirements: they ask only whether the relationship is monotone, i.e. consistently increasing or consistently decreasing.

7.3.1 Spearman’s \(\rho\)

Spearman’s rho replaces each variable’s values with their ranks (1 for the smallest, 2 for the next smallest, and so on, up to \(n\) for the largest) and then computes the ordinary correlation coefficient \(r\) on those ranks: \[ \rho = r(\text{rank}(x), \text{rank}(y)). \] Because it’s a correlation, \(\rho\) is always between \(-1\) and \(1\), with the same sign convention as \(r\). Ranking the data has two effects: a non-linear but still monotone relationship in the original scatterplot becomes closer to a straight line once both variables are replaced by ranks (so \(\rho\) isn’t limited to linear relationships the way \(r\) is), and an extreme outlier can no longer dominate the calculation because however far out it sits, its rank is still just 1 or \(n\).

Below is an example of a monotone but non-linear relationship.

On the left, \(x\) and \(y\) follow an increasing trend, but the relationship is “curved” rather than linear. The correlation coefficient is only \(r = 0.80\). On the right, both variables have been replaced by their ranks: the same monotone trend now looks close to a straight line, and \(\rho = 0.94\) picks up almost all of it.

7.3.2 Kendall’s \(\tau\)

Kendall’s tau looks at every pair of observations and asks only whether the line joining them slopes upward, downward, or is flat. Tau is the difference between the number of upward-sloping pairs and downward-sloping pairs, divided by the total number of pairs (pairs tied on \(x\) don’t count): \[ \tau = \frac{(\text{\# positive-slope pairs}) - (\text{\# negative-slope pairs})}{\text{total \# of pairs}}. \] If every pair slopes upward, \(\tau = 1\); if every pair slopes downward, \(\tau = -1\); if positive- and negative-sloping pairs roughly balance out, \(\tau\) is close to 0. Kendall’s \(\tau\) has a nice interpretation: for two observations \((x_1, y_1)\) and \((x_2, y_2)\) with \(x_1 > x_2\), \(\tau\) is the probability of \(y_1 > y_2\) minus the probability of \(y_2 > y_1\). Like \(\rho\), \(\tau\) only cares about the direction of each pairwise slope, not how steep it is. As a result, it is more robust to outliers than \(r\).

Warning

\(\rho\) and \(\tau\) trade away some of what \(r\) offers:

Both are always between \(-1\) and \(1\), and interpreted the same way as \(r\): close to \(\pm 1\) means a strong, consistent trend; close to 0 means little to no trend.
Both work on numerical and ordinal variables.
They measure monotone relationships, which are more general than linear.
They are more robust than \(r\) to outliers.
\(\rho\) and \(\tau\) typically aren’t equal to each other, or to \(r\), on the same data: they’re answering related but different questions (is it linear? vs. is it monotone?).

Back to the Pokémon example. Recall height and weight from the outlier example above, where a single very short, very heavy Pokémon pulled \(r\) down to 0.66 from 0.69 (its value with that point removed):

rho = 0.84
tau = 0.69

\(\rho = 0.84\) and \(\tau = 0.69\). Unlike \(r\), they barely move after dropping the outlier (to \(\rho = 0.85\) and \(\tau = 0.70\)).

Which one should I use?

Use \(r\) when the relationship looks linear, and the data are clean. Also, for historical reasons, \(r\) is the most common measure. If you only want to detect a monotone relationship, use \(\rho\) or \(\tau\). For small sample sizes, \(\tau\) has better statistical properties and a simpler interpretation. See (Arndt et al. 1999) and references therein.

7.4 Correlation is not causation

A lurking variable is a variable outside the analysis that affects the apparent relationship between two others. Correlation makes lurking variables especially dangerous, because a strong \(r\) is tempting to read as proof that one variable causes the other.

Take that run_time vs rotten_tomatoes correlation from the table above, \(r = -0.22\): longer Pixar films tend to score a bit lower with critics. It’s tempting to conclude that the runtime itself drags a film’s score down, maybe long movies drag on and lose critics’ interest. But watch what happens when we bring in each film’s release year:

run_time vs year: r = 0.51
rotten_tomatoes vs year: r = -0.21

The dark points (early films: Toy Story, Toy Story 2, Finding Nemo, …) cluster in the short-runtime, high-score corner. The lighter points (recent films: Cars 2, Cars 3, Onward, …) are more spread toward longer runtimes and more mixed scores. Runtime has increased over the years (\(r = 0.51\) with year), while scores have drifted down slightly (\(r = -0.21\) with year). Pixar’s films were shorter, on average, during the run of the studio’s most acclaimed films, and have run longer, on average, since. Release year is the lurking variable: it affects both runtime and how well a film tends to score, which is enough to produce a negative correlation between the two, without runtime causing anything.

Warning

When discussing in statistical terms, don’t say “correlation” when you mean “association.” Correlation is the specific, numerical measure \(r\); association is the broader term for any relationship between two variables, linear or not.
Don’t compute a correlation between categorical variables. If a variable isn’t quantitative, \(r\) doesn’t apply to it.
A high \(r\) does not guarantee a straight relationship, and a low \(r\) does not guarantee there’s no relationship at all. A strong curve can produce a small \(r\). Always check the scatterplot.
Correlation never proves causation. A lurking variable can entirely explain a strong, real, reproducible correlation. Establishing causation takes more than an observed association. Interested readers can find out more about causality in statistics in (Pearl and MacKenzie 2017)