Datasets

Author

Charles Truong

Descriptions of the datasets used in class.

US Billboard Hot 100 lyrics, 1973–2023

Collected by Foramitti et al. (2025) to study long-term trends in stress, negativity, and simplicity in popular music lyrics and how societal crises affect those.

Observations. 266,087 entries, one per song per chart week, from January 6, 1973 to December 30, 2023 (51 years)

Variables.

Name Description
Date Chart week (weekly, 1973-01-06 to 2023-12-30)
Rank Position on that week’s Billboard Hot 100 chart (1–100)
Song Title Song title
Artist Performing artist
Lyrics_Sentiment VADER compound sentiment score of the lyrics, from −1 (most negative) to +1 (most positive)
Stress Frequency of stress-related words in the lyrics, from LIWC

Lyrics_Sentiment and Stress are missing for about 13,600 rows: these correspond to songs excluded from the analysis for being instrumental, having incorrect scraping returns, or having non-English lyrics.

How it was collected. The Billboard Hot 100 chart was scraped weekly with BeautifulSoup; lyrics for each charted song were retrieved via the Lyrics.ovh and Genius APIs, cleaned of metadata/inserts (e.g. “[Chorus]”), and filtered down to English-language lyrics only. Sentiment was scored with the VADER sentiment analysis tool, and stress-related language with the LIWC tool; each song’s score is the average across all its lyrics’ words.

Links.

Loading the data in pandas. Download lyrics_OSF.csv from the OSF repository above, then:

import pandas as pd

df = pd.read_csv("lyrics_OSF.csv", index_col=0, parse_dates=["Date"])
df.head()

Spotify songs

Audio features and metadata for ~32,800 Spotify tracks across 6 playlist genres, compiled by researcher Kaylin Pavlik using Spotify’s API (via the R spotifyr package) to explore genre classification patterns, and distributed through the TidyTuesday project.

Observations. 32,833 rows, one per song per playlist it appears in (28,356 unique tracks; a song can appear in multiple playlists, hence multiple rows). Album release dates range from 1957 to early 2020.

Variables.

Name Description
track_id, track_name, track_artist Track identifier, title, and performing artist
track_popularity Popularity score (0–100)
track_album_id, track_album_name, track_album_release_date Album identifier, title, and release date
playlist_name, playlist_id Playlist title and identifier
playlist_genre, playlist_subgenre Primary genre (edm, latin, pop, r&b, rap, rock) and subgenre
danceability, energy, key, loudness, mode, speechiness, acousticness, instrumentalness, liveness, valence, tempo Spotify audio features, mostly on a 0–1 scale (see Spotify’s audio features docs)
duration_ms Track length in milliseconds

Links.

Loading the data in pandas. Load it directly from TidyTuesday’s raw CSV, no download needed:

import pandas as pd

url = "https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2020/2020-01-21/spotify_songs.csv"
df = pd.read_csv(url)
df.head()

Pokémon

Information on 949 Pokémon (English and Brazilian Portuguese), compiled from the {pokemon} R package by William Amorim and distributed through the TidyTuesday project (2025-04-01 dataset, curated by Frank Hull).

Observations. 949 rows, one per Pokémon.

Variables.

Name Description
id, pokemon, species_id Unique ID, name, and species ID
height, weight Height and weight
base_experience Base experience
type_1, type_2 Primary and secondary type
hp, attack, defense, special_attack, special_defense, speed Battle stats
color_1, color_2, color_f Primary, secondary, and final color
egg_group_1, egg_group_2 Primary and secondary egg group
url_icon, url_image Icon and image URLs (url_icon is missing its leading “https:”)
generation_id Generation ID

Links.

Loading the data in pandas. Load it directly from TidyTuesday’s raw CSV:

import pandas as pd

url = "https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2025/2025-04-01/pokemon_df.csv"
df = pd.read_csv(url)
df.head()

Pixar films

Release, runtime, and critical-reception data for Pixar’s feature films, from the {pixarfilms} R package by Eric Leung, distributed through the TidyTuesday project.

Observations. 27 rows, one per Pixar film (release ratings are missing for the most recent films, not yet scored by all outlets). The upstream data is missing the title for film #27 (Elemental, released 2023-06-16); the bundled copy in the dashboard fills it in.

Variables.

Name Description
number Order of release
film Film title
release_date Premiere date
run_time Film length in minutes
film_rating MPA film rating
rotten_tomatoes Rotten Tomatoes score (0–100)
metacritic Metacritic score (0–100)
cinema_score CinemaScore letter grade (A–F)
critics_choice Critics’ Choice Movie Awards score (0–100)

Links.

Loading the data in pandas.

import pandas as pd

films = pd.read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2025/2025-03-11/pixar_films.csv")
response = pd.read_csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/main/data/2025/2025-03-11/public_response.csv")
df = films.merge(response, on="film", how="left")
df.head()

Titanic passengers and crew

This is a well-known dataset about the passengers and crew aboard the Titanic’s 1912 maiden voyage. Refer to Dawson (1995) for information on how this dataset was collected.

Observations. 2,201 rows, one per person aboard.

Variables.

Name Description
Class Ticket class (1st, 2nd, 3rd) or Crew
Sex Male or Female
Age Child or Adult
Survived No or Yes

Loading the data in pandas.

import pandas as pd

url = "https://vincentarelbundock.github.io/Rdatasets/csv/datasets/Titanic.csv"
counts = pd.read_csv(url, index_col=0)
df = counts.loc[counts.index.repeat(counts["Freq"])].drop(columns="Freq").reset_index(drop=True)
df.head()

Old Faithful geyser

Continuous measurements of eruptions from the Old Faithful geyser in Yellowstone National Park, Wyoming, from August 1 to August 15, 1985, collected by Azzalini and Bowman (1990) and distributed as the geyser dataset in the R MASS package (Ripley and Venables 2009).

Observations. 299 rows, one per eruption. Some nocturnal durations were only recorded as “short”, “medium”, or “long” and are coded here as 2, 3, or 4 minutes.

Variables.

Name Description
duration Eruption time, in minutes
waiting Waiting time before this eruption, in minutes

Loading the data in pandas.

import pandas as pd

url = "https://vincentarelbundock.github.io/Rdatasets/csv/MASS/geyser.csv"
df = pd.read_csv(url, index_col=0)
df.head()

Breakfast cereals

Nutritional and shelf-placement information for 77 US breakfast cereals, based on the F&DA food label, collected for the American Statistical Association’s 1993 Statistical Graphics Exposition (“Serial Correlation or Cereal Correlation??”).

Observations. 77 rows, one per cereal.

Variables.

Name Description
name Cereal name
mfr Manufacturer: A = American Home Food Products, G = General Mills, K = Kelloggs, N = Nabisco, P = Post, Q = Quaker Oats, R = Ralston Purina
type cold or hot
calories Calories per serving
protein Protein (g)
fat Fat (g)
sodium Sodium (mg)
fiber Dietary fiber (g)
carbo Complex carbohydrates (g)
sugars Sugars (g)
shelf Display shelf (1, 2, or 3, counting from the floor)
potass Potassium (mg)
vitamins Vitamins & minerals (0 = none added, 25 = enriched, 100 = 100% FDA recommended)
weight Weight of one serving (oz)
cups Cups per serving

None of the cereals had any cholesterol. Missing values (coded -1 in the original data) are encoded as NaN.

Links.

Loading the data in pandas.

import pandas as pd
import numpy as np

cols = ["name", "mfr", "type", "calories", "protein", "fat", "sodium", "fiber",
        "carbo", "sugars", "shelf", "potass", "vitamins", "weight", "cups"]
df = pd.read_csv(
    "https://raw.githubusercontent.com/friendly/candisc/master/data-raw/cereal/cereal.dat",
    sep=r"\s+", names=cols, na_values=-1,
)
df["name"] = df["name"].str.replace("_", " ")
df["type"] = df["type"].map({"C": "cold", "H": "hot"})
df.head()

Galton’s family heights

Heights of parents and their adult children, recorded by Francis Galton and analysed in Galton (1886) to demonstrate regression toward the mean. This is Galton’s original notebook data (205 families, 962 adult children — 486 sons and 476 daughters), recovered and transcribed from University College London’s archives; after excluding entries recorded non-numerically (e.g. “tall”, “medium”, “short”), 934 children (481 sons, 453 daughters) remain.

Observations. 934 rows, one per child.

Variables.

Name Description
family Family ID
father Father’s height (in)
mother Mother’s height (in)
midparentHeight Mid-parent height, (father + 1.08 * mother) / 21
children Number of children in this family
childNum This child’s number within the family (children are listed tallest to shortest, boys before girls)
gender male or female
childHeight Child’s height (in)

Links.

Loading the data in pandas.

import pandas as pd

url = "https://vincentarelbundock.github.io/Rdatasets/csv/HistData/GaltonFamilies.csv"
df = pd.read_csv(url, index_col=0)
df.head()

Cavendish’s determinations of the density of the Earth

Henry Cavendish’s 1798 experiments to determine the mean density of the Earth (relative to water) were used to indirectly calculate the gravitational constant in Newton’s law of gravitation. Distributed as the Cavendish dataset in the R HistData package (Friendly 2021).

Observations. 29 rows, one per determination.

Variables.

Name Description
density Cavendish’s 29 measurements, as he originally recorded and analysed them
density2 Same as density, but with the third value corrected from 4.88 to 5.88 — Cavendish himself treated 4.88 as a recording error and analysed it as 5.88

The modern accepted value is 5.517.

Links.

  • Article: Cavendish, H. (1798), Experiments to Determine the Density of the Earth, Philosophical Transactions of the Royal Society of London, 88.
  • Analysis: Stephen M. Stigler (1977), Do robust estimators work with real data?, Annals of Statistics, 5.

Loading the data in pandas.

import pandas as pd

url = "https://vincentarelbundock.github.io/Rdatasets/csv/HistData/Cavendish.csv"
df = pd.read_csv(url, index_col=0)
df.head()

Shakespeare’s Complete Works

The full text of Shakespeare’s plays and sonnets, used to study word prediction through n-gram statistics and to introduce probability concepts.

Observations. 970,821 words extracted from the public-domain Project Gutenberg edition of Shakespeare’s Complete Works (file #100). Stage directions and front matter removed.

Variables.

Name Description
bigrams Two consecutive words (context, next_word) and their count across the corpus (368,664 unique bigrams)
trigrams Three consecutive words (context_1, context_2, next_word) and their count (794,095 unique trigrams)

These are derived files, not raw Shakespeare text. See the probability chapter for examples of how to use them to compute word-prediction probabilities: P(next word = w | context) = count(context → w) / count(context → anything).

Links.

  • Source (Project Gutenberg): https://www.gutenberg.org/ebooks/100
  • Derived bigrams and trigrams: stored in course-notes/assets/shakespeare/bigrams.csv and course-notes/assets/shakespeare/trigrams.csv

Loading the data in pandas.

import pandas as pd

bigrams = pd.read_csv("course-notes/assets/shakespeare/bigrams.csv")
trigrams = pd.read_csv("course-notes/assets/shakespeare/trigrams.csv")
bigrams.head()

References

Azzalini, A., and A. W. Bowman. 1990. “A Look at Some Data on the Old Faithful Geyser.” Applied Statistics 39 (3): 357. https://doi.org/10.2307/2347385.
Dawson, Robert J. MacG. 1995. “The ‘Unusual Episode’ Data Revisited.” Journal of Statistics Education 3 (3). https://doi.org/10.1080/10691898.1995.11910499.
Foramitti, Markus, Urs Markus Nater, Claus Lamm, and Maurício Martins. 2025. “Societal Crises Disrupt Long-Term Increases in Stress, Negativity, and Simplicity in US Billboard Song Lyrics from 1973 to 2023.” Scientific Reports 15 (1). https://doi.org/10.1038/s41598-025-28327-5.
Friendly, Michael. 2021. “HistData: Data Sets from the History of Statistics and Data Visualization.” In CRAN: Contributed Packages. The R Foundation. https://doi.org/10.32614/cran.package.histdata.
Galton, Francis. 1886. “Regression Towards Mediocrity in Hereditary Stature.” The Journal of the Anthropological Institute of Great Britain and Ireland 15: 246. https://doi.org/10.2307/2841583.
Ripley, Brian, and Bill Venables. 2009. “MASS: Support Functions and Datasets for Venables and Ripley’s MASS.” In CRAN: Contributed Packages. The R Foundation. https://doi.org/10.32614/cran.package.mass.

Footnotes

  1. The 1.08 factor is the author’s attempt to rescale mothers’ heights to compensate for the average sex difference.↩︎