Various data checks

Goals

Check for problems in the combined current archives

Setup

library(tidyverse)
library(janitor)
library(lubridate)

Hot 100

Import

hot100 <- read_csv("data-out/hot-100-current.csv")

Warning: One or more parsing issues, call `problems()` on your data frame for details,
e.g.:
  dat <- vroom(...)
  problems(dat)

Rows: 355600 Columns: 8
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (2): title, performer
dbl  (4): current_week, last_week, peak_pos, wks_on_chart
lgl  (1): wks_at_no1
date (1): chart_week

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

hot100 %>%  glimpse()

Rows: 355,600
Columns: 8
$ chart_week   <date> 2022-01-01, 2022-01-01, 2022-01-01, 2022-01-01, 2022-01-…
$ current_week <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
$ title        <chr> "All I Want For Christmas Is You", "Rockin' Around The Ch…
$ performer    <chr> "Mariah Carey", "Brenda Lee", "Bobby Helms", "Burl Ives",…
$ last_week    <dbl> 1, 2, 4, 5, 3, 7, 9, 11, 6, 13, 15, 17, 18, 0, 8, 25, 19,…
$ peak_pos     <dbl> 1, 2, 3, 4, 1, 5, 7, 6, 1, 10, 11, 8, 12, 14, 7, 16, 12, …
$ wks_on_chart <dbl> 50, 44, 41, 25, 11, 26, 24, 19, 24, 15, 31, 18, 14, 1, 49…
$ wks_at_no1   <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…

Check most recent chart

counts_hot100 <- hot100 %>% 
  group_by(chart_week) %>% 
  summarise(cnt = n()) %>% 
  arrange(chart_week %>% desc())

counts_hot100 %>% head()

Look for short counts

counts_hot100 %>% 
  filter(cnt != 100)

Billboard 200

Import

bb200 <- read_csv("data-out/billboard-200-current.csv")

Rows: 620591 Columns: 8
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr  (2): title, performer
dbl  (4): current_week, last_week, peak_pos, wks_on_chart
lgl  (1): wks_at_no1
date (1): chart_week

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

bb200 |> glimpse()

Rows: 620,591
Columns: 8
$ chart_week   <date> 2021-01-02, 2021-01-02, 2021-01-02, 2021-01-02, 2021-01-…
$ current_week <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17…
$ title        <chr> "Evermore", "McCartney III", "Music To Be Murdered By", "…
$ performer    <chr> "Taylor Swift", "Paul McCartney", "Eminem", "Michael Bubl…
$ last_week    <dbl> 1, 0, 199, 4, 10, 8, 6, 3, 13, 12, 7, 17, 2, 11, 23, 9, 1…
$ peak_pos     <dbl> 1, 2, 1, 1, 3, 6, 5, 1, 7, 10, 1, 12, 2, 1, 14, 1, 2, 18,…
$ wks_on_chart <dbl> 2, 1, 48, 90, 99, 56, 13, 22, 18, 88, 25, 15, 2, 8, 46, 4…
$ wks_at_no1   <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…

See most recent charts

bb200 |> 
  count(chart_week) |> 
  arrange(chart_week |> desc()) |> 
  head()

Check counts less than 200

bb200 |> 
  count(chart_week) |> 
  filter(n != 200)

The 175 weeks are how they appear online today. Could be that was all there were as that was the chart’s birth.
The 1967-09-16 data is an error in my collected data that needs to be fixed, but I haven’t decided how yet.

Assignment file

assign_hot100 <- read_csv("data-out/hot100_assignment.csv")

Rows: 355600 Columns: 7
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (3): CHART WEEK, TITLE, PERFORMER
dbl (4): THIS WEEK, LAST WEEK, PEAK POS., WKS ON CHART

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Check the date:

assign_hot100 |> 
  clean_names() |> 
  mutate(chart_week = mdy(chart_week)) |> 
  summary()

   chart_week           this_week           title            performer     
 Min.   :1958-08-04   Min.   :  1.0   Length   :355600   Length   :355600  
 1st Qu.:1975-08-21   1st Qu.: 26.0   N.unique : 27095   N.unique : 11308  
 Median :1992-09-01   Median : 51.0   N.blank  :     0   N.blank  :     0  
 Mean   :1992-09-01   Mean   : 50.5   Min.nchar:     1   Min.nchar:     1  
 3rd Qu.:2009-09-13   3rd Qu.: 75.0   Max.nchar:    75   Max.nchar:   113  
 Max.   :2026-09-26   Max.   :100.0                                        
                                                                           
   last_week        peak_pos       wks_on_chart    
 Min.   :  0.0   Min.   :  1.00   Min.   :  1.000  
 1st Qu.: 22.0   1st Qu.: 13.00   1st Qu.:  4.000  
 Median : 46.0   Median : 37.00   Median :  7.000  
 Mean   : 46.9   Mean   : 40.38   Mean   :  9.468  
 3rd Qu.: 71.0   3rd Qu.: 65.00   3rd Qu.: 13.000  
 Max.   :100.0   Max.   :100.00   Max.   :112.000  
 NAs    :32460