---
title: First Tutorial for R4DS
author: Luke Li and David Kane
tutorial:
  id: r4ds-1
output: 
  learnr::tutorial:
    df_print: default
    progressive: true
    allow_skip: true
runtime: shiny_prerendered
description: 'This tutorial for "R for Data Science" covers Chapter 1: Data visualization, Chapter 3: Data transformation, Chapter 5: Data tidying, and Chapter 7: Data import.'
---

```{r setup, include = FALSE}
library(learnr)
library(tutorial.helpers)
library(tidyverse)
library(quarto) # Total hack to shut up renv! Really need to put this in the package help file.

knitr::opts_chunk$set(echo = FALSE)
options(tutorial.exercise.timelimit = 60, 
        tutorial.storage = "local") 

line_graph <- billboard |>
  filter(track %in% c("Dancing Queen", "Loser", "Try Again")) |>
  pivot_longer(
    cols = wk1:wk5,
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  ggplot(aes(x = week, y = rank, color = track, group = track)) +
  geom_line() + 
  facet_wrap(~ track)

billboard_clean <- billboard |>
  filter(wk1 < 101) |>
  arrange(wk1) |>
  mutate(improvement = wk1 - wk2, .after = wk2) |>
  select(artist:improvement) |>
  rename(week1 = wk1, week2 = wk2)

music_1 <- read_csv(
   "data/music.csv",
   col_types = cols_only(
     artist.familiarity = col_character(),
 artist.hotttnesss = col_character(),
 artist.id = col_character()
   )
)

df <- read_csv(
   "data/music.csv",
   col_types = cols(
     artist.familiarity = col_double()
   )
)

billboard_longer <- billboard |>
  pivot_longer(
    cols = starts_with("wk"),
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  mutate(week = parse_number(week))
```

```{r info-section, child = system.file("child_documents/info_section.Rmd", package = "tutorial.helpers")}
```

## Introduction
###

This section covers key concepts from [Chapter 1: Data visualization](https://r4ds.hadley.nz/data-visualize.html), [Chapter 3: Data transformation](https://r4ds.hadley.nz/data-transform.html), [Chapter 5: Data tidying](https://r4ds.hadley.nz/data-tidy.html), and [Chapter 7: Data import](https://r4ds.hadley.nz/data-import.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund.

You will learn about importing, organizing, and visualizing data using core packages such as **[readr](https://readr.tidyverse.org/)**, **[dplyr](https://dplyr.tidyverse.org/)**, and **[ggplot2](https://ggplot2.tidyverse.org/)**. Important functions include: [`read_csv()`](https://readr.tidyverse.org/reference/read_delim.html), [`filter()`](https://dplyr.tidyverse.org/reference/filter.html), [`ggplot()`](https://ggplot2.tidyverse.org/reference/ggplot.html), and [`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html).

### Exercise 1

Create a Github repo called `r4ds-1`. Set the "Add README" toggle switch to "On."

Connect the repo to a project on your computer using `File -> New Folder from Git ...`.  Make sure to select the "Open in a new window" box. Start an R interpreter, if necessary.

You need two Positron windows: this one for running the tutorial and the one you just created for writing your code and interacting with the Console.

In the new window, select `File -> New File -> Quarto Document ...`. Provide a title -- `"Analyzing the Billboard 100"` -- and an author (you). Render the document and save it as `analysis.qmd`.

Create a `.gitignore` file with `analysis_files` on the first line and then a blank line. Save and push.

In the Console, run:

```         
show_file(".gitignore")
```

If that fails, it is probably because you have not yet loaded `library(tutorial.helpers)` in the Console.

CP/CR.

```{r introduction-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

### 

Professionals keep their data science work in the cloud because laptops fail.

### Exercise 2

In your QMD, put `library(tidyverse)` in a new code chunk. Render the file using `Cmd/Ctrl + Shift + K`.

Notice that the file does not look good because the code is visible and there are annoying messages. To take care of this, add `#| message: false` to remove all the messages in this `setup` chunk. Also, add the following to the YAML header to remove all code echoes from the HTML:

```         
execute: 
  echo: false
```

Using `Cmd/Ctrl + Shift + K`, render the file again. Only the title and author should appear in the HTML.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r introduction-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 6)
```

### 

You may get a warning like:

```
Warning message:
In readLines(path) : incomplete final line found on 'analysis.qmd'
```

This means that `analysis.qmd` lacks an empty last line. Add one. Text files should usually have a blank line at the end.

### Exercise 3

Place your cursor in the QMD file on the `library(tidyverse)` line. Use `Cmd/Ctrl + Enter` to execute that line.

Note how this command causes `library(tidyverse)` to be copied down to the Console and then executed. 

CP/CR.

```{r introduction-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 8)
```

###

You should practice moving from QMD World to Console World and back.

Note that **ggplot2** is loaded with **tidyverse**.

R has several systems for making graphs, but [**ggplot2**](https://ggplot2.tidyverse.org/index.html) is the best. **ggplot2** implements the *grammar of graphics*, a coherent system for describing and building graphs. 

### Exercise 4

From the Console, run these three commands:

`getwd()`  
`dir.create("data")`  
`list.files()`  

These commands organize your project directories: `getwd()` shows your working directory (WD), which is your current location. `dir.create("data")` creates a data folder in your WD and `list.files()` displays files in WD when parentheses are empty.

CP/CR.

```{r introduction-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

Your answer should look something like this, although your path will be different.

````
> getwd()
[1] "/Users/dkane/Desktop/projects/r4ds-1"
> dir.create("data")
> list.files()
 [1] "analysis_files" "analysis.html"  "analysis.qmd"   "data"           "README.md"  
>
````


## Billboard 100
###

The [`billboard`](https://tidyr.tidyverse.org/reference/billboard.html) tibble is a dataset in the [**tidyr**](https://tidyr.tidyverse.org/index.html) package. `billboard` contains data from 317 songs and their weekly rankings in the Billboard top 100 for 76 weeks after they were released.

### Exercise 1

Loading the **tidyverse** package automatically loaded the packages which are part of the "core" *Tidyverse*, including **tidyr**, **ggplot2**, and **dplyr**.

Type `billboard` into the Console.

CP/CR.

```{r billboard-100-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

You should get a tibble that looks like the below. Note that the code that appears after your submit a question is our answer to the question. It is OK if it differs from your answer.

```{r billboard-100-1-test, echo = TRUE}
billboard
```

###

A "variable" represents a quantity, quality, or property that can be measured or observed. Variables can be numeric or categorical, continuous or discrete, qualitative or quantitative.

**Tabular data** is data organized in a table. A table is a group of cells, organized in rows and columns. 

### Exercise 2

Choose an AI to use and in the chat, provide a tibble of `billboard`. You only need the top 3 lines or so.

Ask AI: Create a simple graph plotting `wk1` against `wk2` while only using the functions `ggplot()`, `aes()`, and `geom_point()`. 

Copy and paste the AI generated code into a new code chunk with your QMD. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-2-test, echo = TRUE}
ggplot(data = billboard, 
	   mapping = aes(x = wk1, y = wk2)) +
  geom_point()
```

###

The warning about 'removed rows containing missing values' means `ggplot()` automatically excluded NA entries. This is normal behavior when working with real-world data that has missing values.

### Exercise 3

Ask AI: Continue the code with `+` and add a trendline with `geom_smooth()`. Add a `title`, `subtitle`, `caption`, and axis labels using the `labs()` function.

Copy and paste this code to continue the previous code and save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

```{r billboard-100-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-3-test, echo = TRUE}
ggplot(billboard, aes(x = wk1, y = wk2)) +
  geom_point() +
  geom_smooth() +
  labs(
    title = "Billboard Rankings: Week 1 vs. Week 2",
    subtitle = "Shows how a song’s debut ranking relates to its second week performance",
    caption = "Source: Billboard dataset (2000)",
    x = "Week 1 Rank",
    y = "Week 2 Rank"
  )
```

###

`geom_smooth()` adds a trend line to your scatterplot to show the general pattern in your data. The gray shaded area around the line shows how uncertain the trend is - a wider area means more uncertainty. Use `method = "lm"` for linear regression (straight line) or LOESS for curved trends.

### Exercise 4

In a new prompt for AI provide a tibble of `billboard` again.

Ask AI: Create a simple graph plotting a line graph showing the rank of 3 songs over the first 5 weeks, mainly using the functions `ggplot()`, `aes()`, and `geom_line()`. 

Copy and paste the AI generated code into a new code chunk with your QMD. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-4-test, echo = TRUE}
billboard |>
  filter(track %in% c("Dancing Queen", "Loser", "Try Again")) |>
  pivot_longer(
    cols = wk1:wk5,
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  ggplot(aes(x = week, y = rank, color = track, group = track)) +
  geom_line()
```

###

The code that the AI produced like contained the symbol `%>%` or `|>`, which are pipe operators. **Replace** `%>%` with `|>` using `Cmd/Ctrl + H` because `|>` is a base R function, which helps ensure better performance. The pipe `|>` passes the left side as input to the right side function. Remember: use `|>` between data steps and `+` between ggplot layers.


### Exercise 5

Ask AI: Continue the code to split the graph into 3 graphs, 1 per track, using `facet_wrap()`. 

Copy and paste this code to continue the previous code and save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-5-test, echo = TRUE}
billboard |>
  filter(track %in% c("Dancing Queen", "Loser", "Try Again")) |>
  pivot_longer(
    cols = wk1:wk5,
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  ggplot(aes(x = week, y = rank, color = track, group = track)) +
  geom_line() + 
  facet_wrap(~ track)
```

###

For categorical variables, splitting your plot into **facets** creates subplots that each display one subset of the data. The first argument of `facet_wrap()` is a formula, which you create with `~` followed by a variable name. The variable that you pass to `facet_wrap()` should be categorical.

### Exercise 6

Provide the AI with your current code.

Ask AI: Continue the code by assigning the plot to the variable `line_graph`, and then save `line_graph` with `ggsave()`.

Update your code in `analysis.qmd`. Place your cursor on the first line of this code and run `Cmd/Ctrl + Enter` in order to run the code in the Console. You may need to do this twice so that both the plot creation code and `ggsave()` are executed.

CP/CR.

```{r billboard-100-6}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

Your code should look something like this:

```
line_graph <- billboard |>
  filter(track %in% c("Dancing Queen", "Loser", "Try Again")) |>
  pivot_longer(
    cols = wk1:wk5,
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  ggplot(aes(x = week, y = rank, color = track, group = track)) +
  geom_line() + 
  facet_wrap(~ track)
```

###

This creates the plot object `line_graph` which you can view by typing its name in the Console. You can use `ggsave(filename = "billboard_tracks.png", plot = line_graph)` to save the image as a PNG file.

### Exercise 7

In a new prompt for AI provide a tibble of `billboard` again.

Ask AI: Create a simple graph with boxplots showing the distribution of ranks of 3 songs, mainly using the functions `ggplot()`, `aes()`, and `geom_boxplot()`. 

Copy and paste the AI generated code into a new code chunk with your QMD. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-7}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-7-test, echo = TRUE}
billboard |>
  filter(track %in% c("Kryptonite", "Loser", "Try Again")) |>
  pivot_longer(
    cols = wk1:wk76,
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  ggplot(aes(x = track, y = rank)) +
  geom_boxplot()
```

###

To visualize the relationship between a numerical and a categorical variable, we can use side-by-side box plots. A boxplot shows the distribution's key statistics: the median (middle line), quartiles (box edges), and potential outliers (individual points). The box represents the middle 50% of data, whiskers show the data range, and points beyond indicate unusual values.

### Exercise 8

In a new prompt for AI provide a tibble of `billboard` again.

Ask AI: Create a simple bar graph with bars showing the count of weeks 3 songs were ranked in the top 100, mainly using the functions `ggplot()`, `aes()`, and `geom_bar()`. 

Copy and paste the AI generated code into a new code chunk with your QMD. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-8}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r billboard-100-8-test, echo = TRUE}
billboard |>
  filter(track %in% c("Kryptonite", "Loser", "Try Again")) |>
  pivot_longer(
    cols = wk1:wk76,
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  filter(rank <= 100) |>
  ggplot(aes(x = track)) +
  geom_bar()
```

###

`geom_bar()` makes the height of the bar proportional to the number of cases in each group. If you want the heights of the bars to represent values in the data, use `geom_col()` instead. `geom_bar()` uses `stat_count()` by default: it counts the number of cases at each x position. `geom_col()` uses `stat_identity()`: it leaves the data as is.

### Exercise 9

Now, we'll move onto functions from the **dplyr** package.

In a new prompt for AI provide a tibble of `billboard` again.

Ask AI: Create a pipe with `billboard` to filter for the rows where the values in `wk1` are less than 101. Continue the pipe with `billboard` to arrange the rows by `wk1` rank values.

Copy and paste the AI generated code into a new code chunk with your QMD. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-9}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-9-test, echo = TRUE}
billboard |>
  filter(wk1 < 101) |>
  arrange(wk1)
```

###

`filter()` selects rows based on conditions without changing their order. Use boolean operators like `&` (and), `|` (or), and `%in%` to combine multiple conditions. This function returns a new tibble with only the rows that match your criteria. `arrange()` changes the order of the rows based on the value of the columns. 

### Exercise 10

Ask AI: Continue the pipe with `billboard` to create a new column titled `improvement` that contains the difference between `wk1` and `wk2` rankings, and place it after the `wk2` column.

If you're using a different AI, provide the `billboard` tibble again and the previous code.

Copy and paste the AI generated code in the place to continue the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-10}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-10-test, echo = TRUE}
billboard |>
  filter(wk1 < 101) |>
  arrange(wk1) |>
  mutate(improvement = wk1 - wk2, .after = wk2)
```

###

`mutate()` creates new columns derived from existing columns. The `.after` argument positions new columns after specified variables.

### Exercise 11

Ask AI: Continue the pipe with `billboard` to select the columns `artist` through `improvement`.

If you're using a different AI, provide the `billboard` tibble again and the previous code.

Copy and paste the AI generated code in the place to continue the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-11}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r billboard-100-11-test, echo = TRUE}
billboard |>
  filter(wk1 < 101) |>
  arrange(wk1) |>
  mutate(improvement = wk1 - wk2, .after = wk2) |>
  select(artist:improvement)
```

###

`select()` chooses which columns to keep in your data frame. It uses special operators like `&`, `!`, and `|` to select columns by name, position, or pattern. This function helps you focus on the variables relevant to your analysis.

### Exercise 12

Ask AI: Continue the pipe with `billboard` to rename the columns `wk1` and `wk2` to be `week1` and `week2` respectively.

If you're using a different AI, provide the `billboard` tibble again and the previous code.

Copy and paste the AI generated code in the place to continue the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r billboard-100-12}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-12-test, echo = TRUE}
billboard |>
  filter(wk1 < 101) |>
  arrange(wk1) |>
  mutate(improvement = wk1 - wk2, .after = wk2) |>
  select(artist:improvement) |>
  rename(week1 = wk1, week2 = wk2)
```

###

`rename()` changes the names of the columns with no impact on the other variables.

You can also rename variables as you `select()` them by using `=`. The new name appears on the left hand side of the `=`, and the old variable appears on the right hand side. 

### Exercise 13

Before creating a plot, we need to ensure that your data matches our data. In the QMD, replace your code from the previous exercise with our code.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r billboard-100-13}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 6)
```

If `show_file()` gives an error, add `library(tutorial.helpers)` to your first code chunk where `library(tidyverse)` is located.

###

`summarize()` calculates group statistics and returns one row per group. Use `na.rm = TRUE` to ignore missing values in calculations. The `.by` argument groups data directly within `summarize()`, making `group_by()` unnecessary for most cases. `slice_*()` allows you to extract specific rows.

### Exercise 14

Within the latest code chunk, add the option: `#| cache: true`. Assign the result of the pipe to `billboard_clean`. 

`Cmd/Ctrl + Shift + K`. By including `#| cache: true` you cause Quarto to cache the results of the chunk. The next time you render your QMD, as long as you have not changed the code, Quarto will just load up the saved object.

If you have not done so already, you should add `analysis_cache` to the `.gitginore`. The content of the cache file does not belong on GitHub.

Place your cursor on the line where the pipe is assigned to `billboard_clean`, run `Cmd/Ctrl + Enter`, thus ensuring that the workspace also includes a copy of `billboard_clean`.

CP/CR.

```{r billboard-100-14}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 8)
```

###



```{r, echo = TRUE}
billboard_clean <- billboard |>
  filter(wk1 < 101) |>
  arrange(wk1) |>
  mutate(improvement = wk1 - wk2, .after = wk2) |>
  select(artist:improvement) |>
  rename(week1 = wk1, week2 = wk2)
```

###

The pipe takes the thing on its left and passes it along to the function on its right so that `billboard |> filter(wk1 < 101)` is equivalent to `filter(billboard, wk1 < 101)`, and `billboard |> filter(wk1 < 101) |> arrange(wk1)` is equivalent to `arrange(filter(billboard, wk1 < 101), wk1)`. The easiest way to pronounce the pipe is “then”.

### Exercise 15

Within the Console, type `billboard_clean`, which we previously assigned to a pipe and ran in the Console. Hit `Enter`.

CP/CR.

```{r billboard-100-15}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 8)
```

###



```{r, echo=TRUE}
billboard_clean
```

###

`billboard` is a tibble, a special type of data frame used by the tidyverse to avoid some common gotchas, which are unexpected behaviors that lead to confusion. The most important difference between tibbles and data frames is the way tibbles print; they are designed for large datasets, so they only show the first few rows and only the columns that fit on one screen.

`billboard_clean` is also a tibble since **tidyverse** functions return tibbles by default.

### Exercise 16

Ask AI: Generate R code that uses `billboard_clean` to plot a basic graph showing a histogram for `improvement`. Mention you want to use the data from `billboard_clean` and copy/paste the `billboard_clean` you ran in the Console with the resulting tibble. You only need the top 3 lines, mainly to include column names.

Within `labs()`, edit or add a proper title, subtitle, and caption. If axis labels would be useful, add them, but if unnecessary, don't bother. Don't assign the code for the plot to any variable. Put the plot code in a new code chunk. Run `Cmd/Ctrl + Shift + K` to ensure that everything works. 

Make your plot look nice. Edit the appearance by hand or get AI to help. The titles should be legible and everything should have proper capitalization.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r billboard-100-16}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 12)
```

###

```{r, echo=TRUE}
ggplot(billboard_clean, aes(x = improvement)) +
  geom_histogram(binwidth = 5, fill = "skyblue", color = "black") + 
  labs(
	title = "Distribution of Billboard Top 100 Song Ranking Improvements From Week 1 to Week 2",
	subtitle = "Most songs either had no improvement or improved their ranking by up to 20 places",
	x = "Song Rank Improvement",
	y = "Count",
	caption = "Source: Song rankings for Billboard top 100 in the year 2000"
  )
```

###

A histogram divides the x-axis into equally spaced bins and then uses the height of a bar to display the number of observations that fall in each bin. You can set the width of the intervals in a histogram with the `binwidth` argument, which is measured in the units of the x variable. 

### Exercise 17

Ask AI: Continue the code for the plot with `theme()` in order to make the plot look nice. Make the `title` and `subtitle` fit on the graph. If necessary, manually edit the sizes. Place this code as a continuation of the graph code, in the end the chunk should just have the final graph. Run `Cmd/Ctrl + Shift + K` to ensure that everything works. 

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r billboard-100-17}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r billboard-100-17-test, echo = TRUE}
ggplot(billboard_clean, aes(x = improvement)) +
  geom_histogram(binwidth = 5, fill = "skyblue", color = "black") + 
  labs(
	title = "Distribution of Billboard Top 100 Song Ranking Improvements From Week 1 to Week 2",
	subtitle = "Most songs either had no improvement or improved their ranking by up to 20 places",
	x = "Song Rank Improvement",
	y = "Count",
	caption = "Source: Song rankings for Billboard top 100 in the year 2000"
  ) +
  theme(
    plot.title = element_text(size = 7.5, face = "bold"),
    plot.subtitle = element_text(size = 8)
  )
```

###

Many `theme_*()` functions exist within **ggplot2** that create different appearance themes for plots while not affecting the data display. Click this [link](https://ggplot2.tidyverse.org/reference/ggtheme.html) to find out the different themes.

## Music
###

The **music** CSV file comes from the [**Million Song Dataset**](https://corgis-edu.github.io/corgis/csv/music/), which used a company called the Echo Nest to derive data points about one million popular contemporary songs. **Music** only contains 10,000 rows of data of the songs from the **Million Song Dataset**.

Key functions you'll learn:

[`read_csv()`](https://readr.tidyverse.org/reference/read_delim.html) - Reads and loads CSV files

[`write_csv()`](https://readr.tidyverse.org/reference/write_delim.html) - Writes a CSV file

[`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html) - Transforms data from a "wide" to a "long" format

[`pivot_wider()`](https://tidyr.tidyverse.org/reference/pivot_wider.html) - Transforms data from a "long" to a "wide" format


### Exercise 1

We begin by downloading our CSV file `music.csv` to our the `data` directory directly from the internet using `download.file()`. 

In the Console, run:

```         
download.file(
  "https://corgis-edu.github.io/corgis/datasets/csv/music/music.csv",
  destfile = "data/music.csv",
  mode = "wb"
)
```

CP/CR.

```{r music-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

"CSV" stands for **C**omma-**S**eparated **V**alues, meaning that the variable names and data values are separated by commas in the file. CSV files are just one type of text file. A text file is any file which includes plain text. The contents of such files are easy to look at in any text editor, or in RStudio.

### Exercise 2

Ask AI: Read a `.csv` dataset called `data/music.csv` using `read_csv()`. 

Add this code to a new code chunk in your QMD. Make sure the result of the function is not assigned to a variable. 

Place your cursor at the beginning of the line and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

###


```{r music-2-hint-1, echo = TRUE, eval = FALSE}
read_csv("data/music.csv")
```

###

When you run `read_csv()`, it prints information about the rows, columns, and data types you used. The **readr** package includes similar functions for other file formats like `read_tsv()` for tabs and `read_delim()` for custom separators. These functions help read different types of structured data files.

### Exercise 3

Provide AI with the line of code from the question above, as well as the tibble of data.

Ask AI: Edit the code so that the `read_csv()` skips the first 5 lines. Continue the code so that the `read_csv()` will also not read the first row as headings.

Copy and paste the AI generated code in the place to edit the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-3-test, echo = TRUE}
read_csv("data/music.csv", skip = 5, col_names = FALSE)
```

###

The argument `skip` is used to skip rows, but to skip columns, you can use the `col_only()` function as the argument to `col_types` in order to read in only the columns which you want. The argument `col_names` can also be used to create custom column names. 

### Exercise 4

Ask AI: Edit the code so that the `read_csv()` is assigned to an object `df` and `read_csv()` is told with `col_types` that the column type of `artist.familiarity` is a double and to check if it fails. Then run `problems(df)`.

If you're using a different AI, provide the `music.csv` tibble again and the previous code.

Copy and paste the AI generated code in the place to edit the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-4-test, echo = TRUE}
df <- read_csv(
   "data/music.csv",
   col_types = cols(
     artist.familiarity = col_double()
   )
)
 
problems(df)
```

###

CSV files don't specify data types, so **readr** automatically guesses whether columns contain text, numbers, or other values. The `col_types` argument lets you manually specify column types when readr guesses incorrectly. This ensures your data imports with the correct formats for analysis.

### Exercise 5

Now delete `df` along with the code. The chunk should be empty.

Ask AI: Edit the code `read_csv("data/music.csv")` so that the `read_csv()` only selects the columns `artist.familiarity`, `artist.hotttnesss`, and `artist.id` with `cols_only()`, and then overrides the code type to be a string.

If you're using a different AI, provide the `music.csv` tibble again and the previous code.

Copy and paste the AI generated code into the empty chunk. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-5-test, echo = TRUE}
read_csv(
  "data/music.csv",
  col_types = cols_only(
    artist.familiarity = col_character(),
	artist.hotttnesss = col_character(),
	artist.id = col_character()
  )
)
```

###

Assign this code to the object `music_1`.

###

The **readr** package provides column type functions like `col_character()`, `col_double()`, `col_integer()`, and `col_logical()` to control data import. Use these when readr guesses incorrectly or you need specific types.

The [help page](https://readr.tidyverse.org/reference/cols.html) for `cols()` includes more details and discussion.

### Exercise 6

Provide AI with a tibble of `music_1`.

Ask AI: Based off of the `music_1` tibble previously provided, create a similar tibble with only 3 rows of data using `tibble()`, and with the same 3 columns. Don't assign the tibble to an object.

Copy and paste the AI generated code below the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-6}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-6-test, echo = TRUE}
tibble(
  artist.familiarity = c("0.85", "0.67", "0.92"),
  artist.hotttnesss  = c("0.78", "0.55", "0.88"),
  artist.id          = c("AR123", "AR456", "AR789")
)
```

###

Sometimes you’ll need to assemble a tibble “by hand,” doing a little data entry in your R script. There are two useful functions to help you do this which differ in whether you layout the tibble by columns [`tibble()`](https://tibble.tidyverse.org/reference/tibble.html) or by rows [`tribble()`](https://tibble.tidyverse.org/reference/tribble.html).

### Exercise 7

Ask AI: Continue the previous code with the new tibble so that it is fed through a pipe and is written into a new CSV file titled `music_fake.csv` within the `data` directory.

If you're using a different AI, provide the previous code.

Manually, using the same command, do the same for the `music_1` tibble.

Copy and paste the AI generated code into the appropriate place. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-7}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r music-7-test, echo = TRUE}
tibble(
  artist.familiarity = c("0.85", "0.67", "0.92"),
  artist.hotttnesss  = c("0.78", "0.55", "0.88"),
  artist.id          = c("AR123", "AR456", "AR789")
) |>
  write_csv("data/music_fake.csv")

write_csv(music_1, "data/music_1.csv")
```

###

**readr** also comes with two useful functions for writing data back to disk: `write_csv()` and `write_tsv()`. The most important arguments to these functions are `x` (the data frame or tibble to write) and `file` (the location to write it to). You can also specify how missing values are written with `na`, and if you want to `append` to an existing file.

### Exercise 8

Ask AI: Creating a vector of both CSV files, use `read_csv()` to read both CSV files `"music_1.csv"` and `"music_fake.csv"` and create a new column showing the file that the data comes from with the argument `id`. Don't assign the combined data frame to an object.

If you're using a different AI, provide the `music.csv` tibble again and the previous code.

Copy and paste the AI generated code in the place following the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-8}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-8-test, echo = TRUE}
read_csv(c("data/music_1.csv", "data/music_fake.csv"), id = "file")
```

###

With `read_csv()` you can read these multiple CSV files in at once and stack them on top of each other in a single data frame. The code above will only work if you have the CSV files in a data folder in your project.

###

Now, delete all code regarding the CSV file.

We will now move on to learn how to make our data tidy.

### Exercise 9

We are returning to our previous data frame `billboard`. Within your AI, copy/paste the first 3 rows of the tibble to remind it of what we are now working with.

Ask AI: Using the `billboard` tibble, create a pipe to change the columns starting with `wk` to all go in a single column titled `week`, and another column titled `rank` with all the values from the `wk` columns.

Copy and paste the AI generated code in the place following the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-9}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-9-test, echo = TRUE}
billboard |>
  pivot_longer(
    cols = starts_with("wk"),
    names_to = "week",
    values_to = "rank"
  )
```

###

[`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html) "lengthens" data, increasing the number of rows and decreasing the number of columns.

There are three key arguments:

`cols` specifies which columns need to be pivoted, i.e. which columns aren’t variables. This argument uses the same syntax as `select()` so here we could use `!c(artist, track, date.entered)` or `starts_with("wk")`.
`names_to` names the variable stored in the column names, we named that variable week.
`values_to` names the variable stored in the cell values, we named that variable rank.


### Exercise 10

Ask AI: Continue the previous code with `billboard` so that NA values are dropped, and the pipe continues so that the number within the week column is parsed.

If you're using a different AI, provide the `billboard` tibble again and the previous code.

Copy and paste the AI generated code to continue the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-10}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-10-test, echo = TRUE}
billboard |>
  pivot_longer(
    cols = starts_with("wk"),
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  mutate(week = parse_number(week))
```

###

This data is now tidy as NA values are removed, but we could make future computation a bit easier by converting values of week from character strings to numbers using `mutate()` and `readr::parse_number()`. `parse_number()` is a handy function that will extract the first number from a string, ignoring all other text.

### Exercise 11

In the Console, type out `household` and press enter. The result should be a tibble that is a part of **tidyverse**.

Provide AI with the first 3 lines of the `household` tibble.

Ask AI: Using the `household` tibble, create code that changes the columns so that there are only 4 columns, `family`, `child`, `dob`, and `name`.

Copy and paste the AI generated code in the place following the previous code. Run `Cmd/Ctrl + S` to save the file. Place your cursor on the first line of code and run `Cmd/Ctrl + Enter`.

CP/CR.

```{r music-11}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###



```{r music-11-test, echo = TRUE}
household |>
  pivot_longer(
    cols = !family,
    names_to = c(".value", "child"),
    names_sep = "_child"
  )
```

###

`pivot_longer()` makes data longer by turning column names into variables, with `.value` letting part of a name become a variable.  
`pivot_wider()` does the reverse, spreading rows into columns using `values_from`, `names_from`, and `id_cols` to keep rows unique.

### Exercise 12

Before creating a plot, we need to ensure that your data matches our data. In the QMD, replace your code from exercise 12 with our code and place it within a new code chunk.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r music-12}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 6)
```

###

There are three interrelated rules that make a dataset tidy:

Each variable is a column; each column is a variable.
Each observation is a row; each row is an observation.
Each value is a cell; each cell is a single value.

### Exercise 13

Within the latest code chunk, add the option: `#| cache: true`. Assign the result of the pipe to `billboard_longer`. 

`Cmd/Ctrl + Shift + K`. By including `#| cache: true` you cause Quarto to cache the results of the chunk. The next time you render your QMD, as long as you have not changed the code, Quarto will just load up the saved object.

If you have not done so already, you should add `analysis_cache` to the `.gitginore`. The content of the cache file does not belong on GitHub.

Place your cursor on the line where the pipe is assigned to `billboard_longer`, run `Cmd/Ctrl + Enter`, thus ensuring that the workspace also includes a copy of `billboard_longer`.

CP/CR.

```{r music-13}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 8)
```

###



```{r, echo = TRUE}
billboard_longer <- billboard |>
  pivot_longer(
    cols = starts_with("wk"),
    names_to = "week",
    values_to = "rank",
    values_drop_na = TRUE
  ) |>
  mutate(week = parse_number(week))
```

###

Why ensure that your data is tidy? There are two main advantages:

* Consistent data structure makes it easier to learn the tools related to it.

* Column-based variables work better with R's vectorized functions.

### Exercise 14

Within the Console, type `billboard_longer`, which we previously assigned to a pipe and ran in the Console. Hit `Enter`.

CP/CR.

```{r music-14}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 8)
```

###



```{r, echo=TRUE}
billboard_longer
```

###

Most real-world data is untidy because it's often organized for data entry rather than analysis. Many people creating datasets aren't familiar with tidy data principles, making data cleaning a common first step in analysis.

### Exercise 15

Ask AI to generate R code that uses `billboard_longer` to plot a basic graph showing the Billboard rank of each track over the weeks after they were released. Mention you want to use the data from `billboard_longer` and copy/paste the `billboard_longer` you ran in the Console with the resulting tibble. You only need the top 3 lines, mainly to include column names.

Within `labs()`, edit or add a proper title, subtitle, and caption. If axis labels would be useful, add them, but if unnecessary, don't bother. Don't assign the code for the plot to any variable. Put the plot code in a new code chunk. Run `Cmd/Ctrl + Shift + K` to ensure that everything works. Make your plot look nice.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r music-15}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 12)
```

###



```{r, echo=TRUE}
billboard_longer |>
  ggplot(aes(x = week, y = rank, group = track)) +
  geom_line(alpha = 0.2) +
  scale_y_reverse() +
  labs(
    title = "Billboard Track Rankings Over Time",
	subtitle = "Most tracks rise in Billboard rankings in their first 10 weeks",
    x = "Weeks Since Release",
    y = "Billboard Rank",
	caption = "Source: billboard from tidyr"
  )
```

###

Most real analyses will require at least a little tidying. You’ll begin by figuring out what the underlying variables and observations are. Sometimes this is easy; other times you’ll need to consult with the people who originally generated the data. Then, you’ll pivot your data into a tidy form, with variables in the columns and observations in the rows.


## Summary
###

This section covered key concepts from [Chapter 1: Data visualization](https://r4ds.hadley.nz/data-visualize.html), [Chapter 3: Data transformation](https://r4ds.hadley.nz/data-transform.html), [Chapter 5: Data tidying](https://r4ds.hadley.nz/data-tidy.html), and [Chapter 7: Data import](https://r4ds.hadley.nz/data-import.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund.

You will learn about importing, organizing, and visualizing data using core packages such as **[readr](https://readr.tidyverse.org/)**, **[dplyr](https://dplyr.tidyverse.org/)**, and **[ggplot2](https://ggplot2.tidyverse.org/)**. Important functions include: [`read_csv()`](https://readr.tidyverse.org/reference/read_delim.html), [`filter()`](https://dplyr.tidyverse.org/reference/filter.html), [`ggplot()`](https://ggplot2.tidyverse.org/reference/ggplot.html), and [`pivot_longer()`](https://tidyr.tidyverse.org/reference/pivot_longer.html).

### Exercise 1

`Cmd/Ctrl + Shift + K` to ensure that everything works.  The resulting HTML page should be attractive, showing clean versions of your two plots.

At the Console, run:

```
tutorial.helpers::show_file("analysis.qmd")
```

CP/CR.

```{r summary-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 30)
```

### 

Tabular data is considered *tidy* if and only if it satisfies the following rules:

1.  Each variable is a column; each column is a variable.
2.  Each observation is a row, each row is an observation.
3.  Each value is a cell, each cell is a single value.

### Exercise 2

Publish your rendered QMD to GitHub Pages. In the Terminal --- not the Console! --- run:

````
quarto publish gh-pages analysis.qmd
````

If the Terminal is still running from rendering, stop it first using `Ctrl/Cmd + C`.

Copy/paste the resulting URL below.

```{r summary-2}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 1)
```

### 

Publishing the QMD allows for there to be a link on the internet that anyone can access to view the rendered project.

### Exercise 3

Commit and push all your files. Copy/paste the URL to your Github repo.

```{r summary-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

### 

Github repositories are where people can view the code you used to create your project.

###

`pivot_longer()` and `pivot_wider()` are complicated and unique functions, and you can learn more about how they work step by step with the book we used to create this tutorial. The sections for [`pivot_longer()`](https://r4ds.hadley.nz/data-tidy.html#how-does-pivoting-work) and [`pivot_wider()`](https://r4ds.hadley.nz/data-tidy.html#how-does-pivot_wider-work) will explain in depth.

```{r download-answers, child = system.file("child_documents/download_answers.Rmd", package = "tutorial.helpers")}
```
