---
title: Exploratory data analysis
author: David Kane
tutorial:
  id: 10-exploratory-data-analysis
output:
  learnr::tutorial:
    progressive: yes
    allow_skip: yes
runtime: shiny_prerendered
description: 'Tutorial for Chapter 10: Exploratory data analysis'
---

```{r setup, include = FALSE}
library(learnr)
library(tutorial.helpers)
library(knitr)

library(tidyverse)
library(tidymodels)
library(hexbin)

knitr::opts_chunk$set(echo = FALSE)
knitr::opts_chunk$set(out.width = '90%')
options(tutorial.exercise.timelimit = 60, 
        tutorial.storage = "local") 

smaller <- diamonds |> 
  filter(carat < 3)

p1 <- nycflights13::flights |> 
  mutate(
    cancelled = is.na(dep_time),
    sched_hour = sched_dep_time %/% 100,
    sched_min = sched_dep_time %% 100,
    sched_dep_time = sched_hour + (sched_min / 60)
  ) |> 
  ggplot(aes(x = sched_dep_time)) + 
  geom_freqpoly(aes(color = cancelled), binwidth = 1/4) + 
  labs(title = "Frequency of Cancelled and Non-cancelled Flights for Scheduled Flights",
       subtitle = "There are a lot of non-cancelled flights at 3 PM",
       x = "Scheduled Departure Time",
       y = "Frequency",
       color = "Cancelled or Non-Cancelled")

p2 <- mpg |> 
  ggplot(aes(x = hwy, y = fct_reorder(class, hwy, median))) +
  geom_boxplot() + 
  labs(
    title = "Variation of Highway Mileage over Classes", 
    subtitle = "The compact and midsized cars are tied for the highest highway mileage.",
    x = "Highway Mileage",
    y = "Car Classes"
  )

p3 <- diamonds |> 
  count(color, cut) |>  
  ggplot(aes(x = color, y = cut)) +
  geom_tile(aes(fill = n)) +
  labs(
    title = "Covariation between Diamond Cut and the Color of the Diamond",
    subtitle = "There are more ideally cut diamonds than other cut types of diamonds",
    x = "Color of diamond",
    y = "Cut of diamond"
  )

p4 <- smaller |> 
  ggplot( aes(x = carat, y = price)) + 
  geom_boxplot(aes(group = cut_width(carat, 0.1))) +
  labs(title = "Covariation between Number of Carats and Price of Diamonds",
       subtitle = "As the number of carats of a diamond increases the price also increases",
       x = "Number of Carats",
       y = "Price of the Diamond")
  
diamonds3 <- diamonds |>
  mutate(
    log_price = log(price),
    log_carat = log(carat)
  )

diamonds_fit <- linear_reg() |>
  fit(log_price ~ log_carat, data = diamonds3)

diamonds_aug <- augment(diamonds_fit, new_data = diamonds3) |>
  mutate(.resid = exp(.resid))

p5 <- diamonds_aug |> 
  ggplot(aes(x = cut, y = .resid)) + 
  geom_boxplot() + 
  labs(
    title = "Prediction Model between Carat and Residual Price",
    subtitle = "As the number of carat increases the Residual Price decreases",
    x = "Number of Carats",
    y = "Residual Price"
  )
```

```{r copy-code-chunk, child = system.file("child_documents/copy_button.Rmd", package = "tutorial.helpers")}
```

```{r info-section, child = system.file("child_documents/info_section.Rmd", package = "tutorial.helpers")}
```


<!-- The below links should be dropped at least once, if not twice. -->

<!-- https://ggplot2.tidyverse.org/ -->
<!-- https://ggplot2-book.org/ -->
<!-- https://socviz.co/ -->
<!-- https://clauswilke.com/dataviz/ -->

<!-- This tutorial took 100 minutes in June 2024, but I have since deleted the big Portland weather graphic, which was lovely but took too long. Should be appropriately sized going forward. -->

<!-- Revisit the Patterns and models section. It is a mess! -->

## Introduction
### 

This tutorial covers [Chapter 10: Exploratory data analysis](https://r4ds.hadley.nz/eda.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. We will be making use of the [**ggplot2**](https://ggplot2.tidyverse.org/) and the [**dplyr**](https://dplyr.tidyverse.org/) packages to learn about variation and patterns. Some key commands include [`coord_cartesian()`](https://ggplot2.tidyverse.org/reference/coord_cartesian.html), [`geom_freqpoly()`](https://ggplot2.tidyverse.org/reference/geom_histogram.html), [`fct_reorder()`](https://forcats.tidyverse.org/reference/fct_reorder.html), [`geom_tile()`](https://ggplot2.tidyverse.org/reference/geom_tile.html), and [`geom_count()`](https://ggplot2.tidyverse.org/reference/geom_count.html). 

<!-- There is a Question section which includes no code. I have incorporated that material as knowledge drops in the Variation section. -->


## Variation
### 

<!-- DK: Section is OK, but could be cleaned up. For example, why bother creating `smaller`. Just pipe it. -->

Your goal during EDA is to develop an understanding of your data. The easiest way to do this is to use questions as tools to guide your investigation. When you ask a question, the question focuses your attention on a specific part of your dataset and helps you decide which graphs, models, or transformations to make.

### Exercise 1

Load in the **tidyverse** package using `library()`.

```{r variation-1, exercise = TRUE}

```

```{r variation-1-hint-1, eval = FALSE}
library(...)
```

```{r variation-1-test, include = FALSE}
library(tidyverse)
```

### 

In this tutorial, we’ll combine what you’ve learned about the **dplyr** package and the **ggplot2** package to interactively ask questions, answer them with data, and then ask new questions.


### Exercise 2

Type in `diamonds` and hit "Run Code"

```{r variation-2, exercise = TRUE}

```

```{r variation-2-hint-1, eval = FALSE}
diamonds
```

```{r variation-2-test, include = FALSE}
diamonds
```

### 

The `diamonds` dataset contains the prices and other attributes of `r nrow(diamonds)` diamonds, such as the `price`, `carat`, `cut`, `color`, and so on.

### Exercise 3

With a new pipeline, pipe `diamonds` to `ggplot()`.

```{r variation-3, exercise = TRUE}

```

```{r variation-3-hint-1, eval = FALSE}
diamonds |> 
  ...
```

```{r variation-3-test, include = FALSE}
diamonds |>
  ggplot()
```

Note that this plot is blank. `ggplot()` has simply created an empty plot object which, when printed, shows as a blank recatangle.

### 

You can see variation easily in real life; if you measure any continuous variable twice, you will get two different results. Each of your measurements will include a small amount of error that varies from measurement to measurement.

### Exercise 4

Within `ggplot()`, using `aes()`, set the `x` argument to `carat`.

```{r variation-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-4-hint-1, eval = FALSE}
diamonds |> 
  ggplot(aes(x = ...))
```

```{r variation-4-test, include = FALSE}
diamonds |>
  ggplot(aes(x = carat))
```

### 

Variables can also vary if you measure across different subjects (e.g., the eye colors of different people) or at different times (e.g., the energy levels of an electron at different moments). 

### Exercise 5

Add `geom_histogram()` to the pipeline.

```{r variation-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-5-hint-1, eval = FALSE}
... + 
  geom_histogram()
```

```{r variation-5-test, include = FALSE}
diamonds |>
  ggplot(aes(x = carat)) + 
  geom_histogram()
```

### 

Every variable has its own pattern of variation, which can reveal interesting information about how that it varies between measurements on the same observation as well as across observations. The best way to understand that pattern is to visualize the distribution of the variable’s values.

### Exercise 6

Within `geom_histogram()`, add the `binwidth` argument and set it equal to `0.5`. 

```{r variation-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-6-hint-1, eval = FALSE}
... +
  geom_histogram(binwidth = ...)
```

```{r variation-6-test, include = FALSE}
diamonds |>
  ggplot(aes(x = carat)) + 
  geom_histogram(binwidth = 0.5)
```

### 

We’ll start our exploration by visualizing the distribution of weights (carat) of ~54,000 diamonds from the diamonds dataset. Since carat is a numerical variable, we can use a histogram.

### Exercise 7

Create a new variable called `smaller` and set it equal to `diamonds`. 

```{r variation-7, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-7-hint-1, eval = FALSE}
smaller <- ...
```

```{r variation-7-test, include = FALSE}
smaller <- diamonds
```

### 

Some valuable questions to ask yourself when dealing with variation are the following:

- Which values are the most common? Why?
- Which values are rare? Why? Does that match your expectations?
- Can you see any unusual patterns? What might explain them?

### Exercise 8

We are still creating the `smaller` tibble. Copy your previous code and create a pipeline with `diamonds`. Pipe `diamonds` with `filter()` and filter out rows for which `carat` is less than `3`. 

```{r variation-8, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-8-hint-1, eval = FALSE}
smaller <- diamonds |> 
  filter(carat < ...)
```

```{r variation-8-test, include = FALSE}
smaller <- diamonds |>
  filter(carat < 3)
```

### 

*“There are no routine statistical questions, only questionable statistical routines.” — Sir David Cox*

### Exercise 9

With a new pipeline, pipe `smaller` to `ggplot()`. 

```{r variation-9, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-9-hint-1, eval = FALSE}
smaller |> 
  ...
```

```{r variation-9-test, include = FALSE}
smaller |>
  ggplot()
```

### 

EDA, or exploratory data analysis, is an iterative cycle. You:

1. Generate questions about your data.

2. Search for answers by visualizing, transforming, and modelling your data.

3. Use what you learn to refine your questions and/or generate new questions.

### Exercise 10

Within `ggplot()`, using `aes()`, set the `x` argument to `carat`. 

```{r variation-10, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-10-hint-1, eval = FALSE}
smaller |> 
  ggplot(aes(x = ...))
```

```{r variation-10-test, include = FALSE}
smaller |>
  ggplot(aes(x = carat))
```

### 

EDA is not a formal process with a strict set of rules. More than anything, EDA is a state of mind. During the initial phases of EDA you should feel free to investigate every idea that occurs to you. Some of these ideas will pan out, and some will be dead ends.

### Exercise 11

Add `geom_histogram()` to the pipeline.

```{r variation-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-11-hint-1, eval = FALSE}
... + 
  geom_histogram()
```

```{r variation-11-test, include = FALSE}
smaller |>
  ggplot(aes(x = carat)) +
  geom_histogram()
```

### 

This histogram suggests several interesting questions:

- Why are there more diamonds at whole carats and common fractions of carats?

- Why are there more diamonds slightly to the right of each peak than there are slightly to the left of each peak?

### Exercise 12

Within `geom_histogram()`, set the `binwidth` argument to `0.01`. 

```{r variation-12, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-12-hint-1, eval = FALSE}
... +
  geom_histogram(binwidth = ...)
```

```{r variation-12-test, include = FALSE}
smaller |>
  ggplot(aes(x = carat)) +
  geom_histogram(binwidth = 0.01)
```

### 

<!-- DK: Does this knowledge drop really belong here? -->

This visualization also revealed clusters, which suggest that subgroups exist in your data. To understand the subgroups, ask:

- How are the observations within each subgroup similar to each other?

- How are the observations in separate clusters different from each other?

- How can you explain or describe the clusters?

- Why might the appearance of clusters be misleading?

### Exercise 13

Let's observe and take note of unusual values in the dataset. With a new pipeline, pipe `diamonds` to `ggplot()`. 

```{r variation-13, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-13-hint-1, eval = FALSE}
diamonds |> 
  ...
```

```{r variation-13-test, include = FALSE}
diamonds |> 
  ggplot()
```

### 

Outliers are observations that are unusual; data points that don’t seem to fit the pattern.

### Exercise 14

Within `ggplot()`, using `aes()`, set the `x` argument to `y`. 

```{r variation-14, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-14-hint-1, eval = FALSE}
diamonds |> 
  ggplot(aes(x = ...))
```

```{r variation-14-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = y))
```

### 

Be careful! The `x` in this code is an argument, the usual usage with `aes()`. The `y`, on the other hand, is not an argument! It is not the usual `y` for which we provide a value when using `aes()`. In this case, `y` is a variable in the `diamonds` tibble. 

Always be careful when variable names happen to be the same as argument names!

Sometimes outliers are data entry errors, sometimes they are simply values at the extremes that happened to be observed in this data collection, and other times they suggest important new discoveries.

### Exercise 15

Add `geom_histogram()` to the pipeline

```{r variation-15, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-15-hint-1, eval = FALSE}
... +
  geom_histogram()
```

```{r variation-15-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = y)) +
    geom_histogram()
```

### 

When you have a lot of data, outliers are sometimes difficult to see in a histogram. We know that they are there because the x-axis extends all the way to 60.

### Exercise 16

Within `geom_histogram()`, add the `binwidth` argument and set it equal to `0.5`. 

```{r variation-16, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-16-hint-1, eval = FALSE}
... +
  geom_histogram(binwidth = ...)
```

```{r variation-16-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = y)) +
    geom_histogram(binwidth = 0.5)
```

### 

There are so many observations in the common bins that the rare bins are very short, making it very difficult to see them (although maybe if you stare intently at 0 you’ll spot something). To make it easy to see the unusual values, we need to zoom to small values of the y-axis with `coord_cartesian()`.

### Exercise 17

Add `coord_cartesian()` to the pipeline. Within `coord_cartesian()`, add the `ylim` argument and set it equal to `c(0, 50)`. 

```{r variation-17, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-17-hint-1, eval = FALSE}
... + 
  coord_cartesian(ylim = ...)
```

```{r variation-17-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = y)) +
    geom_histogram(binwidth = 0.5) +
    coord_cartesian(ylim = c(0, 50))
```

### 

The function, `coord_cartesian()`, also has an `xlim` argument for when you need to zoom into the x-axis. **ggplot2** also has `xlim()` and `ylim()` functions that work slightly differently: they throw away the data outside the limits. This allows us to see that there are three unusual values: `0`, `~30`, and `~60`. 

### Exercise 18

Pipe `diamonds` to `filter()` and filter `y` to be less than `3`.

```{r variation-18, exercise = TRUE}

```

```{r variation-18-hint-1, eval = FALSE}
diamonds |> 
  filter(y < ...)
```

```{r variation-18-test, include = FALSE}
diamonds |> 
  filter(y < 3)
```

### 

The `y` variable measures one of the three dimensions of these diamonds, in `mm`. We know that diamonds can’t have a width of 0 mm, so these values must be incorrect. 


### Exercise 19

Modify the pipe by adding, to the `filter()` command,  `|` and then `y` greater than `20`.

```{r variation-19, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-19-hint-1, eval = FALSE}
diamonds |> 
  filter(y < 3 | y > ...)
```

```{r variation-19-test, include = FALSE}
diamonds |> 
  filter(y < 3 | y > 20)
```

### 

EDA is an important part of any data analysis, even if the primary research questions are handed to you on a platter, because you always need to investigate the quality of your data.

### Exercise 20

Add `select()` to the pipeline. Within `select()`, select the variables `price`, `x`, `y`, and `z`.

```{r variation-20, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-20-hint-1, eval = FALSE}
... |> 
  select(price, x, ...)
```

```{r variation-20-test, include = FALSE}
diamonds |> 
  filter(y < 3 | y > 20) |> 
  select(price, x, y, z)
```

### Exercise 21

Add `arrange()` to the pipeline. Within `arrange()`, add the variable `y`  

```{r variation-21, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r variation-21-hint-1, eval = FALSE}
... |> 
  arrange(...)
```

```{r variation-21-test, include = FALSE}
diamonds |> 
  filter(y < 3 | y > 20) |> 
  select(price, x, y, z) |> 
  arrange(y)
```

This arranges the data by the width of the diamonds. By doing EDA, we have discovered missing data that was coded as `0`, which we never would have found by simply searching for `NAs`. Going forward we might choose to re-code these values as `NAs` in order to prevent misleading calculations. We might also suspect that measurements of `32mm` and `59mm` are implausible: those diamonds are over an inch long, but don’t cost hundreds of thousands of dollars!


## Unusual Values
### 

In this section we will learn about the ways to deal with unusual values in a dataset. One way is to drop the entire row with the strange values. By the end of this section we will be creating the plot below:

<!-- DK: This plot is awful ugly, although it is also ugly in the book. Ought to make it nicer, or even replace it? -->

<!-- DK: Make this more like the chapter. -->

```{r}
p1 <- nycflights13::flights |> 
  mutate(
    cancelled = is.na(dep_time),
    sched_hour = sched_dep_time %/% 100,
    sched_min = sched_dep_time %% 100,
    sched_dep_time = sched_hour + (sched_min / 60)
  ) |> 
  ggplot(aes(x = sched_dep_time)) + 
  geom_freqpoly(aes(color = cancelled), binwidth = 1/4) + 
  labs(title = "Frequency of Cancelled and Non-cancelled Flights for Scheduled Flights",
       subtitle = "There are a lot of non-cancelled flights at 3 PM",
       x = "Scheduled Departure Time",
       y = "Frequency",
       color = "Cancelled or Non-Cancelled")

p1
```

### Exercise 1

With a new pipeline, pipe `diamonds` to `filter()`. Within `filter()`, type `between(y, 3, 20)`. 

```{r unusual-values-1, exercise = TRUE}

```

```{r unusual-values-1-hint-1, eval = FALSE}

```

```{r unusual-values-1-test, include = FALSE}

```

### 

This just keeps the rows with the normal values and drops all of the rows with the strange values. We don’t recommend this option because one invalid value doesn’t imply that all the other values for that observation are also invalid. Additionally, if you have low quality data, by the time that you’ve applied this approach to every variable you might find that you don’t have any data left!


### Exercise 2

Create a new variable called `diamonds2` and set it equal to `diamonds`.


```{r unusual-values-2, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-2-hint-1, eval = FALSE}
diamonds2 <- ...
```

```{r unusual-values-2-test, include = FALSE}

```

### 

The function `if_else()` is a vectorized *if-else*. Compared to the base R equivalent, `ifelse()`, this function allows you to handle missing values in the `condition` with `missing` and always takes `true`, `false`, and `missing` into account when determining what the output type should be.

### Exercise 3

Copy your previous code and pipe `diamonds` to `mutate()`. Within `mutate()`, create a new variable, `y_new`, and set it equal to `if_else(y < 3 | y > 20, NA, y)`.

```{r unusual-values-3, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-3-hint-1, eval = FALSE}
diamonds2 <- diamonds |> 
  mutate(y_new = ...)
```

```{r unusual-values-3-test, include = FALSE}

```

### 

Here we replaced the unusual values with missing values. Here, we use the `mutate()` function to replace the variable with a modified copy. We used the `if_else()` function to replace unusual values with `NA`:

### Exercise 4

<!-- DK: Bad directions. -->

Add `ggplot()` to the pipeline.

```{r unusual-values-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-4-hint-1, eval = FALSE}
... |> 
  ggplot()
```

```{r unusual-values-4-test, include = FALSE}

```

### Exercise 5

Within `ggplot()`, using `aes()`, set the `x` argument to `x` and the `y` argument to `y_new`.

```{r unusual-values-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-5-hint-1, eval = FALSE}
... +
  ggplot(aes(x = ..., y = ...))
```

```{r unusual-values-5-test, include = FALSE}

```

### Exercise 6

Add `geom_point()` to the pipeline.

```{r unusual-values-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-6-hint-1, eval = FALSE}
... +
  geom_point()
```

```{r unusual-values-6-test, include = FALSE}

```

### 

It’s not obvious where you should plot missing values, so **ggplot2** doesn’t include them in the plot, but it does warn that they’ve been removed. To suppress that warning, set `na.rm = TRUE`.

### Exercise 7

Within `geom_point()`, add the `na.rm` argument and set it equal to `TRUE`. 

```{r unusual-values-7, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-7-hint-1, eval = FALSE}
... + 
  geom_point(na.rm = ...)
```

```{r unusual-values-7-test, include = FALSE}

```

### 

### Exercise 8

Add a `title`, `subtitle`, `x` axis title, and a `y` axis title using `labs()`. 

```{r unusual-values-8, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-8-hint-1, eval = FALSE}
... + 
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = ...
    )
```

```{r unusual-values-8-test, include = FALSE}

```

### 

Other times you want to understand what makes observations with missing values different to observations with recorded values. For example, in `nycflights13::flights`, missing values in the `dep_time` variable indicate that the flight was cancelled. So you might want to compare the scheduled departure times for cancelled and non-cancelled times. You can do this by making a new variable, using `is.na()` to check if `dep_time` is missing. Let's make a plot using `geom_freqpoly()`. 

### Exercise 9

With a new pipeline, pipe `nycflights13::flights` to `mutate()`. Within `mutate()`, create a new variable called `cancelled` and set it equal to `is.na(dep_time)`.

```{r unusual-values-9, exercise = TRUE}

```

```{r unusual-values-9-hint-1, eval = FALSE}
nycflights13::flights |> 
  mutate(
    cancelled = ...
  )
```

```{r unusual-values-9-test, include = FALSE}

```

### 

In the dataset `nycflights13::flights1`, missing values in the `dep_time` variable indicate that the flight was cancelled. So you might want to compare the scheduled departure times for cancelled and non-cancelled times. You can do this by making a new variable, using `is.na()` to check if `dep_time` is missing.


### Exercise 10

Within `mutate()`, create the `sched_hour` variable and set it equal to `sched_dep_time %/% 100`. 

```{r unusual-values-10, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-10-hint-1, eval = FALSE}
nycflights13::flights |> 
  mutate(
    cancelled = is.na(dep_time),
    sched_hour = ...
  )
```

```{r unusual-values-10-test, include = FALSE}

```

### 

Data cleaning is just one application of EDA: you ask questions about whether your data meets your expectations or not. To do data cleaning, you’ll need to deploy all the tools of EDA: visualization, transformation, and modelling.

### Exercise 11

<!-- DK: Explain %% and %/%. -->

Within `mutate()`, create the `sched_min` variable and set it equal to `sched_dep_time %% 100`.

```{r unusual-values-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-11-hint-1, eval = FALSE}
nycflights13::flights |> 
  mutate(
    cancelled = is.na(dep_time),
    sched_hour = sched_dep_time %/% 100,
    sched_min = ...
  )
```

```{r unusual-values-11-test, include = FALSE}

```

### 

*“Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.” — John Tukey*

### Exercise 12

Within `mutate()`, create the `sched_dep_time` variable and set it equal to `sched_hour` plus `(sched_min / 60)`.

```{r unusual-values-12, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-12-hint-1, eval = FALSE}
nycflights13::flights |> 
  mutate(
    cancelled = is.na(dep_time),
    sched_hour = sched_dep_time %/% 100,
    sched_min = sched_dep_time %% 100,
    sched_dep_time = ...
  )
```

```{r unusual-values-12-test, include = FALSE}

```

### 

EDA is fundamentally a creative process. And like most creative processes, the key to asking quality questions is to generate a large quantity of questions. It is difficult to ask revealing questions at the start of your analysis because you do not know what insights can be gleaned from your dataset.

### Exercise 13

Add `ggplot()` to the pipeline.

```{r unusual-values-13, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-13-hint-1, eval = FALSE}
... |> 
  ggplot()
```

```{r unusual-values-13-test, include = FALSE}

```

### 

Each new question that you ask will expose you to a new aspect of your data and increase your chance of making a discovery. You can quickly drill down into the most interesting parts of your data—and develop a set of thought-provoking questions—if you follow up each question with a new question based on what you find.

### Exercise 14

Within `ggplot()`, using `aes()`, set the `x` argument to the `sched_dep_time` variable. 

```{r unusual-values-14, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-14-hint-1, eval = FALSE}
... + 
  ggplot(aes(x = ...))
```

```{r unusual-values-14-test, include = FALSE}

```

### 

There is no rule about which questions you should ask to guide your research. However, two types of questions will always be useful for making discoveries within your data. You can loosely word these questions as:

1. What type of variation occurs within my variables?

2. What type of covariation occurs between my variables?

### Exercise 15

Add `geom_freqpoly()` to the pipeline.

```{r unusual-values-15, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-15-hint-1, eval = FALSE}
... + 
  geom_freqpoly()
```

```{r unusual-values-15-test, include = FALSE}

```

### Exercise 16

Within `geom_freqpoly()`, using `aes()`, set the `color` argument equal to the `cancelled` variable.

```{r unusual-values-16, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-16-hint-1, eval = FALSE}
... + 
  geom_freqpoly(aes(color = ...))
```

```{r unusual-values-16-test, include = FALSE}

```

### Exercise 17

Within `geom_freqpoly()`, add the `binwidth` argument and set it equal to `1/4`. 

```{r unusual-values-17, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-17-hint-1, eval = FALSE}
... +
  geom_freqpoly(aes(color = cancelled), binwidth = ...)
```

```{r unusual-values-17-test, include = FALSE}

```

### Exercise 18

Add a `title`, `subtitle`, `x` axis title, and a `y` axis title by adding `labs()` to the pipeline. 

```{r unusual-values-18, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r unusual-values-18-hint-1, eval = FALSE}
... + 
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = ...
  )
```

```{r unusual-values-18-test, include = FALSE}

```

### 

However this plot isn’t great because there are many more non-cancelled flights than cancelled flights. In the next section we’ll explore some techniques for improving this comparison.

### 

Reminder: This is what our graph should look like

```{r}
p1

```

## Covariation
### 

If variation describes the behavior within a variable, covariation describes the behavior between variables. By the end of this section we will have made this graph:

```{r}
p2 <- mpg |> 
  ggplot(aes(x = hwy, y = fct_reorder(class, hwy, median))) +
  geom_boxplot() + 
  labs(
    title = "Variation of Highway Mileage over Classes", 
    subtitle = "The compact and midsized cars are tied for the highest highway mileage.",
    x = "Highway Mileage",
    y = NULL
  )

p2
```

### Exercise 1

With a new pipeline, pipe `diamonds` to `ggplot()`.

```{r covariation-1, exercise = TRUE}

```

```{r covariation-1-hint-1, eval = FALSE}
diamonds |> 
  ...
```

```{r covariation-1-test, include = FALSE}
diamonds |> 
  ggplot()
```

### 

Covariation is the tendency for the values of two or more variables to vary together in a related way. The best way to spot covariation is to visualize the relationship between two or more variables.

### Exercise 2

Within `ggplot()`, using `aes()`, set the `x` argument to the variable `price`. 

```{r covariation-2, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-2-hint-1, eval = FALSE}
diamonds |> 
  ggplot(aes(x = ...))
```

```{r covariation-2-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = price))
```

### 

Every variable has its own pattern of variation, which can reveal interesting information about how that it varies between measurements on the same observation as well as across observations. The best way to understand that pattern is to visualize the distribution of the variable’s values.

### Exercise 3

Add `geom_freqpoly()` to the pipeline.

```{r covariation-3, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-3-hint-1, eval = FALSE}
... + 
  geom_freqpoly()
```

```{r covariation-3-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = price)) +
    geom_freqpoly()
```

### 

The function, `geom_freqpoly()`, visualizes the distribution of a single continuous variable by dividing the x-axis into bins and counting the number of observations in each bin. Frequency polygons display the counts with lines. Frequency polygons are more suitable when you want to compare the distribution across the levels of a categorical variable.

### Exercise 4

Within `geom_freqpoly()`, using `aes()`, set the `color` argument to the `cut` variable. 

```{r covariation-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-4-hint-1, eval = FALSE}
... + 
  geom_freqpoly(aes(color = ...))
```

```{r covariation-4-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = price)) +
    geom_freqpoly(aes(color = cut))
```

### 

Note that **ggplot2** uses an ordered color scale for `cut` because it’s defined as an ordered factor variable in the data.

### Exercise 5

Within `geom_freqpoly()`, set `binwidth` to `500` and `linewidth` to `0.75`. 

```{r covariation-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-5-hint-1, eval = FALSE}
... + 
  geom_freqpoly(aes(color = cut), 
                binwidth = ...,
                ... = 0.75)
```

```{r covariation-5-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = price)) +
    geom_freqpoly(aes(color = cut), 
                  binwidth = 500,
                  linewidth = 0.75)
```

### 

The default appearance of `geom_freqpoly()` is not that useful here because the height, determined by the overall count, differs so much across `cut`s, making it hard to see the differences in the shapes of their distributions. To make the comparison easier we need to swap what is displayed on the y-axis. Instead of displaying count, we’ll display the **density**, which is the count standardized so that the area under each frequency polygon is one.

### Exercise 6

Within `ggplot()`, within `aes()`, set the `y` argument to `after_stat(density)`. 

```{r covariation-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-6-hint-1, eval = FALSE}
diamonds |> 
  ggplot(aes(x = price, y = ...) + 
  geom_freqpoly(aes(color = cut), binwidth = 500, linewidth = 0.75)
```

```{r covariation-6-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = price, y = after_stat(density))) +
    geom_freqpoly(aes(color = cut), 
                  binwidth = 500,
                  linewidth = 0.75)
```

### 

Note that we’re mapping the `density` to `y`, but since `density` is not a variable in the `diamonds` dataset, we need to first calculate it. We use the `after_stat()` function to do so.

### Exercise 7

Add a `title`, `subtitle`, `x` axis title, a `y` axis title, and a legend title (`color`) by adding `labs()` to the pipeline. 

```{r covariation-7, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-7-hint-1, eval = FALSE}
... + 
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = ...
  )
```

### 

There’s something rather surprising about this plot - it appears that fair diamonds (the lowest quality) have the highest average price! But maybe that’s because frequency polygons are a little hard to interpret - there’s a lot going on in this plot. A visually simpler plot for exploring this relationship is using side-by-side boxplots.

### Exercise 8

With a new pipeline, pipe `diamonds` to `ggplot()`, setting `x` to `cut` and `y` to `price` within `aes()`. Add `geom_boxplot()` at the end.

```{r covariation-8, exercise = TRUE}

```

```{r covariation-8-hint-1, eval = FALSE}
... |> 
  ggplot(...(x = cut, y = price)) +
    ...
```

```{r covariation-8-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = cut, y = price)) +
    geom_boxplot()
```

### 

We see much less information about the distribution, but the boxplots are much more compact so we can more easily compare them (and fit more on one plot). It supports the counter-intuitive finding that better quality diamonds are typically cheaper! In the exercises, you’ll be challenged to figure out why.

### Exercise 9

Add a `title`, `subtitle`, `x` axis title, and a `y` axis title by adding `labs()` to the pipeline. 

```{r covariation-9, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-9-hint-1, eval = FALSE}
... + 
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = ...
  )
```

### 

`cut` is an ordered factor: Fair is worse than Good, which is worse than Very Good and so on. Many categorical variables don’t have such an intrinsic order, so you might want to reorder them to make a more informative display. One way to do that is with `fct_reorder()`. Let's use a new dataset to explore this approach.

### Exercise 10

Type in `mpg` and hit "Run Code". 

```{r covariation-10, exercise = TRUE}

```

```{r covariation-10-hint-1, eval = FALSE}
mpg
```

```{r covariation-10-test, include = FALSE}

```

### 

This dataset contains a subset of the fuel economy data that the EPA makes available on https://fueleconomy.gov/. It contains only models which had a new release every year between 1999 and 2008 - this was used as a proxy for the popularity of the car.

### Exercise 11

With a new pipeline, pipe `mpg` to `ggplot()`, setting `x` to `class` and `y` to `hwy` within `aes()`. Add `geom_boxplot()` at the end.

```{r covariation-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-11-hint-1, eval = FALSE}
mpg |> 
  ggplot(aes(... = class, y = ...)) +
    ...
```

```{r covariation-11-test, include = FALSE}
mpg |> 
  ggplot(aes(x = class, y = hwy)) +
    geom_boxplot()
```

### 

The trend is pretty hard to see, so to make it easier on the eyes, we can reorder `class` based on the `median` value of `hwy`. 

### Exercise 12

Within `ggplot()`, within `aes()`, set the `x` argument to `fct_reorder(class, hwy, median)`.

```{r covariation-12, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-12-hint-1, eval = FALSE}
mpg |> 
  ggplot(aes(x = ..., y = hwy)) +
    geom_boxplot()
```

```{r covariation-12-test, include = FALSE}
mpg |> 
  ggplot(aes(x = fct_reorder(class, hwy, median), y = hwy)) +
    geom_boxplot()
```

### 

The function, `fct_reorder()`, is useful for 1d displays where the factor is mapped to position and the function `fct_reorder2()` is for 2d displays where the factor is mapped to a non-position aesthetic. 

### Exercise 13

Since we have long variable names, `geom_boxplot()` will work better if you flip it `90°`. You can do that by exchanging the `x` and `y` aesthetic mappings. Within `ggplot()`, within `aes()`, swap the aesthetic mappings of `x` and `y`.

```{r covariation-13, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-13-hint-1, eval = FALSE}
mpg |> 
  ggplot(aes(y = fct_reorder(class, hwy, median), ... = ...)) +
    geom_boxplot()
  ...
```

```{r covariation-13-test, include = FALSE}
mpg |> 
  ggplot(aes(y = fct_reorder(class, hwy, median), x = hwy)) +
    geom_boxplot()
```

### 

If you have a small dataset, it’s sometimes useful to use `geom_jitter()` to avoid overplotting to more easily see the relationship between a continuous and categorical variable. The [**ggbeeswarm**](https://r-charts.com/distribution/ggbeeswarm/) package provides a number of methods similar to `geom_jitter()`.

### Exercise 14

Add a `title`, `subtitle` and `x` axis title by adding `labs()` to the pipeline. Set the y-axis label to `NULL`. 

```{r covariation-14, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-14-hint-1, eval = FALSE}
... + 
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = NULL
  )
```

```{r covariation-14-test, include = FALSE}
mpg |> 
  ggplot(aes(x = hwy, y = fct_reorder(class, hwy, median))) +
  geom_boxplot() + 
  labs(
    title = "Variation of Highway Mileage over Classes", 
    subtitle = "The compact and midsized cars are tied for the highest highway mileage.",
    x = "Highway Mileage",
    y = NULL
  )
```

Reminder this is what our plot should look like: 

```{r}
p2
```

## Covariation with two categorical variables
### 

In this section we will learn how to visualize the covariation between categorical variables. 

### 

By the end of this section, we will be creating this plot:

```{r}
p3 <- diamonds |> 
  count(color, cut) |>  
  ggplot(aes(x = color, y = cut)) +
  geom_tile(aes(fill = n)) +
  labs(
    title = "Covariation between Diamond Cut and the Color of the Diamond",
    subtitle = "There are more ideally cut diamonds than other cut types of diamonds",
    x = "Color of diamond",
    y = "Cut of diamond"
  )

p3
```

### Exercise 1

With a new pipeline, pipe the `diamonds` dataset to `ggplot()` with `aes(x = cut, y = color)`.

```{r covariation-with-two-categoric-1, exercise = TRUE}

```

```{r covariation-with-two-categoric-1-hint-1, eval = FALSE}
diamonds |> 
  ggplot(aes(... = cut, y = ...))
```

```{r covariation-with-two-categoric-1-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = cut, y = color))
```

### 

To visualize the covariation between categorical variables, you’ll need to count the number of observations for each combination of levels of these categorical variables.

### Exercise 2

One way to visualize two categorical variables is by using the function `geom_count()`. This function counts the number of observations at each location. Add `geom_count()` to the pipeline.

```{r covariation-with-two-categoric-2, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-categoric-2-hint-1, eval = FALSE}
... + 
  geom_count()
```

```{r covariation-with-two-categoric-2-test, include = FALSE}
diamonds |> 
  ggplot(aes(x = cut, y = color)) +
    geom_count()
```

### 

The size of each circle in the plot displays how many observations occurred at each combination of values. Covariation will appear as a strong correlation between specific x values and specific y values.





### Exercise 3

Another approach for exploring the relationship between these variables is computing the counts with **dplyr**.

With a new pipeline, pipe the `diamonds` dataset to `count()`. Within `count` add the variables `color` and `cut`. 

```{r covariation-with-two-categoric-3, exercise = TRUE}

```

```{r covariation-with-two-categoric-3-hint-1, eval = FALSE}
diamonds |> 
  count(color, ...)
```

```{r covariation-with-two-categoric-3-test, include = FALSE}
diamonds |> 
  count(color, cut)
```

### 

Now, let's visualize this with `geom_tile()` and the fill aesthetic.


### Exercise 4

Add `ggplot()` to the pipeline, using `aes()`, with `x` set to `color` and `y` set to `cut`. 

```{r covariation-with-two-categoric-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-categoric-4-hint-1, eval = FALSE}
... + 
  ggplot(aes(x = color, y = ...))
```

```{r covariation-with-two-categoric-4-test, include = FALSE}
diamonds |> 
  count(color, cut) |> 
  ggplot(aes(x = color, y = cut))
```

### 

If the categorical variables are unordered, you might want to use the [**seriation**](https://cran.r-project.org/web/packages/seriation/index.html) package to simultaneously reorder the rows and columns in order to more clearly reveal interesting patterns.

### Exercise 5

To the pipeline, add `geom_tile()`, within which using `aes()`, setting the `fill` argument to the `n` variable.

```{r covariation-with-two-categoric-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-categoric-5-hint-1, eval = FALSE}
... + 
  geom_tile()
```

```{r covariation-with-two-categoric-5-test, include = FALSE}
diamonds |> 
  count(color, cut) |> 
  ggplot(aes(x = color, y = cut)) +
    geom_tile(aes(fill = n))
```

### 

For larger plots, you might want to try the [**heatmaply**](https://cran.r-project.org/web/packages/heatmaply/vignettes/heatmaply.html) package, which creates interactive plots.

### Exercise 6

Add a `title`, `subtitle`, `x` axis title, and a `y` axis title by adding `labs()` to the pipeline.

```{r covariation-with-two-categoric-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-categoric-6-hint-1, eval = FALSE}
... + 
  labs(title = ...,
       subtitle = ...,
       x = ...,
       y = ...)
```

```{r covariation-with-two-categoric-6-test, include = FALSE}
diamonds |> 
  count(color, cut) |> 
  ggplot(aes(x = color, y = cut)) +
    geom_tile(aes(fill = n)) +
    labs(
      title = "Covariation between Diamond Cut and the Color of the Diamond",
      subtitle = "There are more ideally cut diamonds than other cut types of diamonds",
      x = "Color of diamond",
      y = "Cut of diamond"
      )
```

Reminder this is what your plot should look like: 

```{r}
p3
```

## Covariation with two numerical variables
### 

In this section, we will learn how to visualize covariation between two numerical variables. 

```{r}
p4 <- smaller |> 
  ggplot(aes(x = carat, y = price)) + 
  geom_boxplot(aes(group = cut_width(carat, 0.1))) +
  labs(title = "Covariation between Number of Carats and Price of Diamonds",
       subtitle = "As the number of carats of a diamond increases the price also increases",
       x = "Number of Carats",
       y = "Price of the Diamond")

p4
```

### 

Let's build this plot.

### Exercise 1

Type in `smaller` and hit "Run Code".

```{r covariation-with-two-numerical-1, exercise = TRUE}

```

```{r covariation-with-two-numerical-1-hint-1, eval = FALSE}
smaller
```

```{r covariation-with-two-numerical-1-test, include = FALSE}
smaller
```

### 

This tibble only includes diamonds that have less than 3 carats. 

### Exercise 2

You’ve already seen one great way to visualize the covariation between two numerical variables: draw a scatterplot with `geom_point()`. You can see covariation as a pattern in the points. Pipe `smaller` into `ggplot()`, using `aes(x = carat, y = price))` as the value for `mapping`. Add `geom_point()` at the end.

```{r covariation-with-two-numerical-2, exercise = TRUE}

```

```{r covariation-with-two-numerical-2-hint-1, eval = FALSE}
smaller |> 
  ggplot(aes(x = ..., y = price)) +
    ...
```

```{r covariation-with-two-numerical-2-test, include = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    geom_point()
```

### 

For example, you can see a positive relationship between the carat size and the price of a diamond: diamonds with more carats have a higher price. The relationship seems exponential.

### Exercise 3

Scatterplots become less useful as the size of your dataset grows, because points begin to overplot, and pile up into areas of uniform black, making it hard to judge differences in the density of the data across the 2-dimensional space as well as making it hard to spot the trend. 

Take the previous code and add `alpha = 1 / 100` to the call to `geom_point()`.

```{r covariation-with-two-numerical-3, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-numerical-3-hint-1, eval = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    geom_point(alpha = ...)
```

```{r covariation-with-two-numerical-3-test, include = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    geom_point(alpha = 1 / 100)
```

### 

Using the `alpha` aesthetic to add transparency is one way to deal with too much data.

### Exercise 4

Using transparency can be challenging for very large datasets. Another solution is to "bin" the data, using functions like `geom_bin2d()` and `geom_hex()`. Replace the call to `geom_point()` with `geom_bin2d()`.

```{r covariation-with-two-numerical-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-numerical-4-hint-1, eval = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    ...
```

```{r covariation-with-two-numerical-4-test, include = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    geom_bin2d()
```

### 

`geom_bin2d()` and `geom_hex()` divide the coordinate plane into 2d bins and then use a fill color to display how many points fall into each bin. `geom_bin2d()` creates rectangular bins. `geom_hex()` creates hexagonal bins. You will need to install the **hexbin** package to use `geom_hex()`.


### Exercise 5

Another option is to bin one continuous variable so it acts like a categorical variable. Then you can use one of the techniques for visualizing the combination of a categorical and a continuous variable that you learned about. For example, you could bin carat and then for each group, display a boxplot. 

To create such a plot, begin by replacing `geom_hex()` with `geom_boxplot()` from the previous pipeline.

```{r covariation-with-two-numerical-5, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-numerical-5-hint-1, eval = FALSE}
... +
  geom_boxplot()
```

```{r covariation-with-two-numerical-5-test, include = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    geom_boxplot()
```

### 

Not a particularly informative plot! We need to change `carat` from a continuous numeric variable to a category variable, each value of which would cover a range of `carat`.


### Exercise 6

Within `geom_boxplot()`, using `aes()`, set the `group` argument to `cut_width(carat, 0.1)`. 

```{r covariation-with-two-numerical-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-numerical-6-hint-1, eval = FALSE}
... +
  geom_boxplot(aes(... = cut_width(..., 0.1)))
```

```{r covariation-with-two-numerical-6-test, include = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) +
    geom_boxplot(aes(group = cut_width(carat, 0.1)))
```

### 

`cut_width(x, width)`, as used above, divides x into bins of `width` width. By default, boxplots look roughly the same (apart from number of outliers) regardless of how many observations there are, so it’s difficult to tell that each boxplot summarizes a different number of points.

### Exercise 7

Add a `title`, `subtitle`, `x` axis title, and a `y` axis title by adding `labs()` to the pipeline. 

```{r covariation-with-two-numerical-7, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r covariation-with-two-numerical-7-hint-1, eval = FALSE}
... + 
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = ...,
  )
```

```{r covariation-with-two-numerical-7-test, include = FALSE}
smaller |> 
  ggplot(aes(x = carat, y = price)) + 
  geom_boxplot(aes(group = cut_width(carat, 0.1))) +
  labs(title = "Covariation between Number of Carats and Price of Diamonds",
       subtitle = "As the number of carats of a diamond increases the price also increases",
       x = "Number of Carats",
       y = "Price of the Diamond")
```

This is a reminder that our plot should look like this:

```{r}
p4
```

### 

One way to deal with different number of observations in each bin is to make the width of the boxplot proportional to the number of points with `varwidth = TRUE`.



## Patterns and models
### 

<!-- DK: This is not bad and does a good job of using all the text from the chapter. But is that text well-organized? Also, whole thing is sort of disjoint and tough to follow. Do we really expect students to understand this plot? Introducing tidymodels, and variable transformations, and residuals is a lot of stuff! Also, sure would be nice if we could do all this in one pipe, rather than creating a bunch of new objects. First line would start with diamonds, then select just the three variables we need, proving a bicg picture on what we are doing; then log-transform, and so on. -->

In this section, we will discuss how to find and display patterns in your data. We will create this plot:

```{r}
p5 <- diamonds_aug |> 
  ggplot(aes(x = cut, y = .resid)) + 
  geom_boxplot() + 
  labs(
    title = "Cut Quality and (adjusted) Price",
    subtitle = "A better cut means higher prices, adjusted for the number of carats",
    x = "Quality of the Cut",
    y = "Price (adjusted for carats)"
  )

p5
```

### 

If a systematic relationship exists between two variables it will appear as a pattern in the data. If you spot a pattern, you can ask yourself the following questions:

* Could this pattern be due to coincidence (i.e. random chance)?

* How can you describe the relationship implied by the pattern?

### Exercise 1

Load in the `tidymodels` package by using `library()`.

```{r patterns-and-models-1, exercise = TRUE}

```


```{r patterns-and-models-1-hint-1, eval = FALSE}
library(...)
```

```{r patterns-and-models-1-test, include = FALSE}
library(tidymodels)
```

### 

Other relevant questions of data patterns include:

* How strong is the relationship implied by the pattern?

* What other variables might affect the relationship?

* Does the relationship change if you look at individual subgroups of the data?

The [**tidymodels**](https://www.tidymodels.org/) package provides tools for answering these questions.

### Exercise 2

Create a new variable `diamonds3` and set it equal to `diamonds`. 

```{r patterns-and-models-2, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-2-hint-1, eval = FALSE}
diamonds3 <- diamonds
```

```{r patterns-and-models-2-test, include = FALSE}
diamonds3 <- diamonds
```

### 

Patterns in your data provide clues about relationships, i.e., they reveal covariation. If you think of variation as a phenomenon that creates uncertainty, covariation is a phenomenon that reduces it.

### Exercise 3

Let's manipulate the variables in `diamonds3`.  Retain the code from the previous question but pipe `diamonds` to `mutate()`. Within `mutate()`, create a new variable `log_price` and set it equal to `log(price)`. 

```{r patterns-and-models-3, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-3-hint-1, eval = FALSE}
diamonds3 <- diamonds |> 
  mutate(
    log_price = ...
  )
```

```{r patterns-and-models-3-test, include = FALSE}
diamonds3 <- diamonds |> 
  mutate(
    log_price = log(price)
  )
```

### 

If two variables covary, you can use the values of one variable to make better predictions about the values of the second. If the covariation is due to a causal relationship (a special case), then you can use the value of one variable to control the value of the second.

### Exercise 4

Within `mutate()`, create another variable called `log_carat` and set it equal to `log(carat)`. 

```{r patterns-and-models-4, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-4-hint-1, eval = FALSE}
diamonds3 <- diamonds |> 
  mutate(
    log_price = log(price),
    log_carat = ...
  )
```

```{r patterns-and-models-4-test, include = FALSE}
diamonds3 <- diamonds |> 
  mutate(
    log_price = log(price),
    log_carat = log(carat)
  )
```

### 

We will be using `diamonds3`, with these two new variables,  to make a model that predicts `log_price` from `log_carat` and then computes the residuals (the difference between the predicted value and the actual value).

### Exercise 5

Create a new variable called `diamonds_fit` and set it equal to `linear_reg()`. 

```{r patterns-and-models-5, exercise = TRUE}

```

```{r patterns-and-models-5-hint-1, eval = FALSE}
diamonds_fit <- ...
```

```{r patterns-and-models-5-test, include = FALSE}
diamonds_fit <- linear_reg()
```

### 

Models are a tool for extracting patterns out of data. For example, consider the diamonds data. It’s hard to understand the relationship between cut and price, because cut and carat, and carat and price are tightly related.

### Exercise 6

Using your previous code, pipe `linear_reg()` with the following code `fit(log_price ~ log_carat, data = diamonds3)`.

```{r patterns-and-models-6, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-6-hint-1, eval = FALSE}
diamonds_fit <- linear_reg() |> 
  ...
```

```{r patterns-and-models-6-test, include = FALSE}
diamonds_fit <- linear_reg() |> 
  fit(log_price ~ log_carat, data = diamonds3)
```

### 

The functon [`linear_reg()`](https://parsnip.tidymodels.org/reference/linear_reg.html) defines a model that can predict numeric values from predictors using a linear function. This function can fit regression models. There are different ways to fit this model, and the method of estimation is chosen by setting the model `engine`.

### Exercise 7

Create a new variable called `diamonds_aug` and set it equal to `augment(diamonds_fit, new_data = diamonds3)`. 

```{r patterns-and-models-7, exercise = TRUE}

```

```{r patterns-and-models-7-hint-1, eval = FALSE}
diamonds_aug <- ...
```

```{r patterns-and-models-7-test, include = FALSE}
diamonds_aug <- augment(diamonds_fit, new_data = diamonds3)
```

### 

If you look at `diamonds_aug`, you will notice a new variable: `.resid`. The residuals give us a view of the `log(price)` of the diamond, once the effect of `log(carat)` has been removed.

### Exercise 8

Using your previous code, pipe `augment(diamonds_fit, new_data = diamonds3)` to `mutate()`. Within `mutate()`, create a new variable `.resid` and set it equal to `exp(.resid)`. 

```{r patterns-and-models-8, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-8-hint-1, eval = FALSE}
diamonds_aug <- augment(diamonds_fit, new_data = diamonds3) |>
  ...
```

```{r patterns-and-models-8-test, include = FALSE}
diamonds_aug <- augment(diamonds_fit, new_data = diamonds3) |>
  mutate(.resid = exp(.resid))
```

### 

Note that instead of using the raw values of `price` and `carat`, we log transformed them first, and fitted a model to the log-transformed values. Then, we exponentiate the residuals to put them back on the scale of raw prices.

### Exercise 9

With a new pipeline, pipe `diamonds_aug` to `ggplot()`.

```{r patterns-and-models-9, exercise = TRUE}

```

```{r patterns-and-models-9-hint-1, eval = FALSE}
diamonds_aug |> 
  ggplot()
```

```{r patterns-and-models-9-test, include = FALSE}
diamonds_aug |> 
  ggplot()
```

Note that this plot is empty, as usual when we call `ggplot()` without specifying a value for `mapping`.

### Exercise 10

Within `ggplot()`, using `aes()`, set the `x` argument to the `carat` variable.

```{r patterns-and-models-10, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-10-hint-1, eval = FALSE}
diamonds_aug |> 
  ggplot(aes(x = ...))
```

```{r patterns-and-models-10-test, include = FALSE}
diamonds_aug |> 
  ggplot(aes(x = carat))
```

### 

Big picture: We are sure that there is a major relationship between `price` and `carat`. Diamonds with more carats are much more expensive. If we want to study the relationship between `price` and `cut`, we first need to "adjust" for the number of carats.

### Exercise 11

Within `ggplot()`, using `aes()`, set the `y` argument to the `.resid` variable.

```{r patterns-and-models-11, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-11-hint-1, eval = FALSE}
diamonds_aug |> 
  ggplot(aes(x = carat, y = ...))
```

```{r patterns-and-models-11-test, include = FALSE}
diamonds_aug |> 
  ggplot(aes(x = carat, y = .resid))
```

### 

<!-- DK: This is was too concise to be meaningful to new students! -->

If we had not log-transformed `price` and `carat`, this adjustment process would not have worked nearly as well.

### Exercise 12

Add `geom_point()` to the pipeline.

```{r patterns-and-models-12, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-12-hint-1, eval = FALSE}
... +
  geom_point()
```

```{r patterns-and-models-12-test, include = FALSE}
diamonds_aug |> 
  ggplot(aes(x = carat, y = .resid)) +
    geom_point()
```

### 

<!-- DK: This is word-for-word from the book, but I don't understand it! -->

Once you’ve removed the strong relationship between carat and price, you can see what you expect in the relationship between cut and price: relative to their size, better quality diamonds are more expensive.

### Exercise 13

In the previous pipeline, replace `carat` with `cut` and replace `geom_point()` with `geom_boxplot()`. 

```{r patterns-and-models-13, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-13-hint-1, eval = FALSE}
diamonds_aug |> 
  ggplot(aes(x = ..., y = .resid)) +
    ...
```

```{r patterns-and-models-13-test, include = FALSE}
diamonds_aug |> 
  ggplot(aes(x = cut, y = .resid)) +
    geom_boxplot()
```

### 

A boxplot is the best way to see the relationship between the type of diamond cut and the price of the diamond, adjusted for `carat`. 

### Exercise 14

Add a `title`, `subtitle`, `x` axis title, and a `y` axis title by adding `labs()` to the pipeline.

```{r patterns-and-models-14, exercise = TRUE}

```

<button onclick = "transfer_code(this)">Copy previous code</button>

```{r patterns-and-models-14-hint-1, eval = FALSE}
... +
  labs(
    title = ...,
    subtitle = ...,
    x = ...,
    y = ...
  )
```

```{r patterns-and-models-14-test, include = FALSE}
diamonds_aug |> 
  ggplot(aes(x = cut, y = .resid)) + 
  geom_boxplot() + 
  labs(
    title = "Cut Quality and (adjusted) Price",
    subtitle = "A better cut means higher prices, adjusted for the number of carats",
    x = "Quality of the Cut",
    y = "Price (adjusted for carats)"
  )
```

### 

Reminder this is what our plot should look like.

```{r}
p5
```


## Summary
### 

This tutorial covered [Chapter 10: Exploratory data analysis](https://r4ds.hadley.nz/eda.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. We made use of the [**ggplot2**](https://ggplot2.tidyverse.org/) and the [**dplyr**](https://dplyr.tidyverse.org/) packages to learn about variation and patterns. Some key commands included [`coord_cartesian()`](https://ggplot2.tidyverse.org/reference/coord_cartesian.html), [`geom_freqpoly()`](https://ggplot2.tidyverse.org/reference/geom_histogram.html), [`fct_reorder()`](https://forcats.tidyverse.org/reference/fct_reorder.html), [`geom_tile()`](https://ggplot2.tidyverse.org/reference/geom_tile.html), and [`geom_count()`](https://ggplot2.tidyverse.org/reference/geom_count.html).


```{r download-answers, child = system.file("child_documents/download_answers.Rmd", package = "tutorial.helpers")}
```
