---
title: Third Tutorial for R4DS
author: Luke Li, David Kane, and Sruthi Gandhi
tutorial:
  id: r4ds-3
output: 
  learnr::tutorial:
    df_print: default
    progressive: true
    allow_skip: true
runtime: shiny_prerendered
description: 'This tutorial for "R for Data Science" covers Chapter 12: Logical vectors, Chapter 13: Numbers, Chapter 14: Strings, Chapter 15: Regular expressions, and Chapter 21: Databases.'
---

```{r setup, include = FALSE}
library(learnr)
library(tutorial.helpers)

library(tidyverse)
library(babynames)
library(nycflights13)
library(DBI)
library(duckdb)
library(dbplyr)

knitr::opts_chunk$set(echo = FALSE)
options(tutorial.exercise.timelimit = 600, 
        tutorial.storage = "local") 

# duck db connections
con_names <- dbConnect(duckdb::duckdb(), 
				 dbdir = "data/nameby_year.duckdb")
con_air <- dbConnect(duckdb::duckdb(), 
				 dbdir = "data/nycflights13.duckdb")

#making lazy table for dbplyr use
dbplyr_names <- tbl(con_names, "babynames")
dbplyr_fly <- tbl(con_air, "flights")

#babynames plot
x <-  dbplyr_names |>
filter(str_starts(name, "A"), year > 2000, n >= 10) |>
mutate(length_class = if_else(str_length(name) <= 5, "Short", "Long"))

data <- flights |>
  select(dep_time, sched_dep_time, dep_delay, tailnum) |>
  mutate(tail_number = parse_number(tailnum)) |>
  filter(!is.na(dep_time))

#flights plot
departure_data <- data |> 
  mutate(
    dep_hour = dep_time %/% 100,
    dep_minute = dep_time %% 100,
    sched_hour = sched_dep_time %/% 100,
    sched_minute = sched_dep_time %% 100,
    true_dep_delay = pmax(dep_delay, 0)
  ) |>
  count(sched_hour, dep_hour, wt = true_dep_delay, name = "total_delay")
```

```{r info-section, child = system.file("child_documents/info_section.Rmd", package = "tutorial.helpers")}
```

## Introduction
### 

This section covers key concepts from [Chapter 13: Numbers](https://r4ds.hadley.nz/numbers.html), [Chapter 14: Strings](https://r4ds.hadley.nz/strings.html), [Chapter 15: Regular Expressions](https://r4ds.hadley.nz/regexps.html), [Chapter 12: Logical Vectors](https://r4ds.hadley.nz/logicals.html), and [Chapter 21: Databases](https://r4ds.hadley.nz/databases.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund.

You will learn about working with numeric data and text patterns using core *Tidyverse* packages like **readr**, **dplyr**, and **stringr**, as well as database interaction using **dbplyr**. Important functions include: [parse_number()](https://readr.tidyverse.org/reference/parse_number.html), [is.na()](https://rdrr.io/r/base/NA.html), [str_detect()](https://stringr.tidyverse.org/reference/str_detect.html), [pmax()](https://rdrr.io/r/base/Extremes.html), [`str_c()`](https://stringr.tidyverse.org/reference/str_c.html), [`str_flatten()`](https://stringr.tidyverse.org/reference/str_flatten.html), [`dbWriteTable()`](https://rdrr.io/cran/DBI/man/dbWriteTable.html), and [`dbConnect()`](https://rdrr.io/cran/DBI/man/dbConnect.html). 

You will also explore logical operations with `if_else()`, and concepts like integer division and modulo. This knowledge will help you manipulate numbers, strings, logical vectors, and connect with DuckDB databases using SQL from within R.

### Exercise 1

Create a Github repo called `r4ds-3`. Set the "Add README" toggle switch to "On."

Connect the repo to a project on your computer using `File -> New Folder from Git ...`. Make sure to select the "Open in a new window" box.

You need two Positron windows: this one for running the tutorial and the one you just created for writing your code and interacting with the Console.

Select `File -> New File -> Quarto Document ...`. Provide a title -- `"Analyzing Baby Names and Flights"` -- and an author (you). Render the document and save it as `analysis.qmd`.

Create a `.gitignore` file with `analysis_files` on the first line and then a blank line. Save and push.

In the Console, run:

```         
show_file(".gitignore")
```

If that fails, it is probably because you have not yet loaded `library(tutorial.helpers)` in the Console.

CP/CR.

```{r introduction-1}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 3)
```

### 
We will be working with databases in this tutorial. Two key differences between data frames and database tables:

* Database tables are stored on disk and can be arbitrarily large. Data frames are stored in memory and are fundamentally limited.

* Database tables almost always have indexes, making it possible to quickly find rows without looking at every single row. Data frames and tibbles don't have indexes.

### Exercise 2

In your QMD, put `library(tidyverse)`, `library(DBI)`, `library(duckdb)`, and `library(dbplyr)` in a new code chunk. Render the file using `Cmd/Ctrl + Shift + K`.

Notice that the file does not look good because the code is visible and there are annoying messages. To take care of this, add `#| message: false` to remove all the messages in this `setup` chunk. Also add the following to the YAML header to remove all code echoes from the HTML:

```         
execute: 
  echo: false
```

Render the file using `Cmd/Ctrl + Shift + K`. In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r introduction-2}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 6)
```

### 
Databases are run by database management systems (DBMS's), which come in three basic forms:

* Client-server DBMS's run on a powerful central server that you connect to from your computer. Great for sharing data with multiple people in an organization.

* Cloud DBMS's run in the cloud, easily handling extremely large datasets and automatically scaling compute resources as needed.

* In-process DBMS's run entirely on your computer. Great for working with large datasets where you're the primary user.

### Exercise 3

Place your cursor in the QMD file at the start of the `library(tidyverse)` line. Use `Cmd/Ctrl + Shift + Enter` to execute all the libraries in the code chunk.

Note that this causes `library(tidyverse)` to be copied down to the Console and then executed.

CP/CR.

```{r introduction-3}
question_text(NULL,
    answer(NULL, correct = TRUE),
    allow_retry = TRUE,
    try_again_button = "Edit Answer",
    incorrect = NULL,
    rows = 3)
```

### 

For this tutorial, we will be working with the babynames database. At the simplest level, you can think about a database as a collection of data frames, called tables in database terminology. Like a data frame, a database table is a collection of named columns, where every value in the column is the same type.

### Exercise 4

From the Console, run these three commands:

`getwd()`
`dir.create("data")`
`list.files()`

This will create a `data` directory in your project. This is a good place to store any data that you are working with.

CP/CR

```{r introduction-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

Your answer should look something like, although your path will be different.

````
> getwd()
[1] "/Users/dkane/Desktop/projects/XX"
> dir.create("data")
> list.files()
 [1] "analysis.qmd"        "data"    "README.md"
>
````


## Baby names
### 

In this section we are going to focus on generating code to create a graph based on a database called `nameby_year.duckdb`. The data in this database comes from `babynames`, which is a dataset that contains names used for American babies from 1880 to 2017. If you want to explore related data, checkout [popular baby names](https://www.ssa.gov/oact/babynames/limits.html). We will be using AI to generate the code for us and then we will break down the functions within the code and how they work.

Key functions you'll learn:

`DBI::dbConnect()` - Creates a connection to a database

`dbReadTable()` - Reads a complete table from a database into a data frame

`as_tibble()` - Converts data frames to tibbles for better printing and consistency

`tbl()` - Creates a lazy reference to a database table for use with dbplyr

`dbGetQuery()` - Executes a SQL query and returns the results as a data frame


### Exercise 1

We begin by downloading a DuckDB version of the babynames dataset directly from GitHub using `download.file()`. 

This stores the database file, `data/nameby_year.duckdb`, to our the `data` directory so we can use it in this tutorial.

In the Console, run:

```         
download.file(
  "https://github.com/PPBDS/positron.tutorials/raw/refs/heads/main/inst/tutorials/r4ds-3/data/nameby_year.duckdb",
  destfile = "data/nameby_year.duckdb",
  mode = "wb"
)
```

CP/CR.

```{r baby-names-1}
 
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

`nameby_year.duckdb` is currently a collection of binary files and is not ready to use until we create a database connection using `DBI::dbConnect()`. The first argument selects the DBMS, then the second and subsequent arguments describe how to connect to it (i.e. where it lives and the credentials that you need to access it).

### Exercise 2

Ask AI: how to connect the `nameby_year.duckdb` database located in the "data" directory to `duckdb` using `DBI::dbConnect()` by assigning it to a variable `con_names`, so it is accessible to use with the rest of the code in the document. Add the code to your QMD in a new chunk. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

CP/CR.

  
```         
show_file("analysis.qmd", chunk = "Last")
```

```{r baby-names-2}
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r baby-names-2-hint-1, echo = TRUE, eval = FALSE}
con_names <- DBI::dbConnect(
    duckdb::duckdb(),
    dbdir = "data/nameby_year.duckdb"
  )
```

This creates a connection to the database and assigns it to the variable con, short for connection. This connection acts like a bridge between R and the DuckDB file. **DBI** is a low-level interface that connects to databases and executes SQL; **dbplyr** is a high-level interface that translates your **dplyr** code to SQL queries then executes them with **DBI**.

### Exercise 3

Before we can read data from a database, we need to know what tables it contains.

Ask AI: How to use `dbListTables()` with `con_names` to see what tables are available in the `nameby_year.duckdb` database. Add this code in a new code chunk to the QMD. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

```{r baby-names-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 3)
```

###
```{r baby-names-3-test, echo = TRUE}
dbListTables(con_names)
```

###

The database contains one table: `babynames`.

### Exercise 4

Now that we know the table name, let's read its contents.

Ask AI: Generate SQL code that takes `babynames` and filters the dataset to show all names that have a first name starting with A after the year 2000. Make sure to provide some lines of **babynames** for the AI to use. If necessary, change the code so that the result of the pipe is assigned to a variable called "sql". 

Add the code as a replacement to the current code in your current chuck in the QMD. 

Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r baby-names-4}

 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r baby-names-4-test, echo = TRUE}

sql <- "
  SELECT name, year, sex, n, prop
  FROM babynames
  WHERE name LIKE 'A%' AND year > 2000
"
```

A query is made up of **clauses**. There are five important clauses: `SELECT`, `FROM`, `WHERE`, `ORDER BY`, and `GROUP BY`. Every query must have the `SELECT` and `FROM` clauses and the simplest query is `SELECT * FROM table`, which selects all columns from the specified table. `WHERE` and `ORDER BY` control which rows are included and how they are ordered, and `GROUP BY` converts the query to a summary, causing aggregation to happen.

### Exercise 5

You can use `dbGetQuery()` to get the results of running a query on the database. 

Ask AI: how to get the results of the query by using `dbGetQuery()`. Make sure to provide your code of the DBI connection and the SQL query.  Add this code to the same code chunk in your QMD and run `Cmd/Ctrl + Enter`.

```{r baby-names-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r baby-names-5-test, echo = TRUE}
sql <- "
  SELECT name, year, sex, n, prop
  FROM babynames
  WHERE name LIKE 'A%' AND year > 2000
"

as_tibble(dbGetQuery(con_names, sql))
```

###

`as_tibble()` turns an existing object, such as a data frame or matrix, into a tibble, a data frame with class `tbl_df`. This is in contrast with `tibble()`, which builds a tibble from individual columns.

### Exercise 6

You just used SQL to filter the babynames database directly and ran a query to see your results. Although, instead of writing raw SQL yourself, we can do the exact same task with R code using `dbplyr`. `dbplyr` is a `dplyr` backend, which means that you keep writing `dplyr` code but the backend executes it differently. In this, `dbplyr` translates to SQL; other backends include `dbplyr` which translates to `data.table`, and `multidplyr` which executes your code on multiple cores. 

Replace the code in your current code chunk in your new QMD by pasting this code instead:

```         
 dbplyr_names <- tbl(con_names, "babynames")
```

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r baby-names-6}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

To use `dbplyr`, you must first use `tbl()` to create an object that represents a database table. This creates a lazy table - operations on it are translated into SQL queries and executed in the database, not in R's memory. Results are only retrieved into R when explicitly requested using `collect()`. 

### Exercise 7

Ask AI: how to use `dbplyr` to filter the data to show all names start with A after the year 2000 using `str_starts()`. Paste this code into the same code chunk and place your cursor at the start of `dbplyr_names` and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r baby-names-7}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r baby-names-7-test, echo = TRUE}
dbplyr_names |>
  filter(name %like% "A%", year > 2000, n >= 10)
```

### 

**dplyr**’s `case_when()` is inspired by SQL’s `CASE` statement and provides a flexible way of performing different computations for different conditions. It has a special syntax that unfortunately looks like nothing else you’ll use in the tidyverse. It takes pairs that look like `condition ~ output`. `condition` must be a logical vector; when it’s `TRUE`, `output` will be used.

### Exercise 8

Paste your current pipe into AI and ask it for a continuation that uses `if_else()`, `str_length()`, and `length_class()` to create a new column of names classified as "Short" or "Long", where anything less than or equal to 5 is classified as short. Add this code as a continuation of your pipe to the same chunk in your QMD. Since the `babynames` is a data set which contains names used for babies from 1880 to 2017 in America, we are using the truncated average length of an American name during 2009: 5. 

Place your cursor on the first line of the code in your current code chunk at the start of `dbplyr_names` and run `Cmd/Ctrl + Enter`. 

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r baby-names-8}
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r baby-names-8-test, echo = TRUE}
 dbplyr_names |>
  filter(name %like% "A%", year > 2000, n >= 10) |>
  mutate(length_class = if_else(str_length(name) <= 5, "Short", "Long"))
```

`>=` is used when you want to find a variable that that is greater than or equal to a number. Conversely, you could use `<=` to find a variable that that is less than or equal to a number. Finally, `==` can be used to find a variable that is exactly equal to a number.

### Exercise 9

Prompt AI to continue the pipe by using `.by()` to summarize the data by grouping by `year` and `length_class`, calculating the total number of babies in each group by summing `n` using `sum()`. Add the code as a continuation of your pipe to your QMD in the same chunk. Place your cursor on the first line of the code in your current code chunk at the start of `dbplyr_names` and run `Cmd/Ctrl + Enter`. 

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

  
```{r baby-names-9}
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r baby-names-9-test, echo = TRUE}
 dbplyr_names |>
  filter(name %like% "A%", year > 2000, n >= 10) |>
  mutate(length_class = if_else(str_length(name) <= 5, "Short", "Long")) |> 
  summarize(total = sum(n), .by = c(year, length_class)) 
```

### 

The `.by` argument is a selection of columns to group by for just this operation, functioning as an alternative to `group_by()`. For details and examples, see `?dplyr_by`.

### Exercise 10

Prompt the AI to arrange the summarized results in ascending order by `year`, and collect the results from the database into a local R tibble for further use, such as plotting. Add the code to your QMD in the same chunk. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`. Now, prompt the AI to arrange the summarized results in ascending order by year, and collect the results from the database into a local R tibble for further use, such as plotting. Add the code to your QMD in the same chunk. Place your cursor on the first line of the code in your current code chunk at the start of `dbplyr_names` and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

  
```{r baby-names-10}
 
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r baby-names-10-test, echo = TRUE}
dbplyr_names |>
	filter(name %like% "A%", year > 2000, n >= 10) |>
  	mutate(length_class = if_else(str_length(name) <= 5, "Short", "Long")) |> 
  	summarize(total = sum(n), .by = c(year, length_class)) |> 
	arrange(year) 
```

### 

Ordering rows involves a straightforward translation from `arrange()` to the SQL `ORDER BY` clause.

### Exercise 11

Now that we have filtered our dataset through a pipe, we need to be on track with the same, correct, code. Replace your last code chunk with our code in your QMD. Add `collect()` to the end of the pipe.

**Change the start letter from `A` to a letter of your choice.**

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

  
```{r baby-names-11}
 
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r baby-names-11-test, echo = TRUE}
dbplyr_names |>
	filter(name %like% "A%", year > 2000, n >= 10) |>
  	mutate(length_class = if_else(str_length(name) <= 5, "Short", "Long")) |> 
  	summarize(total = sum(n), .by = c(year, length_class)) |> 
	arrange(year) |>
   	collect()
```

### 

To get all the data back into R, you call `collect()`. Behind the scenes, this generates the SQL, calls `dbGetQuery()` to get the data, then turns the result into a tibble. 

Again, you should filter for names which start with any other letter except for `A`.

### Exercise 12

Within the recent code chunk, add the option: `#| cache: true`. Assign the result of our pipe to `x`.

`Cmd/Ctrl + Shift + K`. By including `#| cache: true` you cause Quarto to cache the results of the chunk. The next time you render your QMD, as long as you have not changed the code, Quarto will just load up the saved fitted object.

Place your cursor on the line where the pipe is assigned to x, run `Cmd/Ctrl + Enter`. Now, the workspace also includes a copy of `x`.

CP/CR.

```{r baby-names-12}
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r baby-names-12-test, echo = TRUE}
 x <-  dbplyr_names |>
	filter(name %like% "A%", year > 2000, n >= 10) |>
  	mutate(length_class = if_else(str_length(name) <= 5, "Short", "Long")) |> 
  	summarize(total = sum(n), .by = c(year, length_class)) |> 
	arrange(year) |>
   	collect()
```

### 

The `collect()` function is the bridge between database operations and in-memory R operations. Everything before `collect()` in a `dbplyr` pipeline runs as SQL in the database. Once you call `collect()`, the results transfer into R as a tibble, allowing you to use any R function on the data - but be mindful of the data size you're bringing into memory.

### Exercise 13

Within the Console, type `x`, which we previously assigned to a pipe and ran in the Console. Hit `Enter`.

CP/CR.   

```{r baby-names-13}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r baby-names-13-test, echo = TRUE}
x
```

### 

The `.by` argument in `summarize()` performs grouped operations without modifying your data structure. Unlike `group_by()`, which creates a grouped data frame that stays grouped for subsequent operations, `.by` only groups for that specific operation. This makes your code clearer and prevents accidentally carrying grouping structure through your analysis pipeline.

### Exercise 14

Ask AI to generate R code that plots a basic line graph showing the proportion of "Short" vs. "Long" names starting with a specific letter --- the letter you choose to replace `A`--- over the years. Mention you want to use the data from `x` and copy and paste the `x` you ran in the Console with the resulting dataframe. You only need the top 3 lines, mainly to include column names.

Within `labs()` edit or add a proper title, subtitle, and caption. If axis labels are appropriate, add them, but if unnecessary, don't bother. Don't assign the code to any variable. Put this code into a new code chunk. Run `Cmd/Ctrl + Shift + K`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r baby-names-14}
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r baby-names-14-test, fig.keep = FALSE, echo = TRUE}
ggplot(x, aes(x = year, y = total, color = length_class)) +
  geom_line(linewidth = 1.2) +
  labs(
    title = "Trends in Short vs Long A-Names Since 2000",
	subtitle = "Short baby names starting with A are becoming more popular",
    x = "Year", 
    y = "Total Babies",
    color = "Name Length",
	caption = "Source: Babynames"
  ) +
	scale_y_continuous(labels = scales::label_comma()) +
  theme_minimal()
```

### 

Creating categorical variables like `length_class` transforms continuous data (name length in characters) into discrete groups that are easier to interpret and visualize. This classification reveals broader patterns - like whether short or long names are trending - that might be obscured by analyzing individual name lengths.

### Exercise 15

Publish your rendered QMD to GitHub Pages. In the Terminal --- not the Console! --- run:

```         
quarto publish gh-pages analysis.qmd
```

Copy/paste the resulting URL below.

  
```{r baby-names-15}
 
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

Commit/push everything.


## Flights
### 

This section covers key concepts from [Chapter 13: Numbers](https://r4ds.hadley.nz/numbers.html) and [Chapter 15: Regular Expressions](https://r4ds.hadley.nz/regexps.html) from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund. You will learn about some of the power of working with numeric data and text patterns using core Tidyverse packages: readr, dplyr, and stringr. 

Key functions you'll learn: 

[`parse_number()`](https://readr.tidyverse.org/reference/parse_number.html) - Parses the first number it finds

[`is.na()`](https://rdrr.io/r/base/NA.html) - Indicates which elements are missing

[`str_detect()`](https://stringr.tidyverse.org/reference/str_detect.html) - Returns a logical vector with TRUE for each element of string that matches pattern

[`pmax()`](https://rdrr.io/r/base/Extremes.html) - return a single vector giving the maxima of the argument vectors

We will also learn about integer division and moduluo.

In this section we are going to focus on generating code to create a graph based on a dataset called `nycflights13`. The `nycflights13` is a dataset which contains information about all flights departing from New York City airports in 2013. We will be using AI to generate the code for us and then we will break down the functions within the code and how they work. 


### Exercise 1

We begin by downloading a DuckDB version of the flights dataset directly from GitHub using `download.file()`. 

This stores the database file, `data/nycflights13.duckdb`, to our the `data` directory so we can use it in this tutorial.

In the Console, run:

```         
download.file(
  "https://github.com/PPBDS/positron.tutorials/raw/refs/heads/main/inst/tutorials/r4ds-3/data/nycflights13.duckdb",
  destfile = "data/nycflights13.duckdb",
  mode = "wb"
)
```

CP/CR.

```{r flights-1}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###


### Exercise 2

Ask AI: how to connect the `flights` table from the `nycflights13` database located in the "data" directory to `duckdb` using `DBI::dbConnect()` by assigning it to a variable `con_air`, so it is accessible to use with the rest of the code in the document. Add the code to your QMD in a new chunk. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r flights-2}
 
 question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-2-test, echo = TRUE, eval = FALSE}
con_air <- DBI::dbConnect(
    duckdb::duckdb(),
    dbdir = "data/nycflights13.duckdb"
  )
```

### 

Run `dbListTables()` on `con_air` to list all tables in the database.


### Exercise 3

The `nycflights13.duckdb` database has multiple tables: `airlines`, `airports`, `flights`, `planes`, and `weather`.

Ask AI: how to start a pipe from `con_air` to retrieve the contents of the `flights` table from the `nycflights13.duckdb` database using `dbReadTable()` and convert it to a tibble using `as_tibble()`. Make sure the result is not assigned to a variable. Add this code in a new code chunk to the QMD. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`. 

```{r flights-3}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

###

```{r flights-3-test, echo = TRUE}
con_air |>
    dbReadTable("flights") |> 
    as_tibble()
```

###

`dbGetQuery()` is a function which takes a connection and an SQL query as a text string as inputs. It returns the result of that query as a data frame. 

### Exercise 4

To use `dbplyr`, you must first use `tbl()` to create an object that represents a database table, called a lazy table.

Paste this code as a replacement to your current code in the same chunk in your QMD:

```         
 dbplyr_fly <- tbl(con_air, "flights")
```

 View the data by putting `dbplyr_fly` in your Console. Copy/paste the resulting lazy table.

```{r flights-4}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-4-test, echo = TRUE}
 dbplyr_fly <- tbl(con_air, "flights")
 dbplyr_fly
```

### 

Lazy tables let you explore database structure without loading data. When you print a lazy table, dbplyr shows you column names, types, and a preview of rows - all retrieved with a lightweight SQL query. This is crucial for understanding unfamiliar databases before writing complex queries.

### Exercise 5

View the `tailnum` column by pasting this in your console:

```         
class(pull(dbplyr_fly, tailnum))
```

Ask AI: how to check the column type of `tailnum` from the `flights` table using `class()`. Add the code to your Console and hit `Enter`.

CP/CR.

```{r flights-5}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-5-test, echo = TRUE}
class(pull(dbplyr_fly, tailnum))
```

### 

[[ and $ can be used to extract columns out of a data frame. [[ can access by position or by name, and $ is specialized for access by name:

### Exercise 6

The column type of `tailnum` is character. 

Ask AI: Clean the `dbplyr_fly` table by selecting the columns `dep_time`, `sched_dep_time`, `dep_delay`, and `tailnum`. Then, create a new column called `tail_number` by extracting numeric values from the character column `tailnum` using the `parse_number()` function. Add the code to your QMD in a new chunk. Assign this pipe to `departure_data`. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

```{r flights-6}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-6-test, echo = TRUE}
departure_data <- dbplyr_fly |>
  select(dep_time, sched_dep_time, dep_delay, tailnum) |>
  mutate(tail_number = parse_number(tailnum)) 
```

### 

Use `parse_number()` when the string contains non-numeric text that you want to ignore. This is particularly useful for currency data and percentages.

### Exercise 7

 Ask your favorite AI to write code that summarizes how many missing values are in the `dep_time` and `arr_time` columns using the function `is.na()`. Add the code to your QMD in a new chunk. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

```{r flights-7}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-7-test, echo = TRUE}
dbplyr_fly |> 
  summarise(
    missing_dep = sum(is.na(dep_time)),
    missing_arr = sum(is.na(arr_time))
  )
```

Delete this code cell.
 <!--SG: where should I put these -->


We have a lot of missing values that needs to be cleaned.  `is.na(x)` works with any type of vector and returns TRUE for missing values and FALSE for everything else, which we can use to find all the rows with a missing `dep_time`. To check for `NA` values, use `any()` function with argument `is.na()`, argument of which should be the dataset babynames.

### Exercise 8

Ask AI: how you can continue your previous pipe and filters out rows where dep_time is missing using `!is.na()`. Add the code to your QMD in the same chunk as an addition to the pipe that is assigned to `departure_data`. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`. View your new cleaned data.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

```{r flights-8}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r flights-8-test, echo = TRUE}
departure_data <- dbplyr_fly |>
  select(dep_time, sched_dep_time, dep_delay, tailnum) |>
  mutate(tail_number = as.numeric(regexp_extract(tailnum, '[0-9]+'))) |>
  filter(!is.na(dep_time)) |>
  collect()

departure_data
```

### 

`is.na()` can also be useful in `arrange()`. `arrange()` usually puts all the missing values at the end but you can override this default by first sorting with `is.na()`.The `arrange()` function is used to order rows in a data set based on a column. It allows you to sort the data set in either ascending or descending order.

### Exercise 9

Ask AI: how you can find tail numbers starting with "N1". Add the code to your QMD in a new code chunk. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.
In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

```{r flights-9}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-9-test, echo = TRUE}
departure_data |>
  filter(str_detect(tailnum, "^N1")) |>
  count(tailnum, sort = TRUE)
```

### 
Delete this code cell. 

Since `str_detect()` returns a logical vector of the same length as the initial vector, it pairs well with `filter()` There are two functions that are closely related to `str_detect()`:\
\* `str_subset()` returns a character vector containing only the strings that match. 
\* `str_which()` returns an integer vector giving the positions of the strings that match.

Also, `n_distinct(x)` counts the number of distinct (unique) values of one or more variables. It is part of the same family of functions as `count()` and `n()`.

### Exercise 10

The scheduled departure time and the actual departure time for flights is written in the 24-hr system. Let's change this to the 12-hr system. Ask your favorite AI how to break military time into hours and minutes in R using integer division and modulo operators for the `sched_dep_time` and `dep_time` columns. Add the code to your QMD in a new code chunk. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

```{r flights-10}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-10-test, echo = TRUE}
departure_data |>
  mutate(
    dep_hour = dep_time %/% 100,
    dep_minute = dep_time %% 100,
    sched_hour = sched_dep_time %/% 100,
    sched_minute = sched_dep_time %% 100
  )
```

### 

When combined with`%%` or `%/%`, `row_number()` can be a useful tool for dividing data into similarly sized groups. `row_number()` can also be used without any arguments when inside a `dplyr` verb.

### Exercise 11

Let's add a column for positive delays only. Ask the same AI how to continue the pipe and add `true_dep_delay` using `pmax(dep_delay, 0)` to treat early departures (negative delays) as 0. Then, how to use `count()` to group by `sched_hour` and `dep_hour`, weighing by `true_dep_delay` to get total delay minutes for each hour combination. Add the code to your QMD in the same code chunk as a continuation of your pipe. Place your cursor on the first line of the code and run `Cmd/Ctrl + Enter`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

```{r flights-11}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-11-test, echo = TRUE}
departure_data |>
  mutate(
    dep_hour = dep_time %/% 100,
    dep_minute = dep_time %% 100,
    sched_hour = sched_dep_time %/% 100,
    sched_minute = sched_dep_time %% 100,
    true_dep_delay = pmax(dep_delay, 0)
  ) |>
  count(sched_hour, dep_hour, wt = true_dep_delay, name = "total_delay")
```

### 

Note that these are different to the summary functions `min()` and `max()` which take multiple observations and return a single value. You can tell that you’ve used the wrong form when all the minimums and all the maximums have the same value.

### Exercise 12

Now that we have filtered our dataset through a pipe, we need to be on track with the same, correct, code. Replace your latest code chunk with our code in your QMD.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r flights-12}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r flights-12-test, echo = TRUE}
departure_data |>
  mutate(
    dep_hour = dep_time %/% 100,
    dep_minute = dep_time %% 100,
    sched_hour = sched_dep_time %/% 100,
    sched_minute = sched_dep_time %% 100,
    true_dep_delay = pmax(dep_delay, 0)
  ) |>
  count(sched_hour, dep_hour, wt = true_dep_delay, name = "total_delay")
```

### 

The `pmax()` function computes the parallel maximum across vectors element-wise, unlike `max()` which finds the single maximum value. By using `pmax(dep_delay, 0)`, we replace any negative delays (early departures) with 0, ensuring we only sum actual delay time rather than letting early departures offset late ones.

### Exercise 13

Within the recent code chunk, add the option: `#| cache: true`. Assign the result of our pipe to `x`.

`Cmd/Ctrl + Shift + K`. By including `#| cache: true` you cause Quarto to cache the results of the chunk. The next time you render your QMD, as long as you have not changed the code, Quarto will just load up the saved fitted object.

Place your cursor on the line where the pipe is assigned to `x`, run `Cmd/Ctrl + Enter`. Now, the workspace also includes a copy of `x`.

CP/CR.

```{r flights-13}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 

```{r flights-13-test, echo = TRUE}
x <- departure_data |>
  mutate(
    dep_hour = dep_time %/% 100,
    dep_minute = dep_time %% 100,
    sched_hour = sched_dep_time %/% 100,
    sched_minute = sched_dep_time %% 100,
    true_dep_delay = pmax(dep_delay, 0)
  ) |>
  count(sched_hour, dep_hour, wt = true_dep_delay, name = "total_delay")
```

### 

Weighted counts with `count(wt = ...)` transform a frequency table into a sum table. Instead of counting how many flights fall into each scheduled/actual hour combination, we're summing the total minutes of delay. This aggregation reveals which time combinations contribute most to overall delays.

### Exercise 14

Within the Console, type `x`, which we previously assigned to a pipe and ran in the Console. Hit `Enter`.

CP/CR.

```{r flights-14}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-14-test, echo = TRUE}
x
```

### 

In the next exercise, you will see a crosstabulation structure - with two grouping variables creating a grid of values - is ideal for heatmap visualization. Each cell represents a unique combination of scheduled and actual departure hours, and the fill color will reveal patterns like whether certain scheduled hours consistently lead to later departures.

### Exercise 15

Ask AI to generate R code that creates a beautiful heatmap using `geom_tile()`. (You can make a different plot than a heatmap, if you like.) The plot must make sense. It must be something which tells us about the world. It can't just be nonsense. You need to look at the output.

Mention you want to use the data from `x` and copy and paste the `x` you ran in the Console with the resulting tibble. You only need the top 3 lines, mainly to include column names.

Within `labs()` edit or add a proper title, subtitle, and caption. If axis labels are appropriate, add them, but if unnecessary, don't bother. Don't assign the code for the plot to any variable. Add this code to a new code cell. Run `Cmd/Ctrl + Shift + K`.

In the Console, run:

```         
show_file("analysis.qmd", chunk = "Last")
```

CP/CR.

```{r flights-15}
question_text(NULL,
	answer(NULL, correct = TRUE),
	allow_retry = TRUE,
	try_again_button = "Edit Answer",
	incorrect = NULL,
	rows = 5)
```

### 



```{r flights-15-test, echo = TRUE}
ggplot(x, aes(x = sched_hour, y = dep_hour, fill = total_delay)) +
  geom_tile(color = "white") +
  scale_fill_viridis_c(option = "plasma", name = "Total Delay\n(minutes)") +
  labs(
    title = "Flight Delay Patterns: Scheduled vs Actual Departure Hours",
    subtitle = "Darker regions show higher total delay concentrations",
    x = "Scheduled Departure Hour",
    y = "Actual Departure Hour", 
    caption = "Source: nycflights13 dataset"
  ) +
  theme_minimal() +
  theme(
    panel.grid = element_blank(),
    plot.title = element_text(face = "bold", size = 16),
    plot.subtitle = element_text(size = 12)
  )
```

### 

Once you map an aesthetic, **ggplot2** takes care of the rest. It selects a reasonable scale to use with the aesthetic, and it constructs a legend that explains the mapping between levels and values. For x and y aesthetics, **ggplot2** does not create a legend, but it creates an axis line with tick marks and a label. The axis line provides the same information as a legend; it explains the mapping between locations and values.

## Summary

### 

This tutorial covered [Chapter 12: Logical vectors](https://r4ds.hadley.nz/logicals.html), [Chapter 13: Numbers](https://r4ds.hadley.nz/numbers.html), [Chapter 14: Strings](https://r4ds.hadley.nz/strings.html), [Chapter 15: Regular expressions](https://r4ds.hadley.nz/regexps.html), and [Chapter 16: Factors](https://r4ds.hadley.nz/factors.html), from [*R for Data Science (2e)*](https://r4ds.hadley.nz/) by Hadley Wickham, Mine Çetinkaya-Rundel, and Garrett Grolemund.

You learned about how to work with numbers using commands such as [`pmin()`](https://stat.ethz.ch/R-manual/R-devel/library/base/html/Extremes.html) with the **nycflights13** package.

You learned about some of the power of the [**stringr**](https://stringr.tidyverse.org/) package: how to create, combine, and extract strings with the **babynames** package.

```{r download-answers, child = system.file("child_documents/download_answers.Rmd", package = "tutorial.helpers")}
```
