---
title: "Example Rmarkdown using data language engine"
author: "David M. Kaplan"
date: "`r Sys.Date()`"
output: pdf_document

# Set run_gpg to TRUE to turn on GPG part of document  
params:
  run_gpg: TRUE
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(echo = TRUE)

# If package not installed, install it
if (!requireNamespace("knitrdata")) {
  remotes::install_github("dmkaplan2000/knitrdata",build_vignettes = TRUE)  
}
library(knitrdata) # load package
```

## Change doc root for CRAN

CRAN requires that all package code writes to a temporary directory, so I am changing the document root to the temporary directory. In a real example, one would almost certainly not do this.

```{r}
knitr::opts_knit$set(root.dir = tempdir(check=TRUE))
```

## Conceptual overview

Sometimes it would be useful to make completely standalone Rmarkdown documents that do not depend on data in external files. One important example of this is scientific publications written in Rmarkdown for which we often would like to supply the source document with data to ensure results are reproducible.

This package provides for a new `data` language engine to facilitate the creation of standalone Rmarkdown documents. Instead of putting code inside `data` chunks, one puts the contents of the data file that one wishes to use in your Rmarkdown document. 

For text files (e.g., CSV data tables), the chunk will typically just include the contents of the file itself. For binary data files (e.g., RDS files, images, NetCDF files), the data must be first encoded as text before including it in a `data` chunk. Two encoding formats are currently implemented: `base64`, used for non-sensitive data; and `gpg`, allowing one to encrypt data so that only users with the decryption key have access to the data. The latter option requires that a GPG keyring be installed and properly configured.

`data` chunks do not typically produce output as most code chunks do. Instead, the decoded contents of the chunk are either returned as a variable in the R workspace or saved to an external file that can then be loaded for use in the document.

## Encoding data

Both text and binary data files can be encoded, but this is required for binary data so that it can be incorporated in a (text) Rmarkdown document. Two encoding formats are currently implemented: `base64`, used for non-sensitive data; and `gpg`, allowing one to encrypt data so that only users with the decryption key have access to the data. The latter option requires that a GPG keyring be installed and properly configured.

Two helper functions, `data_encode` and `data_decode` are included in the package to facilitate encoding and decoding of data files. These are basically wrapper functions for functionality provided by the [xfun](https://cran.r-project.org/package=xfun) and [gpg](https://cran.r-project.org/package=gpg) packages.

In the examples below, we will use the following simple data frame that exists in both text and binary formats:

```{r}
D = data.frame(a=1:3,b=letters[1:3])
D
write.csv(D,"test.csv", row.names = FALSE, quote = FALSE)
saveRDS(D,"test.RDS")
```

For text format chunks, the contents of file itself are included directly in the data chunk:

```{r}
cat(readLines("test.csv"),sep="\n")
```

### Base64

Base64 encoding is the standard encoding to be used for all non-sensitive binary data. Encoding works as follows:

```{r}
b64 = data_encode("test.RDS","base64")
```

By default this function will silently return the encoded data as a character string and `cat` the contents of that character string to the command line so that it can be copied and pasted directly into a `data` chunk. This is only practical for relatively small data files, so for larger files, one can place the output in a file:

```{r}
data_encode("test.RDS","base64",output="test.RDS.base64")
cat(readLines("test.RDS.base64"),sep="\n")
```

Data can be decoded as follows:

```{r}
rds = data_decode(b64,"base64",as_text=FALSE)
writeBin(rds,"test_output.RDS")
y = readRDS("test_output.RDS")
y
```

### GPG

GPG encoding requires a properly configured GPG keyring. Numerous websites explain how to do this, included the main [gpg](https://gnupg.org/) website.

For the purposes of this vignette, I will generate a test GPG private-public key pair using the [gpg](https://cran.r-project.org/package=gpg) package, however in real use scenarios proper keys would need to be generated using the `gpg` command line tool.

```{r eval=requireNamespace("gpg") && params$run_gpg}
id = gpg::gpg_keygen("test","test@test.org")
```

Next one uses this key to encode a data file:

```{r eval=requireNamespace("gpg") && params$run_gpg}
enc = data_encode("test.csv","gpg",options = list(receiver=id))
```

Note that the receiver ID must be supplied in the `options` list input argument.

Decoding works as follows:

```{r eval=requireNamespace("gpg") && params$run_gpg}
t = data_decode(enc,"gpg",as_text=TRUE)
cat(t)
```

Note that there is no need to supply the receiver ID when decoding because the appopriate private key is in the keyring. The `as_text` input argument would be set to `FALSE` if the contents were a binary file.

We can delete the text key from our keyring as follows:

```{r eval=requireNamespace("gpg") && params$run_gpg}
gpg::gpg_delete(id,secret=TRUE)
```

## Text data chunks

Textual data can be directly placed into a data chunks.

```{data output.var="t"}
a,b
1,a
2,b
3,c
```

Text contents of the chunk will be read into the variable `t`. This can then be converted into a data.frame using `read.csv`:

```{r}
read.csv(text=t)
```

You can also use a loader function:


```{data output.var="t",loader.function=read.csv}
a,b
1,a
2,b
3,c
```

```{r}
t
```


## Base64-encoded binary data chunks

```{data output.var="b",format="binary"}
H4sIAAAAAAAAA4vgYmBgYGZgZgNiViCTgTU0xE3XAigmDOQwAfE7KM3IwMLACaT5
kvNzCxKTS+Iz80qKUwvRZFmSEotToWK8YHEI/Q+kE2SVAwcDGNh/QKUhaph5wc6B
6GOCsFmQzWfLSS1LzSkGsgTAshBRxkQYIwnGSEbTyJqck1gM0wc3LQ3okfwiiN2o
yvMSc1NhypmwWANWzvIfzUCulMSSRL20IqBmNAM5i/LL9WCGgnzJ1AAk/v///xdk
OwBVQunahwEAAA==
```

The decoded contents of the chunk will be placed into a raw vector `b`. Note that `format` must be specified as `'binary'` and that we can optionally choose `echo=FALSE` to avoid including lots of ugly encoded content in our formatted document.

The contents of the raw vector `b` must be written to a file before they can be read back into the Rmarkdown session:

```{r}
writeBin(b,"test_output.RDS")
```

We can skip this writing step by specifying the `output.file` chunk option instead of `output.var`:

```{data output.file="test_output.RDS",format="binary"}
H4sIAAAAAAAAA4vgYmBgYGZgZgNiViCTgTU0xE3XAigmDOQwAfE7KM3IwMLACaT5
kvNzCxKTS+Iz80qKUwvRZFmSEotToWK8YHEI/Q+kE2SVAwcDGNh/QKUhaph5wc6B
6GOCsFmQzWfLSS1LzSkGsgTAshBRxkQYIwnGSEbTyJqck1gM0wc3LQ3okfwiiN2o
yvMSc1NhypmwWANWzvIfzUCulMSSRL20IqBmNAM5i/LL9WCGgnzJ1AAk/v///xdk
OwBVQunahwEAAA==
```

Specifying the `output.file` chunk option is generally the most practical way of working with binary data.

Finally, this file can be read back into the Rmarkdown session:

```{r}
readRDS("test_output.RDS")
```

And here too you can use a loader function:

```{data output.var="b",format="binary",loader.function=readRDS}
H4sIAAAAAAAAA4vgYmBgYGZgZgNiViCTgTU0xE3XAigmDOQwAfE7KM3IwMLACaT5
kvNzCxKTS+Iz80qKUwvRZFmSEotToWK8YHEI/Q+kE2SVAwcDGNh/QKUhaph5wc6B
6GOCsFmQzWfLSS1LzSkGsgTAshBRxkQYIwnGSEbTyJqck1gM0wc3LQ3okfwiiN2o
yvMSc1NhypmwWANWzvIfzUCulMSSRL20IqBmNAM5i/LL9WCGgnzJ1AAk/v///xdk
OwBVQunahwEAAA==
```

```{r}
b
```

## GPG-encoded data chunks

GPG chunks work similar to base64 chunks except that one must specify `encoding='gpg'`. To demonstrate this functionality, we first import into the GPG keyring the private key previously used to encode some data. We would never do this in a real use case, but this is practical for this vignette and it demonstrates another use of `data` chunks.

```{data key,output.file="key",eval=requireNamespace("gpg") && params$run_gpg}
-----BEGIN PGP PRIVATE KEY BLOCK-----

[encoded data, 4940 characters]
Ng9Af4cf3nf9UzesAVbSy1gtvlZIyX0HwtZNVLNJSS4=
=C6UF
-----END PGP PRIVATE KEY BLOCK-----
```

This key can then be imported into the keyring:

```{r importkey,eval=requireNamespace("gpg") && params$run_gpg}
gpg::gpg_import("key")
```

Now that we have this private key, we can decode some binary data encrypted using this key:

```{data t3,format="binary",encoding="gpg",output.var="d",loader.function=readRDS,eval=requireNamespace("gpg") && params$run_gpg}
-----BEGIN PGP MESSAGE-----
Version: GnuPG v2

hQGMA9TPonHna5j3AQv7BIPNOSR/024iE0Gj3DCo3DvLvj/oEJ29XORHBkn4nul1
+zaRV5E/K4LCKxkkAEx/+FdM72x1hV5FF5Vf0FSet1RHiOOXPuChEEzRHOubkh/U
gw44Q72d8Dp6TOJ+1KT5k/fdkVKsOZRSttL8hvxqC4nyfObF0CkIoG+Kfx+kkYqu
araVWqNtcb3/FbtT+ZC0Hip0Ws6IJ8mGOhZdRxZ3S8KUtgf/t7S3Wa75c6L1wolT
R1/WhPgcWB4epLTvHdSmv9qcu/vFXE8SmNE5MV4V2aSTRU7y9WdPW/+XzU2Et4BK
kGyzhkI6q7QzAXFOeD1sn0uaUeH9/BDwn3AJZEXkwN4qaarPpDKjZ9GVE9Gg8521
BYe7AIZwq7sfnF+v1WyxamFYpSSAiNHze00MHPWot3Db+4SpRFYIWlYlZF00HIMo
Qspb4AmIfnNo9zj0RGG7GJoyod8ZrW4RF5iOEUWtyQ5z6LzymGTSdArWOq1fDgYW
tvEgbkbdYsJA6usJ3Zxc0sBJARfA6gDCFF72nGiAoNS98zoFjtD7hznY9DBvCOoF
jzJ8kHfPQPK9/bRVuofUDP+jOoJyf8/7eB6kANNq+XrzoZL0N42zHR2n47xupqYN
GSdsRljTra8a9zWIs9k8E5/79qRvV25c/wPeysulWkzhLCDaCMMVYJvQQ0JgT2L+
/eBtDeAuqkCqiDjeEGGB0Q4Q81IOIHMxUXFPJHvWJE4eJhnRjLmuPCBDQdL0JYTS
Bc48A/eA/cbfUr+4RluY9RLcaUHRPjKT8e7X98VdnSBPGvikVpSjR3zZhPNQs7Vb
C7H5lml4B8FpRgQBFwt2ou8URLRYR82tUa/OcsByW9jxf988YZx57a1hAw==
=m4sT
-----END PGP MESSAGE-----
```

```{r eval=requireNamespace("gpg") && params$run_gpg}
d
```

```{r deletekey,eval=requireNamespace("gpg") && params$run_gpg}
id = gpg::gpg_list_keys("test@test.org")$id
gpg::gpg_delete(id,secret=TRUE)
```

## Reading a data chunk from an external file

One disadvantage of using `data` chunks is that it can make Rmarkdown files long and difficult to navigate if you use lots of data. To facilitate the initial construction of standalone Rmarkdown documents, data chunks can be read from external files using the `external.file` chunk option. The external file must have the text-encoded chunk contents as they would appear in a `data` chunk. The intended use of this option is that large data chunks would be placed initially in external files, but the contents of these files would be placed directly in the `data` chunks before sharing the document with others. 

```{r}
writeLines(c("This is from an external file.","It has two lines."),
           "test_external.txt")
```

```{data ext,output.var="ext",external.file="test_external.txt"}
Content will be ignored with a warning!
```

```{r}
cat(ext)
```

## When not to use data chunks

Though the `knitrdata` package can be a powerful tool, it can also be easily abused by placing very large amounts of data inside Rmarkdown files. This will make the documents very large and difficult to navigate. To avoid this, it is recommended that data files exceeding ~1 MB not be included in Rmarkdown documents, though there is no formal limit on the size of data chunks.

```{r eval=TRUE}
# Clean up test files
file.remove("test.csv","key","test_external.txt","test_output.RDS",
            "test_output2.RDS","test.RDS.base64","test.RDS")
```

