# Introduction to the tinkr package

    library(tinkr)

The goal of tinkr is to convert (R)Markdown files to XML and back to allow their editing with xml2 (XPath!) instead of numerous complicated regular expressions. If these words mean nothing to you, see our list of resources to get started.

## Use Cases

Possible applications are R scripts using tinkr, and XPath via xml2 to:

- change levels of headers, cf our `roweb2_headers.R` script and pull request #279 to roweb2;
- change chunk labels and options;
- extract all runnable code (including inline code);
- insert arbitrary Markdown elements;
- modify link URLs;
- your idea, please report use cases!

## Workflow

Only the body of the (R) Markdown file is cast to XML, using the Commonmark specification via the `commonmark` package. Frontmatter metadata can be stored in the YAML, TOML or JSON format. YAML-formatted metadata could be edited using the `yaml` package, JSON-formatted metadata could be edited using the `jsonlite` package. which is not the goal of this package.

We have created an R6 class object called yarn to store the representation of both the frontmatter and the XML data, both of which are accessible through the `$body` and `$frontmatter` elements. The field called `$frontmatter_format` contains the format of the metadata (YAML, TOML, or JSON). In addition, the namespace prefix is set to “md” in the `$ns` element.

You can perform XPath queries using the `$body` and `$ns` elements:

    library("tinkr")
    library("xml2")
    path <- system.file("extdata", "example1.md", package = "tinkr")
    head(readLines(path))
    #> [1] "---"
    #> [2] "title: \"What have these birds been studied for? Querying science outputs with R\""
    #> [3] "slug: birds-science"
    #> [4] "authors:"
    #> [5] "  - name: Maëlle Salmon"
    #> [6] "    url: https://masalmon.eu/"
    ex1 <- tinkr::yarn$new(path)
    # find all ropensci.org blog links
    xml_find_all(
      x = ex1$body,
      xpath = ".//md:link[contains(@destination,'ropensci.org/blog')]",
      ns = ex1$ns
    )
    #> {xml_nodeset (7)}
    #> [1] <link destination="https://ropensci.org/blog/2018/08/21/birds-radolfzell/ ...
    #> [2] <link destination="https://ropensci.org/blog/2018/09/04/birds-taxo-traits ...
    #> [3] <link destination="https://ropensci.org/blog/2018/08/21/birds-radolfzell/ ...
    #> [4] <link destination="https://ropensci.org/blog/2018/08/14/where-to-bird/" t ...
    #> [5] <link destination="https://ropensci.org/blog/2018/08/21/birds-radolfzell/ ...
    #> [6] <link destination="https://ropensci.org/blog/2018/08/28/birds-ocr/" title ...
    #> [7] <link destination="https://ropensci.org/blog/2018/09/04/birds-taxo-traits ...

## Examples

### Markdown

This is a basic example. We read “example1.md”, change all headers 3 to headers 1, and save it back to md. Because the xml2 objects are passed by reference, manipulating them does not require reassignment.

    library("magrittr")
    library("tinkr")
    # From Markdown to XML
    path <- system.file("extdata", "example1.md", package = "tinkr")
    # Level 3 header example:
    cat(tail(readLines(path, 40)), sep = "\n")
    #> ### Getting a list of 50 species from occurrence data
    #>
    #> For more details about the following code, refer to the [previous post
    #> of the series](https://ropensci.org/blog/2018/08/21/birds-radolfzell/).
    #> The single difference is our adding a step to keep only data for the
    #> most recent years.
    ex1  <- tinkr::yarn$new(path)
    # transform level 3 headers into level 1 headers
    ex1$body %>%
      xml2::xml_find_all(xpath = ".//md:heading[@level='3']", ex1$ns) %>%
      xml2::xml_set_attr("level", 1)

    # Back to Markdown
    tmp <- tempfile(fileext = "md")
    ex1$write(tmp)
    # Level three headers are now Level one:
    cat(tail(readLines(tmp, 40)), sep = "\n")
    #> # Getting a list of 50 species from occurrence data
    #>
    #> For more details about the following code, refer to the [previous post
    #> of the series](https://ropensci.org/blog/2018/08/21/birds-radolfzell/).
    #> The single difference is our adding a step to keep only data for the
    #> most recent years.
    unlink(tmp)

### R Markdown

For R Markdown files, to ease editing of chunk label and options, `to_xml` munges the chunk info into different attributes. E.g. below you see that `code_blocks` can have a `language`, `name`, `echo` attributes.

    path <- system.file("extdata", "example2.Rmd", package = "tinkr")
    rmd <- tinkr::yarn$new(path)
    rmd$body
    #> {xml_document}
    #> <document xmlns="http://commonmark.org/xml/1.0">
    #>  [1] <code_block xml:space="preserve" language="r" name="setup" include="FALS ...
    #>  [2] <heading level="2">\n  <text xml:space="preserve">R Markdown</text>\n</h ...
    #>  [3] <paragraph>\n  <text xml:space="preserve">This is an </text>\n  <striket ...
    #>  [4] <paragraph>\n  <text xml:space="preserve">When you click the </text>\n   ...
    #>  [5] <code_block xml:space="preserve" language="r" name="" eval="TRUE" echo=" ...
    #>  [6] <heading level="2">\n  <text xml:space="preserve">Including Plots</text> ...
    #>  [7] <paragraph>\n  <text xml:space="preserve">You can also embed plots, for  ...
    #>  [8] <code_block xml:space="preserve" language="python" name="" fig.cap="&quo ...
    #>  [9] <code_block xml:space="preserve" language="python" name="">plot(pressure ...
    #> [10] <paragraph>\n  <text xml:space="preserve">Non-RMarkdown blocks are also  ...
    #> [11] <code_block info="bash" xml:space="preserve" name="">echo "this is an un ...
    #> [12] <code_block xml:space="preserve" name="">This is an ambiguous code block ...
    #> [13] <paragraph>\n  <text xml:space="preserve">Note that the </text>\n  <code ...
    #> [14] <table>\n  <table_header>\n    <table_cell align="left">\n      <text xm ...
    #> [15] <paragraph>\n  <text xml:space="preserve">blabla</text>\n</paragraph>

Note that all of the features in tinkr work for both Markdown and R Markdown.

### Inserting new Markdown elements

Inserting new nodes into the AST is surprisingly difficult if there is a default namespace, so we have provided a method in the yarn object that will take plain Markdown and translate it to XML nodes and insert them into the document for you. For example, you can add a new code block:

    path <- system.file("extdata", "example2.Rmd", package = "tinkr")
    rmd <- tinkr::yarn$new(path)
    xml2::xml_find_first(rmd$body, ".//md:code_block", rmd$ns)
    #> {xml_node}
    #> <code_block space="preserve" language="r" name="setup" include="FALSE" eval="TRUE">

    new_code <- c(
      "```{r xml-block, message = TRUE}",
      "message(\"this is a new chunk from {tinkr}\")",
      "```")
    new_table <- data.frame(
      package = c("xml2", "xslt", "commonmark", "tinkr"),
      cool = TRUE
    )
    # Add chunk into document after the first chunk
    rmd$add_md(new_code, where = 1L)
    # Add a table after the second chunk:
    rmd$add_md(knitr::kable(new_table), where = 2L)
    # show the first 21 lines of modified document
    rmd$head(21)
    #> ---
    #> title: "Untitled"
    #> author: "M. Salmon"
    #> date: "September 6, 2018"
    #> output: html_document
    #> ---
    #>
    #> ```{r setup, include=FALSE, eval=TRUE}
    #> knitr::opts_chunk$set(echo = TRUE)
    #> ```
    #>
    #> ```{r xml-block, message=TRUE}
    #> message("this is a new chunk from {tinkr}")
    #> ```
    #>
    #> | package                    | cool                |
    #> | :------------------------- | :------------------ |
    #> | xml2                       | TRUE                |
    #> | xslt                       | TRUE                |
    #> | commonmark                 | TRUE                |
    #> | tinkr                      | TRUE                |

## Background / pre-requisites

If you are not closely following one of the examples provided, what background knowledge do you need before using tinkr?

- That XPath, a language for querying XML & HTML, exists, and some basics.
- Basics of how xml2 works: how to find, replace, remove nodes etc.
- How to use R6 classes… although reading the examples should help you get the gist.
- If you are not happy with our default stylesheet, then understanding XSLT will help you create your own. Refer to this good resource on XSLT for XML transformations.

## Loss of Markdown style

### General principles and solution

The (R)md to XML to (R)md loop on which `tinkr` is based is slightly lossy because of Markdown syntax redundancy, so the loop from (R)md to R(md) via `to_xml` and `to_md` will be a bit lossy. For instance

- lists can be created with either “+”, “-” or “*“. When using `tinkr`, the (R)md after editing will only use”-” for lists.

- Links built like `[word][smallref]` with a bottom anchor `[smallref]: URL` will have the anchor moved to the bottom of the document.

- Characters are escaped (e.g. “[” when not for a link).

- GitHub tickboxes are preserved (only for `yarn` objects)

- Block quotes lines all get “>” whereas in the input only the first could have a “>” at the beginning of the first line.

- For tables see the next subsection.

Such losses make your (R)md different, and the git diff a bit harder to parse, but should not change the documents your (R)md is rendered to. If it does, report a bug in the issue tracker!

A solution to not loose your Markdown style, e.g. your preferring “*” over “-” for lists is to tweak our XSL stylesheet and provide its filepath as `stylesheet_path` argument to `to_md`.

### The special case of tables

- Tables are supposed to remain/become pretty after a full loop `to_xml` + `to_md`. If you notice something amiss, e.g. too much space compared to what you were expecting, please open an issue.

### LaTeX equations

While Markdown parsers like pandoc know what LaTeX is, commonmark does not, and that means LaTeX equations will end up with extra markup due to commonmark’s desire to escape characters.

However, if you have LaTeX equations that use either `$` or `$$` to delimit them, you can protect them from formatting changes with the `$protect_math()` method (for users of the `yarn` object) or the `protect_math()` function (for those using the output of `to_xml()`). Below is a demonstration using the `yarn` object:

    path <- system.file("extdata", "math-example.md", package = "tinkr")
    math <- tinkr::yarn$new(path)
    math$tail() # malformed
    #>
    #> $$
    #> Q\_{N(norm)}=\\frac{C\_N +C\_{N-1}}2\\times
    #> \\frac{\\sum *{i=N-n}^{N}Q\_i} {\\sum*{j=N-n}^{N}{(\\frac{C\_j+C\_{j-1}}2)}}
    #> $$
    math$protect_math()$tail() # success!
    #>
    #> $$
    #> Q_{N(norm)}=\frac{C_N +C_{N-1}}2\times
    #> \frac{\sum _{i=N-n}^{N}Q_i} {\sum_{j=N-n}^{N}{(\frac{C_j+C_{j-1}}2)}}
    #> $$

Note, however, that there are a few caveats for this:

1. The dollar notation for inline math must be adjacent to the text. E.G. `$\alpha$` is valid, but `$ \alpha$` and `$\alpha $` are not valid.

2. We do not currently have support for bracket notation

3. If you use a postfix dollar sign in your prose (e.g. BASIC commands or a Burroughs-Wheeler Transformation demonstration), you must be sure to either use punctuation after the trailing dollar sign OR format the text as code. (i.e. ``INKEY$`` is good, but `INKEY$` by itself is not good and will be interpreted as LaTeX code, throwing an error:

       path <- system.file("extdata", "basic-math.md", package = "tinkr")
       math <- tinkr::yarn$new(path)
       math$head(15) # malformed
       #> ---
       #> title: basic math
       #> ---
       #>
       #> BASIC programming can make things weird:
       #>
       #> - Give you $2 to tell me what INKEY$ means.
       #> - Give you $2 to *show* me what INKEY$ means.
       #> - Give you $2 to *show* me what `INKEY$` means.
       #>
       #> Postfix dollars mixed with prefixed dollars can make things weird:
       #>
       #> - We write $2 but say 2$ verbally.
       #> - We write $2 but *say* 2$ verbally.
       math$protect_math() #error
       #> Error: Inline math delimiters are not balanced.
       #>
       #> HINT: If you are writing BASIC code, make sure you wrap variable
       #>       names and code in backtics like so: `INKEY$`.
       #>
       #> Below are the pairs that were found:
       #>            start...end
       #>            -----...---
       #>  Give you $2 to ... me what INKEY$ means.
       #>  Give you $2 to ... 2$ verbally.
       #> We write $2 but ...

4. use of `$` as currency will still work, but there is a caveat that mixing this inline math broken across lines will cause problems:

       # this will be mal-formed
       bad <- "It's 5:45 and I've got $5.45 in my pocket.\nThe __area of a circle__ is $A =\n \\pi r^2$, where\n$\\pi$ is irrational when it hasn't had its coffee."
       fails <- tinkr::yarn$new(textConnection(bad))
       fails$show()
       #> It's 5:45 and I've got $5.45 in my pocket.
       #> The **area of a circle** is $A =
       #> \\pi r^2$, where
       #> $\\pi$ is irrational when it hasn't had its coffee.
       fails$
         protect_math()$
         show()
       #> Error: Inline math delimiters are not balanced.
       #>
       #> HINT: If you are writing BASIC code, make sure you wrap variable
       #>       names and code in backtics like so: `INKEY$`.
       #>
       #> Below are the pairs that were found:
       #>                                      start...end
       #>                                      -----...---
       #> It's 5:45 and I've got $5.45 in my pocket....\pi r^2$, where
       #>                                    is $A =...
       # This works
       good <- "It's 5:45 and I've got $5.45 in my pocket.\nThe __area of a circle__ is $A = \\pi r^2$, where\n$\\pi$ is irrational when it hasn't had its coffee."
       works <- tinkr::yarn$new(textConnection(good))
       works$show()
       #> It's 5:45 and I've got $5.45 in my pocket.
       #> The **area of a circle** is $A = \\pi r^2$, where
       #> $\\pi$ is irrational when it hasn't had its coffee.
       works$
         protect_math()$
         show()
       #> It's 5:45 and I've got $5.45 in my pocket.
       #> The **area of a circle** is $A = \pi r^2$, where
       #> $\pi$ is irrational when it hasn't had its coffee.