# TidyMultiqc

## Test Data

In this example we’ll be using the WGS example report used in the MultiQC documentation. You can find view the report in your browser here, and find the associated `multiqc_data.json` file here. Feel free to download this file and use it to follow along with this vignette.

In the rest of the vignette we will use the `multiqc_data_path` variable to indicate the path to this `multiqc_data.json` file. Feel free to set this variable to the path to this file on your system.

## Setup

First, install the package using:

    install.packages("TidyMultiqc")

Then if you want, you can load the package:

    library(TidyMultiqc)

However, for the sake of this tutorial we will explicitly use namespaced functions, so we won’t be using `library` a great deal.

## Basic Usage

The main entry point to the `TidyMultiqc` package is the function `load_multiqc`. A basic invocation of this function looks like this:

(note the arrow for scrolling through columns)

    df = TidyMultiqc::load_multiqc(multiqc_data_path)
    df

We’ve now generated a `tibble` (a kind of data frame), whose rows are samples in the QC report, and whose columns are QC data and metadata about these samples.

By default this function only returns the “general” statistics, which are the ones in the “General Statistics” table at the top of the MultiQC report. In TidyMultiqc, these statistics are all prefixed by `general.` We can also extract the “raw” statistics, which includes some fields normally hidden from the report. These statistics will have the prefix `raw.<toolname>.` where `<toolname>` is the QC tool used to calculate it.

    TidyMultiqc::load_multiqc(multiqc_data_path, sections = 'raw')

Often you won’t care about fields like `raw.qualimap_bamqc_genome_results.bam_file`, the path to the original BAM file, but ‘raw’ at least provides this option.

You can also combine both `general` and `raw` sections by passing in a longer vector:

    df_both = TidyMultiqc::load_multiqc(multiqc_data_path, sections = c('raw', 'general'))
    ncol(df_both)
    #> [1] 388

That’s a lot of columns!

## Uses

This section will briefly talk about some downstream use-cases for this package.

### Plotting

One use for this data frame is creating QC plots. For example, to visualise the duplication rate per sample:

    library(magrittr)

    df %>%
      ggplot2::ggplot(ggplot2::aes(x=metadata.sample_id, y=general.percent_duplication)) +
      ggplot2::geom_col()

Of course, this is basically just replicating a plot already in the MultiQC report, but now we can customise it how we like!

### Hypothesis Testing

With all this data, we might also want to test a specific hypothesis! For example, we might want to test the hypothesis that the mean GC content is the same as the mean GC content in the human genome (41%). If we assume that GC content is normally distributed, we can do the following:

    t.test(df$general.percent_gc, mu=41)
    #>
    #>  One Sample t-test
    #>
    #> data:  df$general.percent_gc
    #> t = -1, df = 5, p-value = 0.3632
    #> alternative hypothesis: true mean is not equal to 41
    #> 95 percent confidence interval:
    #>  40.40490 41.26176
    #> sample estimates:
    #> mean of x
    #>  40.83333

It seems that we cannot reject this hypothesis, so these may well be human samples!

## Extracting Metadata

It may be the case that your samples have important metadata that you want in your data frame. For example, it is common for the sample names or file names to be composed of a number of metadata fields, and indeed the report we are working with has IDs such as `P4107_1003`, which is composed of two identifiers.

To include this metadata in our output, we need to provide the `find_metadata` argument to `load_multiqc`, which is a function that is called for each sample, and which returns a named vector of metadata fields for that sample. It also gets passed the entire parsed MultiQC JSON report, so the function can traverse the structure as it wants to extract metadata.

### From the File Name

Here is an example that parses the input file names to annotate additional metadata. Notice that the first argument our function is passed is a string which is the sample identifier for a sample, and how in this example we ignore the `parsed` argument. Also notice that the names we give to our return value (“batch” and “sample”) are prefixed by “metadata” to become the final names in the data frame.

    TidyMultiqc::load_multiqc(
      multiqc_data_path,
      find_metadata = function(sample, parsed) {
        # Split the sample ID to obtain some metadata
        segments <- stringr::str_split(sample, "_")[[1]]
        c(
          batch = segments[[1]],
          sample = segments[[2]]
        )
      }
    )

### From the File Path

We can extend this approach, but this time actually look up the file paths within the `report_data_sources` section of the MultiQC report:

    TidyMultiqc::load_multiqc(
      multiqc_data_path,
      find_metadata = function(sample, parsed) {
        # This gives us the path to the fastqc output file
        filepath = parsed$report_data_sources$FastQC$all_sections[[sample]]
        # Split into path segments
        path_segments = stringr::str_split(filepath, "/")[[1]]
        # The filename is the last path segment
        filename = dplyr::last(path_segments)
        # Split the filename using dots and underscores
        name_segments = stringr::str_split(filename, "[_\\.]")[[1]]
        # Arbitrarily assign names for the outputs
        name_segments %>% purrr::set_names(LETTERS[1:length(name_segments)])
      }
    )

Of course in a real application we would choose specific names for each field.

### From MultiQC Configuration

Finally, we might want to include metadata that doesn’t relate to the sample at all. For example, MultiQC has a number of report fields prefixed by `config_` that we might want to store:

    TidyMultiqc::load_multiqc(
      multiqc_data_path,
      find_metadata = function(sample, parsed) {
        parsed[c(
          "config_creation_date",
          "config_version"
        )]
      }
    )

## Extracting Plot Data

### Motivation

It is occasionally useful to extract QC data from the MultiQC plots. For example, let’s say we want to calculate the median quality score of every base in each sample. Unfortunately, MultiQC provides no numerical summary statistic for the read quality, it only has mapping quality and pass/fails for the per-base sequence quality:

    df_both %>% dplyr::select(dplyr::contains('quality'))

However, our MultiQC report does have plots that contain this information. In particular, let’s look at the “Per Sequence Quality Scores” plot.

### Listing Plots

Firstly, we need the ID of the plot we want. This isn’t necessarily obvious from just looking at the report, so we can use a utility function here:

    TidyMultiqc::list_plots(multiqc_data_path)

Now, we know we want the “Per Sequence Quality Scores” plot, and by looking at the data frame above we can tell that the corresponding ID is `fastqc_per_sequence_quality_scores_plot`.

### Loading Plots

Now that we have the plot ID, we can load the plot data. First, we need to tell `TidyMultiqc` to load include some plots by using `sections = "plot"` (you can load other sections at the same time, as explained above). Also, we need to pass the plot ID from the previous step into the `plots` argument:

    df = TidyMultiqc::load_multiqc(
      multiqc_data_path,
      sections = 'plot',
      plots = "fastqc_per_sequence_quality_scores_plot"
    )
    df

We now have the plot data, but it’s not in a very usable form! This is because each sample has an entire data frame of plot data. At this point if you’re comfortable using `dplyr` and `tidyr` to deal with nested data frames, you probably know what to do. Otherwise, read on.

### Converting Plot Data

Recall that we are after the median quality score of each sample. First, we should look at the plot data for a single sample to know what we’re dealing with:

    df$plot.fastqc_per_sequence_quality_scores_plot[[1]]

So each data frame is a set of x, y pairs. As it’s a histogram plot, we know that the `x` value is the quality score, and `y` is the number of times that score has been counted.

### Unnesting

One possible way to process this nested data is to use `tidyr`:

    df %>%
      tidyr::unnest(cols = plot.fastqc_per_sequence_quality_scores_plot)

As you can see, if we unnest in this way, we now have multiple rows for the same sample, which is a bit confusing (and not Tidy). However, if we use `group_by` and then `summarise`, this can be a useful way to calculate summary statistics. For example, if we want the total number of reads, we could do the following:

    df %>%
      tidyr::unnest(cols = plot.fastqc_per_sequence_quality_scores_plot) %>%
      dplyr::group_by(metadata.sample_id) %>%
      dplyr::summarise(total_reads = sum(y))

### Mapping with Purrr

Although unnesting worked well in this example, it can get a bit awkward for more complex operations. In these cases we can use `purrr`. Refer to the next section for an example that compares both approaches. The below example sums the number of reads for each sample, as we have done above, but this time it uses `purrr::map_dbl` to map over the list of data frames:

    df %>%
      dplyr::mutate(
        total_reads = purrr::map_dbl(plot.fastqc_per_sequence_quality_scores_plot, ~sum(.$y)),
        plot.fastqc_per_sequence_quality_scores_plot = NULL
      )

### Properly Handling Histogram Data

Of course, we actually want to find the median here, which is a bit harder. Luckily there exists a package called `HistDat` for generating summary statistics from histogram-type data. You can check out the package’s manual and vignettes here, but in brief, we want to convert each of these plot data frames into a `HistDat` object, which we can do using the same strategies as before. Then, using `HistDat`, we can calculate our summary statistics in one of the two ways mentioned above.

Using the `tidyr` approach, we can unnest the plot data, group it, create a `HistDat` object for each group, and then produce new columns using the new `hist` column:

    df %>%
      tidyr::unnest(cols = plot.fastqc_per_sequence_quality_scores_plot) %>%
      dplyr::group_by(metadata.sample_id) %>%
      dplyr::mutate(hist = list(HistDat::HistDat(vals = x, counts = y)), .keep = "unused") %>%
      dplyr::mutate(
        mean_coverage = hist %>% dplyr::first() %>% mean(),
        median_coverage = hist %>% dplyr::first() %>% median(),
        max_coverage = hist %>% dplyr::first() %>% max(),
        hist= NULL
      ) %>%
      dplyr::slice(1)

Alternatively, using the `purrr` method, we can just map each plot data frame into a row of summary statistics. Much neater!

    df %>%
      dplyr::mutate(
        purrr::map_dfr(plot.fastqc_per_sequence_quality_scores_plot, function(plot_df){
          hist = HistDat::HistDat(vals=plot_df$x, counts = plot_df$y)
          list(
            mean_coverage = mean(hist),
            median_coverage = median(hist),
            max_coverage = max(hist)
          )
        }),
        plot.fastqc_per_sequence_quality_scores_plot = NULL
      )

### Custom Plot Parsers

So far, we have used `TidyMultiqc`’s built-in parsers. Each parser is a function that converts the plot JSON into a list of data frames. However, it is possible that we might encounter a new type of plot that is not yet implemented in `TidyMultiqc`. If this happens, the first thing you should do is file an issue against this package.

Then, if you’re daring, you can try to implement a parser. For your reference, you can refer to the existing parsers in the source code. Then, you can pass in your custom parser using the `plot_parsers` argument to `load_multiqc`:

    TidyMultiqc::load_multiqc(
      multiqc_data_path,
      sections = 'plot',
      plots = "fastqc_per_sequence_quality_scores_plot",
      plot_parsers = list(
        # This fake parser function takes a plot and just returns the iris dataset
        xy_line = function(plot_data, name){
          list(
            sample_1 = list(
              plot_name = list(iris)
            )
          )
        }
      )
    )

Finally, if your parser works, please submit a pull request to `TidyMultiqc` to share this with everyone!

Using the strategies and patterns explained in this section, you should be in good stead to handle whatever plots MultiQC throws at you.