---
title: "Longitudinal analysis pipeline with longdat_disc()"
subtitle: "Time (the proxy for treatment) as a discrete variable"
output: 
  rmarkdown::html_vignette:
    toc: true 
vignette: >
  %\VignetteIndexEntry{longdat_disc_tutorial}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---
## Introduction

This is an example of running ```longdat_disc()```. Note that the time variable (proxy of treatment) here should be discrete. If the time variable is continuous, please apply ```longdat_cont()``` instead. 

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  message = FALSE,
  warning = FALSE,
  rmarkdown.html_vignette.check_title = FALSE
)
```

```{r message=FALSE}
# Load the packages
library(LongDat)
library(kableExtra)
```

## Explaining the input data frame format

The input data frame (called master table) should have the same format as the example data "LongDat_disc_master_table". If you have metadata and feature (eg. microbiome, immunome) data stored in separate tables, you can go to the section [Preparing the input data frame with make_master_table()] below. The function ```make_master_table()``` helps you to create master table from metadata and feature tables.

Now let's have a look at the required format for the input master table. The example below is a dummy longitudinal data set with 3 time points (1, 2, 3). Here we want to see if the treatment has a significant effect on gut microbial abundance or not. 

```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
# Read in the data frame. LongDat_disc_master_table is already lazily loaded.
master <- LongDat_disc_master_table
master %>%
  kableExtra::kbl() %>%
  kableExtra::kable_paper(bootstrap_options = "responsive", font_size = 12) %>% 
  kableExtra::scroll_box(width = "700px", height = "200px")
```
As you can see, the "Individual" is at the first column, and the features (dependent variables), which are gut microbial abundances in this case, are at the end of the table. Any column apart from individual, test_var (e.g. Time_point) and dependent variables will be taken as potential covariates (could be confounder or mediator). For example, here the potential covariates are sex, age, drug A and drug B. **Please avoid using characters that don't belong to [ASCII printable characters](https://www.w3schools.com/charsets/ref_html_ascii.asp) for the column names in the input data frame.**

## Preparing the input data frame with make_master_table()

If you have your input master table prepared already, you can skip this section and go to [Run longdat_disc()] directly. If your metadata and feature (eg. microbiome, immunome) data are stored in two tables, you can create a master table out of them easily with the function ```make_master_table()```.

First, let's take a look at an example of the metadata table. Metadata table should be a data frame whose columns consist of sample identifiers (sample_ID, unique for each sample), individual, time point and other meta data. Each row corresponds to one sample_ID. 
```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
# Read in the data frame. LongDat_disc_metadata_table is already lazily loaded.
metadata <- LongDat_disc_metadata_table
metadata %>%
  kableExtra::kbl() %>%
  kableExtra::kable_paper(bootstrap_options = "responsive", font_size = 12) %>% 
  kableExtra::scroll_box(width = "700px", height = "200px")
```
This example is a dummy longitudinal meatadata with 3 time points for each individual. Besides sample_ID, individual, time point columns, there are also information of sex, age and drugs that individuals take. Here we want to see if the treatment has a significant effect on gut microbial abundance or not. 

Then, let's see how a feature table looks like. Feature table should be a data frame whose columns only consist of sample identifiers (sample_ID) and features (dependent variables, e.g. microbiome). Each row corresponds to one sample_ID. Please do not include any columns other than sample_ID and features in the feature table.
```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
# Read in the data frame. LongDat_disc_feature_table is already lazily loaded.
feature <- LongDat_disc_feature_table
feature %>%
  kableExtra::kbl() %>%
  kableExtra::kable_paper(bootstrap_options = "responsive", font_size = 12) %>% 
  kableExtra::scroll_box(width = "700px", height = "200px")
```
This example is a dummy longitudinal feature data. It stores the gut microbial abundance of each sample.

**To enable the joining process of metadata and feature tables, please pay attention to the following rules.** 

1. The row numbers of metadata and feature tables should be the same.
2. Sample_IDs are unique for each sample (i.e. no repeated sample_ID)
3. Metadata and feature tables have the same sample_IDs. If sample_IDs don't match between the two tables, the joining process will fail.
4. As mentioned above, feature table should include only the columns of sample_ID and features.
5. Avoid using characters that don't belong to [ASCII printable characters](https://www.w3schools.com/charsets/ref_html_ascii.asp) for the column names.

Now let's create a master table and take a look at the result!
```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
master_created <- make_master_table(
metadata_table = LongDat_disc_metadata_table,
feature_table = LongDat_disc_feature_table,
sample_ID = "Sample_ID",
individual = "Individual")

master_created %>%
  kableExtra::kbl() %>%
  kableExtra::kable_paper(bootstrap_options = "responsive", font_size = 12) %>% 
  kableExtra::scroll_box(width = "700px", height = "200px")
```

The table "master_created" is just the same as the table "master" or "LongDat_disc_master_table" in the previous section, with the "Individual" as the first column, and the features (dependent variables), which are gut microbial abundances in this case, are at the end of the table. Any column apart from individual, test_var (e.g. Time_point) and dependent variables will be taken as potential covariates (could be confounder or mediator). For the details of the arguments, please read the help page of this function by using ```?make_master_table```.


OK, now we're ready to run ```longdat_disc()```!

## Run longdat_disc()

The input is the example data frame LongDat_disc_master_table (same as "master" or "master_created" in the previous sections), and the data_type is "count" since the dependent variables (features, in this case they're gut microbial abundance) are count data. The "test_var" is the independent variable you're testing, and here we're testing "Time_point" (time as the proxy for treatment). The variable_col is 7 because the dependent variables start at column 7. And the fac_var mark the columns that aren't numerical. For the details of the arguments, please read the help page of this function by using ```?longdat_disc```.

The run below takes less than a minute to complete. When data_type equals to "count", please remember to set seed (as shown below) so that you'll get reproducible randomized control test.

```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
# Run longdat_disc() on LongDat_disc_master_table
set.seed(100)
test_disc <- longdat_disc(input = LongDat_disc_master_table,
data_type = "count",
test_var = "Time_point",
variable_col = 7, 
fac_var = c(1:3))
```
If you have completed running the function successfully, you'll see the message "Finished successfully!" at the end. The results are stored in list format.

## Results

The major output from ```longdat_disc()``` include a result table and a covariate table. If you have count data (data_type equals to "count"), then there are chances that you get a third table "randomized control table". If you specify data_type as either "measurement" or "others", then you'll get a "Normalize_method" table. For more details about the "randomized control table" and "Normalize_method" table, please read the help page of this function by using ```?longdat_disc```.

### Result table

Let's have a look at the result table first. 
```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
# The first dataframe in the list is the result table
result_table <- test_disc[[1]]
result_table %>%
  kableExtra::kbl() %>%
  kableExtra::kable_paper(bootstrap_options = "responsive", font_size = 12, position = "center") %>% 
  kableExtra::scroll_box(width = "700px")
```

The second and third columns show the prevalence and mean abundance of each feature According to the "Signal" column, treatment is a significant predictor for BacteriumA as it shows "OK_nrc" (which represents "OK and not reducible to covariate"), meaning that there are potential covariates, however there is an effect of time (proxy of treatment) and it is independent of those of covariates. To find out what the covariates are, you'll need to see the covariate table, but we'll get to that later. On the other hand, the signal for BacteriumC is "OK_nc" (which represents OK and no covariate), meaning that the abundance of BacteriumC alters significantly throughout time (proxy of treatment), and that there is no potential covariate. As for BacteriumB, time (proxy of treatment) has no effect on its abundance.

The following columns "Effect_1_2", "Effect_1_3" and "Effect_2_3" describe how features (dependent variables) change from time point a to b. Here we can tell that BacteriumA decreases from time point 1 to 2, and then enriched from time point 2 to 3. From the columns "EffectSize_1_2" and "EffectSize_2_3", we know that the effect size are -0.6 and 0.8. The most relevant information for users ends here, which are listed from the first column to "EffectSize" columns. 

Then the following columns contain the details of model test p values ("Null_time_model_q"), the post-hoc test p values (Post.hoc_q_1_2, Post.hoc_q_1_3 and Post.hoc_q_2_3). For more detailed information of the columns in the result table, please refer to the help page by using ```?longdat_disc```. 

The explanation of each type of "Signal" is listed below.

|Signal|Meaning|Explanation|          
|---|--------|-------------|
|NS| Non-significant | There’s no effect of time.|
|OK_nc |OK and no covariate | There’s an effect of time and there’s no potential covariate.|
|OK_d| OK but doubtful |  There’s an effect of time and there’s no potential covariate, however the confidence interval of the test_var estimate in the model test includes zero, and thus it is doubtful. Please check the raw data (e.g., plot feature against time) to confirm if there is real effect of time.|
|OK_nrc| OK and not reducible to covariate | There are potential covariates, however there’s an effect of time and it is independent of those of covariates. |
|EC| Entangled with covariate | There are potential covariates, and it isn’t possible to conclude whether the effect is resulted from time or covariates.|
|RC| Effect reducible to covariate | There’s an effect of time, but it can be reduced to the covariate effects.|

### Covariate table

Next, let's take a look at the covariate table.

```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
# The second dataframe in the list is the covariate table
covariate_table <- test_disc[[2]]
covariate_table %>%
  kableExtra::kbl() %>%
  kableExtra::kable_paper(bootstrap_options = "responsive", font_size = 12, position = "center") %>% 
  kableExtra::scroll_box(width = "700px")
```

The columns of this covariate table are grouped every three columns. "Covariate1" is the name of the covariate, while "Covariate_type1" is the covariate type of covariate1, that is, if the effect of time is reducible to covariate1. "Effect_size1" is the effect size of the dependent variable values between different levels of covariate1. Here, the effect of time isn't reducible to covariate1, so we don't need to worry about the covariate effect of covariate1 on bacteriumA. If there are more than one covariates, they will be listed along the rows of each dependent variable.

### Result interpretation

From the result above, we see that the potential covariate for BacteriumA is DrugB, but we don't need to worry about the effect of time (proxy for the treatment) being able to reduce to DrugB since it's covariate type is "not reducible to covariate", meaning that the effect of time is independent from the effect of DrugB. With this information, we confirm that the treatment can solely explain the changes in BacteriumA abundance. All together, the treatment induces significant changes on the abundance of BacteriumA and BacteriumC, while causing no alteration in that of BacteriumB.

## Plotting the result

Finally, we can plot the result with the function ```cuneiform_plot()```. The required input is a result table from ```longdat_disc()``` (or any table with the same format as a result table does).

```{r message=FALSE, tidy=TRUE, tidy.opts=list(width.cutoff=80)}
test_plot <- cuneiform_plot(result_table = test_disc[[1]],
                            x_axis_order = c("Effect_1_2", "Effect_2_3",                             "Effect_1_3"),
                            title_size = 15)
```

```{r fig.asp = 0.6, fig.width = 8, out.width = "80%", dpi = 600}
test_plot
```


Here we can see the result clearly from the cuneiform plot. It shows the features whose signals are not "NS". The left panel displays the effects in each time interval. Red represents positive effect size while blue describes negative one (colors can be customized by users). Signficant signals are indicated by solid shapes, whereas insignificant signals are denoted by transparent ones. The right panel displays the covariate status of each feature, and users can remove it by specifying ```covariate_panel = FALSE```. For more details of the arguments, please read the help page of this function by using ```?cuneiform_plot```.

### Wrap-up

This tutorial ends here! If you have any further questions and can't find the answers in the vignettes or help pages, please contact the author (Chia-Yu.Chen@mdc-berlin.de).


