# Molecular Acquisition, Cleaning, and Evaluation in R (MACER) information and example.

# Description:

This repository contains the MACER package located at rgyoung6/MACER. The Molecular Acquisition, Cleaning, and Evaluation in R (MACER) is a tool to assemble, align, trim, and evaluate molecular sequence datasets from BOLD and GenBank.

# Citation

Young RG, Gill R, Gillis D, Hanner RH (2021) Molecular Acquisition, Cleaning and Evaluation in R (MACER) - A tool to assemble molecular marker datasets from BOLD and GenBank. Biodiversity Data Journal 9: e71378. <https://doi.org/10.3897/BDJ.9.e71378>

# Functions:
**auto_seq_download()**<br/>
**create_fastas()**<br/>
**align_to_ref()**<br/>
**barcode_clean()**<br/>

# Installation 

MACER can be installed three ways.

## 1. Install from CRAN

install.packages('MACER’)

## 2. Install via GitHub
Run the following commands in your R terminal...<br/>

> install.packages("devtools")<br/>
> library(devtools)<br/>
> devtools::install_github("rgyoung6/MACER")<br/>

**Note:** the first command to install the "devtools" may not be necessary if already installed.<br/>

## 3. Install through download from GitHub.
Navigate to the [MACER](https://github.com/rgyoung6/MACER) GitHub page. Download the files associated with this page to your local computer and place them somewhere in the main file folder named MACER. Then run the following command pointing to that location on your local computer by replacing the HERE with the path in the below command...<br/>
> library("MACER", lib.loc="HERE")<br/>

# Function Descriptions:

## auto_seq_download()
This function takes a list of genera, as supplied by the user, and searches and downloads molecular sequence data from BOLD and Genbank.

## create_fastas()
This function takes an output file from the auto_seq_download() function and a user supplied table of taxa and marker names desired and outputs fasta files with target records.

## align_to_ref()
This function takes a FASTA file with target sequences and aligns them against a reference sequence submitted to the program. The output is an aligned fasta file that is trimmed to the length of the reference sequence. Sequences without full coverage (records having sequences with leading or trailing gaps) are removed. Finally, internal gaps are removed from the sequence based on the submitted multiple sequence alignment (MSA) percent coverage of the character position as provided in the pigl argument supplied by the user.

## barcode_clean()
This function takes an input fasta file and removes genus level outliers and species outliers based on the 1.5 x greater than the interquartile range both at the genus and species level. It also, if selected, checks the sequence using amino acid translation and has the option to eliminate sequences that have non-IUPAC codes. Finally, the program calculates the barcode gap for the species in the submitted dataset.

# Running MACER 

## NOTE: When running MACER scripts the paths for the files selected cannot have whitespace! Any file with whitespace in the file naming may cause the program to not run and end in an error, or may result in unexpected output.

# Package pipeline example:
An example walk through of the MACER functions is available in the associated GitHub repository rgyoung6/MACER_example. 

# Package Function Details

## Download – auto_seq_download()

### Input 
File with list of genera you want to download. Place these genera in a file in a single column.

### Arguments
BOLD_database – TRUE is to include, FALSE is to exclude; default TRUE<br />
NCBI_database – TRUE is to include, FALSE is to exclude; default TRUE<br />
search_str – NULL uses the default string, anything other than NULL then that string will be used for the GenBank search; default NULL<br />
Default string…<br />
(genus[ORGN]) NOT (shotgun[ALL] OR genome[ALL] OR assembled[ALL] OR microsatellite[ALL])<br />
Note: When using a custom search string for NCBI only a single genus at a time can be used.<br />
input_file –  NULL prompts the user to indicate the location of the input file through point and click prompts, anything other than NULL then the string supplied will be used for the location; default NULL<br />
output_file –  NULL prompts the user to indicate the location of the output file through point and click prompts, anything other than NULL then the string supplied will be used for the location; default NULL<br />
seq_min - Lower sequence length value. Sequences with fewer than this number of nucleotide characters will be flagged in the final output. Default 100<br/>
seq_max - Upper sequence length value. Sequences with more than this number of nucleotide characters will be flagged in the final output. Default 100<br/>

### Output
One main folder containing three other folders.<br />
Main folder - Seq_auto_dl_TTTTTT_MMM_DD<br />
Three subfolders<br />
BOLD - Contains a file for every genus downloaded with the raw data from the BOLD system.<br />
NCBI - Contains a file for every genus downloaded with the raw data from GenBank.<br />
Total_tables - Contains files for the running of the script these include.<br />
A_Summary.txt - This file contains information about the downloads.<br />
A_Total_Table.tsv – This is a file with a single table (tab delimited) containing the accumulated data obtained for all genera searched.<br />

Note: the A_Total_Table.tsv file contains all of the records obtained and the final column is a flagged file of the results of the auto_seq_download() function

### Dependencies
rentrez used to access and download sequences from NCBI’s GenBank

## Table to Fasta – create_fastas()

### Input 
File with list of genera with the molecular markers names below the taxa. The information to create this parameters file can be obtained from A_Summary.txt file from the download script results. For example see below…<br />
Tamias<t/> Tamias<br />
CYTB<t/> COI-5P<br />
CYTOCHROMEB<t/>	CYTOCHROMECOXIDASESUBUNIT1<br />
CYTOCHROME-B<t/>	CYTOCHROMECOXIDASESUBUNITI<br />

### Arguments
data_file – NULL prompts the user to indicate the location of the data file in the format of the auto_seq_download output, anything other than NULL then the string supplied will be used for the location; default NULL<br />
input_file – NULL prompts the user to indicate the location of the input file used to select through point and click prompts, anything other than NULL then the string supplied will be used for the location; default NULL<br />
output_folder – NULL prompts the user to indicate the location of the output file through point and click prompts, anything other than NULL then the string supplied will be used for the location; default NULL<br />

no_marker – If set to TRUE then will include records filtered out due to no marker data. Default is FALSE to not include records with no marker data.<br />
no_taxa – If set to TRUE then will include records filtered out due to no taxa data. Default is FALSE to not include records with no taxa data.<br />
no_seq – If set to TRUE then will include records filtered out due to no sequence data. Default is FALSE to not include records with no sequence data.<br />
name_issue – If set to TRUE then will include records filtered out due to genus and species names with more than two terms. Default is FALSE to not include records with taxonomic naming issues.<br />
taxa_digits – If set to TRUE then will include records filtered out due to genus or species names containing digits. Default is FALSE to not include records with digits in the taxonomic naming.<br />
taxa_punct – If set to TRUE then will include records filtered out due to the presence of punctuation in the genus or species names. Default is FALSE to not include records with punctuation in the taxonomic naming.<br />
wrong_taxa – If set to TRUE then will include records filtered out due to the incorrect genera based on the list of taxa initially submitted to the download program. Default is FALSE to not include records of non-target taxa.<br />

### Output
This script outputs a fasta file of sequences for each column in the submitted parameters file. These files are named with the genera of interest and the first marker name in the column of the parameters file. These files are located in the folder where the Total_tables.tsv file is located.

### Dependencies
N/A

## Align - align_to_ref()

### Input 
A file folder location with the fasta files that need to be aligned and trimmed using the supplied reference sequence. Please note that any and all fasta files (named *.fas) in this folder will be analyzed.<br />
A reference sequence file with a sequence or MSA with all sequences having the same length. <br />
The location of the MAFFT executable file (https://mafft.cbrc.jp/alignment/software/)<br />

### Arguments
data_folder – This variable can be used to provide a location for the file containing all of the fasta files wanting to be aligned. The default value is set to NULL where the program will prompt the user to select the folder through point-and-click.<br />
ref_seq_file – This variable can be used to provide a location for the reference sequence file. The default value is set to FALSE where the program will prompt the user to select the folder through point-and-click.<br />
MAFFT_loc – This variable can be used to provide a location for the MAFFT program. The default value is set to NULL where the program will prompt the user to select the folder through point-and-click.<br />
output_file – This variable can be used to set the location of the output files from the program. The default value is set to NULL where the program will place the output files in the same location as the target files.<br />

pigl – This is the percent internal gap loop argument. This provides a percent that will remove records causing internal gaps is more than the percent value assigned to this argument is reached. If this value is set to 0 then internal gaps are not removed. The default for this value is 0.95.<br />
op – This is the gap opening penalty for the use of MAFFT. The higher the value the larger penalty in the alignment. The default for this value is set to 10. The default value in the MAFFT program is 1.53. For alignment of highly conserved regions where no gaps are expected this should be set to a much higher number and 10 is recommended for barcode regions like the COI-5P.<br />

### Output
In the submitted file folder location there will be a log file titled “MAFFT_log”.<br />
The sequence output files from this script are placed into two subfolders. These folders are in the submitted file location where the fasta files of interest are located. The two folders created are MAFFT and MAFFT_trimmed. In the MAFFT folder there will be files with name of the files in the submitted file folder appended with “_MAFFT”. <br />
The MAFFT_trimmed file will contain files with the same naming convention as the files in the submitted folder and appended with “MAFFT_trimmed”.<br />

### Dependencies
While not an R dependency, the use of the MAFFT program as installed on the local computer is required. 

## Clean - barcode_clean()

### Input 
A file folder with one or more fasta files of interest

### Arguments
AA_code – This is the amino acid translation matrix used to check the sequences for stop codons. The following codes are available. The Invertebrate matrix, 2, is the default.
  - std (1 in ape) is standard code
  - vert (2 in ape) is vertebrate mitochondrial
  - invert (5 in ape) is invertebrate mitochondrial
  - FALSE skips the AA clean section
AGCT_only – This only keeps sequences with AGCT exclusively, not IUPAC characters.
  - TRUE is on 
  - FALSE is accepting all IUPAC characters
data_folder This variable can be used to provide a location for the file containing all of the fasta files wanting to be aligned. The default value is set to NULL where the program will prompt the user to select the folder through point-and-click.
dist_model - This is the model of nucleotide evolution that the ape program will use (see ape documentation for options. Default is "raw"
replicates - This is the number of replicates that the bootstrapping will perform. Note: more replicates will take longer. Default is 1000
replacement - This indicates that the replacement of MSA nucleotide columns will be replaced in the random resampling. Default is set to TRUE
conf_level - This is a percentage of the initial MSA nucleotide length. When set to 1 the bootstrapped resampling will have the same length as the initial MSA. Default is set to 1
numCores - This is the number of cores that the user would like to use where multithreading is available. Default is set to 1, indicating only a single thread will be used.


### Output
A single log file for the running of the function with the name A_Clean_File_YYYY-DD-TTTTTTTT. The function will also output three files for each fasta file submitted. The first is the distance matrix that was calculated and used to assess the DNA barcode gaps. This file is named the same as the input file with ‘_dist_table.dat” appended to the end of the name. The second file is the total data table file which provides a table of all submitted records for each data set accompanied with the results from each section of the analysis. This file is named the same as the input fasta with “_data_table.dat” appended to the end, Finally, a fasta file with all outliers and flagged records removed is generated for each input fasta file. This output file is named the same as the input fasta with “_no_outlier.fas” appended to the end. The flags that are possible are non_AGCT, Stop_Codon, Genus_Outlier, Species_Outlier, and '-'.

### Dependencies
ape is required for distance matrix construction.
ggplot2 is required for figure generation
parallel is required for use of multiple processors
pbapply is required to apply the function with a time bar
