Author: Kesondra Key
Date: 2025-09-18

# Introduction to Fluxtools

#### Kesondra Key

#### 2025-09-18

    library(fluxtools)

    ## Welcome to fluxtools V0.7.1

    ##
    ## To start the app: run_fluxtools()

    ## To apply Physical Range Module (PRM) filters: use apply_prm()

    ##
    ## For additional help: see ?run_fluxtools, ?apply_prm, browseVignettes('fluxtools')

    ## To view citation: citation('fluxtools')

# Overview

fluxtools is an R package that provides an interactive Shiny‐based QA/QC environment to explore or remove data in the AmeriFlux BASE (or Fluxnet) format. In just a few clicks, you can:

1. Upload eddy covariance data in a .csv format (AmeriFlux standard naming and timestamp conventions; up to ~1 GB)
2. Visualize any numeric variable against time (default: TIMESTAMP_START) or against another variable
3. Detect outliers with the ±σ slider (residuals from a simple linear fit) and stage them for removal
4. Manually flag points using box/lasso selection, time ranges, or custom min/max thresholds
5. Overlay or compare: Plot multiple Y variables in the same dataset, or upload a second dataset (Dataset B) for side-by-side comparison with Dataset A
6. Add smoothers (LOESS) with adjustable span and optional 95% CI; optionally show only smoothed line(s)
7. Apply PRM (Physical Range Module) to automatically replace values outside published physical bounds
8. Generate reproducible code: fluxtools builds dplyr code that sets flagged values to NA_real_. Copy current or accumulated snippets directly
9. Download results: export a ZIP with cleaned CSV(s), replay scripts, and (if PRM applied) summary/audit files

This vignette shows you how to install, launch, and use the main Shiny app—`run_fluxtools()`—and walks through a typical workflow.

# Installation

You can install fluxtools from CRAN, or directly from GitHub:

    # Install from CRAN
    install.packages("fluxtools")

    # Install from GitHub
    library(devtools)
    devtools::install_github("kesondrakey/fluxtools")

# Launching the Shiny App

Load fluxtools and launch the QA/QC application:

    library(fluxtools)

    # Run the app
    run_fluxtools()

Example workflow

- Upload: Select your AmeriFlux-style CSV (e.g., `US_VT1_HH_202401010000_202501010000.csv`). Files can be up to 1GB (larger file sizes might be harder on the Shiny interface)

- Choose Year(s): By default “all” is selected, but you can subset to specific years

- Choose variables: `TIMESTAMP_START` is on the x-axis by default. Change the y-axis to your variable of interest (e.g., `FC_1_1_1`). The generated R code focuses on removing the y-axis variable

- Compare Two Datasets Upload a second dataset (Dataset B) to compare against your primary upload (Dataset A).

- ⚠️ Note: Flagging and removals only apply to Dataset A

- You can assign custom labels for each dataset, which appear in the legend

- Choose custom colors per dataset

- All advanced options (smoothers, lines vs. points, opacity, etc.) work in comparison mode, except global themes and single-variable color overrides (which apply to Dataset A only)

- Plot Multiple Variables (Overlay) Enable Plot multiple variables to overlay several Y variables from the same dataset in one plot

- Select variables in the Overlay variables menu.

- Use Include current Y-variable if you want the currently selected variable included automatically

- Colors are assigned from the active palette; you can override them manually in Advanced Options

- Advanced Options (Flag Style & Markers)

- Switch between scatter and line plots

- Adjust opacity and line width

- In single-variable mode, markers can be hollow circles for better visibility of flagged rings

- Smoother Overlay Add a LOESS smoother to any plot:

- Adjustable span parameter to control curve smoothness

- Option to display a 95% confidence interval

- Toggle Show only smoothed line(s) to hide base scatter/lines and keep just the LOESS curves

- Color Options

- For single-variable or overlay mode, colors follow the chosen palette (Okabe–Ito, Tableau10, viridis, etc.) or manual overrides

- For dataset comparison, colors are set separately under the Compare two datasets menu

- Time Subsetting Subset your dataset by year, month(s), day(s), or time of day

- Example: Select 06:00–18:00 to restrict analysis to daytime values

- Icons update dynamically: a ☀️ sun for daytime, 🌙 moon for nighttime

- Select data: Use the box or lasso to select points. This populates the “Current” code box with something like:

       df <- df %>%
         mutate(
           FC_1_1_1 = case_when(
             TIMESTAMP_START == '202401261830' ~ NA_real_,
             TIMESTAMP_START == '202401270530' ~ NA_real_,
             …
             TRUE ~ FC_1_1_1
           )
         )

- Flag data and Accumulate code: With points still selected, click “Flag data.” Selected points turn orange, and code is appended to the “Accumulated” box, allowing multiple selections per session.

- Unflag data: Use the box or lasso to de-select points and remove from the Accumulated code box.

- Clear Selection: To reset all selections from the current y-variable, click “Clear Selection” to reset the current view.

- Switch variables: Change y to any other variable (e.g., `SWC_1_1_1`) and select more points. Click “Flag data” Code for both variables to appear:

       df <- df %>%
         mutate(
           FC_1_1_1 = case_when(
             TIMESTAMP_START == '202401261830' ~ NA_real_,
             TIMESTAMP_START == '202401270530' ~ NA_real_,
             …
             TRUE ~ FC_1_1_1
           )
         )

       df <- df %>%
         mutate(
           SWC_1_1_1 = case_when(
             TIMESTAMP_START == '202403261130' ~ NA_real_,
             TIMESTAMP_START == '202403270800' ~ NA_real_,
             …
             TRUE ~ SWC_1_1_1
           )
         )

- Compare variables: Change to variables you would like to compare (e.g., change y to `TA_1_1_1` and x to `T_SONIC_1_1_1`). The app computes an R² via simple linear regression. The top R² is based on points before removals, and once data is selected, a second R² will pop up - calculating the linear regression assuming the selected points have been removed

- Highlight outliers: Use the slider to select ±σ residuals. Click “Select all ±σ outliers” to append them to the Accumulated code. Click “Clear ±σ outliers” to deselect and remove from the code box

- Copy all: Click the Copy Icon to the right of the current or accumulated code box and paste into your own R script for documentation

- Apply Removals: Click “Apply Removals” to remove each selected data points, from the current y-variable, to replace points with `NA` in a new .csv (raw data is unaffected), available using ‘export cleaned data’ and remove these values from view

- Reload original data: Make a mistake or want a fresh start? Click Reload original data to reload the .csv from above to start over

- Export cleaned data: Download a ZIP containing:

- A cleaned CSV (with applied NAs)

- An R script that reproduces your removals

- Optional PRM summary when used (see below)

- Physical Boundary Module (PRM) function: The Physical Range Module (PRM) removes out-of-range values to `NA` based on similar variables using patterns like `^SWC($|_)` or `^P($|_)`.
  Columns containing `"QC"` are skipped by default. No columns are removed.

Source of ranges: AmeriFlux Technical Documents, Table A1 (Physical Range Module).

## Quick start

    # tiny demo dataset with a few out-of-range values
    set.seed(1)
    df <- tibble::tibble(
      TIMESTAMP_START = seq.POSIXt(as.POSIXct("2024-01-01", tz = "UTC"),
                                   length.out = 10, by = "30 min"),
      SWC_1_1_1 = c(10, 20, 150, NA, 0.5, 99, 101, 50, 80, -3),  # bad: 150, 101, -3; 0.5 triggers SWC unit note
      P         = c(0, 10, 60, NA, 51, 3, 0, 5, 100, -1),        # bad: 60, 51, 100, -1
      RH_1_1_1  = c(10, 110, 50, NA, 0, 100, -5, 101, 75, 30),   # bad: 110, -5, 101
      SWC_QC    = sample(0:2, 10, replace = TRUE)                # QC col should be ignored
    )

    # To see the Physical Boundary Module (PRM) rules:
    get_prm_rules()

    #Apply filter to all relevant variables
    res <- apply_prm(df)

    # PRM summary (counts and % replaced per column)
    res$summary

    # Only set range for SWC
    df_filtered_swc <- apply_prm(df, include = "SWC")

    # Only set range for SWC + P
    df_filtered_swc_P <- apply_prm(df, include = c("SWC", "P"))

## Physical Range Module Values

Variable | Min | Max | Description | Units
--- | --- | --- | --- | ---
COND_WATER | 0 | 10000 | Conductivity of water | uS cm^-1
DO | 0 | NA | Dissolved oxygen in water | umol L^-1
PCH4 | 0 | NA | Dissolved methane (CH4) in water | nmolCH4 mol^-1
PCO2 | 0 | 10000 | Dissolved carbon dioxide (CO2) in water | umolCO2 mol^-1
PN2O | 0 | NA | Dissolved nitrous oxide (N2O) in water | nmolN2O mol^-1
PPFD_UW_IN | 0 | 2400 | Photosynthetic photon flux density, underwater, incoming | umolPhotons m^-2 s^-1
TW | -20 | 50 | Water temperature | deg C
DBH | 0 | 500 | Tree diameter at breast height | cm
LEAF_WET | 0 | 100 | Leaf wetness (0-100) | %
SAP_DT | -10 | 10 | Sapflow probe temperature difference | deg C
T_BOLE | -50 | 70 | Bole temperature | deg C
T_CANOPY | -50 | 70 | Canopy/surface temperature | deg C
CH4 | 0 | 15000 | Methane (CH4) mole fraction (wet air) | nmolCH4 mol^-1
CH4_MIXING_RATIO | 0 | 15000 | Methane (CH4) mole fraction (dry air) | nmolCH4 mol^-1
CO | 0 | NA | Carbon monoxide (CO) mole fraction (wet air) | nmolCO mol^-1
CO2 | 150 | 1200 | Carbon dioxide (CO2) mole fraction (wet air) | umolCO2 mol^-1
CO2_SIGMA | 0 | 150 | Std. dev. of CO2 mole fraction (wet air) | umolCO2 mol^-1
CO2C13 | NA | -6 | Stable isotope delta13C of CO2 (permil) | permil
FC | -100 | 100 | CO2 turbulent flux (no storage correction) | umolCO2 m^-2 s^-1
FCH4 | -500 | 4000 | CH4 turbulent flux (no storage correction) | nmolCH4 m^-2 s^-1
H2O | 0 | 100 | Water vapor in mole fraction (wet air) | mmolH2O mol^-1
H2O_MIXING_RATIO | 0 | 100 | Water vapor in mole fraction (dry air) | mmolH2O mol^-1
H2O_SIGMA | 0 | 15 | Std. dev. of water vapor mole fraction | mmolH2O mol^-1
N2O | 0 | NA | N2O mole fraction (wet air) | nmolN2O mol^-1
N2O_MIXING_RATIO | 0 | NA | N2O mole fraction (dry air) | nmolN2O mol^-1
NO | 0 | NA | NO mole fraction (wet air) | nmolNO mol^-1
NO2 | 0 | NA | NO2 mole fraction (wet air) | nmolNO2 mol^-1
O3 | 0 | NA | O3 mole fraction (wet air) | nmolO3 mol^-1
SC | -100 | 100 | CO2 storage flux | umolCO2 m^-2 s^-1
SO2 | 0 | NA | SO2 mole fraction (wet air) | nmolSO2 mol^-1
FH2O | -10 | 20 | Water vapor (H2O) turbulent flux (no storage correction) | mmolH2O m^-2 s^-1
G | -250 | 400 | Soil heat flux | W m^-2
H | -450 | 900 | Sensible heat flux (no storage correction) | W m^-2
LE | -450 | 900 | Latent heat flux (no storage correction) | W m^-2
SG | -100 | 250 | Soil heat storage flux above plates | W m^-2
SH | -150 | 150 | Sensible heat storage flux | W m^-2
SLE | -150 | 150 | Latent heat storage flux | W m^-2
PA | 60 | 105 | Atmospheric pressure | kPa
PBLH | 0 | 3000 | Planetary boundary layer height | m
RH | 0 | 100 | Relative humidity (0-100) | %
T_SONIC | -50 | 50 | Sonic temperature | deg C
T_SONIC_SIGMA | 0 | 5 | Std. dev. of sonic temperature | deg C
TA | -50 | 50 | Air temperature | deg C
VPD | 0 | 80 | Vapor pressure deficit | hPa
D_SNOW | 0 | 500 | Snow depth | cm
P | 0 | 50 | Precipitation | mm
P_RAIN | 0 | 50 | Rainfall | mm
P_SNOW | 0 | 50 | Snowfall | mm
RUNOFF | 0 | 200 | Runoff | mm
STEMFLOW | 0 | 200 | Stemflow | mm
THROUGHFALL | 0 | 20 | Throughfall | mm
ALB | 0 | 100 | Albedo (0-100) | %
APAR | 0 | 2300 | Absorbed PAR | umolPhoton m^-2 s^-1
EVI | -1 | 1 | Enhanced Vegetation Index | nondimensional
FAPAR | 0 | 100 | Fraction of absorbed PAR (0-100) | %
FIPAR | 0 | 100 | Fraction of intercepted PAR (0-100) | %
LW_BC_IN | 50 | 600 | Longwave radiation, below canopy incoming | W m^-2
LW_BC_OUT | 100 | 750 | Longwave radiation, below canopy outgoing | W m^-2
LW_IN | 50 | 600 | Longwave radiation, incoming | W m^-2
LW_OUT | 100 | 750 | Longwave radiation, outgoing | W m^-2
MCRI | 0 | 10 | Carotenoid Reflectance Index | nondimensional
MTCI | 0 | 10 | MERIS Terrestrial Chlorophyll Index | nondimensional
NDVI | -1 | 1 | Normalized Difference Vegetation Index | nondimensional
NETRAD | -200 | 1100 | Net radiation | W m^-2
NIRV | 0 | 2 | Near Infrared Vegetation Index | W m^-2 sr^-1 nm^-1
PPFD_BC_IN | 0 | 2400 | PPFD, below canopy incoming | umolPhoton m^-2 s^-1
PPFD_BC_OUT | 0 | 2000 | PPFD, below canopy outgoing | umolPhoton m^-2 s^-1
PPFD_DIF | 0 | 1400 | PPFD, diffuse incoming | umolPhoton m^-2 s^-1
PPFD_DIR | 0 | 2400 | PPFD, direct incoming | umolPhoton m^-2 s^-1
PPFD_IN | 0 | 2400 | PPFD, incoming | umolPhoton m^-2 s^-1
PPFD_OUT | 0 | 2000 | PPFD, outgoing | umolPhoton m^-2 s^-1
PRI | -1 | 1 | Photochemical Reflectance Index | nondimensional
R_UVA | 0 | 85 | UVA radiation, incoming | W m^-2
R_UVB | 0 | 20 | UVB radiation, incoming | W m^-2
REDCI | 0 | 10 | Red-Edge Chlorophyll Index | nondimensional
REP | 400 | 800 | Red-Edge Position | nm
SPEC_NIR_IN | 0 | 2 | NIR band radiation, incoming (hemispherical) | W m^-2 nm^-1
SPEC_NIR_OUT | 0 | 2 | NIR band radiation, outgoing | W m^-2 sr^-1 nm^-1
SPEC_NIR_REFL | 0 | 1 | NIR band reflectance | nondimensional
SPEC_PRI_REF_IN | 0 | 2 | PRI reference band radiation, incoming | W m^-2 nm^-1
SPEC_PRI_REF_OUT | 0 | 2 | PRI reference band radiation, outgoing | W m^-2 sr^-1 nm^-1
SPEC_PRI_REF_REFL | 0 | 1 | PRI reference band reflectance | nondimensional
SPEC_PRI_TGT_IN | 0 | 2 | PRI target band radiation, incoming | W m^-2 nm^-1
SPEC_PRI_TGT_OUT | 0 | 2 | PRI target band radiation, outgoing | W m^-2 sr^-1 nm^-1
SPEC_PRI_TGT_REFL | 0 | 1 | PRI target band reflectance | nondimensional
SPEC_RED_IN | 0 | 2 | Red band radiation, incoming (hemispherical) | W m^-2 nm^-1
SPEC_RED_OUT | 0 | 2 | Red band radiation, outgoing | W m^-2 sr^-1 nm^-1
SPEC_RED_REFL | 0 | 1 | Red band reflectance | nondimensional
SR | 0 | 10 | Simple Ratio | nondimensional
SW_BC_IN | 0 | 1300 | Shortwave radiation, below canopy incoming | W m^-2
SW_BC_OUT | 0 | 800 | Shortwave radiation, below canopy outgoing | W m^-2
SW_DIF | 0 | 750 | Shortwave radiation, diffuse incoming | W m^-2
SW_DIR | 0 | 1300 | Shortwave radiation, direct incoming | W m^-2
SW_IN | 0 | 1300 | Shortwave radiation, incoming | W m^-2
SW_OUT | 0 | 800 | Shortwave radiation, outgoing | W m^-2
TCARI | 0 | 10 | Transformed Chlorophyll Absorption in Reflectance Index | nondimensional
SWC | 0 | 100 | Soil water content (volumetric, 0-100) | %
SWP | -750 | 0 | Soil water potential | kPa
TS | -40 | 65 | Soil temperature | deg C
TSN | -40 | 4 | Snow temperature | deg C
WTD | -10 | 10 | Water table depth | m
TAU | -10 | 2 | Momentum flux | kg m^-1 s^-2
U_SIGMA | 0 | 12 | Std. dev. of along-wind velocity | m s^-1
USTAR | 0 | 8 | Friction velocity | m s^-1
V_SIGMA | 0 | 10 | Std. dev. of cross-wind velocity | m s^-1
W_SIGMA | 0 | 5 | Std. dev. of vertical velocity | m s^-1
WD | 0 | 360 | Wind direction | degree
WD_SIGMA | 0 | 180 | Std. dev. of wind direction | degree
WS | 0 | 40 | Wind speed | m s^-1
WS_MAX | 0 | 50 | Max wind speed in averaging period | m s^-1
GPP | -30 | 100 | Gross primary productivity | umolCO2 m^-2 s^-1
NEE | -100 | 100 | Net ecosystem exchange | umolCO2 m^-2 s^-1
RECO | -20 | 50 | Ecosystem respiration | umolCO2 m^-2 s^-1

Fluxtools is an independent project and is not affiliated with or endorsed by the AmeriFlux Network. “AmeriFlux” is a registered trademark of Lawrence Berkeley National Laboratory and is used here for identification purposes only.