# Custom Activation Function

# Rationale

The biggest strength of `{kindling}` when modelling neural networks is its versatility — it inherits `{torch}`’s versatility while being human friendly, including the ability to apply custom optimizer functions, loss functions, and per-layer activation functions. Learn more: https://kindling.joshuamarie.com/articles/special-cases.

With `act_funs()`, you are not limited to the activation functions available in `{torch}`’s namespace. Use `new_act_fn()` to wrap any compatible function into a validated custom activation. This feature, however, is only available on version 0.3.0 and above.

# Function to use

To do this, use `new_act_fn()`. It takes a user-supplied function, validates it against a small dummy tensor at definition time (a dry-run probe), and wraps it in a call-time type guard. This means errors surface early — before your model ever starts training.

The function you supply must:

- Accept at least one argument (the input tensor).
- Return a `torch_tensor`.

## Basic Usage

Currently, `nnf_tanh` doesn’t exist in the `{torch}` namespace, so `tanh` is not a valid argument to `act_funs()`. With `new_act_fn()`, you can wrap `torch::torch_tanh()` to make it usable.

Here’s a basic example that wraps `torch::torch_tanh()` as a custom activation:

    hyper_tan = new_act_fn(\(x) torch::torch_tanh(x))

You can also pass it directly into `act_funs()`, just like any built-in activation:

    act_funs(relu, elu, new_act_fn(\(x) torch::torch_tanh(x)))

## Using Custom Activations in a Model

Naturally, functions for modelling, like `ffnn()`, accept `act_funs()` into the `activations` argument. Again, you can pass a custom activation function within `new_act_fn()`, then pass it through `act_funs()`.

Here’s a basic example:

    model = ffnn(
        Sepal.Length ~ .,
        data = iris[, 1:4],
        hidden_neurons = c(64, 32, 16),
        activations = act_funs(
            relu,
            silu,
            new_act_fn(\(x) torch::torch_tanh(x))
        ),
        epochs = 50
    )
    model

Each element of `act_funs()` corresponds to one hidden layer, in order. Here, the first hidden layer uses ReLU, the second uses SiLU (Swish), and the third uses Tanh.

You can also use a single custom activation recycled across all layers:

    ffnn(
        Sepal.Length ~ .,
        data = iris[, 1:4],
        hidden_neurons = c(64, 32),
        activations = act_funs(new_act_fn(\(x) torch::torch_tanh(x))),
        epochs = 50
    )

## Skipping the Dry-Run Probe

By default, `new_act_fn()` runs a quick dry-run with a small dummy tensor to validate your function before training. You can disable this with `probe = FALSE`, though this is generally not recommended:

    my_act = new_act_fn(\(x) torch::torch_tanh(x), probe = FALSE)

## Naming Your Custom Activation

You can provide a human-readable name via `.name`, which is used in print output and diagnostics:

    my_act = new_act_fn(\(x) torch::torch_tanh(x), .name = "my_tanh")

Here’s a simple application:

    ffnn(
        Sepal.Length ~ .,
        data = iris[, 1:4],
        hidden_neurons = c(64, 32),
        activations = act_funs(
            relu,
            new_act_fn(\(x) torch::torch_tanh(x), .name = "hyper_tanh")
        ),
        epochs = 50
    )

# Error Handling

`new_act_fn()` is designed to fail loudly and early. Common errors include:

1. Function returns a non-tensor. This will error at definition time:

       new_act_fn(\(x) as.numeric(x))

2. Function accepts no arguments. This will error immediately:

       new_act_fn(function() torch::torch_zeros(2))

These checks ensure your model’s architecture is valid before any data ever flows through it.

# Summary

Feature | Details
--- | ---
Wraps any R function | Must accept a tensor, return a tensor
Dry-run probe | Validates at definition time (`probe = TRUE` by default)
Call-time guard | Type-checks output on every forward pass
Compatible with `act_funs()` | Use alongside built-in activations freely
Closures supported | Parametric activations work naturally