# Introduction to RenyiExtropy

## Overview

RenyiExtropy provides a unified interface for computing entropy and extropy measures for discrete probability distributions. All functions accept a probability vector (or matrix for joint measures) and return a single numeric value measured in nats (natural-logarithm base).

The package covers three families of measures:

- Entropy — Shannon, Renyi, Tsallis, normalised, joint, conditional
- Extropy — classical (Shannon), Renyi, conditional Renyi, maximum Renyi
- Divergences — KL, Jensen-Shannon, cross-entropy

## Probability vectors

Every function that accepts a vector `p` requires it to satisfy:

- numeric, no `NA`/`NaN`
- all elements in \[0, 1\]
- elements sum to 1 (within tolerance 1e-8)
- length ≥ 2

    p <- c(0.2, 0.5, 0.3)   # valid 3-outcome distribution

## Shannon entropy

The Shannon entropy is defined as \[H(\mathbf{p}) = -\sum_{i=1}^n p_i \log p_i\] and equals zero for a degenerate distribution and \(\log n\) for the uniform distribution.

    shannon_entropy(p)               # three-outcome distribution
    #> [1] 1.029653
    shannon_entropy(rep(0.25, 4))    # uniform: H = log(4)
    #> [1] 1.386294
    shannon_entropy(c(1, 0, 0))      # degenerate: H = 0
    #> [1] 0
    normalized_entropy(p)            # H(p) / log(n), always in [0, 1]
    #> [1] 0.9372306

## Renyi entropy

The Renyi entropy of order \(q > 0\) is \[H_q(\mathbf{p}) = \frac{1}{1-q} \log\!\left(\sum_i p_i^q\right)\]

For \(q \to 1\) it reduces to the Shannon entropy; the function automatically returns the Shannon limit when \(|q - 1| < 10^{-8}\).

    renyi_entropy(p, q = 2)         # collision entropy
    #> [1] 0.967584
    renyi_entropy(p, q = 0.5)       # Renyi entropy, q = 0.5
    #> [1] 1.063659
    renyi_entropy(p, q = 1)         # limit: equals shannon_entropy(p)
    #> [1] 1.029653
    shannon_entropy(p)               # same value
    #> [1] 1.029653

## Tsallis entropy

The Tsallis entropy is \[S_q(\mathbf{p}) = \frac{1 - \sum_i p_i^q}{q - 1}\] a non-extensive generalisation of Shannon entropy widely used in statistical physics.

    tsallis_entropy(p, q = 2)
    #> [1] 0.62
    tsallis_entropy(p, q = 1)       # limit: equals shannon_entropy(p)
    #> [1] 1.029653

## Extropy

Extropy is the dual complement of entropy, measuring information via complementary probabilities: \[J(\mathbf{p}) = -\sum_{i=1}^n (1-p_i)\log(1-p_i)\]

`extropy()` and `shannon_extropy()` compute the same quantity.

    extropy(p)
    #> [1] 0.7747609
    shannon_extropy(p)               # identical
    #> [1] 0.7747609

## Renyi extropy

The Renyi extropy of order \(q\) is \[J_q(\mathbf{p}) = \frac{-(n-1)\log(n-1) + (n-1)\log\!\left(\sum_i(1-p_i)^q\right)}{1-q}\]

For \(n = 2\) it equals the Renyi entropy. For \(q \to 1\) it returns the classical extropy.

    renyi_extropy(p, q = 2)
    #> [1] 0.7421274
    renyi_extropy(p, q = 1)          # limit: equals extropy(p)
    #> [1] 0.7747609

    # n = 2: Renyi extropy == Renyi entropy
    renyi_extropy(c(0.4, 0.6), q = 2)
    #> [1] 0.6539265
    renyi_entropy(c(0.4, 0.6), q = 2)
    #> [1] 0.6539265

    # Maximum Renyi extropy over n outcomes (uniform distribution)
    max_renyi_extropy(3)
    #> [1] 0.8109302
    renyi_extropy(rep(1/3, 3), q = 2)
    #> [1] 0.8109302

## Joint and conditional entropy

`joint_entropy()` accepts a matrix of joint probabilities; rows correspond to outcomes of \(X\), columns to outcomes of \(Y\).

    Pxy <- matrix(c(0.2, 0.3, 0.1, 0.4), nrow = 2, byrow = TRUE)

    joint_entropy(Pxy)               # H(X, Y)
    #> [1] 1.279854
    conditional_entropy(Pxy)         # H(Y | X) = H(X,Y) - H(X)
    #> [1] 0.586707

    # Independent distributions: H(X,Y) = H(X) + H(Y)
    px <- c(0.4, 0.6)
    py <- c(0.3, 0.7)
    Pxy_indep <- outer(px, py)
    joint_entropy(Pxy_indep)
    #> [1] 1.283876
    shannon_entropy(px) + shannon_entropy(py)
    #> [1] 1.283876

## Conditional Renyi extropy

    conditional_renyi_extropy(Pxy, q = 2)
    #> [1] 0.1039623
    conditional_renyi_extropy(Pxy, q = 1)   # limit: conditional Shannon extropy
    #> [1] 0.13636

## Divergences

### KL divergence

The Kullback-Leibler divergence \(D_\text{KL}(P \| Q)\) is asymmetric and can be infinite when \(q_i = 0\) but \(p_i > 0\) (a warning is emitted in that case).

    q_dist <- c(0.3, 0.4, 0.3)
    kl_divergence(p, q_dist)         # KL(P || Q)
    #> [1] 0.03047875
    kl_divergence(q_dist, p)         # KL(Q || P) -- asymmetric
    #> [1] 0.03238211
    kl_divergence(p, p)              # 0
    #> [1] 0

### Jensen-Shannon divergence

The Jensen-Shannon divergence is the symmetrised, always-finite version, bounded in \([0, \log 2]\):

    js_divergence(p, q_dist)         # symmetric
    #> [1] 0.0078174
    js_divergence(q_dist, p)         # same value
    #> [1] 0.0078174
    js_divergence(p, p)              # 0
    #> [1] 0
    js_divergence(c(1, 0), c(0, 1)) # maximum: log(2)
    #> [1] 0.6931472
    log(2)
    #> [1] 0.6931472

### Cross-entropy

Cross-entropy relates to KL divergence via \(H(P, Q) = H(P) + D_\text{KL}(P \| Q)\):

    cross_entropy(p, q_dist)
    #> [1] 1.060132
    shannon_entropy(p) + kl_divergence(p, q_dist)   # same value
    #> [1] 1.060132
    cross_entropy(p, p)                              # equals shannon_entropy(p)
    #> [1] 1.029653

## Input validation

All functions produce informative errors when given invalid inputs:

    shannon_entropy(c(0.2, 0.3, 0.1))   # does not sum to 1
    #> Error in `shannon_entropy()`:
    #> ! Elements of 'p' must sum to 1 (tolerance 1e-8).
    renyi_entropy(p, q = NA)            # NA not allowed
    #> Error in `renyi_entropy()`:
    #> ! 'q' must not be NA or NaN.
    max_renyi_extropy(1)                # n must be >= 2
    #> Error in `max_renyi_extropy()`:
    #> ! 'n' must be at least 2.
    kl_divergence(p, c(0.5, 0.5))      # length mismatch
    #> Error in `kl_divergence()`:
    #> ! 'p' and 'q' must have the same length.