# SPDX-License-Identifier: EPL-2.0 OR GPL-2.0-or-later
# SPDX-FileCopyrightText: Bradley M. Bell <bradbell@seanet.com>
# SPDX-FileContributor: 2003-22 Bradley M. Bell
# ----------------------------------------------------------------------------
{xrst_begin exp_2.hpp}

exp_2: Implementation
#####################

{xrst_literal
   introduction/exp_2.hpp
   // BEGIN C++
   // END C++
}

{xrst_end exp_2.hpp}
-----------------------------------------------------------------------------
{xrst_begin exp_2.cpp}

exp_2: Test
###########

{xrst_literal
   introduction/exp_2.cpp
   // BEGIN C++
   // END C++
}

{xrst_end exp_2.cpp}
-----------------------------------------------------------------------------
{xrst_begin exp_2_for0}

exp_2: Operation Sequence and Zero Order Forward Mode
#####################################################

Mathematical Form
*****************
The operation sequence (see below) corresponding to
the algorithm :ref:`exp_2.hpp-name` is the same for all values of *x* .
The mathematical form for the corresponding function is

.. math::

   f(x) = 1 + x + x^2 / 2

An algorithmic differentiation package
does not operate on the mathematical function :math:`f(x)`
but rather on the particular algorithm used to compute the function
(in this case :ref:`exp_2.hpp-name` ).

Zero Order Expansion
********************
In general, a zero order forward sweep is given a vector
:math:`x^{(0)} \in \B{R}^n` and it returns the corresponding vector
:math:`y^{(0)} \in \B{R}^m` given by

.. math::

   y^{(0)} = f( x^{(0)} )

The superscript :math:`(0)` denotes zero order derivative;
i.e., it is equal to the value
of the corresponding variable.
For the example we are considering here,
both :math:`n` and :math:`m` are equal to one.

Operation Sequence
******************
An atomic *Type* operation is an operation
that has a *Type* result and is not made up of
other more basic operations.
A sequence of atomic *Type* operations is called a
*Type* operation sequence.
Given an C++ algorithm and its inputs,
there is a corresponding *Type* operation sequence for each type.
If *Type* is clear from the context,
we drop it and just refer to the operation sequence.

We consider the case where :ref:`exp_2.hpp-name` is executed with
:math:`x^{(0)} = .5`.
The table below contains the corresponding operation sequence
and the results of a zero order sweep.

Index
=====
The Index column contains the index in the operation sequence
of the corresponding atomic operation and variable.
A Forward sweep starts with the first operation
and ends with the last.

Code
====
The Code column contains the C++ source code corresponding
to the corresponding atomic operation in the sequence.

Operation
=========
The Operation column contains the
mathematical function corresponding to each atomic operation in the sequence.

Zero Order
==========
The Zero Order column contains the zero order derivative for
the corresponding variable in the operation sequence.
Forward mode refers to the fact that
these coefficients are computed in the same order as the original algorithm;
i.e, in order of increasing index in the operation sequence.

Sweep
=====

.. csv-table::
   :widths: auto

   **Index**,,**Code**,,**Operation**,,**Zero Order**
   1,,``Type v1  = x;``,,:math:`v_1 = x`,,:math:`v_1^{(0)} = 0.5`
   2,,``Type v2  = Type(1) + v1;``,,:math:`v_2 = 1 + v_1`,,:math:`v_2^{(0)} = 1.5`
   3,,``Type v3  = v1 * v1;``,,:math:`v_3 = v_1 * v_1`,,:math:`v_3^{(0)} = 0.25`
   4,,``Type v4  = v3 / Type(2);``,,:math:`v_4 = v_3 / 2`,,:math:`v_4^{(0)} = 0.125`
   5,,``Type v5  = v2 + v4;``,,:math:`v_5 = v_2 + v_4`,,:math:`v_5^{(0)} = 1.625`

Return Value
************
The return value for this case is

.. math::

   1.625 =
   v_5^{(0)} =
   f( x^{(0)} )

{xrst_toc_hidden
   introduction/exp_2_for0.cpp
}

Verification
************
The file :ref:`exp_2_for0.cpp-name` contains a routine
that verifies the values computed above.

Exercises
*********

#. Suppose that :math:`x^{(0)} = .2`,
   what is the result of a zero order forward sweep for
   the operation sequence above;
   i.e., what are the corresponding values for

   .. math::

      v_1^{(0)} , v_2^{(0)} , \cdots , v_5^{(0)}

#. Create a modified version of
   :ref:`exp_2_for0.cpp-name` that verifies the values you obtained
   for the previous exercise.
#. Create and run a main program that reports the result of calling
   the modified version
   of :ref:`exp_2_for0.cpp-name` in the previous exercise.

{xrst_end exp_2_for0}
------------------------------------------------------------------------------
{xrst_begin exp_2_for1}

exp_2: First Order Forward Mode
###############################

First Order Expansion
*********************
We define :math:`x(t)` near :math:`t = 0` by the first order expansion

.. math::

   x(t) = x^{(0)} + x^{(1)} * t

it follows that :math:`x^{(0)}` is the zero,
and :math:`x^{(1)}` the first,
order derivative of :math:`x(t)`
at :math:`t = 0`.

Purpose
*******
In general, a first order forward sweep is given the
:ref:`zero order derivative<exp_2_for0@Zero Order Expansion>`
for all of the variables in an operation sequence,
and the first order derivatives for the independent variables.
It uses these to compute the first order derivatives,
and thereby obtain the first order expansion,
for all the other variables in the operation sequence.

Mathematical Form
*****************
Suppose that we use the algorithm :ref:`exp_2.hpp-name` to compute

.. math::

   f(x) = 1 + x + x^2 / 2

The corresponding derivative function is

.. math::

   \partial_x f (x) =   1 + x

An algorithmic differentiation package
does not operate on the mathematical form of the function,
or its derivative,
but rather on the
:ref:`exp_2_for0@Operation Sequence`
for the for the algorithm that is used to evaluate the function.

Operation Sequence
******************
We consider the case where :ref:`exp_2.hpp-name` is executed with
:math:`x = .5`.
The corresponding operation sequence and
zero order forward mode values
(see :ref:`zero order sweep<exp_2_for0@Operation Sequence@Sweep>` )
are inputs and are used by a first order forward sweep.

Index
=====
The Index column contains the index in the operation sequence
of the corresponding atomic operation.
A Forward sweep starts with the first operation
and ends with the last.

Operation
=========
The Operation column contains the
mathematical function corresponding to each atomic operation in the sequence.

Zero Order
==========
The Zero Order column contains the zero order derivatives
for the corresponding variable in the operation sequence
(see :ref:`zero order sweep<exp_2_for0@Operation Sequence@Sweep>` ).

Derivative
==========
The Derivative column contains the
mathematical function corresponding to the derivative
with respect to :math:`t`,
at :math:`t = 0`, for each variable in the sequence.

First Order
===========
The First Order column contains the first order derivatives
for the corresponding variable in the operation sequence; i.e.,

.. math::

   v_j (t) = v_j^{(0)} + v_j^{(1)} t

We use :math:`x^{(1)} = 1` so that differentiation
with respect to :math:`t`,
at :math:`t = 0`, is the same as partial differentiation
with respect to :math:`x` at :math:`x = x^{(0)}`.

Sweep
=====

.. csv-table::
   :widths: auto

   **Index**,,**Operation**,,**Zero Order**,,**Derivative**,,**First Order**
   1,,:math:`v_1 = x`,,0.5,,:math:`v_1^{(1)} = x^{(1)}`,,:math:`v_1^{(1)} = 1`
   2,,:math:`v_2 = 1 + v_1`,,1.5,,:math:`v_2^{(1)} = v_1^{(1)}`,,:math:`v_2^{(1)} = 1`
   3,,:math:`v_3 = v_1 * v_1`,,0.25,,:math:`v_3^{(1)} = 2 * v_1^{(0)} * v_1^{(1)}`,,:math:`v_3^{(1)} = 1`
   4,,:math:`v_4 = v_3 / 2`,,0.125,,:math:`v_4^{(1)} = v_3^{(1)} / 2`,,:math:`v_4^{(1)} = 0.5`
   5,,:math:`v_5 = v_2 + v_4`,,1.625,,:math:`v_5^{(1)} = v_2^{(1)} + v_4^{(1)}`,,:math:`v_5^{(1)} = 1.5`

Return Value
************
The derivative of the return value for this case is

.. math::
   :nowrap:

   \begin{eqnarray}
      1.5
      & = &
      v_5^{(1)} =
      \left[ \D{v_5}{t} \right]_{t=0} =
      \left[ \D{}{t} f ( x^{(0)} + x^{(1)} t ) \right]_{t=0}
      \\
      & = &
      f^{(1)} ( x^{(0)} ) * x^{(1)} =
      f^{(1)} ( x^{(0)} )
   \end{eqnarray}

(We have used the fact that :math:`x^{(1)} = 1`.)
{xrst_toc_hidden
   introduction/exp_2_for1.cpp
}
Verification
************
The file :ref:`exp_2_for1.cpp-name` contains a routine
which verifies the values computed above.

Exercises
*********

#. Which statement in the routine defined by :ref:`exp_2_for1.cpp-name` uses
   the values that are calculated by the routine
   defined by :ref:`exp_2_for0.cpp-name` ?
#. Suppose that :math:`x = .1`,
   what are the results of a zero and first order forward sweep for
   the operation sequence above;
   i.e., what are the corresponding values for
   :math:`v_1^{(0)}, v_2^{(0)}, \cdots , v_5^{(0)}` and
   :math:`v_1^{(1)}, v_2^{(1)}, \cdots , v_5^{(1)}` ?
#. Create a modified version of :ref:`exp_2_for1.cpp-name` that verifies
   the derivative values from the previous exercise.
   Also create and run a main program that reports the result
   of calling the modified version of
   :ref:`exp_2_for1.cpp-name` .

{xrst_end exp_2_for1}
------------------------------------------------------------------------------
{xrst_begin exp_2_rev1}
{xrst_spell
   preform
   rcll
}

exp_2: First Order Reverse Mode
###############################

Purpose
*******
First order reverse mode uses the
:ref:`exp_2_for0@Operation Sequence` ,
and zero order forward sweep values,
to compute the first order derivative
of one dependent variable with respect to all the independent variables.
The computations are done in reverse
of the order of the computations in the original algorithm.

Mathematical Form
*****************
Suppose that we use the algorithm :ref:`exp_2.hpp-name` to compute

.. math::

   f(x) = 1 + x + x^2 / 2

The corresponding derivative function is

.. math::

   \partial_x f (x) =   1 + x

f_5
***
For our example, we chose to compute the derivative
of the value returned by :ref:`exp_2.hpp-name`
which is equal to the symbol :math:`v_5` in the
:ref:`exp_2 operation sequence<exp_2_for0@Operation Sequence>` .
We begin with the function :math:`f_5` where :math:`v_5`
is both an argument and the value of the function; i.e.,

.. math::
   :nowrap:

   \begin{eqnarray}
   f_5 ( v_1 , v_2 , v_3 , v_4 , v_5 ) & = & v_5
   \\
   \D{f_5}{v_5} & = & 1
   \end{eqnarray}

All the other partial derivatives of :math:`f_5` are zero.

Index 5: f_4
************
Reverse mode starts with the last operation in the sequence.
For the case in question, this is the operation with index 5,

.. math::

   v_5 = v_2 + v_4

We define the function
:math:`f_4 ( v_1 , v_2 , v_3 , v_4 )`
as equal to :math:`f_5`
except that :math:`v_5` is eliminated using
this operation; i.e.

.. math::

   f_4  =
   f_5 [  v_1 , v_2 , v_3 , v_4 , v_5 ( v_2 , v_4 ) ]

It follows that

.. math::

   \begin{array}{rcll}
   \D{f_4}{v_2}
   & = & \D{f_5}{v_2} +
      \D{f_5}{v_5} * \D{v_5}{v_2}
   & = 1
   \\
   \D{f_4}{v_4}
   & = & \D{f_5}{v_4} +
      \D{f_5}{v_5} * \D{v_5}{v_4}
   & = 1
   \end{array}

All the other partial derivatives of :math:`f_4` are zero.

Index 4: f_3
************
The next operation has index 4,

.. math::

   v_4 = v_3 / 2

We define the function
:math:`f_3 (  v_1 , v_2 , v_3 )`
as equal to :math:`f_4`
except that :math:`v_4` is eliminated using this operation; i.e.,

.. math::

   f_3 =
   f_4 [ v_1 , v_2 , v_3 , v_4 ( v_3 ) ]

It follows that

.. math::

   \begin{array}{rcll}
   \D{f_3}{v_1}
   & = & \D{f_4}{v_1}
   & = 0
   \\
   \D{f_3}{v_2}
   & = & \D{f_4}{v_2}
   & = 1
   \\
   \D{f_3}{v_3}
   & = & \D{f_4}{v_3} +
      \D{f_4}{v_4} * \D{v_4}{v_3}
   & = 0.5
   \end{array}

Index 3: f_2
************
The next operation has index 3,

.. math::

   v_3 = v_1 * v_1

We define the function
:math:`f_2 ( v_1 , v_2 )`
as equal to :math:`f_3`
except that :math:`v_3` is eliminated using this operation; i.e.,

.. math::

   f_2 =
   f_3 [ v_1 , v_2 , v_3 ( v_1 ) ]

Note that the value of :math:`v_1` is equal to :math:`x`
which is .5 for this evaluation.
It follows that

.. math::

   \begin{array}{rcll}
   \D{f_2}{v_1}
   & = & \D{f_3}{v_1} +
      \D{f_3}{v_3} * \D{v_3}{v_1}
   & = 0.5
   \\
   \D{f_2}{v_2}
   & = & \D{f_3}{v_2}
   & = 1
   \end{array}

Index 2: f_1
************
The next operation has index 2,

.. math::

   v_2 = 1 + v_1

We define the function
:math:`f_1 ( v_1 )`
as equal to :math:`f_2`
except that :math:`v_2` is eliminated using this operation; i.e.,

.. math::

   f_1 =
   f_2 [ v_1 , v_2 ( v_1 ) ]

It follows that

.. math::

   \begin{array}{rcll}
   \D{f_1}{v_1}
   & = & \D{f_2}{v_1} +
      \D{f_2}{v_2} * \D{v_2}{v_1}
   & = 1.5
   \end{array}

Note that :math:`v_1` is equal to :math:`x`,
so the derivative of this is the derivative of
the function defined by :ref:`exp_2.hpp-name` at :math:`x = .5`.
{xrst_toc_hidden
   introduction/exp_2_rev1.cpp
}
Verification
************
The file :ref:`exp_2_rev1.cpp-name` contains a routine
which verifies the values computed above.
It only tests the partial derivatives of
:math:`f_j` that might not be equal to the corresponding
partials of :math:`f_{j+1}`; i.e., the
other partials of :math:`f_j` must be equal to the corresponding
partials of :math:`f_{j+1}`.

Exercises
*********

#. Which statement in the routine defined by :ref:`exp_2_rev1.cpp-name` uses
   the values that are calculated by the routine
   defined by :ref:`exp_2_for0.cpp-name` ?
#. Consider the case where :math:`x = .1`
   and we first preform a zero order forward sweep
   for the operation sequence used above.
   What are the results of a
   first order reverse sweep; i.e.,
   what are the corresponding derivatives of
   :math:`f_5 , f_4 , \ldots , f_1`.
#. Create a modified version of
   :ref:`exp_2_rev1.cpp-name`
   that verifies the values you obtained for the previous exercise.
   Also create and run a main program that reports the result
   of calling the modified version of
   :ref:`exp_2_rev1.cpp-name` .

{xrst_end exp_2_rev1}
------------------------------------------------------------------------------
{xrst_begin exp_2_for2}

exp_2: Second Order Forward Mode
################################

Second Order Expansion
**********************
We define :math:`x(t)` near :math:`t = 0` by the second order expansion

.. math::

   x(t) = x^{(0)} + x^{(1)} * t + x^{(2)} * t^2 / 2

It follows that for :math:`k = 0 , 1 , 2`,

.. math::

   x^{(k)} = \dpow{k}{t} x (0)

Purpose
*******
In general, a second order forward sweep is given the
:ref:`exp_2_for1@First Order Expansion`
for all of the variables in an operation sequence,
and the second order derivatives for the independent variables.
It uses these to compute the second order derivative,
and thereby obtain the second order expansion,
for all the variables in the operation sequence.

Mathematical Form
*****************
Suppose that we use the algorithm :ref:`exp_2.hpp-name` to compute

.. math::

   f(x) = 1 + x + x^2 / 2

The corresponding second derivative function is

.. math::

   \Dpow{2}{x} f (x) =   1

Operation Sequence
******************
We consider the case where :ref:`exp_2.hpp-name` is executed with
:math:`x = .5`.
The corresponding operation sequence,
zero order forward sweep values,
and first order forward sweep values
are inputs and are used by a second order forward sweep.

Index
=====
The Index column contains the index in the operation sequence
of the corresponding atomic operation.
A Forward sweep starts with the first operation
and ends with the last.

Zero
====
The Zero column contains the zero order sweep results
for the corresponding variable in the operation sequence
(see :ref:`zero order sweep<exp_2_for0@Operation Sequence@Sweep>` ).

Operation
=========
The Operation column contains the
first order sweep operation for this variable.

First
=====
The First column contains the first order sweep results
for the corresponding variable in the operation sequence
(see :ref:`first order sweep<exp_2_for1@Operation Sequence@Sweep>` ).

Derivative
==========
The Derivative column contains the
mathematical function corresponding to the second derivative
with respect to :math:`t`,
at :math:`t = 0`, for each variable in the sequence.

Second
======
The Second column contains the second order derivatives
for the corresponding variable in the operation sequence; i.e.,
the second order expansion for the *i*-th variable is given by

.. math::

   v_i (t) = v_i^{(0)} + v_i^{(1)} * t +  v_i^{(2)} * t^2 / 2

We use :math:`x^{(0)} = 1`, and :math:`x^{(2)} = 0`
so that second order differentiation
with respect to :math:`t`, at :math:`t = 0`,
is the same as the second partial differentiation
with respect to :math:`x` at :math:`x = x^{(0)}`.

Sweep
=====

.. csv-table::
   :widths: auto

   **Index**,,**Zero**,,**Operation**,,**First**,,**Derivative**,,**Second**
   1,,0.5,,:math:`v_1^{(1)} = x^{(1)}`,,1,,:math:`v_1^{(2)} = x^{(2)}`,,:math:`v_1^{(2)} = 0`
   2,,1.5,,:math:`v_2^{(1)} = v_1^{(1)}`,,1,,:math:`v_2^{(2)} = v_1^{(2)}`,,:math:`v_2^{(2)} = 0`
   3,,0.25,,:math:`v_3^{(1)} = 2 * v_1^{(0)} * v_1^{(1)}`,,1,,:math:`v_3^{(2)} = 2 * (v_1^{(1)} * v_1^{(1)} + v_1^{(0)} * v_1^{(2)} )`,,:math:`v_3^{(2)} = 2`
   4,,0.125,,:math:`v_4^{(1)} = v_3^{(1)} / 2`,,.5,,:math:`v_4^{(2)} = v_3^{(2)} / 2`,,:math:`v_4^{(2)} = 1`
   5,,1.625,,:math:`v_5^{(1)} = v_2^{(1)} + v_4^{(1)}`,,1.5,,:math:`v_5^{(2)} = v_2^{(2)} + v_4^{(2)}`,,:math:`v_5^{(2)} = 1`

Return Value
************
The second derivative of the return value for this case is

.. math::
   :nowrap:

   \begin{eqnarray}
      1
      & = &
      v_5^{(2)} =
      \left[ \Dpow{2}{t} v_5 \right]_{t=0} =
      \left[ \Dpow{2}{t} f( x^{(0)} + x^{(1)} * t ) \right]_{t=0}
      \\
      & = &
      x^{(1)} * \Dpow{2}{x} f ( x^{(0)} ) * x^{(1)} =
      \Dpow{2}{x} f ( x^{(0)} )
   \end{eqnarray}

(We have used the fact that
:math:`x^{(1)} = 1` and :math:`x^{(2)} = 0`.)
{xrst_toc_hidden
   introduction/exp_2_for2.cpp
}
Verification
************
The file :ref:`exp_2_for2.cpp-name` contains a routine
which verifies the values computed above.

Exercises
*********

#. Which statement in the routine defined by :ref:`exp_2_for2.cpp-name` uses
   the values that are calculated by the routine
   defined by :ref:`exp_2_for1.cpp-name` ?
#. Suppose that :math:`x = .1`,
   what are the results of a zero, first, and second order forward sweep for
   the operation sequence above;
   i.e., what are the corresponding values for
   :math:`v_i^{(k)}` for :math:`i = 1, \ldots , 5` and :math:`k = 0, 1, 2`.
#. Create a modified version of :ref:`exp_2_for2.cpp-name` that verifies
   the derivative values from the previous exercise.
   Also create and run a main program that reports the result
   of calling the modified version of
   :ref:`exp_2_for2.cpp-name` .

{xrst_end exp_2_for2}
------------------------------------------------------------------------------
{xrst_begin exp_2_rev2}
{xrst_spell
   preform
   rcll
}

exp_2: Second Order Reverse Mode
################################

Purpose
*******
In general, a second order reverse sweep is given the
:ref:`exp_2_for1@First Order Expansion`
for all of the variables in an operation sequence.
Given a choice of a particular variable,
it computes the derivative,
of that variables first order expansion coefficient,
with respect to all of the independent variables.

Mathematical Form
*****************
Suppose that we use the algorithm :ref:`exp_2.hpp-name` to compute

.. math::

   f(x) = 1 + x + x^2 / 2

The corresponding second derivative is

.. math::

   \Dpow{2}{x} f (x) =   1

f_5
***
For our example, we chose to compute the derivative of :math:`v_5^{(1)}`
with respect to all the independent variable.
For the case computed for the
:ref:`first order sweep<exp_2_for1@Operation Sequence@Sweep>` ,
:math:`v_5^{(1)}` is the derivative
of the value returned by :ref:`exp_2.hpp-name` .
This the value computed will be the second derivative of
the value returned by  :ref:`exp_2.hpp-name` .
We begin with the function :math:`f_5` where :math:`v_5^{(1)}`
is both an argument and the value of the function; i.e.,

.. math::
   :nowrap:

   \begin{eqnarray}
   f_5 \left(
      v_1^{(0)}, v_1^{(1)} , \ldots  , v_5^{(0)} , v_5^{(1)}
   \right)
   & = & v_5^{(1)}
   \\
   \D{f_5}{v_5^{(1)}} & = & 1
   \end{eqnarray}

All the other partial derivatives of :math:`f_5` are zero.

Index 5: f_4
************
Second order reverse mode starts with the last operation in the sequence.
For the case in question, this is the operation with index 5.
The zero and first order sweep representations of this operation are

.. math::
   :nowrap:

   \begin{eqnarray}
      v_5^{(0)} & = & v_2^{(0)} + v_4^{(0)}
      \\
      v_5^{(1)} & = & v_2^{(1)} + v_4^{(1)}
   \end{eqnarray}

We define the function
:math:`f_4 \left( v_1^{(0)} , \ldots , v_4^{(1)} \right)`
as equal to :math:`f_5`
except that :math:`v_5^{(0)}` and :math:`v_5^{(1)}` are eliminated using
this operation; i.e.

.. math::

   f_4  =
   f_5 \left[   v_1^{(0)} , \ldots , v_4^{(1)} ,
      v_5^{(0)} \left( v_2^{(0)} , v_4^{(0)} \right) ,
      v_5^{(1)} \left( v_2^{(1)} , v_4^{(1)} \right)
   \right]

It follows that

.. math::

   \begin{array}{rcll}
   \D{f_4}{v_2^{(1)}}
   & = & \D{f_5}{v_2^{(1)}} +
      \D{f_5}{v_5^{(1)}} * \D{v_5^{(1)}}{v_2^{(1)}}
   & = 1
   \\
   \D{f_4}{v_4^{(1)}}
   & = & \D{f_5}{v_4^{(1)}} +
      \D{f_5}{v_5^{(1)}} * \D{v_5}{v_4^{(1)}}
   & = 1
   \end{array}

All the other partial derivatives of :math:`f_4` are zero.

Index 4: f_3
************
The next operation has index 4,

.. math::
   :nowrap:

   \begin{eqnarray}
      v_4^{(0)} & = & v_3^{(0)} / 2
      \\
      v_4^{(1)} & = & v_3^{(1)} / 2
   \end{eqnarray}

We define the function
:math:`f_3 \left(  v_1^{(0)} , \ldots , v_3^{(1)} \right)`
as equal to :math:`f_4`
except that :math:`v_4^{(0)}` and :math:`v_4^{(1)}`
are eliminated using this operation; i.e.,

.. math::

   f_3 =
   f_4 \left[ v_1^{(0)} , \ldots , v_3^{(1)} ,
      v_4^{(0)} \left( v_3^{(0)} \right) ,
      v_4^{(1)} \left( v_3^{(1)} \right)
   \right]

It follows that

.. math::

   \begin{array}{rcll}
   \D{f_3}{v_2^{(1)}}
   & = & \D{f_4}{v_2^{(1)}}
   & = 1
   \\
   \D{f_3}{v_3^{(1)}}
   & = & \D{f_4}{v_3^{(1)}} +
      \D{f_4}{v_4^{(1)}} * \D{v_4^{(1)}}{v_3^{(1)}}
   & = 0.5
   \end{array}

All the other partial derivatives of :math:`f_3` are zero.

Index 3: f_2
************
The next operation has index 3,

.. math::
   :nowrap:

   \begin{eqnarray}
      v_3^{(0)} & = & v_1^{(0)} * v_1^{(0)}
      \\
      v_3^{(1)} & = & 2 * v_1^{(0)} * v_1^{(1)}
   \end{eqnarray}

We define the function
:math:`f_2 \left(  v_1^{(0)} , \ldots , v_2^{(1)} \right)`
as equal to :math:`f_3`
except that :math:`v_3^{(0)}` and :math:`v_3^{(1)}` are
eliminated using this operation; i.e.,

.. math::

   f_2 =
   f_3 \left[ v_1^{(0)} , \ldots , v_2^{(1)} ,
      v_3^{(0)} ( v_1^{(0)} ) ,
      v_3^{(1)} ( v_1^{(0)} , v_1^{(1)} )
   \right]

Note that, from the
:ref:`first order forward sweep<exp_2_for1@Operation Sequence@Sweep>` ,
the value of :math:`v_1^{(0)}` is equal to :math:`.5`
and  :math:`v_1^{(1)}` is equal 1.
It follows that

.. math::

   \begin{array}{rcll}
   \D{f_2}{v_1^{(0)}}
   & = &
   \D{f_3}{v_1^{(0)}} +
      \D{f_3}{v_3^{(0)}} * \D{v_3^{(0)}}{v_1^{(0)}}  +
      \D{f_3}{v_3^{(1)}} * \D{v_3^{(1)}}{v_1^{(0)}}
   & = 1
   \\
   \D{f_2}{v_1^{(1)}}
   & = &
   \D{f_3}{v_1^{(1)}} +
      \D{f_3}{v_3^{(1)}} * \D{v_3^{(1)}}{v_1^{(1)}}
   & = 0.5
   \\
   \D{f_2}{v_2^{(0)}}
   & = & \D{f_3}{v_2^{(0)}}
   & = 0
   \\
   \D{f_2}{v_2^{(1)}}
   & = & \D{f_3}{v_2^{(1)}}
   & = 1
   \end{array}

Index 2: f_1
************
The next operation has index 2,

.. math::
   :nowrap:

   \begin{eqnarray}
      v_2^{(0)} & = & 1 + v_1^{(0)}
      \\
      v_2^{(1)} & = & v_1^{(1)}
   \end{eqnarray}

We define the function
:math:`f_1 ( v_1^{(0)} , v_1^{(1)} )`
as equal to :math:`f_2`
except that :math:`v_2^{(0)}` and  :math:`v_2^{(1)}`
are eliminated using this operation; i.e.,

.. math::

   f_1 =
   f_2 \left[ v_1^{(0)} , v_1^{(1)} ,
      v_2^{(0)} ( v_1^{(0)} )  ,
      v_2^{(1)} ( v_1^{(1)} )
   \right]

It follows that

.. math::

   \begin{array}{rcll}
   \D{f_1}{v_1^{(0)}}
   & = & \D{f_2}{v_1^{(0)}} +
      \D{f_2}{v_2^{(0)}} * \D{v_2^{(0)}}{v_1^{(0)}}
   & = 1
   \\
   \D{f_1}{v_1^{(1)}}
   & = & \D{f_2}{v_1^{(1)}} +
      \D{f_2}{v_2^{(1)}} * \D{v_2^{(1)}}{v_1^{(1)}}
   & = 1.5
   \end{array}

Note that :math:`v_1` is equal to :math:`x`,
so the second derivative of
the function defined by :ref:`exp_2.hpp-name` at :math:`x = .5`
is given by

.. math::

   \Dpow{2}{x} v_5^{(0)}
   =
   \D{ v_5^{(1)} }{x}
   =
   \D{ v_5^{(1)} }{v_1^{(0)}}
   =
   \D{f_1}{v_1^{(0)}} = 1

There is a theorem about Algorithmic Differentiation that explains why
the other partial of :math:`f_1` is equal to the
first derivative of
the function defined by :ref:`exp_2.hpp-name` at :math:`x = .5`.
{xrst_toc_hidden
   introduction/exp_2_rev2.cpp
}
Verification
************
The file :ref:`exp_2_rev2.cpp-name` contains a routine
which verifies the values computed above.
It only tests the partial derivatives of
:math:`f_j` that might not be equal to the corresponding
partials of :math:`f_{j+1}`; i.e., the
other partials of :math:`f_j` must be equal to the corresponding
partials of :math:`f_{j+1}`.

Exercises
*********

#. Which statement in the routine defined by :ref:`exp_2_rev2.cpp-name` uses
   the values that are calculated by the routine
   defined by :ref:`exp_2_for0.cpp-name` ?
   Which statements use values that are calculate by the routine
   defined in :ref:`exp_2_for1.cpp-name` ?
#. Consider the case where :math:`x = .1`
   and we first preform a zero order forward sweep,
   then a first order sweep,
   for the operation sequence used above.
   What are the results of a
   second order reverse sweep; i.e.,
   what are the corresponding derivatives of
   :math:`f_5 , f_4 , \ldots , f_1`.
#. Create a modified version of
   :ref:`exp_2_rev2.cpp-name`
   that verifies the values you obtained for the previous exercise.
   Also create and run a main program that reports the result
   of calling the modified version of
   :ref:`exp_2_rev2.cpp-name` .

{xrst_end exp_2_rev2}
------------------------------------------------------------------------------
