Stan Math Library
4.4.0
Automatic Differentiation

The automatic differentiation testing framework tests the value and derivatives (up to third order) of a function against results derived from the primitive implementation with finite differences. Multivariate functions are tested as unary functions by projecting to each output dimension. All that needs to be provided is a working primitive implementation and the inputs to test.
Derivatives are tested
var
(gradient), fvar<double>
tested using gradient()
functionalsfvar<var>
, fvar<fvar<double>>
tested using hessian()
functionalsfvar<fvar<var>>
tested using grad_hessian()
functionalint
or double
) and automatic differentiation variables, andExceptions are tested insofar as if one version of the primitive or autodiff versions of the function throws an exception they must all throw an exception That is, each autodiff call must throw exceptions for exactly the same argument values as the primitive implementation.
The framework can test unary and binary functions with any of the following input and output types.
The basic types that can be handled by the testing framework are
int
: integerdouble
: doubleprecision floating pointEigen::Matrix<double, 1, 1>
: column vectorEigen::Matrix<double, 1, 1>
: row vectorEigen::Matrix<double, 1, 1>
: matrixFor any type T
that can be handled, the framework can also handle the corresponding standard vector type (used for arrays in Stan)
std::vector<T>
: array of elements of type T
The tests are instrumented to test relative error between the values and derivatives produced by the function being tested and those produced by finite differences. For nonzero inputs, the relative error is defined to be the error divided by the average absolute values of the arguments:
If one of the inputs is zero, error is used rather than relative error (because relative error devolves if one or both inputs are zero)
The following table lists the complete set of tests, with their default tolerances, the type of automatic differentiation variable used, the name of the functional used to perform the test, and the order of the test. The first column lists the names of the member variables in the simple struct stan::test::ad_tolerances
. Every test accepts an ad_tolerances
object as a first argument, so the defaults may be overridden for particular tests. The default tolerances are set relatively high because of the inaccuracy of simple finite difference calculations as used by these tests.
function  tolerance  autodiff type  functional  (order) test 

gradient_val_  1e8  var  gradient  (0) value 
gradient_grad_  1e4  var  gradient  (1) gradient 
gradient_fvar_val_  1e8  fvar<double>  gradient  (0) value 
gradient_fvar_grad_  1e4  fvar<double>  gradient  (1) gradient 
hessian_val_  1e8  fvar<var>  hessian  (0) value 
hessian_grad_  1e4  fvar<var>  hessian  (1) gradient 
hessian_hessian_  1e3  fvar<var>  hessian  (2) Hessian 
hessian_fvar_val_  1e8  fvar<fvar<double>>  hessian  (0) value 
hessian_fvar_grad_  1e4  fvar<fvar<double>>  hessian  (1) gradient 
hessian_fvar_hessian_  1e3  fvar<fvar<double>>  hessian  (2) Hessian 
grad_hessian_val_  1e8  fvar<fvar<var>>  grad_hessian  (0) value 
grad_hessian_hessian_  1e3  fvar<fvar<var>>  grad_hessian  (2) Hessian 
grad_hessian_grad_hessian_  1e2  fvar<fvar<var>>  grad_hessian  (3) gradient of Hessian 
These examples are all included in the unit tests for the test framework in file test/unit/math/test_ad_test.cpp
Let's start by seeing how to test matrix inverse. In Stan, that's a unary function inverse
that takes a matrix input and returns a matrix output.
We'll need to include the test framework (which includes both the Google test framework and all of the Stan math library (by including `<stan/math.hpp>). The entire library is included to ensure that all functions work under the standard include system when all higherorder autodiff is included.
Each test is scoped within a Google test framework TEST
macro. Thus all of our test files will look like this:
If there is more than one test, there will be more than one TEST
macro invocation, each with a unique combination of test name and subtest name.
The body of a test will define a polymorphic functor to test, then a sequence of arguments to test with thest framework. Although a functor is necessary in order for there to be a single type encapsulating all possible overloads, it can be a simple closure.
Here's what the first test looks like for inverse
, repeating the macro wrapper and include
That's it. The inverse_functor
is defined using a lambda (with no closure — the [ ]
bit) taking a const auto&
argument and returning the result of applying stan::math::inverse
. Generic lambdas are a C++14 feature (in std=c++1y
for the compiler). They're convenient here because they define a polymorphic functor defined for any argument for which the body is defined, which here means any argument u
for which stan::math::inverse(u)
is defined. Bare functions aren't enough for the testing framework functions like expect_ad
to perform type inference as they do not have a single type. The functor defined by the lambda closure has a single type.
The second bit of the code defines the Matrix to test, here an invertible positivedefinite matrix with first row [1.9 0.3]
and second row [0.3 1.9]
.
The final bit of the code simply invokes the test framework for the functor and argument(s) being tested. The expect_ad
function is both variadic and overloaded to deal with arbitrary Stanrelevant input and output types.
If we wanted to lower the tolerance for the gradient test using reversemode autodiff, we could use
Binary functions are handled the same way as unary functions. The framework tests all combinations of input autodiff types, e.g., f(double, var)
, f(var, double)
, and f(var, var)
for testing reversemode instantiations. When there is more than one autodiff variable argument, all autodiff arguments must be of the same type.
The lambda here abstracts two arguments to which it applies the multiply
function. There are two tests. Here The first takes y = 3
and the second y = NaN
.
Most of the unary scalar functions in Stan are vectorized in that they can be applied elementwise to any container type.
The only difference in calls is that the expect_ad_vectorized
function is used in place of expect_ad
. The log10
function is vectorized to operate elementwise, and the following code will test it.
The following function has an autodiff implementation that differs by a negation from the primitive version, throwing off value and derivative tests.
Uncommenting the test above in the test file and running produces the exception diagnostic: