stansummary: MCMC Output Analysis

The CmdStan stansummary program reports statistics for one or more sampler chains over all sampler and model parameters and quantities of interest. The statistics reported include both summary statistics of the estimates and diagnostic statistics on the sampler chains, reported in the following order:

  • Mean - sample mean
  • MCSE - Monte Carlo Standard Error, a measure of the amount of noise in the sample
  • StdDev - sample standard deviation - the standard deviation around the sample mean.
  • MAD - Median Absolute Deviation - the median absolute deviation around the sample median.
  • Quantiles - default 5%, 50%, 95%
  • ESS_bulk
  • ESS_tail
  • R_hat - \(\hat{R}\) statistic, a MCMC convergence diagnostic

When reviewing the stansummary output, it is important to check the final three output columns first - these are the diagnostic statistics on MCMC convergence and effective sample size. A \(\hat{R}\) statistic of greater than \(1\) indicates potential convergence problems and that the sample is not presentative of the target posterior, thus the estimates of the mean and all other summary statistics are likely to be invalid. A value \(1.01\) can be used as generic threshold to decide whether more iterations or further convergence analysis is needed, but other thresholds can be used depending on the specific use case.

Estimation by sampling produces an approximate value for the model parameters; the MCSE statistic indicates the amount of uncertainty in the estimate. Therefore MCSE column is placed next to the sample mean column, in order to make it easy to compare this sample with others.

For more information, see the Posterior Analysis chapter of the Stan Reference Manual which describes both the theory and practice of MCMC estimation techniques.

The statistics - Mean, StdDev, MAD, and Quantiles - are computed directly from all draws across all chains. The diagnostic statistics - ESS_bulk, ESS_tail, and R_hat are computed from the rank-normalized, folded, and splitted chains according to the definitions by Vehtari et al. (2021). the MCSE statistic is computed using split chain R_hat and autocorrelations. The summary statistics and the algorithms used to compute them are described in sections Notation for samples and Effective Sample Size.

Building the stansummary command

The CmdStan makefile task build compiles the stansummary utility into the bin directory. It can be compiled directly using the makefile as follows:

> cd <cmdstan-home>
> make bin/stansummary

Running the stansummary program

The stansummary utility processes one or more output files from a set of chains from one run of the HMC sampler. To run stansummary on the output file or files generated by a run of the sampler, on Mac or Linux:

<cmdstan-home>/bin/stansummary <file_1.csv> ... <file_N.csv>

On Windows, use backslashes to call the stansummary.exe.

<cmdstan-home>\bin\stansummary.exe <file_1.csv> ... <file_N.csv>

For example, after running 4 chains to fit the example model eight_schools.stan to the supplied example data file, we run stansummary on the resulting Stan CSV output files to get the following report:

> bin/stansummary eight_*.csv
Inference for Stan model: eight_schools_model
4 chains: each with iter=1000; warmup=1000; thin=1; 1000 iterations saved.

Warmup took (0.031, 0.031, 0.038, 0.031) seconds, 0.13 seconds total
Sampling took (0.032, 0.039, 0.026, 0.029) seconds, 0.13 seconds total

                 Mean   MCSE  StdDev    MAD     5%   50%   95%  ESS_bulk  ESS_tail  R_hat

lp__              -18   0.36     5.3    5.5    -26   -18  -8.7       214       217    1.0
accept_stat__    0.80  0.021    0.29  0.068  0.033  0.95  1.00       268       314    1.0
stepsize__       0.20    nan   0.040  0.029   0.14  0.21  0.25       nan       nan    nan
treedepth__       3.7   0.22    0.88    1.5    2.0   4.0   5.0        26        18    1.1
n_leapfrog__       17    3.1      10     12    3.0    15    31        30        21    1.1
divergent__     0.022    nan    0.15   0.00   0.00  0.00  0.00       nan       nan    nan
energy__           23   0.38     5.7    5.8     13    23    32       223       286    1.0

mu                7.9   0.16     5.3    4.9  -0.35   7.9    16       977       472    1.0
theta[1]           12   0.32     8.8    7.4  -0.19    11    28       727       525    1.0
theta[2]          7.8   0.17     6.4    6.0   -2.8   7.8    18      1298      2137    1.0
theta[3]          5.9   0.20     8.1    6.8   -8.0   6.5    18      1493      1720    1.0
theta[4]          7.5   0.17     6.9    6.3   -3.4   7.6    18      1517      1935    1.0
theta[5]          4.9   0.20     6.3    6.1   -6.1   5.1    15       998      1357    1.0
theta[6]          5.8   0.18     6.8    6.1   -6.3   6.2    16      1434      1431    1.0
theta[7]           11   0.25     7.0    6.4   0.42    10    23       741       562    1.0
theta[8]          8.6   0.21     8.1    7.2   -3.6   8.3    22      1444      1850    1.0
tau               7.3   0.31     5.8    4.4    1.5   5.7    18       226      1230    1.0

Samples were drawn using hmc with nuts.
For each parameter, ESS_bulk and ESS_tail measure the effective sample size for the entire sample
(bulk) and for the .05 and .95 tails (tail),
and R_hat measures the potential scale reduction on split chains. At convergence R_hat will be
very close to 1.00.

The console output information consists of

  • Model, chains, and timing summaries
  • Sampler parameter statistics
  • Model parameter statistics
  • Sampling algorithm - either nuts (shown here) or static HMC.

There is one row per parameter and the row order in the summary report corresponds to the column order in the Stan CSV output file.

Sampler parameters

The initial Stan CSV columns provide information on the sampler state for each draw:

  • lp__ - the total log probability density (up to an additive constant) at each sample
  • accept_stat__ - the average Metropolis acceptance probability over each simulated Hamiltonian trajectory
  • stepsize__ - integrator step size
  • treedepth__ - depth of tree used by NUTS (NUTS sampler)
  • n_leapfrog__ - number of leapfrog calculations (NUTS sampler)
  • divergent__ - has value 1 if trajectory diverged, otherwise 0. (NUTS sampler)
  • energy__ - value of the Hamiltonian
  • int_time__ - total integration time (static HMC sampler)

Because we ran the NUTS sampler, the above summary reports sampler parameters treedepth__, n_leapfrog__, and divergent__; the static HMC sampler would report int_time__ instead.

Model parameters and quantities of interest

The remaining Stan CSV columns report the values of all parameters, transformed parameters, and generated quantities in the order in which these variables are declared in the Stan program. For container variables, i.e., vector, row_vector, matrix, and array variables, the statistics for each element are reported separately, in row-major order. The eight_schools.stan program parameters block contains the following parameter variable declarations:

  real mu;
  array[J] real theta;
  real<lower=0> tau;

In the example data, J is \(8\); therefore the stansummary listing reports on theta[1] through theta[8].

Command-line options

The stansummary command syntax provides a set of flags to customize the output which must precede the list of filenames. When invoked with no arguments or with the -h or --help option, the program prints the usage message to the console and exits.

Report statistics for one or more Stan CSV files from a HMC sampler run.
Example:  stansummary model_chain_1.csv model_chain_2.csv
Options:
  -a, --autocorr [n]          Display the chain autocorrelation for the n-th
                              input file, in addition to statistics.
  -c, --csv_filename [file]   Write statistics to a CSV file.
  -h, --help                  Produce help message, then exit.
  -p, --percentiles [values]  Percentiles to report as ordered set of
                              comma-separated numbers from (0.1,99.9), inclusive.
                              Default is 5,50,95.
  -s, --sig_figs [n]          Significant figures reported. Default is 2.
                              Must be an integer from (1, 18), inclusive.
  -i, --include_param [name]  Include the named parameter in the summary output.
                              By default, all parameters in the file are summarized,
                              passing this argument one or more times will filter
                              the output down to just the requested arguments.

Both short an long option names are allowed. Short names are specified as -<o> <value>; long option names can be specified either as --<option>=<value> or --<option> <value>.

The --percentiles argument can also be passed an empty string "", which results in no percentiles being displayed in the output of the command.

The amount of precision in the sampler output limits the amount of real precision in the summary report. CmdStan’s command line interface also has output argument sig_figs. The default sampler output precision is 6. The --sig_figs argument to the stansummary program should not exceed the sig_figs argument to the sampler.

Back to top

References

Vehtari, Aki, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. 2021. “Rank-Normalization, Folding, and Localization: An Improved \(\widehat{R}\) for Assessing Convergence of MCMC.” Bayesian Analysis 16: 667–718.