This is an old version, view current version.

16 stansummary: MCMC Output Analysis

The CmdStan stansummary program reports statistics for one or more sampler chains over all sampler and model parameters and quantities of interest. The statistics reported include both summary statistics of the estimates and diagnostic statistics on the sampler chains, reported in the following order:

  • Mean - sample mean
  • MCSE - Monte Carlo Standard Error, a measure of the amount of noise in the sample
  • StdDev - sample standard deviation
  • Quantiles - default 5%, 50%, 95%
  • N_eff - effective sample size - the number of independent draws in the sample
  • N_eff/S - the number of independent draws per second
  • R_hat - \(\hat{R}\) statistic, a measure of chain equilibrium, must be within \(0.05\) of \(1.0\).

When reviewing the stansummary output, it is important to check the final three output columns first - these are the diagnostic statistics on chain convergence and number of independent draws in the sample. A \(\hat{R}\) statistic of greater than \(1.05\) indicates that the chain has not converged and therefore the sample is not drawn from the posterior, thus the estimates of the mean and all other summary statistics are invalid.

Estimation by sampling produces an approximate value for the model parameters; the MCSE statistic indicates the amount of noise in the estimate. Therefore MCSE column is placed next to the sample mean column, in order to make it easy to compare this sample with others.

For more information, see the Posterior Analysis chapter of the Stan Reference Manual which describes both the theory and practice of MCMC estimation techniques. The summary statistics and the algorithms used to compute them are described in sections Notation for samples and Effective Sample Size.

16.1 Building the stansummary command

The CmdStan makefile task build compiles the stansummary utility into the bin directory. It can be compiled directly using the makefile as follows:

> cd <cmdstan-home>
> make bin/stansummary

16.2 Running the stansummary program

The stansummary utility processes one or more output files from a set of chains from one run of the HMC sampler. To run stansummary on the output file or files generated by a run of the sampler, on Mac or Linux:

<cmdstan-home>/bin/stansummary <file_1.csv> ... <file_N.csv>

On Windows, use backslashes to call the stansummary.exe.

<cmdstan-home>\bin\stansummary.exe <file_1.csv> ... <file_N.csv>

For example, after running 4 chains to fit the example model eight_schools.stan to the supplied example data file, we run stansummary on the resulting Stan CSV output files to get the following report:

> bin/stansummary eight_*.csv
Input files: eight_1.csv, eight_2.csv, eight_3.csv, eight_4.csv
Inference for Stan model: eight_schools_model
4 chains: each with iter=(1000,1000,1000,1000); warmup=(0,0,0,0); thin=(1,1,1,1); 4000 iterations saved.

Warmup took (0.048, 0.060, 0.047, 0.045) seconds, 0.20 seconds total
Sampling took (0.057, 0.058, 0.061, 0.066) seconds, 0.24 seconds total

                 Mean     MCSE  StdDev     5%   50%   95%  N_Eff  N_Eff/s    R_hat

lp__              -18     0.33     5.1    -26   -19  -9.1    233      963      1.0
accept_stat__    0.88  1.6e-02    0.23   0.21  0.98  1.00    203      838  1.0e+00
stepsize__       0.18  2.2e-02   0.031   0.14  0.20  0.22    2.0      8.3  3.9e+13
treedepth__       3.8  5.9e-02    0.78    2.0   4.0   5.0    175      724  1.0e+00
n_leapfrog__       18  1.3e+00     9.4    7.0    15    31     51      212  1.0e+00
divergent__     0.015  4.1e-03    0.12   0.00  0.00  0.00    865     3576  1.0e+00
energy__           23  3.4e-01     5.5     13    23    32    258     1066  1.0e+00

mu                7.9     0.16     5.1  -0.23   7.9    16   1021     4221      1.0
theta[1]           12     0.30     8.6  -0.48    11    28    837     3459      1.0
theta[2]          7.8     0.15     6.4   -2.7   7.7    18   1717     7096     1.00
theta[3]          6.1     0.19     7.7   -6.5   6.5    18   1684     6958      1.0
theta[4]          7.5     0.15     6.7   -3.1   7.4    18   2026     8373      1.0
theta[5]          4.7     0.17     6.4   -6.7   5.3    15   1391     5747     1.00
theta[6]          5.9     0.16     6.7   -5.8   6.2    16   1673     6915     1.00
theta[7]           11     0.22     7.0  0.057    10    23   1069     4419      1.0
theta[8]          8.3     0.20     7.9   -4.2   8.0    22   1503     6209     1.00
tau               7.2     0.26     5.2    1.5   5.9    17    401     1657      1.0

Samples were drawn using hmc with nuts.
For each parameter, N_Eff is a crude measure of effective sample size,
and R_hat is the potential scale reduction factor on split chains (at 
convergence, R_hat=1).

The console output information consists of

  • Model, chains, and timing summaries
  • Sampler parameter statistics
  • Model parameter statistics
  • Sampling algorithm - either nuts (shown here) or static HMC.

There is one row per parameter and the row order in the summary report corresponds to the column order in the Stan CSV output file.

16.2.1 Sampler parameters

The initial Stan CSV columns provide information on the sampler state for each draw:

  • lp__ - the total log probability density (up to an additive constant) at each sample
  • accept_stat__ - the average Metropolis acceptance probability over each simulated Hamiltonian trajectory
  • stepsize__ - integrator step size
  • treedepth__ - depth of tree used by NUTS (NUTS sampler)
  • n_leapfrog__ - number of leapfrog calculations (NUTS sampler)
  • divergent__ - has value 1 if trajectory diverged, otherwise 0. (NUTS sampler)
  • energy__ - value of the Hamiltonian
  • int_time__ - total integration time (static HMC sampler)

Because we ran the NUTS sampler, the above summary reports sampler parameters treedepth__, n_leapfrog__, and divergent__; the static HMC sampler would report int_time__ instead.

16.2.2 Model parameters and quantities of interest

The remaining Stan CSV columns report the values of all parameters, transformed parameters, and generated quantities in the order in which these variables are declared in the Stan program. For container variables, i.e., vector, row_vector, matrix, and array variables, the statistics for each element are reported separately, in row-major order. The eight_schools.stan program parameters block contains the following parameter variable declarations:

  real mu;
  real theta[J];
  real<lower=0> tau;

In the example data, J is \(8\); therefore the stansummary listing reports on theta[1] through theta[8].

16.3 Command-line options

The stansummary command syntax provides a set of flags to customize the output which must precede the list of filenames. When invoked with no arguments or with the -h or --help option, the program prints the usage message to the console and exits.

Report statistics for one or more Stan CSV files from a HMC sampler run.
Example:  stansummary model_chain_1.csv model_chain_2.csv
Options:
  -a, --autocorr [n]          Display the chain autocorrelation for the n-th
                              input file, in addition to statistics.
  -c, --csv_filename [file]   Write statistics to a CSV file.
  -h, --help                  Produce help message, then exit.
  -p, --percentiles [values]  Percentiles to report as ordered set of
                              comma-separated integers from (1,99), inclusive.
                              Default is 5,50,95.
  -s, --sig_figs [n]          Significant figures reported. Default is 2.
                              Must be an integer from (1, 18), inclusive.

Both short an long option names are allowed. Short names are specified as -<o> <value>; long option names can be specified either as --<option>=<value> or --<option> <value>.

The amount of precision in the sampler output limits the amount of real precision in the summary report. CmdStan’s command line interface also has output argument sig_figs. The default sampler output precision is 6. The --sig_figs argument to the stansummary program should not exceed the sig_figs argument to the sampler.