16 stansummary
: MCMC Output Analysis
The CmdStan stansummary
program reports statistics for one or more sampler chains
over all sampler and model parameters and quantities of interest.
The statistics reported include both summary statistics of the estimates and
diagnostic statistics on the sampler chains, reported in the following order:
- Mean - sample mean
- MCSE - Monte Carlo Standard Error, a measure of the amount of noise in the sample
- StdDev - sample standard deviation
- Quantiles - default 5%, 50%, 95%
- N_eff - effective sample size - the number of independent draws in the sample
- N_eff/S - the number of independent draws per second
- R_hat - \(\hat{R}\) statistic, a measure of chain equilibrium, must be within \(0.05\) of \(1.0\).
When reviewing the stansummary
output, it is important to check the final three
output columns first - these are the diagnostic statistics on chain convergence and
number of independent draws in the sample.
A \(\hat{R}\) statistic of greater than \(1.05\) indicates that the chain has not converged and
therefore the sample is not drawn from the posterior, thus the estimates of the mean and
all other summary statistics are invalid.
Estimation by sampling produces an approximate value for the model parameters; the MCSE statistic indicates the amount of noise in the estimate. Therefore MCSE column is placed next to the sample mean column, in order to make it easy to compare this sample with others.
For more information, see the Posterior Analysis chapter of the Stan Reference Manual which describes both the theory and practice of MCMC estimation techniques. The summary statistics and the algorithms used to compute them are described in sections Notation for samples and Effective Sample Size.
16.1 Building the stansummary command
The CmdStan makefile task build
compiles the stansummary
utility
into the bin
directory.
It can be compiled directly using the makefile as follows:
> cd <cmdstan-home>
> make bin/stansummary
16.2 Running the stansummary
program
The stansummary
utility processes one or more output files from a set of
chains from one run of the HMC sampler.
To run stansummary
on the output file or files generated by a run of
the sampler, on Mac or Linux:
<cmdstan-home>/bin/stansummary <file_1.csv> ... <file_N.csv>
On Windows, use backslashes to call the stansummary.exe
.
<cmdstan-home>\bin\stansummary.exe <file_1.csv> ... <file_N.csv>
For example, after running 4 chains to fit the example model eight_schools.stan
to the supplied example data file, we run stansummary
on the resulting Stan CSV
output files to get the following report:
> bin/stansummary eight_*.csv
Input files: eight_1.csv, eight_2.csv, eight_3.csv, eight_4.csv
Inference for Stan model: eight_schools_model
4 chains: each with iter=(1000,1000,1000,1000); warmup=(0,0,0,0); thin=(1,1,1,1); 4000 iterations saved.
Warmup took (0.048, 0.060, 0.047, 0.045) seconds, 0.20 seconds total
Sampling took (0.057, 0.058, 0.061, 0.066) seconds, 0.24 seconds total
Mean MCSE StdDev 5% 50% 95% N_Eff N_Eff/s R_hat
lp__ -18 0.33 5.1 -26 -19 -9.1 233 963 1.0
accept_stat__ 0.88 1.6e-02 0.23 0.21 0.98 1.00 203 838 1.0e+00
stepsize__ 0.18 2.2e-02 0.031 0.14 0.20 0.22 2.0 8.3 3.9e+13
treedepth__ 3.8 5.9e-02 0.78 2.0 4.0 5.0 175 724 1.0e+00
n_leapfrog__ 18 1.3e+00 9.4 7.0 15 31 51 212 1.0e+00
divergent__ 0.015 4.1e-03 0.12 0.00 0.00 0.00 865 3576 1.0e+00
energy__ 23 3.4e-01 5.5 13 23 32 258 1066 1.0e+00
mu 7.9 0.16 5.1 -0.23 7.9 16 1021 4221 1.0
theta[1] 12 0.30 8.6 -0.48 11 28 837 3459 1.0
theta[2] 7.8 0.15 6.4 -2.7 7.7 18 1717 7096 1.00
theta[3] 6.1 0.19 7.7 -6.5 6.5 18 1684 6958 1.0
theta[4] 7.5 0.15 6.7 -3.1 7.4 18 2026 8373 1.0
theta[5] 4.7 0.17 6.4 -6.7 5.3 15 1391 5747 1.00
theta[6] 5.9 0.16 6.7 -5.8 6.2 16 1673 6915 1.00
theta[7] 11 0.22 7.0 0.057 10 23 1069 4419 1.0
theta[8] 8.3 0.20 7.9 -4.2 8.0 22 1503 6209 1.00
tau 7.2 0.26 5.2 1.5 5.9 17 401 1657 1.0
Samples were drawn using hmc with nuts.
For each parameter, N_Eff is a crude measure of effective sample size,
and R_hat is the potential scale reduction factor on split chains (at
convergence, R_hat=1).
The console output information consists of
- Model, chains, and timing summaries
- Sampler parameter statistics
- Model parameter statistics
- Sampling algorithm - either nuts (shown here) or static HMC.
There is one row per parameter and the row order in the summary report corresponds to the column order in the Stan CSV output file.
16.2.1 Sampler parameters
The initial Stan CSV columns provide information on the sampler state for each draw:
lp__
- the total log probability density (up to an additive constant) at each sampleaccept_stat__
- the average Metropolis acceptance probability over each simulated Hamiltonian trajectorystepsize__
- integrator step sizetreedepth__
- depth of tree used by NUTS (NUTS sampler)n_leapfrog__
- number of leapfrog calculations (NUTS sampler)divergent__
- has value1
if trajectory diverged, otherwise0
. (NUTS sampler)energy__
- value of the Hamiltonianint_time__
- total integration time (static HMC sampler)
Because we ran the NUTS sampler, the above summary reports sampler parameters
treedepth__
, n_leapfrog__
, and divergent__
;
the static HMC sampler would report int_time__
instead.
16.2.2 Model parameters and quantities of interest
The remaining Stan CSV columns report the values of all parameters, transformed parameters,
and generated quantities in the order in which these variables are declared in the Stan program.
For container variables, i.e., vector, row_vector, matrix, and array variables,
the statistics for each element are reported separately, in row-major order.
The eight_schools.stan
program parameters block contains the following parameter variable
declarations:
real mu;
real theta[J];
real<lower=0> tau;
In the example data, J
is \(8\); therefore the stansummary listing
reports on theta[1]
through theta[8]
.
16.3 Command-line options
The stansummary
command syntax provides a set of flags to customize the output
which must precede the list of filenames.
When invoked with no arguments or with the -h
or --help
option,
the program prints the usage message to the console and exits.
Report statistics for one or more Stan CSV files from a HMC sampler run.
Example: stansummary model_chain_1.csv model_chain_2.csv
Options:
-a, --autocorr [n] Display the chain autocorrelation for the n-th
input file, in addition to statistics.
-c, --csv_filename [file] Write statistics to a CSV file.
-h, --help Produce help message, then exit.
-p, --percentiles [values] Percentiles to report as ordered set of
comma-separated integers from (1,99), inclusive.
Default is 5,50,95.
-s, --sig_figs [n] Significant figures reported. Default is 2.
Must be an integer from (1, 18), inclusive.
Both short an long option names are allowed. Short names are specified as -<o> <value>
;
long option names can be specified either as --<option>=<value>
or --<option> <value>
.
The amount of precision
in the sampler output limits the amount of real precision in the summary report.
CmdStan’s command line interface also has output argument sig_figs
.
The default sampler output precision is 6.
The --sig_figs
argument to the stansummary program should not exceed the sig_figs
argument to the sampler.