Advanced Topic: Using External C++ Functions#
This is based on the relevant portion of the CmdStan documentation here
Consider the following Stan model, based on the bernoulli example.
[2]:
with open('bernoulli_external.stan', 'r') as f:
stan_code = f.read()
print(stan_code)
functions {
real make_odds(real theta);
}
data {
int<lower=0> N;
array[N] int<lower=0, upper=1> y;
}
parameters {
real<lower=0, upper=1> theta;
}
model {
theta ~ beta(1, 1); // uniform prior on interval 0, 1
y ~ bernoulli(theta);
}
generated quantities {
real odds;
odds = make_odds(theta);
}
As you can see, it features a function declaration for make_odds, but no definition. If we try to compile this, we will get an error.
[3]:
from cmdstanpy import CmdStanModel
model_external = CmdStanModel(stan_file='bernoulli_external.stan', force_compile=True)
12:25:18 - cmdstanpy - INFO - compiling stan file /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.stan to exe file /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[3], line 2
1 from cmdstanpy import CmdStanModel
----> 2 model_external = CmdStanModel(stan_file='bernoulli_external.stan', force_compile=True)
File ~/Dev/py/cmdstanpy/cmdstanpy/model.py:183, in CmdStanModel.__init__(self, stan_file, exe_file, force_compile, stanc_options, cpp_options, user_header)
180 elif path not in self._stanc_options['include-paths']:
181 self._stanc_options['include-paths'].append(path)
--> 183 self._exe_file = compilation.compile_stan_file(
184 str(self.stan_file),
185 force=force_compile,
186 stanc_options=self._stanc_options,
187 cpp_options=cpp_options,
188 user_header=user_header,
189 )
191 # try to detect models w/out parameters, needed for sampler
192 if (not cmdstan_version_before(2, 27)) and cmdstan_version_before(
193 2, 36
194 ):
File ~/Dev/py/cmdstanpy/cmdstanpy/compilation.py:429, in compile_stan_file(src, force, stanc_options, cpp_options, user_header)
422 if 'PCH' in console or 'precompiled header' in console:
423 get_logger().warning(
424 "CmdStan's precompiled header (PCH) files "
425 "may need to be rebuilt."
426 "Please run cmdstanpy.rebuild_cmdstan().\n"
427 "If the issue persists please open a bug report"
428 )
--> 429 raise ValueError(
430 f"Failed to compile Stan model '{src}'. Console:\n{console}"
431 )
432 return str(exe_target)
ValueError: Failed to compile Stan model '/home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.stan'. Console:
--- Translating Stan model to C++ code ---
bin/stanc --filename-in-msg=bernoulli_external.stan --o=/home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.hpp /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.stan
Semantic error in 'bernoulli_external.stan', line 2, column 7 to column 16:
-------------------------------------------------
1: functions {
2: real make_odds(real theta);
^
3: }
4: data {
-------------------------------------------------
Function 'make_odds' is declared without specifying a definition.
make: *** [make/program:66: /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.hpp] Error 1
Command ['make', 'STANCFLAGS+=--filename-in-msg=bernoulli_external.stan', '/home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external']
exited with code '2' No such file or directory
Even enabling the --allow-undefined flag to stanc3 will not allow this model to be compiled quite yet.
[4]:
model_external = CmdStanModel(stan_file='bernoulli_external.stan', force_compile=True, stanc_options={'allow-undefined':True})
12:25:18 - cmdstanpy - INFO - compiling stan file /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.stan to exe file /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[4], line 1
----> 1 model_external = CmdStanModel(stan_file='bernoulli_external.stan', force_compile=True, stanc_options={'allow-undefined':True})
File ~/Dev/py/cmdstanpy/cmdstanpy/model.py:183, in CmdStanModel.__init__(self, stan_file, exe_file, force_compile, stanc_options, cpp_options, user_header)
180 elif path not in self._stanc_options['include-paths']:
181 self._stanc_options['include-paths'].append(path)
--> 183 self._exe_file = compilation.compile_stan_file(
184 str(self.stan_file),
185 force=force_compile,
186 stanc_options=self._stanc_options,
187 cpp_options=cpp_options,
188 user_header=user_header,
189 )
191 # try to detect models w/out parameters, needed for sampler
192 if (not cmdstan_version_before(2, 27)) and cmdstan_version_before(
193 2, 36
194 ):
File ~/Dev/py/cmdstanpy/cmdstanpy/compilation.py:429, in compile_stan_file(src, force, stanc_options, cpp_options, user_header)
422 if 'PCH' in console or 'precompiled header' in console:
423 get_logger().warning(
424 "CmdStan's precompiled header (PCH) files "
425 "may need to be rebuilt."
426 "Please run cmdstanpy.rebuild_cmdstan().\n"
427 "If the issue persists please open a bug report"
428 )
--> 429 raise ValueError(
430 f"Failed to compile Stan model '{src}'. Console:\n{console}"
431 )
432 return str(exe_target)
ValueError: Failed to compile Stan model '/home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.stan'. Console:
ERROR: Missing user header.
Because --allow-undefined is set, we need a C++ header file to include.
We tried to find the user header at:
/home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/user_header.hpp
You can also set the USER_HEADER variable to the path of your C++ file.
make: *** [make/program:52: /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/user_header.hpp] Error 1
Command ['make', 'STANCFLAGS+=--filename-in-msg=bernoulli_external.stan', 'STANCFLAGS+=--allow-undefined', '/home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external']
exited with code '2' No such file or directory
To resolve this, we need to both tell the Stan compiler an undefined function is okay and let C++ know what it should be.
We can provide a definition in a C++ header file by using the user_header argument to either the CmdStanModel constructor or the compile method.
This will enables the allow-undefined flag automatically.
[5]:
model_external = CmdStanModel(stan_file='bernoulli_external.stan', user_header='make_odds.hpp')
12:25:18 - cmdstanpy - INFO - compiling stan file /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external.stan to exe file /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external
12:25:34 - cmdstanpy - INFO - compiled model executable: /home/brian/Dev/py/cmdstanpy/docsrc/users-guide/examples/bernoulli_external
We can then run this model and inspect the output
[6]:
fit = model_external.sample(data={'N':10, 'y':[0,1,0,0,0,0,0,0,0,1]})
fit.stan_variable('odds')
12:25:34 - cmdstanpy - INFO - CmdStan start processing
12:25:34 - cmdstanpy - INFO - CmdStan done processing.
[6]:
array([0.17954691, 0.29004656, 0.29004656, ..., 0.11342197, 0.13338524,
0.18325849])
The contents of this header file are presented without comment:
#include <ostream>
double make_odds(const double& theta, std::ostream *pstream__) {
return theta / (1 - theta);
}
Additional guidance, including on writing functions with known derivatives, can be found in the CmdStan documentation.