Sacred (Python)
A Python package to "configure, organize, log and reproduce computational experiments" according to the documentation.
Experiment Tracking
Suppose we are interested in answering some specific research question, e.g. how does changing the configuration to specific parameters affect the performance for some model. This involves a training loop which fits some model and final output(s) which measures the performance of the fitted model. There are at least 3 objects of interest here:
- The final outputs, e.g. accuracy for each configuration.
- Metrics saved during training, e.g. accuracy every 10 iterations.
- The final model itself.
The final outputs are the return values of the experiment function with the @ex.automain
decorator. Figures/tables produced from these values will find themselves in final manuscript. Every specific configuration {var_1:val_1,...,var_k:val_l}
, and its corresponding return value {output_1:result_1,...,output_p:result_p}
should occupy a row in a table, as per the Tidy Data philosophy.
The metrics saved during training, meant to help in diagnostics. This is done via ex.log_scalar("scalar_name", scalar_val)
.
To save an artifact,