Sacred (Python)

Suppose we are interested in answering some specific research question, e.g. how does changing the configuration to specific parameters affect the performance for some model. This involves a training loop which fits some model and final output(s) which measures the performance of the fitted model. There are at least 3 objects of interest here:

• The final outputs, e.g. accuracy for each configuration.
• Metrics saved during training, e.g. accuracy every 10 iterations.
• The final model itself.

The final outputs are the return values of the experiment function with the @ex.automain decorator. Figures/tables produced from these values will find themselves in final manuscript. Every specific configuration {var_1:val_1,...,var_k:val_l}, and its corresponding return value {output_1:result_1,...,output_p:result_p} should occupy a row in a table, as per the Tidy Data philosophy.

The metrics saved during training, meant to help in diagnostics. This is done via ex.log_scalar("scalar_name", scalar_val).

To save an artifact,