Isaac Slavitt - The 10 Commandments of Reliable Data Science | PyData Global 2022 - 2023

Avoid the "we can clean this latter" attitude. If teams follow guidelines, they are 80% of the way there. Consider Django projects in Python, the structure is the same throughout, the database models will be in models.py, and so on. This makes it easier to collaborate.

One solution is to use cookie cutter data science as a starting template.

Every now and then one may realize there might be for e.g. a CSV file that is critical to the project, but not know how it came from, what to do if it gets out of date etc.

Data should mostly not be kept in version control. Some choices for this are:

• Git Large File Storage.
• Amazon S3.
• Google Cloud Storage.
• DVC.
• Own disk space.

Version control for code enables code review. Code review is the most effective tool for maturing data science teams. Most effective software teams have a code review process where in order for code to be accepted in to the code base, it is read by another human - code should primarily be written for humans to read. nbautoexpert can help here, it converts .ipynb files to .py files for example.

Notebooks make version control harder, and having a human manually open a notebook to run it is not automation. They also encourage copying code from one notebook to another, which can introduce errors.

A more balanced workflow is to use notebooks for quick feedback and get things working.

This is challenging, since:

• Large datasets result in long running test suites.
• Data will change over time, hence downstream values fluctation.
• Using randomness means it is hard to know is changes are meaningful or due to expected variation.
• Visualizations are hard to test.
• Notebooks do not have good tooling for test discovery and running.

If we move often used and common functions out to source code files, we can write tests for them.

Do not try to handle exceptions.

We could get a mean from a pandas dataframe even if it has NaNs - in this case one solution is to do assertion testing like:

assert np.allclose(arr, HISTORICAL_MEAN, atol=1.5 * HISTORICAL_SD)
assert df["transaction_type"].isin(KNOWN_ENUM_TYPES).all()

One package for assertion testing is Bulwark.

Overall, we should stick to the ideas of Defensive Programming.

From the raw data to final outputs, the project run should be automated.

Consider GNU Make or something similar.

Many functions in use will have dozens of parameters. On top of this there are environment variables, Make variables, central configuration variables for the project, command line arguments, constants at the top of files, magic numbers right on top of when they are used in some function.

One tool for this is Hydra.

The typical workflow is to start from a notebook then eyeball the results back and forth.

A more improved workflow is to use the ideas and tools mentioned and then eyeball the results.

Log files should be verbose, and stored all the time.

Get to the end product before it is good, and implement things when we need them and not when we think we need them - paraphrased from Mike Gancarx, Linux and the Unix Philosophy 2E.

This means doing the following for e.g.

• Clean the data, but use a very small subset/collection.
• Choose the relevant features, but take a few columns first.
• Train the model, but on a simple model like logistic regresison depending on the task.
• Make the predictions, do benchmarking
  • Print results on a table.

(Repeat the above until satisfactory).

• Then make the end web app etc.