Rashomon Capacity: A Metric for Predictive Multiplicity in Classification - 2022

Title : Rashomon Capacity: A Metric for Predictive Multiplicity in Classification Author(s): Hsu, Hsiang and Calmon, Flavio du Pin Link(s) : http://arxiv.org/abs/2206.01295

Authors present a new measure for predictive multiplicity, i.e. when classification models with similar performances assign conflicting predictions to individual samples. This metric, called the Rashomon capacity can be applied to probabilistic classifiers which distinguishes itself from previous metrics that work on 0-1 valued classes.

The Rashomon effect was first described in (, ) (#NOTE Need to reread that paper). The set of almost-equally performing models for a given learning problem is called the Rashomon Set.

Main contributions of the paper:

• Introducing desirable properties that any predictive multiplicity set must satisfy.
• Introduce the Rashomon capacity metric for quantifying predictive multiplicity.
• Introduce a methodology to report predictive multiplicity.
• Introduce a procedure for resolving predictive multiplicity in probabilistic classifiers.