Experimental Designs

The general experiment process is as follows:

• Choose a design \(\xi\) which contains all aspects of the experiment that we control.
• Run the experiment with \(\xi\), obtaining the outcome \(y\). We assume the outcomes are generated from some unknown distribution \(p_{true}(y|\xi)\). The distribution may also actually depend on additional covariates, unknown latent variables, fixed inputs etc. which can be absorbed into this distribution.
• Evaluate a utility for \(U(\xi,y)\) of the data \(\{(\xi,y)\}\). This utility could be related to fairness, information gain etc.

Experimental design refers to the task of choosing the design \(\xi\), i.e. setting the controllable parts of the experiment. The common approach to do this is choose the experiment as \[ \xi^* = \text{argmax}_\xi \mathbb{E}_{p_{true}(y|\xi)}[U(\xi,y)] \]

However, we do not know \(p_{true}(y|\xi)\) and instead we get samples from this distribution. Bayesian Optimal Experimental Design (BOED) is a model-based approach to experimental design which makes use of notions in information theory.

One example application is to know the values of \(x\) for the question "Would you prefer USD \(x\) now or USD 100 in 1 year?", Choose \(x=70\) is a better question than choosing \(x=200\). The role of \(x\) here is that of the design \(\xi\).