Causal Parrots: Large Language Models May Talk Causality But Are Not Causal - 2023

Details

Title : Causal Parrots: Large Language Models May Talk Causality But Are Not Causal Author(s): Zečević, Matej and Willig, Moritz and Dhami, Devendra Singh and Kersting, Kristian Link(s) : https://openreview.net/forum?id=tv46tCzs83

Rough Notes

This paper defines and shows a new subgroup of SCMs (meta-SCMs) that can encode causal facts about other SCMs in their variables. (#TODO Define variables explicitly in the SCM definition). Conjecture: When LLMs succeed in doing causal inference (#TODO Define what causal inference means here), there is an underlying meta-SCM that shows correlation between causal facts in the training data. Empirical analysis suggests that LLMs are weak causal parrots.

  • Starts by assuming a pro-causal AI stance.
  • Transformers (thus modern LLMs) are not parametrized SCMs, they do not specify structural equations mimicking causal mechanisms.
  • LLMs sometimes answer causal questions correctly.
  • LLMs encounter causal facts during training, since there is so much data.
  • LLMs can be trained to produce desired content, in this case the right causal answers.
  • Figure 1: Nature generates the true SCM, in this case altitude \(A\) causes temperature \(T\). We do not know this apriori, so we would take measurements and try to see if A->T, A<-T, A<->T, A T. Now, that is a causal model, which if known can answer queries which we cannot answer with just observational data. On the other hand, we could have this information encoded in textual format, so if we ask a query like "Does A cause T" or "What is the causal relationship between A and T" it would happen to give the correct answers.
  • Being universal does not imply its easy to make LLMs causal (#TODO What makes an LLM causal?).
  • CHT: Observational data alone cannot answer causal statements (unless given causal assumptions). This is a wall for the scaling hypothesis. UNLESS: Causal assumptions (needed for causal inference as per CHT) are in the observational data itself.
  • SCM definition
  • 3 types of causal relationships: Direct causes (direct edge), causes (path), confounded variables (bidirectional edge).
  • CHT: Three languages (spaces of queries - observational queries, interventional queries, counterfactual queries).
  • Insight 1: Knowledge about structural equations and the causal graph is knowledge about answering L2, L3 queries.
  • an ML model is causal wrt some query if it can answer that query with the right L2 fact (#TODO What does that mean). (#TODO Reread sentence just before Insight 2)
  • Insight 2: Variables of SCMs doesn't have to be typical measurable concepts, they can be meta-concepts such as causal fact, e.g. knowledge about L2, L3.
  • (#TODO Example showing difference between p(y|x) and p(y|do(x)) - namely P(Alt|Temp) neq P(Alt|do(Temp)) )
  • An SCM Mmeta is meta wrt M if observational distribution of Mmeta is enough to answer interventional queries of M.
  • #TODO What does a "language" mean in the context of L1,L2,L3.
  • "Namely, some machine learning model is ‘causal’ w.r.t. some query if it can answer that query with the right L2 fact." #TODO What does L2 fact mean here?

Questions

  • What makes an LLM causal (or not)?
  • How do we model systems causally?
  • What is the difference between a causal model and a statistical model? (Here, talk about the causal hierarchy theorem, where you cannot move between different query spaces without making assumptions - here, does the "assumption" when moving from L1 to L2 mean assuming the likelihood to factorize to the form of the intervention being performed? In this case, what is the assumption made when jumping from L2 to L3? Also, make sure to talk about the fact that different graphs can explain the same observational data, and the relation to knowing SCMs and computing counterfactuals).
  • What is the difference between understanding and knowing?
  • What is a meta-SCM?

Main notes

  • LLMs are sometimes good at causality, sometimes not.
  • Define what is meant by being good/bad at causality.
  • Meta-SCMs encode causal facts about other SCMs - so LLMs show correlations between causal facts.
  • The authors provide an explanation, but how can we be sure that this is how LLMs manage to get causality correct sometimes?

Tidbits

Mainly things that should be related to this:

  • Stochastic parrots.
  • Causal representation learning.
  • Look into Zhijing Jin's work.

Other

  • Read the TMLR OpenReview correspondence.
  • Rewatch the YouTube video.

Emacs 29.4 (Org mode 9.6.15)