- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether in-context learners that closely match Bayesian predictions also acquire the Bayesian predictor's invariances. It introduces Bayes-equivalent prompts: pairs that differ in raw form but induce exactly the same latent posterior and posterior predictive distribution. The authors study MLPs and Transformers trained on controlled synthetic tasks, plus a public pretrained regression Transformer. The main findings are that output sensitivity to Bayes-irrelevant changes falls during training and is attenuated more in later layers, that the nuisance perturbation remains linearly decodable from final representations, that residual sensitivity persists beyond the improved fit (nuisance sensitivity declines from 0.049 to 0.038 across 200k to 800k updates, remaining above the 0.01 pre-specified benchmark), that a consistency penalty reduces targeted disagreement at a held-out KL cost, and that sampled-target consistency gains do not clearly transfer to a second Bayes-equivalent transformation.
Critical strengths
- The Bayes-equivalent prompt construction is a clean, theoretically motivated probe that directly tests whether learned predictors acquire posterior-preserving invariances, rather than relying on label-format or order perturbations whose Bayesian status is unclear.
- The paper has unusually high evidential hygiene. It distinguishes confirmatory from post-confirmatory analyses, reports failed controls and transfer failures in the main text, and clearly states that descriptive intervals are not replacements for confirmatory coverage. This makes the negative and partial results more credible.
Critical weaknesses
- The central empirical claim that nuisance sensitivity persists is sensitive to a study-specific benchmark (epsilon_0 = 0.01) chosen by the authors, and the continued-training analysis is explicitly post-confirmatory and descriptive. Without an independent, pre-specified test or a larger cohort, the conclusion that "fit does not determine invariance" rests more on the conceptual argument than on a sharp quantitative result.
- The empirical scope is narrow: the main longitudinal results rely on 5 model seeds per architecture for four-rule tasks and 12 seeds for the relevance study, all synthetic, with one public pretrained model as a single extra checkpoint. The consistency-penalty result is one weight, +10k updates, and a sampled-target branch with intervals spanning zero on the transfer test. The paper's "hidden nuisance dependence in ICL" framing is broader than the evidence supports.
Questions for authors
- Would the persistence conclusion at 800k hold under a pre-specified 97.5% confirmatory interval, rather than a descriptive 95% interval, with a stricter familywise control across the continued-training checkpoints?
- Why is epsilon_0 = 0.01 the right benchmark? What would the headline "persistence" claim look like under a benchmark calibrated to held-out predictive or decision impact rather than a study-specific value?
- Can the consistency-penalty result be reported as a confirmatory primary contrast with a single pre-specified penalty weight and a formal test of transfer, rather than descriptive 2x2 diagnostics?
- Are the five- and twelve-seed cohorts sufficient for the "hidden attenuation is late" claim? What is the power for detecting a small depth contrast, and how robust is the result to representation choice (LayerNorm vs block output)?
- What does the probe decodability result establish beyond retention? Please state explicitly whether the nuisance representation is causally used or merely stored, and whether the head-null component is an appropriate causal control.
- How should the conceptual claim generalize to natural-language ICL tasks, where the "Bayesian equivalent" construction is not available? Can the invariance test be extended to approximate posteriors or calibrated predictive distributions?
- For the public pretrained model, are the differences practically meaningful beyond statistical significance, and how do the effect sizes compare with the synthetic-task results?
- Please report raw displacements alongside normalized hidden sensitivity, since the paper emphasizes that normalized sensitivity can decline while raw displacement grows.