acceptodds
Withdrawn from ICLR 2027

Bayes-Equivalent Prompts Reveal Hidden Nuisance Dependence in In-Context Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In-context learners can closely match Bayesian predictions on synthetic tasks. But do they also learn which aspects of the input a Bayesian predictor knows to ignore? We introduce Bayes-equivalent prompts: inputs that differ in raw form but induce exactly the same posterior and posterior predictive distribution, a direct test of whether learned predictors acquire the Bayesian solution's invariances. In multilayer perceptrons (MLPs) and Transformers, sensitivity to Bayes-irrelevant variation falls during training and is attenuated more in later layers, yet the identity of the nuisance perturbation remains linearly decodable from final representations. In a separate twelve-seed Transformer study trained to 800k updates, fit keeps improving and responses to task-relevant changes stay close to the Bayesian reference, while sensitivity to the same physical change when it is Bayes-irrelevant declines but stays above a pre-specified, study-specific benchmark. A consistency penalty (one weight, short continuation) sharply reduces disagreement on the targeted perturbation but raises held-out Kullback–Leibler (KL) divergence; in the sampled-target branch, this gain does not clearly carry over to a second Bayes-equivalent transformation. A public pretrained regression Transformer also shows small but measurable sensitivity to an exactly posterior-preserving recoding. Predictive fit therefore does not determine invariance, and invariance gained on one transformation need not transfer: evaluating whether a model has captured a Bayesian procedure requires testing its invariances, not only its predictions.

Primary area:
interpretability and explainable AI
Keywords:
In-context learning, Bayesian inference, Bayes-optimal prediction, training dynamics, prompt sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C23TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether in-context learners that closely match Bayesian predictions also acquire the Bayesian predictor's invariances. It introduces Bayes-equivalent prompts: pairs that differ in raw form but induce exactly the same latent posterior and posterior predictive distribution. The authors study MLPs and Transformers trained on controlled synthetic tasks, plus a public pretrained regression Transformer. The main findings are that output sensitivity to Bayes-irrelevant changes falls during training and is attenuated more in later layers, that the nuisance perturbation remains linearly decodable from final representations, that residual sensitivity persists beyond the improved fit (nuisance sensitivity declines from 0.049 to 0.038 across 200k to 800k updates, remaining above the 0.01 pre-specified benchmark), that a consistency penalty reduces targeted disagreement at a held-out KL cost, and that sampled-target consistency gains do not clearly transfer to a second Bayes-equivalent transformation.

Critical strengths
  1. The Bayes-equivalent prompt construction is a clean, theoretically motivated probe that directly tests whether learned predictors acquire posterior-preserving invariances, rather than relying on label-format or order perturbations whose Bayesian status is unclear.
  2. The paper has unusually high evidential hygiene. It distinguishes confirmatory from post-confirmatory analyses, reports failed controls and transfer failures in the main text, and clearly states that descriptive intervals are not replacements for confirmatory coverage. This makes the negative and partial results more credible.
Critical weaknesses
  1. The central empirical claim that nuisance sensitivity persists is sensitive to a study-specific benchmark (epsilon_0 = 0.01) chosen by the authors, and the continued-training analysis is explicitly post-confirmatory and descriptive. Without an independent, pre-specified test or a larger cohort, the conclusion that "fit does not determine invariance" rests more on the conceptual argument than on a sharp quantitative result.
  2. The empirical scope is narrow: the main longitudinal results rely on 5 model seeds per architecture for four-rule tasks and 12 seeds for the relevance study, all synthetic, with one public pretrained model as a single extra checkpoint. The consistency-penalty result is one weight, +10k updates, and a sampled-target branch with intervals spanning zero on the transfer test. The paper's "hidden nuisance dependence in ICL" framing is broader than the evidence supports.
Questions for authors
  1. Would the persistence conclusion at 800k hold under a pre-specified 97.5% confirmatory interval, rather than a descriptive 95% interval, with a stricter familywise control across the continued-training checkpoints?
  2. Why is epsilon_0 = 0.01 the right benchmark? What would the headline "persistence" claim look like under a benchmark calibrated to held-out predictive or decision impact rather than a study-specific value?
  3. Can the consistency-penalty result be reported as a confirmatory primary contrast with a single pre-specified penalty weight and a formal test of transfer, rather than descriptive 2x2 diagnostics?
  4. Are the five- and twelve-seed cohorts sufficient for the "hidden attenuation is late" claim? What is the power for detecting a small depth contrast, and how robust is the result to representation choice (LayerNorm vs block output)?
  5. What does the probe decodability result establish beyond retention? Please state explicitly whether the nuisance representation is causally used or merely stored, and whether the head-null component is an appropriate causal control.
  6. How should the conceptual claim generalize to natural-language ICL tasks, where the "Bayesian equivalent" construction is not available? Can the invariance test be extended to approximate posteriors or calibrated predictive distributions?
  7. For the public pretrained model, are the differences practically meaningful beyond statistical significance, and how do the effect sizes compare with the synthetic-task results?
  8. Please report raw displacements alongside normalized hidden sensitivity, since the paper emphasizes that normalized sensitivity can decline while raw displacement grows.
Reviewer_XUzTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper continues on a long line of work that studies the extent to which neural networks (MLPs, Transformers) can be interpreted as Bayesian. They create synthetic tasks, where inputs are different, but the resulting Bayesian posterior and posterior predictive are the same. This allows for testing whether neural nets are sensitive of the input changes the Bayesian predictor would ignore.

The synthetic experiments reveal the following main findings:

  • During training, sensivivity to Bayes-irrelevant input changes decreases, but stays above zero both in MLPs and Transformers. Earlier representations are more sensitive than later ones. They also show a public pretrained Transformer also exhibits this sensitivity.
  • Models are more sensitive to input changes that are relevant to the Bayesian predictor too
  • Information from the Bayes-irrelevant input changes can be extracted from the models' final hidden representations with a linear classifier, indicating that the model retains this irrelevant information
  • With a training penalty, the authors can artificially reduce sensitivity to Bayes-irrelevant input changes, but this comes with worse predictive fit.
Critical strengths

The paper's experiments are quite well designed, and form a consistent set. The experiment on artificially reducing sensitivity, and its outcome on data fit is quite intriguing.

Critical weaknesses

**Clarity, presentation, structure: ** I find this as the main weakness, and without very significant improvements, I believe the paper should not be accepted. The text is very hard to read. The structure is also not very logical: some experiment setups are explained sections before their results, while others are grouped together, with no apparent reason why. I think structuring the paper by experiment, grouping the experimental setup together with the results and conclusion would improve readability a lot.

Significance and positioning: the paper does not position its findings within the "Is ICL Bayesian?" literature beyond somewhat superficial acknowledgement of the debate. They do demonstrate that MLPs/Transformers exhibit input sensitivity a true Bayesian model should not, but it's unclear whether the observed sensitivity might decrease towards zero as models become larger or are trained for longer. The paper makes no statement on whether the findings are interpreted as evidence against ICL not being Bayesian. If so, it's unclear what insight this paper brings beyond Falck et al, 2024 (Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective), who also test for an invariance property Bayesian algorithms should satisfy.

Questions for authors

Questions

  • What's the purpose of the hint in Task 1? Could it happen that the trained model learns to rely mostly on the hint and not on the actual in-context samples?
  • Why do you think it is that artificially reducing sensitivity comes at the cost of data fit?
Reviewer_J1FUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether in-context learners keep invariant with respect to transformations that preserve both the latent-task posterior and the posterior predictive distribution under a known generative model. In a four-rule task, magnitude changes preserve the encoded bits and hence the Bayesian inference problem. Output disagreement decreases during training, normalized hidden sensitivity is attenuated at later representations, and the perturbation endpoint remains linearly decodable from final representations. A separate twelve-seed logistic-task study controls posterior relevance while holding the physical query perturbation fixed. Nuisance sensitivity decreases through 800k updates but remains above a study-specific threshold. A short 2×2 continuation experiment contrasts sampled versus exact targets and consistency off versus on. Consistency suppresses the targeted disagreement while increasing held-out KL. Transfer to demonstration complement recoding is unclear in the sampled-target branch. An additional audit finds small recoding sensitivity in one public pretrained regression Transformer.

Critical strengths
  1. The equivalence criterion is precise and well controlled. Preserving the posterior and posterior predictive avoids ambiguity about whether a transformation has changed the intended task. The equivalence is accompanied by theoretical derivations.

  2. The study distinguishes several quantities that are often conflated. It separates output disagreement, normalized representation sensitivity, probe decodability, and approximation error. This warns the community that it may be not that reliable to claim that Transformers are doing Bayesian prediction. The paired squared-error decomposition is exact and helps distinguish broad approximation improvement from targeted suppression.

Critical weaknesses
  1. Scientific insight is somewhat limited. The observation that good average predictive performance does not establish exact structural Bayesian behavior is already present in related coherence and order-based audits, including Falck et al. (2024) and Chlon et al. (2026, v3). The paper cites this literature appropriately. Its distinctive additions are the posterior-relevance control, non-order equivalences, and the continuation intervention, rather than the broad distinction itself. I think these additions are useful, but the central empirical conclusion remains that finite-accuracy predictors exhibit small residual non-invariance. On the paired distribution, the paper's own identity gives EWITHIN≤EPAIRE_{\mathrm{WITHIN}}\le E_{\mathrm{PAIR}}. Thus, a nonzero residual is compatible with ordinary approximation error, and the comparison against a study-specific threshold does not by itself establish a surprising limitation of ICL.
Questions for authors
  1. Which empirical conclusion specifically requires the new posterior-relevance control and cannot be explained by the existing distinction between structural invariance and finite predictive approximation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.