- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the out-of-distribution (OOD) generalization of probes trained on LLM activations. The authors propose Factorized Axial Latent Attention (FALA), which learns distributed representations by aggregating activations across layers and token positions through two-stage cross-attention, together with sparse layer selection to reduce overfitting. Experiments on three probing tasks—hallucination detection, uncertainty calibration, and training-data familiarity—show that the proposed method improves OOD generalization over conventional probing baselines, particularly for uncertainty calibration. The authors also evaluate several distributionally robust learning methods and find that none consistently outperforms standard empirical risk minimization (ERM). Further experiments suggest that increasing the diversity of training environments is more beneficial than simply adding samples from existing environments.
Critical strengths
Comprehensive experimental evaluation. The authors evaluate probing methods across three tasks and 15 environments, comparing different distributed representation architectures and a broad range of distributionally robust learning objectives. The experiments also include several baselines and ablations.
Critical weaknesses
-
Limited improvements over existing baselines. Although FALA-G achieves substantial improvements on uncertainty calibration (S2), its advantages on hallucination detection (S1) and training-data familiarity (S3) are limited. On S1, FALA-G achieves nearly identical average AUROC to the mean-token baseline (0.776 vs. 0.777). On S3, FALA-G achieves only a modest improvement over mean-token pooling (0.702 vs. 0.676) and even performs slightly worse than the random-initialization control (0.702 vs. 0.714). These results weaken the claim that the proposed architecture consistently improves OOD generalization.
-
Limited methodological novelty. FALA primarily combines existing architectural components, including inducing-point cross-attention, axial attention, and Gumbel-based sparse layer selection. While their application to distributed LLM probing is reasonable, the proposed method introduces limited methodological novelty beyond these established techniques.
-
Limited insights into OOD generalization. Although the authors conduct extensive experiments, the findings provide limited new understanding of why certain probing architectures generalize better under distribution shift. In particular, the observation that distributionally robust objectives fail to consistently outperform ERM is not accompanied by a sufficiently convincing explanation. Although the representation diagnostics suggest that ERM learns relatively invariant predictors, they do not establish that this invariance explains the lack of additional benefits from distributionally robust objectives.
-
Missing ablation on sparse layer selection. The authors demonstrate that Gumbel-based sparse layer selection substantially improves performance, but do not compare it against simpler alternatives, such as selecting a fixed subset of layers throughout training or using deterministic top- selection without Gumbel noise. Such comparisons would help clarify whether the improvements come from sparsity itself or from the proposed stochastic selection mechanism.
Questions for authors
How does the relative performance of ERM and distributionally robust learning objectives change as the number of available training environments decreases? In particular, could robust objectives become more beneficial when training-environment diversity is limited?
2 comments
Response to Reviewer 3 (Reviewer U6VN) (2/2)
3. Limited insight into why some probing architectures generalize better
We agree that understanding the mechanisms underlying probe generalization is important. Although our experiments cannot provide a complete explanation, they offer several complementary insights.
Where the gain comes from. First, the clearest evidence concerns the role of sparse layer selection. Removing Gumbel-based selection reduces coverage on S2 by more than half for both architectures: from 0.468 to 0.224 for FALA, and from 0.153 to 0.073 for WAP.
Second, simply increasing access to hidden representations does not appear sufficient. ACT-ViT processes the entire layer × token tensor with 2.9 times as many parameters, yet performs worse than FALA-G on S2 and on the cross-task MNLI evaluation. Similarly, concatenating all layers yields a Cov80 of only 0.004 on S2.
Together, these results suggest that the benefit of FALA-G comes at least partly from restricting the information available to the probe, rather than maximizing its access to the model's internal states.
More broadly, we explore an important trade-off in activation probing: increasing the amount of accessible internal computation can make more sophisticated predictive strategies available, but can also increase the difficulty of learning these strategies. A standard linear probe occupies one end of this spectrum: its restricted access and low complexity make it relatively easy to train. At the other extreme, architectures such as ACT-ViT access much richer representations, potentially allowing more expressive prediction strategies, but also creating more opportunities to exploit environment-specific (spurious) correlations. FALA-G offers an intermediate approach: it can aggregate information across tokens and layers but restricted to a learned subset. We hypothesize that this balance between expressive access and structural restriction contributes to its generalization performance.
We do not claim that this provides a complete account of OOD generalization in probing, but we believe the experiments identify a meaningful architectural trade-off and offer concrete evidence for the benefits of selective activation access.
4. Missing ablation on sparse layer selection
We thank the reviewer for this suggestion. We have conducted both requested ablations across all seven evaluation folds (Appendix L):
- Fixed layers: We replace the learned Gumbel selector with a fixed, contiguous window of layers, selected through a separate linear-probe validation scan. The selected layers are [13, 14, 15] for S1/S2 and [29, 30, 31] for S3, and remain fixed throughout training.
- Deterministic selection: We retain the learned selector but remove Gumbel noise, using deterministic top-K selection instead.
For the S1 and S2 configurations exhibiting the largest differences in the initial experiments, we additionally evaluated two more random seeds.
On S1 with MNLI, averaged across three seeds, FALA-G achieves an AUROC of 0.675, compared with 0.624 for fixed layers and 0.605 for deterministic selection. These differences correspond to 2.4 and 1.8 standard deviations, respectively. Both alternatives perform at or below the linear-probe baseline (0.625), and every FALA-G seed outperforms every seed of either alternative. Thus, on this particularly challenging distribution shift, merely restricting the probe to a few layers is insufficient: learning which layers to select, with Gumbel noise, provides an additional benefit.
On S2, the results point to a somewhat different conclusion: sparsity itself appears to account for most of the improvement. Both sparse alternatives achieve performance close to FALA-G, whereas removing sparse selection all together reduces Cov80 from 0.468 to 0.224. On S1 NQ and CoQA, the single-seed differences between selection strategies are small (at most 0.014).
On S3, the results are mixed. Given the substantial training variability identified in our multi-seed experiments, we refrain from drawing strong conclusions from these differences.
We have incorporated these results and their interpretation into Appendix L.
Question: Does training-environment diversity matter more when it is scarce?
We are running an additional experiment to test this directly, varying environment diversity while comparing ERM against robust objectives, and will post the results as a follow-up comment if they are ready before the end of the discussion period.
Response to Reviewer 3 (Reviewer U6VN) (1/2)
We thank the reviewer for their detailed assessment and for recognizing the breadth of our experimental evaluation. We address each concern below.
1. Limited improvement on S1 and S3
We agree that the improvements on S1 and S3 are smaller than those observed on S2, and we have revised our claims accordingly. In particular, the additional multi-seed experiments show that the differences on S3 are not statistically significant.
Nevertheless, FALA-G remains clearly the best overall choice: it achieves the best performance or performs comparably to the strongest baseline across all three signals, whereas the competing approaches exhibit more pronounced weaknesses on particular tasks. Its main advantage is therefore not a uniformly large improvement on every signal, but consistently competitive performance across substantially different probing problems and distribution shifts.
More importantly, our paper's contribution extends beyond the introduction of FALA-G. We investigate the broader problem of out-of-distribution generalization in activation probing through a systematic LOO evaluation spanning multiple environments, signals, probing architectures, and training objectives. In particular, we believe that our negative findings concerning distributionally robust training constitute an important contribution. Such methods are a natural candidate for addressing generalization failures in probing, yet our extensive evaluation shows that they do not deliver consistent improvements in this setting.
2. Limited methodological novelty
We agree that the individual building blocks of FALA (inducing-point cross-attention, axial factorization, and Gumbel top-K selection) are established techniques. Our methodological contribution lies in their combination for activation probing, but the main contribution is the empirical insight into which aspects of the activation tensor are most useful for cross-environment generalization.
Our experiments indicate that restricting the probe to a sparse subset of layers can be particularly effective. On S2, introducing Gumbel-based sparse selection substantially improves coverage for both FALA and WAP. On the challenging MNLI cross-task evaluation, learning which layers to select with Gumbel noise also outperforms simpler sparse-selection alternatives.
The new ACT-ViT comparison provides further support for this interpretation. Despite processing the full layer × token tensor with approximately three times as many parameters, ACT-ViT does not outperform FALA-G. This suggests that simply exposing the probe to more of the model's internal computation is insufficient; the way this information is selected and aggregated plays a crucial role.
Finally, we emphasize that FALA is only one component of our work. Our broader objective is to characterize OOD generalization in activation probing, including the effects of architecture, activation access, and distributionally robust training. We believe these empirical findings are valuable independently of the novelty of any individual architectural component.