- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This work investigates and quantifies confabulation in existing Natural Language Autoencoders (NLAs). To address this issue, the authors propose Flow-NLA, which replaces the point-reconstruction reward with a distributional denoising objective that captures richer information about activations. Experimental results show that Flow-NLA substantially reduces confabulation and writing defects while largely preserving the utility gains of conventional NLA training.
Critical strengths
The work is grounded in empirical observations and supported by theoretical analysis, making the proposed methodology intuitive and its effectiveness empirically convincing.
Critical weaknesses
-
Limited empirical support for the theoretical motivation. While Section 4.1 and Figure 4 demonstrate that point reconstruction can theoretically fail to reward informative explanations, the practical significance of this limitation is not investigated, making the motivation for Flow-NLA less convincing. It would be helpful to provide a quantitative or qualitative analysis of how often explanations contain useful information about the activation distribution that is not captured by changes in the conditional mean.
-
Unclear connection between distributional reconstruction and explanation quality. It remains unclear why Flow-NLA should reduce confabulation. The denoising reward encourages explanations that help predict activations but does not explicitly constrain the model to avoid unsupported claims, as acknowledged by the authors in Lines 326–328. Similarly, it is unclear why Flow-NLA substantially improves writing quality, as shown in Figure 6, since the training objective does not directly optimize it. These observations raise the question of whether the improvements reflect more faithful and well-formed explanations or are driven by hacking the evaluation metrics.
-
Insufficient validation of evaluation metrics. The proposed evaluation framework relies heavily on LLM-as-a-judge, and the reliability of some evaluation procedures, such as claim-level verification for confabulation, is not sufficiently established. Consequently, some reported metrics are difficult to interpret, such as the 92.6% assertion risk reported in Table 1. Human validation of these metrics, including their agreement with human judgments and their ability to distinguish high-quality from low-quality explanations, would strengthen the credibility of the evaluation.
-
Computational cost and efficiency. Flow-NLA introduces a different and more complex reconstructor architecture, but the computational overhead relative to point reconstruction is not sufficiently discussed. Reporting training costs and reward computation costs would help assess the practical trade-offs of the proposed method.
I would be willing to raise my score if the authors can provide stronger empirical evidence for the motivation behind Flow-NLA, establish a clearer connection between distributional reconstruction and confabulation reduction, and address the concerns regarding the validity of the evaluation metrics during the rebuttal.
Questions for authors
It would be helpful to include concrete examples of explanations generated by NLA and Flow-NLA to illustrate their qualitative differences.
5 comments
W4 | Computational cost and efficiency.
Flow-NLA introduces extra overhead in the training procedure. The verbalizer architecture remains fixed, thus generating explanations throughout training and inference retains the same computational cost. For reward calculation, we need to evaluate the reconstructor across 32 noise levels for all generated explanations, and these draws run in parallel. We timed both methods on Qwen2.5-7B with four GH200 GPUs, 1024 explanations per batch, averaged over 100 batches after a 10-batch warm-up:
| Operation | Point-NLA (s) | Flow-NLA (s) | Increase | Note |
|---|---|---|---|---|
| Verbalizer forward + reference logprobs | 18.018 | 18.018 | shared | Same actor pipeline |
| Actor forward/backward + Adam | 14.732 | 14.732 | shared | Same actor update |
| Reward forward | 1.656 | 2.245 | +35.6% | Denoising across 32 draws |
| Reconstructor training forward | 3.000 | 3.163 | +5.4% | Denoiser with activation injection |
| Reconstructor backward | 6.010 | 6.114 | +1.7% | Extra denoiser gradients |
The increase is concentrated in reward computation, while reconstructor training adds only 1.7–5.4%. We want to clarify that the NLA training procedure is mainly bottlenecked by sequential computation steps rather than parallelizable ones. Generation and the actor update are shared between the methods and dominate per-batch time, so the overhead is small relative to the cost of RL training. We added this table to Appendix E.
Q1 | Qualitative differences
Thank you for the suggestion. We have added four paired excerpts in Appendix O. Each shows the source context, the released-NLA and Flow-NLA explanations of the same Gemma-3-12B activation, and the claims the judge marks as unsupported or contradicted.
I think the authors' response has addressed most of my concerns. I will raise my soundness and overall rating from 2 to 3. Please include the new experiments and discussions in the revised manuscript.
Thank you for the constructive feedback and for clearly outlining what would strengthen the paper. We address each point below: new real-activation evidence for the motivation, a clearer account of the link to confabulation, and further validation of our evaluation. We are happy to clarify anything further.
W1 | Limited empirical support for the theoretical motivation
We agree that Section 4.1 and Figure 4 establish a possible limitation, but do not quantify its prevalence in real explanations. The relevant intuition is that an additional detail can leave the conditional mean unchanged while improving recovery from noisy activations: point reconstruction then provides no additional reward, whereas denoising can reward that detail. More on this in General Response §3.
Appendix J provides complementary evidence on Qwen2.5-7B-Instruct activations. For our likelihood-bound reward, the reconstructor's off-PC variance ratio approaches 1, and its PCA-64 covariance overlap improved. These diagnostics show that the reconstructor captures distributional structure beyond the conditional mean. We add an experiment on Gemma activations (General Response Exp. 3). For pairs of label-compatible and incompatible activations that receive near-identical rewards from a mean prediction, the mean-based scores perform at chance (~50%), whereas the full Flow score prefers the compatible activation in 74–76% of pairs. Our results show that such information is present in real activations and credited by Flow, and we see testing its relation to source support as an important next step. We now state this open question explicitly in the limitations section.
W2 | Unclear connection between distributional reconstruction and explanation quality
We thank the reviewer for raising this point, which also motivated much of our own analysis. Indeed, our reward does not directly target source support. Like other unsupervised interpretability methods, such as SAEs, we restrict ourselves to learning from the activation alone, and thus unsupervised with respect to the source and evaluator. Rewarding based on the source would bias explanations away from the model's internals. This comes with the limitation that confabulations and other properties of explanations have to be addressed indirectly.
Our proposed method is a shift in learning incentives, point reconstruction responds only to the conditional mean, which can miss distinctions that capture information beyond that mean.
Modeling a complex distribution such as the activation space through its mean encourages the verbalizer to combine plausible candidates into one explanation, much as predicting the mean image yields a blurry combination of candidates. Because the flow reconstructor represents such alternatives, the pressure is plausibly reduced, although not removed. Taking writing quality as an example, we observe a deterioration relative to the SFT starting point (Figure 6), but as the reviewer notes, an improvement over the point-reconstruction recipe. The remaining deterioration is consistent with the pressure being reduced rather than removed, while the gap to point reconstruction reflects the part that the distributional target alleviates. We agree with the reviewer that this is not a causal or direct explanation.
For reward hacking, since the verbalizer's reward never accesses the judge, the metrics cannot be optimized directly and are therefore not reward-hacked in the usual sense. The utility gains further indicate that the improvements do not stem from less specific explanations.
W3 | Insufficient validation of evaluation metrics.
We agree that credibility of the evaluation framework relies on validating the judge more thoroughly. To achieve this, we first test agreement of our DeepSeek judge with GPT-6 Astra and Claude Opus 5.5 (General Response Exp. 2), both with reasoning effort set to low and perform a study of human annotator agreement. Note that to reduce the annotator burden, we annotate four explanations for 11 tasks (Our 12th task contains only foreign language samples) and verify the quality defects instead of marking the locations for the released Gemma-3-12B NLA and our Flow-NLA variant.
The judges agree closely. Assertion risk differs by at most 4.2 points across judges, and all three reproduce the Flow-NLA reduction of 20.0–21.9 points. All three also find fewer writing defects for Flow-NLA, although absolute defect counts depend on the judge, so we interpret them as relative differences; all three find 54–72% fewer defects for Flow-NLA.
The human annotation is now complete (General Response Exp. 2.2). With three annotators per sample, human-rated assertion risk drops by 25.2 points for Flow-NLA, and DeepSeek matches the human majority on 80.2% of claim labels, in line with human–human agreement (79.4%).