acceptodds
Under review as a conference paper at ICLR 2027

Understanding Confabulation and Rethinking Reconstruction in Activation Explanations

Abstract

Natural Language Autoencoders (NLAs) produce unsupervised text explanations of a model's activations: a verbalizer describes an activation and a reconstructor learns to recover it from this text. Under the established point-reconstruction NLA training recipe, explanations become more useful for predicting model behavior while also increasingly introducing unsupported details and exhibiting writing defects. To assess these changes separately, we introduce the first publicly available standardized evaluation framework for unstructured NLA explanations, measuring information recoverable from explanations, contextual support for their claims, and writing quality. To address confabulation and writing defects, we move beyond predicting a single activation: explanations can distinguish distributions of possible activations even when their means and optimal point-reconstruction rewards are identical. We introduce Flow-NLA, which models the distribution of activations compatible with an explanation and trains the verbalizer using a diffusion likelihood bound. Across Qwen, Gemma, and Apertus, this richer signal retains the utility gains of point reconstruction while curbing the growth of confabulation and writing defects, opening up a direction for improving activation-derived training signals to encourage more informative, supported, and readable explanations. The evaluation and training code is available at the anonymous repository: https://anonymous.4open.science/r/nla-flow-ICLR-26F8/

Primary area:
interpretability and explainable AI
Keywords:
Natural Language Autoencoders, Mechanistic Interpretability, Large Language Models, Activation Verbalization, Flow Matching, Confabulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xWpEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work investigates and quantifies confabulation in existing Natural Language Autoencoders (NLAs). To address this issue, the authors propose Flow-NLA, which replaces the point-reconstruction reward with a distributional denoising objective that captures richer information about activations. Experimental results show that Flow-NLA substantially reduces confabulation and writing defects while largely preserving the utility gains of conventional NLA training.

Critical strengths

The work is grounded in empirical observations and supported by theoretical analysis, making the proposed methodology intuitive and its effectiveness empirically convincing.

Critical weaknesses
  1. Limited empirical support for the theoretical motivation. While Section 4.1 and Figure 4 demonstrate that point reconstruction can theoretically fail to reward informative explanations, the practical significance of this limitation is not investigated, making the motivation for Flow-NLA less convincing. It would be helpful to provide a quantitative or qualitative analysis of how often explanations contain useful information about the activation distribution that is not captured by changes in the conditional mean.

  2. Unclear connection between distributional reconstruction and explanation quality. It remains unclear why Flow-NLA should reduce confabulation. The denoising reward encourages explanations that help predict activations but does not explicitly constrain the model to avoid unsupported claims, as acknowledged by the authors in Lines 326–328. Similarly, it is unclear why Flow-NLA substantially improves writing quality, as shown in Figure 6, since the training objective does not directly optimize it. These observations raise the question of whether the improvements reflect more faithful and well-formed explanations or are driven by hacking the evaluation metrics.

  3. Insufficient validation of evaluation metrics. The proposed evaluation framework relies heavily on LLM-as-a-judge, and the reliability of some evaluation procedures, such as claim-level verification for confabulation, is not sufficiently established. Consequently, some reported metrics are difficult to interpret, such as the 92.6% assertion risk reported in Table 1. Human validation of these metrics, including their agreement with human judgments and their ability to distinguish high-quality from low-quality explanations, would strengthen the credibility of the evaluation.

  4. Computational cost and efficiency. Flow-NLA introduces a different and more complex reconstructor architecture, but the computational overhead relative to point reconstruction is not sufficiently discussed. Reporting training costs and reward computation costs would help assess the practical trade-offs of the proposed method.

I would be willing to raise my score if the authors can provide stronger empirical evidence for the motivation behind Flow-NLA, establish a clearer connection between distributional reconstruction and confabulation reduction, and address the concerns regarding the validity of the evaluation metrics during the rebuttal.

Questions for authors

It would be helpful to include concrete examples of explanations generated by NLA and Flow-NLA to illustrate their qualitative differences.

5 comments

Thank you for the constructive feedback and for clearly outlining what would strengthen the paper. We address each point below: new real-activation evidence for the motivation, a clearer account of the link to confabulation, and further validation of our evaluation. We are happy to clarify anything further.

W1 | Limited empirical support for the theoretical motivation

We agree that Section 4.1 and Figure 4 establish a possible limitation, but do not quantify its prevalence in real explanations. The relevant intuition is that an additional detail can leave the conditional mean unchanged while improving recovery from noisy activations: point reconstruction then provides no additional reward, whereas denoising can reward that detail. More on this in General Response §3.

Appendix J provides complementary evidence on Qwen2.5-7B-Instruct activations. For our likelihood-bound reward, the reconstructor's off-PC variance ratio approaches 1, and its PCA-64 covariance overlap improved. These diagnostics show that the reconstructor captures distributional structure beyond the conditional mean. We add an experiment on Gemma activations (General Response Exp. 3). For pairs of label-compatible and incompatible activations that receive near-identical rewards from a mean prediction, the mean-based scores perform at chance (~50%), whereas the full Flow score prefers the compatible activation in 74–76% of pairs. Our results show that such information is present in real activations and credited by Flow, and we see testing its relation to source support as an important next step. We now state this open question explicitly in the limitations section.

W2 | Unclear connection between distributional reconstruction and explanation quality

We thank the reviewer for raising this point, which also motivated much of our own analysis. Indeed, our reward does not directly target source support. Like other unsupervised interpretability methods, such as SAEs, we restrict ourselves to learning from the activation alone, and thus unsupervised with respect to the source and evaluator. Rewarding based on the source would bias explanations away from the model's internals. This comes with the limitation that confabulations and other properties of explanations have to be addressed indirectly.

Our proposed method is a shift in learning incentives, point reconstruction responds only to the conditional mean, which can miss distinctions that capture information beyond that mean.

Modeling a complex distribution such as the activation space through its mean encourages the verbalizer to combine plausible candidates into one explanation, much as predicting the mean image yields a blurry combination of candidates. Because the flow reconstructor represents such alternatives, the pressure is plausibly reduced, although not removed. Taking writing quality as an example, we observe a deterioration relative to the SFT starting point (Figure 6), but as the reviewer notes, an improvement over the point-reconstruction recipe. The remaining deterioration is consistent with the pressure being reduced rather than removed, while the gap to point reconstruction reflects the part that the distributional target alleviates. We agree with the reviewer that this is not a causal or direct explanation.

For reward hacking, since the verbalizer's reward never accesses the judge, the metrics cannot be optimized directly and are therefore not reward-hacked in the usual sense. The utility gains further indicate that the improvements do not stem from less specific explanations.

W3 | Insufficient validation of evaluation metrics.

We agree that credibility of the evaluation framework relies on validating the judge more thoroughly. To achieve this, we first test agreement of our DeepSeek judge with GPT-6 Astra and Claude Opus 5.5 (General Response Exp. 2), both with reasoning effort set to low and perform a study of human annotator agreement. Note that to reduce the annotator burden, we annotate four explanations for 11 tasks (Our 12th task contains only foreign language samples) and verify the quality defects instead of marking the locations for the released Gemma-3-12B NLA and our Flow-NLA variant.

The judges agree closely. Assertion risk differs by at most 4.2 points across judges, and all three reproduce the Flow-NLA reduction of 20.0–21.9 points. All three also find fewer writing defects for Flow-NLA, although absolute defect counts depend on the judge, so we interpret them as relative differences; all three find 54–72% fewer defects for Flow-NLA.

The human annotation is now complete (General Response Exp. 2.2). With three annotators per sample, human-rated assertion risk drops by 25.2 points for Flow-NLA, and DeepSeek matches the human majority on 80.2% of claim labels, in line with human–human agreement (79.4%).

W4 | Computational cost and efficiency.

Flow-NLA introduces extra overhead in the training procedure. The verbalizer architecture remains fixed, thus generating explanations throughout training and inference retains the same computational cost. For reward calculation, we need to evaluate the reconstructor across 32 noise levels for all generated explanations, and these draws run in parallel. We timed both methods on Qwen2.5-7B with four GH200 GPUs, 1024 explanations per batch, averaged over 100 batches after a 10-batch warm-up:

OperationPoint-NLA (s)Flow-NLA (s)IncreaseNote
Verbalizer forward + reference logprobs18.01818.018sharedSame actor pipeline
Actor forward/backward + Adam14.73214.732sharedSame actor update
Reward forward1.6562.245+35.6%Denoising across 32 draws
Reconstructor training forward3.0003.163+5.4%Denoiser with activation injection
Reconstructor backward6.0106.114+1.7%Extra denoiser gradients

The increase is concentrated in reward computation, while reconstructor training adds only 1.7–5.4%. We want to clarify that the NLA training procedure is mainly bottlenecked by sequential computation steps rather than parallelizable ones. Generation and the actor update are shared between the methods and dominate per-batch time, so the overhead is small relative to the cost of RL training. We added this table to Appendix E.

Q1 | Qualitative differences

Thank you for the suggestion. We have added four paired excerpts in Appendix O. Each shows the source context, the released-NLA and Flow-NLA explanations of the same Gemma-3-12B activation, and the claims the judge marks as unsupported or contradicted.

I think the authors' response has addressed most of my concerns. I will raise my soundness and overall rating from 2 to 3. Please include the new experiments and discussions in the revised manuscript.

Reviewer_6smdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Natural Language Autoencoders (NLAs) learn text explanations of model activations via reconstruction: a verbalizer describes an activation, a reconstructor recovers it, and reconstruction quality serves as an unsupervised RL reward. The paper makes two contributions. First, it introduces a standardized, publicly-released evaluation framework that separates three properties of explanations — utility (what a blind judge can recover), source support / confabulation (are claims grounded in the source), and writing quality — and uses it to show that point-reconstruction training increases utility but also increases confabulation and writing defects, across models such as Qwen, Gemma, and Apertus. Second, it argues point reconstruction has an "information blind spot" (the optimal unit-norm point loss depends on the conditional only through its mean) and proposes Flow-NLA, which replaces point reconstruction with a conditional denoising / diffusion-likelihood-bound reward. Empirically Flow-NLA retains utility gains while sharply curbing growth in confabulation and writing defects.

Critical strengths
  • Clear, well-motivated problem. Identification of drawbacks with point-training recipe, proposal of a diffusion like reward. Trying to explore what gets rewarded during training is especially useful for interpretability. The evaluation framework, separating utility from faithfulness/confabulation and also from writing quality also appears to be a right decomposition, and the paper executes it cleanly.

  • ** Strong theoretical observations** Eq. 1 — that the optimal unit-norm point-reconstruction loss depends on p(h|z) only through its conditional mean — is a simple and correct. The Figure 4 four-direction example makes it concrete, and the denoising-reward gap to information gain I(Z; Y) across the noise band is a nice justification.

  • **Reproducibility and honesty **. That this method is derived carefully from the diffusion variational bound, the energy-score reward, the EMA-reconstructor fix for the moving-target problem, and the per-noise-level credit-assignment analysis. Releasing code and reporting reruns that approximately match the released Anthropic NLAs (Table 1) give enough credibility.

Critical weaknesses
  1. The causal claim about the reward is confounded. Flow-NLA differs from the baseline in several ways at once — the distributional reward, an EMA reconstructor (decay 0.99), different reconstructor SFT, and a velocity-prediction architecture — so the reduction in confabulation can't be cleanly attributed to the reward objective alone. Without an ablation isolating the reward from these training-stability and architecture changes, the paper's central thesis that "richer reconstruction feedback" is what curbs confabulation remains under-supported.

  2. Every metric rests on a single, unvalidated LLM judge. Utility, source support, and writing quality are all scored by one judge (DeepSeek V4.1 Flash), with no human-agreement study, no second-judge cross-check, and no bias analysis. Since the entire paper is about measuring confabulation precisely, the reliability of this measurement instrument is load-bearing and currently unproven.

  3. "Retained utility" overstates a mixed result. Table 2 shows Flow-NLA's behavior macro-F1 actually drops versus point reconstruction on Qwen (74.9 → 72.9) and Apertus (73.6 → 69.6), and suffix accuracy is consistently slightly lower. The abstract and conclusion language ("retains substantial utility gains") glosses over the fact that one of the three utility axes moves the wrong way on two of three models.

  4. The core mechanism is proven in a toy setting but not demonstrated on real explanations. The "beyond the mean" argument holds in the four-direction example (Figure 4, Appendices H/J), and the distributional-fit diagnostics show the reconstructor captures more spread, but there is no direct evidence that the reward's multi-modal sensitivity is what actually drives the confabulation reduction on real activations. The link between the elegant theory and the empirical gain is suggestive rather than demonstrated.

Questions for authors
  • Can you ablate the reward in isolation — Flow-NLA reward with the baseline (non-EMA, current) reconstructor update, and/or point-reconstruction with an EMA reconstructor — to attribute the confabulation reduction to the objective rather than the training-stability changes? This helps understnad the real contributor, architecture, reward loss or both.
  • What is the judge's agreement with human annotators on a sample of source-support and writing-defect labels? How sensitive are the main conclusions to the choice of judge?
  • Three models at 7–12B range were experimented. Any reasons that can be expected as to why this holds or does not hold at 70B+?
  • How were layer and token positions selected per model, and how sensitive are the confabulation trends to that choice?
2 comments

Thank you for the detailed review. We address your questions and weaknesses below, including the requested ablations, judge validation, and the revised utility claims. We would be glad to clarify anything further.

Q1 | Reward Ablation in isolation

Thank you for this suggestion. We ran both proposed ablations on Qwen2.5-7B for 1600 steps: Flow-NLA without the EMA reconstructor, and point reconstruction with an EMA reconstructor (General Response Exp. 1; revised Appendix B, Figure 8). Flow-NLA without EMA closely tracks Flow-NLA, but confabulation rises later in training. Point reconstruction with EMA delays the rise in confabulation but reaches a similar level. The confabulation difference is therefore attributable mainly to the reconstruction objective rather than to EMA stabilization. We also observe a decrease in writing defects when using EMA, which prompts further investigation.

Note that the velocity-prediction architecture is part of the distributional formulation itself. A point reconstructor has no noise-conditioned input, meaning this component cannot be ablated separately from the objective. Therefore, the SFT of the reconstructor differs with respect to its training objective, even though the same samples and optimizer settings were used.

Q2 | Judge Sensitivity

Thank you for this question. We re-scored a paired subset of 88 Gemma-3-12B explanations (released NLA vs. Flow-NLA) with GPT-6 Astra and Claude Opus 5.5 (General Response Exp. 2). This was done on the 11 tasks that are in English, for comparison with the subsequent human annotator agreement. Assertion risk differs by at most 4.2 points across judges, and all three reproduce the Flow-NLA reduction of 20.0–21.9 points. All three also find fewer writing defects for Flow-NLA, although absolute defect counts depend on the judge, thus relative differences are more informative. The main conclusions are therefore not sensitive to the choice of judge.

The human annotation is now complete (General Response Exp. 2.2). With three annotators per sample, human-rated assertion risk drops by 25.2 points for Flow-NLA, and DeepSeek matches the human majority on 80.2% of claim labels, in line with human–human agreement (79.4%).

Q3 | Model Scaling

Thank you for raising the point of model size. To provide clarification, our evaluation includes larger released NLAs: Table 1 shows that similar confabulation and writing defects are also present for Gemma-3-27B and Llama-3.3-70B. The conditional-mean limitation of point reconstruction does, in principle, not depend on model size. Our Flow-NLA training experiments were limited to 7–12B models by our computational budget, and extending them to 70B+ is beyond our available resources. We expect the method to apply at larger scale, since confabulation persists there, but this remains to be tested.

Q4 | Layer and Token Choice

Appendix F.1 and Table 5 specify the extraction settings. For Qwen and Gemma, the extraction layers match the released NLA checkpoints. For Apertus, we follow the same rule used for the released checkpoints and extract from the layer at two-thirds of the model's depth (hidden_states[22]). Our point-reconstruction reruns and Flow-NLA share the same extraction site within each target model. Evaluation token positions are task-specific and identical across models: the last source-text token before the evaluation question for classification, the last prefix token for suffix prediction, and the first generated response token for response-behavior prediction.

We do want to point the reader to the original paper's section on layer sensitivity (https://transformer-circuits.pub/2026/nla/#nla-layer-sensitivity), which provides motivation for the layer choice. Sweeping over layers is costly, and we therefore restrict ourselves to the released configuration.

W3 | Utility retention

We agree that "retains utility gains" overstated a mixed result. Behavior macro-F1 decreases on Qwen and Apertus relative to point reconstruction, while classification and suffix prediction differ by at most 1.0 point. We have revised the abstract and conclusion to not overstate this.

W4 | Mechanism beyond the toy setting

We now test the "beyond the mean" argument on real Gemma activations (General Response, Exp. 3). For activation pairs with near-identical mean-based rewards, the full Flow reward prefers the label-compatible activation in 74–76% of pairs, against ~50% for the mean. This confirms that the distinction exists and is rewarded on real activations. We agree with the reviewer, however, that this does not yet demonstrate a direct theoretical link to the confabulation reduction in natural explanations (General Response §3).

AuthorsOpenReview ↗

Follow-Up to Reviewer 6smd

Thank you again for your detailed review. The human annotation you asked about in Q2 is now complete, and you can find the results in the updated General Response (Exp. 2.2).

Please let us know if anything remains unclear; we would be happy to address it in the remaining discussion period.

Reviewer_Nc8fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of reconstruction-based training for natural-language explanations of LLM activations, showing that improved activation reconstruction can lead to increased confabulation and writing defects. To systematically evaluate explanation quality, the authors introduce a framework that separately measures predictive utility, contextual support, and writing quality. They further identify a theoretical limitation of conventional point reconstruction: its optimal reward depends only on the conditional mean of activations, potentially overlooking important distributional information. To address this, they propose Flow-NLA, which replaces point reconstruction with a conditional diffusion-based reconstructor and trains the explanation generator using a denoising reward derived from a diffusion likelihood bound.

Critical strengths
  • The paper identifies a fundamental limitation of point-reconstruction objectives: their optimal reconstruction reward depends only on the conditional mean, potentially overlooking important distributional information. The proposed Flow-NLA addresses this limitation through conditional denoising and a diffusion likelihood-based reward.
  • The paper introduces a systematic evaluation framework that separately measures predictive utility, contextual confabulation, and writing quality, enabling a more nuanced assessment of activation explanations.
Critical weaknesses
  • The paper does not justify why richer reconstruction signals should necessarily reduce confabulation. The proposed objective optimizes activation reconstruction rather than factual correctness, leaving the connection between the theoretical motivation and the observed improvements unclear. Furthermore, Flow-NLA introduces additional training modifications, such as EMA-based reconstructor scoring, without sufficiently isolating their contributions. This makes it difficult to determine whether the improvements arise specifically from the proposed distributional reconstruction objective.

  • The evaluation relies heavily on a single LLM judge (DeepSeek V4.1 Flash) to assess predictive utility, contextual support, and writing quality, without sufficient independent or human validation of the resulting judgments. Moreover, the confabulation metrics measure consistency with the source text rather than whether explanations faithfully reflect the model's internal computations. Consequently, lower confabulation scores do not necessarily indicate more faithful activation explanations.

Questions for authors
  • The theoretical analysis demonstrates that distributional reconstruction captures information beyond the conditional mean. However, it remains unclear why this additional information should lead to fewer unsupported claims, since the reward does not explicitly optimize factual correctness or source support. Could the authors provide further theoretical intuition or empirical evidence explaining the mechanism through which distributional reconstruction reduces confabulation?

  • Flow-NLA introduces several changes beyond replacing point reconstruction, including an EMA-based reconstructor for reward scoring. Could the authors provide controlled ablations that isolate the contribution of the distributional reconstruction objective from these additional training modifications?

  • The evaluation framework relies on DeepSeek V4.1 Flash to assess utility, confabulation, and writing quality. Could the authors validate these metrics through human annotations or evaluations using multiple independent LLM judges? How sensitive are the reported improvements to the choice of evaluator?

  • Although Flow-NLA generally preserves predictive utility, some metrics, particularly response-behavior prediction, decrease relative to point-reconstruction NLA. Could the authors investigate whether reducing confabulation comes at the cost of losing certain activation-relevant information? How does this trade-off vary across models and explanation types?

5 comments

Thank you for the constructive review. Below we address your questions on the confabulation mechanism, the EMA ablation, judge validation, and the utility trade-off. Please let us know whether this resolves your concerns or if further questions remain.

Q1 | Mechanism of confabulation reduction

We agree with the reviewer that additional activation information does not itself imply source support. Our proposed mechanism is an indirect change in relative incentives: point reconstruction can overlook supported distinctions that change the conditional distribution without changing its mean, while still rewarding unsupported details that happen to encode activation information. Distributional reconstruction can credit those supported distinctions, potentially increasing the reward of a supported explanation relative to an unsupported alternative. When such alternatives compete for selection, this provides a possible mechanism for reducing confabulation.

Flow can also reward unsupported information, and supported and unsupported details coexist within an explanation. We therefore do not claim that distributional reconstruction inherently enforces faithfulness. General Response Exp. 3 provides activation evidence that Flow credits distinctions invisible to the mean.

Empirically, Figures 6–7 show that Flow-NLA largely retains utility gains while limiting confabulation growth relative to Point-NLA across three target models. These results are consistent with the proposed incentive mechanism, but do not isolate it as the cause of the improvement. We have added the activation experiment (Appendix D) and expanded the limitations (Section 5).

Q2 | Ablation of method changes

The reviewer is right to address the fact that we introduce the EMA scoring as an additional change, without the proper ablation. To give further insight into the contribution of each component, we have run an ablation for Flow-NLA without EMA and point reconstruction with EMA for 1600 steps on Qwen2.5-7B.

In the revised PDF (Appendix B, Figure 8), we have added a six-panel figure to illustrate this ablation. As can be seen, Flow-NLA without EMA closely tracks Flow-NLA, but with a later rise in confabulation. In contrast, point reconstruction with EMA shows a delayed response across confabulation metrics, but a similar rise in confabulation. These results suggest that the EMA-based reward scoring introduced for training stability is not the main driver for the reduced confabulation. Furthermore, the experiment shows a decrease in writing quality defects for Point NLA with EMA, which warrants further investigation into the relationship between EMA and writing defects.

Q3 | Evaluation framework

To answer the reviewer's question on validation through human annotations or evaluations, we calculate the agreement with state-of-the-art models and human annotators.

We re-scored a paired subset of 88 Gemma-3-12B explanations with GPT-6 Astra and Claude Opus 5.5. This was done on the 11 tasks that are in English, for comparison with the subsequent human annotator agreement. The three judges agree on assertion risk within 4.2 points and on the Flow-NLA reduction within 1.9 points, and all three find fewer writing defects for Flow-NLA (General Response Exp. 2).

The human annotation is now complete (General Response Exp. 2.2). With three annotators per sample, human-rated assertion risk drops by 25.2 points for Flow-NLA, and DeepSeek matches the human majority on 80.2% of claim labels, in line with human–human agreement (79.4%).

Q4 | Utility Preservation Trade-off

Thank you for highlighting this point. The most direct comparison is with our point-reconstruction reruns on the same target models (Table 2). Table 1 illustrates the discrepancies between the rerun and official numbers. For a subset of models, our reruns show lower behavioral prediction scores than the released NLAs but similar utility metrics otherwise. Relative to these reruns, Flow-NLA changes response-behavior macro-F1 by −2.0 points on Qwen, +2.4 on Gemma, and −4.0 on Apertus. Classification and suffix-prediction scores differ by at most 1.0 point, while assertion risk decreases by 8.5–17.0 points across models. Figure 6 complements these checkpoint comparisons: both methods achieve broadly similar utility gains over training, while Flow-NLA exhibits substantially less confabulation growth.

Checkpoint selection also matters when interpreting these comparisons. Table 2 reports results at selected checkpoints, whereas Figure 6 shows the full training trajectories relative to each run’s supervised initialization. Both methods achieve broadly similar utility gains, with fluctuations across training, suggesting that some differences at selected checkpoints may partly reflect the training stage and checkpoint choice. Meanwhile, Flow-NLA consistently exhibits less confabulation growth. We therefore interpret the checkpoint results alongside these trajectories, while acknowledging the model- and task-specific differences in Tables 1 and 2.

Regarding information loss: classification and suffix prediction remain within 1.0 point of point reconstruction, so we see no evidence of a broad loss of activation-relevant information (Figure 7 and Table 2).

AuthorsOpenReview ↗

Follow-Up to Reviewer Nc8f

Thank you again for your thoughtful review. We have now completed the human annotation, and the results are available in the updated General Response (Exp. 2.2).

If any of your concerns remain open, we would be glad to address them during the remaining discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their constructive feedback. We first summarize the new experiments, then address the three concerns shared across reviews.

New experiments

#ExperimentLocationReviewers
1EMA ablation (Flow-NLA w/o EMA, Point-NLA w/ EMA)App. B, Fig. 8Nc8f, 6smd
2Cross-judge validation (GPT-6 Astra, Claude Opus 5.5)App. C, Fig. 9Nc8f, 6smd, xWpE
3Real-activation compatibility probesApp. DNc8f, 6smd, xWpE
4Human annotationApp. CNc8f, 6smd, xWpE
5Computational costApp. ExWpE
6Paired qualitative examplesApp. OxWpE

1. Attributing the gains to the reconstruction objective (Nc8f, 6smd)

We ran an ablation of Qwen2.5-7B for Flow-NLA without EMA and point reconstruction with EMA. Inspecting the new ablation Figure 8 in Appendix B, Flow-NLA without EMA closely tracks Flow-NLA, but later reaches a noticably higher assertion risk. Point reconstruction with EMA delays the rise in confabulation but reaches a similar level. This suggests that the difference in confabulation is mainly attributable to the distributional objective, rather than to EMA stabilization. However, EMA stabilization does benefit Flow-NLA. Point reconstruction with EMA also shows fewer writing defects, which warrants further investigation.

General Response Exp. 1

See Figure 8 in Appendix B, the table below shows the final (update 1600) confabulation metrics.

MethodReward EMAΔ Assertion risk (pp) ↓Δ Confabulated token coverage (pp) ↓
Point NLA (paper)No+13.70+13.09
Flow-NLA (paper trajectory)Yes+3.76+1.03
Point NLA + EMA (new)Yes+12.60+12.34
Flow-NLA w/o EMA (new)No+7.19+2.02

2. Reliability of the LLM judge (Nc8f, 6smd, xWpE)

We re-score a paired subset of 88 Gemma-3-12B explanations (44 released-NLA and 44 Flow-NLA on the same sources, four per task across the 11 English tasks) with GPT-6 Astra and Claude Opus 5.5, both with low reasoning effort. On 556 extracted claims, Astra and Opus agree with DeepSeek on 87.5% and 91.1% of labels. Assertion risk differs by at most 4.2 points across judges, and all three reproduce the Flow-NLA reduction (20.0–21.9 points). Absolute defect counts depend on the judge, so we interpret them as relative differences; all three find 54–72% fewer defects for Flow-NLA.

General Response Exp. 2.1

JudgeReleased assertion riskFlow assertion riskReleased defects/expl.Flow defects/expl.
DeepSeek V4.1 Flash (reference)82.5%62.5%1.820.84
GPT-6 Astra (low)85.6%63.7%1.800.50
Claude Opus 5.5 (low)86.7%64.9%4.892.00

The human annotation of the 88 paired Gemma-3-12B explanations (556 claims) is complete. Six annotators took part, each labeling half of the samples, so that every sample has three annotators. To keep the annotation burden manageable, we measure human agreement with the judge rather than collecting independent annotations: annotators labeled source support for the claims the DeepSeek judge had already extracted, and verified the writing defects it had flagged.

General Response Exp. 2.2

AssessorReleased risk (%)Flow risk (%)Released defectsFlow defects
DeepSeek V4.1 Flash82.562.51.820.84
GPT-6 Astra (low)85.663.71.800.50
Claude Opus 5.5 (low)86.764.94.892.00
Human aggregate76.050.81.360.54
DecisionComparisonAgreement / acceptance
ClaimsIndividual human–DeepSeek78.1%
ClaimsHuman majority–DeepSeek80.2%
ClaimsHuman–human79.4%
WritingIndividual proposal acceptance71.5%
WritingMajority acceptance76.9%
WritingHuman–human65.2%

Human-rated assertion risk drops by 25.2 points for Flow-NLA, compared with 20.0–21.9 points under the three LLM judges. Humans also find 60% fewer writing defects for Flow-NLA, within the 54–72% range of the judges. DeepSeek matches the human majority on 80.2% of claim labels, while two human annotators agree on 79.4%. Astra and Opus match the human majority on 81.6% and 81.9%.

Some limitations we note are that the LLM judges are stricter than the human annotators; human-rated assertion risk is 6.5 points lower than DeepSeek's for the released NLA and 11.7 points lower for Flow-NLA. The judges also agree more with each other (87.5–91.9%) than with humans, which suggests some shared bias. For writing quality, the human majority accepts 76.9% of flagged defects, but human–human agreement is lower (65.2%), and since annotators verified proposed defects only, this measures the judge's precision and not its recall.

AuthorsOpenReview ↗

General Response to All Reviewers 2

3. Link between distributional reconstruction and confabulation (Nc8f, 6smd, xWpE)

We thank the reviewers for raising this important point, which is at the basis of what led us to pursue this research. Indeed, the reward does not directly target source support. This part is deliberate, like other unsupervised interpretability methods, internal monitors aim to learn from the activations alone. Rewarding based on the source would compromise the monitor, as it would drift from the model's internal computation. Any effect on confabulation therefore arises indirectly with respect to the source text.

Our proposed mechanism is a shift in relative incentives. Point reconstruction rewards only changes in the conditional mean. It can therefore overlook supported distinctions that change the distribution but not its mean, while still rewarding unsupported details that encode activation information. Distributional reconstruction can credit information beyond this mean, but also still reward unsupported claims. An intuition is that a mean-based target encourages blending cues from several possible candidates, whereas a distributional reconstructor can represent uncertainty and alternatives. Our method plausibly reduces this pressure, but we acknowledge that a causal link is hard to establish, and that further work is required in that respect. In the revision, we have added elaboration on this in the limitations section of the conclusion.

As an empirical test, we check whether Flow captures information beyond the mean on real Gemma activations. We use 448 activations from four classification datasets with templated label descriptions (e.g. "Company"). For each label-compatible activation, we select the incompatible activation from another class whose mean-based reward under the same description is closest, keeping pairs within 0.05 standard deviations. The mean-based reward comes from either the point reconstructor or the mean of Flow's own samples (the two rows). We then check which of the two activations the full Flow reward prefers:

General Response Exp. 3

Matched onMean-based rewardFull Flow reward
Point reconstructor51.2%76.2%
Flow mean50.4%74.0%

Since selection equalizes the mean-based rewards, they are at chance by construction. The full Flow reward, not used in selection, prefers the compatible activation in 74–76% of pairs in both matchings.

The new experiments are in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.