acceptodds
Under review as a conference paper at ICLR 2027

Layer-Wise Semantic Evidence Rewriting under Unified Mask-Token Training

Abstract

Extending vision-language models to predict masks can also change their answers to semantic questions. What changes in the evidence behind those answers? We study layer-wise semantic evidence rewriting between reference and unified checkpoints. Our framework decomposes a decision margin into input and residual-block contributions. The decomposition accounts for normalization and is exact for a fixed forward pass. Comparing representations under checkpoint-sourced readouts separates representation-side changes from readout-side changes. A calibrated linear transport then maps block evidence between checkpoints. Across five model-benchmark comparisons spanning two backbone families, representation-side drift exceeds readout-side drift. Full transport reduces reconstruction error by 30.6-69.0% relative to uniform rescaling on the original evaluation splits. A retrospective group-disjoint reanalysis finds positive sample-level associations in four settings. Full transport also yields lower observed mean absolute errors than scalar, diagonal, and probe-mean baselines. Its advantage over diagonal transport persists when both include layer-wise intercepts. In HiMTok, selected late-layer groups are sensitive to residual-write rescaling, but no more sensitive than the last three blocks. These findings reveal a layer-wise structure in semantic change that aggregate accuracy alone cannot show.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Unified Multimodal Models, Mask Token Modeling, Representation Analysis, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4D5xOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how layer-wise semantic evidence changes between reference vision-language models and unified checkpoints with mask-prediction capabilities. To characterize these changes, the paper introduces a normalization-aware decomposition of decision margins into input and residual-block contributions that is exact for a fixed forward pass. It then compares representations under checkpoint-sourced readouts to separate representation-side, readout-side, and interaction changes, and fits a calibrated linear transport to map block evidence between checkpoints. Experiments across five model–benchmark comparisons spanning two backbone families show greater representation-side than readout-side drift and improved evidence reconstruction over uniform rescaling.

Critical strengths
  • The paper provides a normalization-aware decomposition of decision margins into input and residual-block contributions that is exact for a fixed forward pass. The checkpoint-sourced comparisons offer a systematic way to distinguish representation-side, readout-side, and interaction changes.

  • The transport analysis covers five model–benchmark comparisons across two backbone families. The group-disjoint reanalysis and affine controls provide additional evidence that the observed predictive structure extends beyond uniform rescaling and layer-wise mean shifts.

  • The evaluation distinguishes evidence reconstruction, sample-level prediction, and behavioral sensitivity. Real-forward interventions recompute downstream states and outputs, while comparisons with simple layer-selection baselines help clarify the scope of the findings.

Critical weaknesses
  • The paper improves evidence reconstruction and sample-level prediction, but how these gains explain specific changes in semantic behavior remains unclear. The additional analyses offer limited support: the transport-selected groups show no greater intervention sensitivity than the last three blocks (Table 20), and the auxiliary controller shows no consistent semantic-retention advantage (Table 26). While these findings do not negate the descriptive contribution, the paper should better establish what new understanding follows from the identified patterns.

  • The main novelty lies in organizing checkpoint differences into a layer-wise evidence analysis that combines representation/readout contrasts, calibrated transport, and behavioral interventions. The underlying technical ingredients are largely established: the fixed-pass decomposition closely follows direct logit attribution, and the transport is regularized linear regression whose summed prediction is equivalent to direct ridge prediction under the same conditions. The contribution therefore appears primarily methodological and empirical, with more limited novelty in the underlying mathematics.

  • The research question is clear, but the motivation for the specific transport analysis and its methodological novelty could be better established. Although the comparisons support a description beyond uniform or independent layer-wise rescaling, it remains unclear which important phenomena this distinction explains beyond existing diagnostic approaches. The fixed-pass decomposition closely follows residual-based logit attribution, while transport is regularized linear regression with structural constraints. Equation (11) explicitly shows that, under matched inputs, loss normalization, and regularization, the summed prediction of full transport is equivalent to direct no-intercept ridge regression. The layer-wise matrix retains information beyond this scalar prediction, but its additional explanatory value remains insufficiently demonstrated, particularly given that the transport-selected layer groups show no greater sensitivity to residual-write rescaling than the last three blocks.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

The Value of Layer-Resolved Transport and Its Empirical Contribution

Thank you for recognizing the source comparisons, group-disjoint evaluation, affine baselines, and real-forward experiments. Your central question is what layer-resolved transport adds beyond existing attribution and regression tools. We have clarified this contribution and added a behavioral experiment testing a concrete use of its layer-level predictions.

1. Why retain the matrix, and where is the contribution? (Weaknesses 2 and 3)

Residual logit attribution and regularized linear regression are established foundations of this work. Our contribution is to compare checkpoints in a common normalization-aware evidence representation, distinguish representation and readout sources, and evaluate the layer-resolved description through structural controls and real-forward interventions. Section 2 and the contribution statement now make this positioning explicit.

The matrix is retained because the question concerns both the total evidence change and its distribution across layers. A scalar predictor cannot distinguish changes with the same total but different layerwise allocations. Section 3.4 formalizes this: when 1⊤K=0\mathbf{1}^{\top}K=0, TT and T+KT+K give identical summed predictions, although their blockwise predictions can differ. Accurate scalar prediction therefore leaves the layer-resolved description to be tested separately.

We use two tests: reconstruction against matrices with different structural constraints, and whether the layerwise predictions help select intervention locations. The off-diagonal terms describe statistical cross-depth dependence, rather than causal connections between blocks.

2. What do the structural results tell us? (Weaknesses 1 and 3)

First, representation-side drift exceeds readout-side drift in all five settings, with a non-negligible interaction. The example in Section 5.1 illustrates the distinction: the unified trajectory retains a negative margin under Base's readout, while Base's trajectory retains a positive margin under the unified readout. This separates representation and readout changes in a way that an accuracy drop alone cannot.

Second, the transport comparisons test whether simpler descriptions account for the observed changes. Full improves reconstruction over Scalar in all five settings (Table 3). In the four affine controls, the advantage over Diagonal persists when each layer has its own intercept (Appendix C.1, Table 17). Uniform scaling and independent layerwise scaling plus mean shifts therefore do not reach the same reconstruction quality on these evaluation data.

These results turn the broad observation that the model has changed internally into comparisons of evidence sources and layerwise statistical structure. They do not identify the training mechanism that produced those changes.

3. What behavioral evidence has been added? (Weaknesses 1 and 3)

In the original rescaling experiment, the high-score group did not outperform the last three blocks. This remains true after excluding identified links to calibration data (Appendix D.4, Table 24).

The new experiment tests a sample-specific use of the predictions: selecting layers for restoration toward the matched Base write and measuring the resulting complete-margin gain. It uses 88 MME records in 74 groups without identified calibration links. Full and Diagonal each select three layers; we intervene on each layer separately, recompute downstream states and outputs, and average the three margin gains (Appendix D.5, Table 25).Full exceeds last-three selection by +0.420360, with a 95% interval of [0.208057, 0.643320] and Holm-adjusted p=0.000800p=0.000800. Against Diagonal, the difference is +0.065840, with interval [−0.009408, 0.137285] and adjusted p=0.075192p=0.075192.

Thus, sample-specific Full selection has a demonstrated use beyond a fixed last-three rule under this restoration operation. An additional behavioral benefit from cross-depth mixing over Diagonal is not yet established. The experiment is retrospective and label-informed, and matches intervention counts rather than depth or perturbation magnitude.

Restoration gain and rescaling sensitivity are different endpoints, so the new result does not conflict with the earlier last-three comparison. The three-seed OECMT result also remains unchanged: no consistent semantic-retention advantage (Appendix E.3, Table 29). The revised Discussion separates the diagnostic findings and restoration use from the further question of how to turn them into a better training policy.

AuthorsOpenReview ↗

The Additional Value of Layer-Resolved Transport

Thank you again for raising the question of what the layer-resolved matrix contributes beyond scalar regression. We agree that the summed prediction alone does not establish that contribution.

The additional object of study is the distribution of evidence changes across layers. Different layerwise changes can produce the same scalar prediction. Our structural comparisons test this distribution directly: Full retains an advantage over Diagonal when both allow layer-specific intercepts, showing that independent layerwise scaling and mean shifts do not capture the checkpoint relation equally well.

The restoration experiment tests a concrete use of these layer-level outputs: selecting intervention locations separately for each question. This differs from ranking fixed layer groups by rescaling sensitivity. The answer-correctness analysis now shows that Full-selected restoration raises average accuracy from 59.09% to 65.15%, whereas last-three restoration gives 57.58% (Appendix D.5, Tables 26–27). These averages cover three separate single-layer restorations per question.

This supports a behavioral use of the layer-resolved predictions beyond scalar fit quality. The additional behavioral advantage of Full over Diagonal remains unresolved; our contribution rests on the source-separated analysis, structural comparisons, and demonstrated restoration use.

Thank you for your rebuttal. The authors’ responses have largely addressed some of my concerns. I will raise my score to 3.

AuthorsOpenReview ↗

Thank You for the Discussion

We sincerely thank the reviewer for the careful reassessment and constructive feedback throughout the discussion. We are glad that our responses and additional results have addressed the reviewer's concerns.

Reviewer_C8vSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares how a reference VLM and a model with masking capabilities distribute decision evidence across different layers. The method decomposes decision margins, fits linear mappings between layer-wise contributions, and evaluates reconstruction, prediction, and intervention results. The study also reports numerical checks and training attempts that did not yield consistent improvements.

Critical strengths
  1. The analysis carefully distinguishes the reconstruction of observed evidence from explanations of causal mechanisms. It also considers input and residual contributions, interactions between representations and readouts, and numerical checks. The discussion of what the fitted mappings can and cannot establish is a strength of the paper.
  2. The control experiments are valuable, and the negative results are clearly reported. Group-disjoint prediction tests and affine baselines rule out some alternative explanations. The paper also acknowledges that intervention-based layer selection does not outperform simply selecting the last three blocks, and that the training attempts do not yield consistent improvements.
Critical weaknesses
  1. The main results remain descriptive fits between specific model checkpoints. The summed predictions from the mappings are equivalent to scalar ridge regression, while the mappings exclude input contributions and depend on the selected answer contrast. Differences in the training histories of the checkpoints further limit causal interpretation. Therefore, the central conclusions should focus on evidence reconstruction rather than explaining why semantic capabilities change.
  2. The behavioral experiments do not independently validate the fitted layer structure. Although group-disjoint evaluation is used for prediction, the interventions still rely on earlier evaluation samples with associations across data splits. The selected layer groups are also insufficiently stable and do not outperform the baseline of selecting the last three blocks. This limits the method's value for identifying behaviorally important layers beyond reconstruction.
Questions for authors

Please address the issues mentioned in the weaknesses.

5 comments
AuthorsOpenReview ↗

Diagnostic Scope, Data Dependence, and Behavioral Controls

Thank you for recognizing the source-separated analysis, controls, and reporting of negative results. Your concerns focus on the scope of the conclusions and on data dependence and simple baselines in the behavioral experiments. The revision clarifies the former and adds two analyses addressing the latter.

1. Scope, ridge equivalence, and the prediction target (Weakness 1)

Transport describes evidence changes calibrated from paired checkpoint data. Section 3.4 and Appendix A.4 retain the equivalence between summed full-transport prediction and no-intercept scalar ridge regression under matched inputs and regularization, including the qualifications at zero regularization. The contribution concerns layer-resolved evidence and its sources, rather than a new scalar regression estimator.

The complete fixed-pass decomposition includes the contribution associated with H0H_0 and the effective bias. Transport uses the 32 Transformer-block contributions, so its target is consistently described as block-evidence change. The intervention experiments instead recompute the complete output margin. Sections 3–4 make this distinction explicit.

The answer contrast, checkpoint histories, and evaluation interfaces remain stated qualifications. We therefore frame the findings as statistical diagnostics of checkpoint differences and do not attribute all differences to mask supervision. The behavioral use of the matrix is assessed through an intervention endpoint separate from reconstruction.

2. Excluding identified links between intervention and calibration data (Weakness 2)

You are correct that group-disjoint prediction does not remove cross-split links in the original intervention data. Appendix D.4 adds an analysis excluding intervention records connected to the original probe or validation data through available question/image identities. This retains:

  • MME: 88 records in 74 groups.
  • MMBench: 196 records in 196 groups.

The original selectors remain fixed, and records are not filtered by intervention outcomes.

The high-score groups remain more sensitive than probe-RMS controls on both benchmarks. On MME, however, they remain less sensitive than the last three blocks: the high-minus-last-three difference is −0.057207, with a 95% interval of [−0.079796, −0.033926]. On MMBench, the high group is exactly the last three blocks, so the two do not provide a separate comparison (Table 24).

Excluding the identified links therefore preserves the main findings, including the unfavorable last-three result. The analysis remains retrospective.

3. Testing sample-specific Base-write restoration (Weakness 2)

We further tested whether sample-level evidence predictions help select locations for restoring the matched Base write (Section 5.4; Appendix D.5).

The experiment covers all 88 eligible MME records. The frozen Full and Diagonal operators each select the three smallest coordinates of the predicted block-evidence change. Each selected layer is intervened on separately, restoring its answer-position residual write toward the matched Base write. We recompute downstream states and outputs and average the three complete-margin gains.

Paired comparisonMean margin-gain difference95% intervalHolm-adjusted pp
Full − Diagonal+0.065840[−0.009408, 0.137285]0.075192
Full − Last-three+0.420360[0.208057, 0.643320]0.000800

Full outperforms fixed last-three selection for this restoration operation, while its advantage over Diagonal remains uncertain. This experiment measures signed restoration gains; the earlier rescaling experiment measures the magnitude of output changes. They address different questions.The restoration test uses known answer contrasts and matches the number of interventions, but not depth or perturbation magnitude. Its conclusion is consequently specific to this operation.

4. Selection stability (Weakness 2)

The historical top-three set recovery rates remain 42% on MME and 78% on MMBench (Appendix D.1). The additional experiments do not change this stability result. Intervention intervals condition on fixed selectors and exclude fitting and selection uncertainty.

The revised Discussion distinguishes structural reconstruction, held-out sample prediction, and intervention-based utility. The evidence supports a concrete use in Base-write restoration, while selection stability and the incremental behavioral value of Full over Diagonal remain limitations.

Thank you for the detailed response and additional experiments. However, an increase in the decision margin does not necessarily mean that the model answers more questions correctly. Your restoration experiment may correct some originally incorrect predictions, but it may also cause some originally correct predictions to become incorrect. Could you report how many predictions were corrected and how many became incorrect after restoration, along with the overall accuracy change? It would also be helpful to break down these results by MME categories to show which types of questions benefit from the restoration and which are negatively affected. This would provide a clearer picture of whether the restoration actually improves the model's semantic performance.

AuthorsOpenReview ↗

Answer Accuracy and Category-Level Effects of Restoration

Thank you for this suggestion. We have added answer-correctness results to Appendix D.5, Tables 26–27 (pp. 28–29). The results show that Full improves average answer accuracy as well as decision margin: corrections outweigh newly introduced errors.

On the same 88 MME questions from 74 groups, the unmodified HiMTok checkpoint answers 52 questions correctly, giving a baseline accuracy of 59.09%. Full produces:

Incorrect → correctCorrect → incorrectAverage restored accuracyAccuracy change
22665.15%+6.06 pp

For each question, we apply restoration separately at each of the three selected layers. The transition counts therefore cover 264 question–layer outcomes. Accuracy averages the three correctness outcomes within each question and then averages equally across questions. Full yields 16 more corrections than newly introduced errors, accounting for the 6.06-percentage-point improvement.

The category breakdown identifies where these gains occur. Full improves average accuracy on celebrity, count, position, posters, and scene. It reduces accuracy on OCR, color, and landmark, while the remaining categories have zero net accuracy change. Table 27 reports the question counts, corrections, harms, and accuracy changes for all 14 categories. Table 26 also includes the complete comparisons with Diagonal and Last-three.

Full’s restoration benefit extends beyond a larger margin to a net increase in correct answer outcomes on the evaluated questions, with both corrections and newly introduced errors explicitly accounted for.

Reviewer_k4zEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper focuses on how the contributions of individual layers to a vision-language model’s outputs change after the model is adapted to generate segmentation masks. It proposes separating the margin between outputs into contributions per layer, in order to study changes that are not directly observable in terms of accuracy.

Critical strengths

The central question seems interesting to me, since simply knowing that a model loses accuracy after being adapted for segmentation says little about what changed internally. Decomposing the contributions by layer can be a useful way to study that difference.

Critical weaknesses
  1. I found difficult to understand the details of the main technical aspects. From the very title, the paper describes concepts and notation without first explaining them or why they are necessary. I had to constantly go back and forward to definitions and details scattered throughout the main text and the appendices just to interpret the results. Even after a careful reading, my understanding is incomplete. I could not clearly follow how each experiment contributes to answering the initial question.

  2. Many papers have studied what VLMs lose when trained for segmentation. For example, F-LMM studies failures in following instructions, and See, Say, and Segment (Wu et al., 2024) examines failures to reject requests involving absent objects. The first gets only a brief mention here, and the second is missing. With so little comparison, is difficult to judge if this paper explains why such failures happen or just measures the changes that come with them.

  3. The work very quickly loses sight of the question that motivates it. Many pages are focused on the accuracy of the decomposition, numerical precision, transport variants, etc. Although I understand the need for certain checks, it’s just too much detail that doesn’t contribute to understand the problem. The results are surrounded by so many definitions and caveats that it is difficult to identify the significant finding.

  4. After finishing, I do not feel that I have a better understanding of the problem. The stated motivation promises to go beyond just observing a drop in accuracy, but there is no explanation of what changes within the model. My concern is not simply that the writing is difficult, the work also fails to demonstrate the deeper level of understanding that would justify the entire analysis.

Questions for authors
  1. What did you learn about why a model’s answers get worse after segmentation training? Please walk through one example showing how the diagnosis helps explain the change.
  2. Which experiment gives the strongest evidence for that explanation? I had trouble following the link between the decomposition, the transport results, and the layer interventions.
5 comments
AuthorsOpenReview ↗

Worked Example, Experimental Logic, and Related Work

Thank you for pointing out that the original presentation made it difficult to connect the experiments to the research question. Your two questions—what we learn from a concrete failure and which experiment provides the strongest evidence—guided our revisions. We added a worked example, clarified the experimental sequence, and expanded the related-work discussion.

1. What do we learn from a concrete failure? (Weakness 4; Question 1)

Section 5.1 and Appendix B.5 examine MME:143, which asks whether an artwork is displayed in the Church of Brou. The correct answer is “No.” Base answers correctly, whereas HiMTok answers “Yes.” The complete No-minus-Yes margin, evaluated in FP32 on the native forward caches, changes from +3.615 to −3.434.

The answers alone reveal the error. Comparing representation and readout sources adds two observations:

  • HiMTok's trajectory evaluated with Base's readout parameters still has a negative margin: −2.714.
  • Base's trajectory evaluated with HiMTok's readout parameters has a positive margin: +4.503.

Thus, changing the readout alone does not account for this answer reversal. The representation-side change remains evident under the base-sourced readout.

The decomposition then shows how support for the correct answer changes across layers. Under each checkpoint's own readout, block 27's contribution falls from +4.012 to −0.017, and block 32's from −0.976 to −3.782, while other blocks increase their support. The change is uneven across layers. The complete accounting also includes the input contribution, which is small in this example (Table 15).

We also report where the diagnosis falls short. Transport predicts a base-readout block-evidence change of −1.917, compared with the observed −6.330, and none of the 74 cached rescaling interventions restores the correct answer. The case describes the internal evidence changes accompanying the error; it does not identify the training process that produced them.

2. How do the experiments connect, and which evidence is most direct? (Weaknesses 1 and 3; Question 2)

The revised Introduction organizes the experiments around three questions:

  1. Where does the evidence change? Complete accounting and representation/readout comparisons address this question. In the example above, the source comparison directly shows that a readout change alone is insufficient to explain the margin reversal.
  2. What structure do the changes have across samples? Scalar, Diagonal, Banded, and Full transport compare uniform scaling, independent layer scaling, and cross-layer statistical mappings.
  3. Can this information help select intervention locations? Real-forward experiments modify a residual write and recompute downstream states, normalization, and outputs.

For the third question, we added sample-specific Base-write restoration (Section 5.4; Appendix D.5). On 88 MME records retained after excluding identified calibration links, Full and Diagonal each select three layers from their predicted block-evidence changes. We intervene on each layer separately and average the three complete-margin gains.

Full exceeds the last-three-block baseline by +0.4204, with a 95% interval of [0.2081, 0.6433]. Its difference from Diagonal is +0.0658, with interval [−0.0094, 0.1373]. This provides direct evidence that diagnostic selection can be useful for this restoration operation, while the additional benefit over Diagonal remains uncertain. These aggregate results do not imply that the worked example was repaired.

3. Related work and presentation (Weakness 2; Weaknesses 1 and 3)

Section 2 expands the discussion of F-LMM's instruction-following findings and adds See, Say, and Segment's analysis of false-premise rejection. These studies characterize behavioral failures. Our analysis complements them by comparing the internal decision evidence across checkpoints for a specified answer contrast.The revised Abstract, Introduction, and Discussion foreground this question, and the worked example connects the measurements to an actual answer change. Detailed numerical checks remain in the appendices, so the main text can focus on what each experiment establishes.

Reviewer_k4zEOpenReview ↗

Response to authors

Thanks for the clarifications and changes to the manuscript. The new example helps to understand what measures the diagnostic, and the restoration experiment brings evidence that was not present in the previous version. However, I still consider that the main weaknesses remain unresolved, therefore I maintain my original score.

With respect to the first point of your response, the example partially addresses my concerns, as I still do not understand what information or capability change caused the model to make that mistake. I do not expect you to rebuild the whole training process to answer, but my still unanswered question is what we learn about besides locating changes in the contributions to the margin.

Regarding the second point, the three questions in the introduction are an improvement and the added restoration shows some of the usefulness of the diagnostic. Nonetheless, I still don't see how these results explain the change in the answers given by the model, and the manuscript is still difficult to follow.

Adding the previously mentioned references also corrects the original omission, but its inclusion is strictly limited to distinguishing the procedures, rather than clarifying what we learn about the phenomenon through that difference. That was my main concern about the comparison with the literature.

AuthorsOpenReview ↗

Clarifying What the Diagnosis Reveals

Thank you again for asking what the analysis teaches us beyond observing an accuracy drop. We would like to make the main finding more explicit.

In the worked example, the answer reversal cannot be explained by changing the readout alone: the Unified trajectory still gives a negative correct-answer margin under the Base readout (−2.714), while the Base trajectory retains a positive margin under the Unified readout (+4.503). The decomposition further shows that support for the correct answer decreases sharply at particular blocks while increasing at others. This identifies both an evidence source associated with the reversal and the uneven layerwise changes behind it.

Transport then tests how these changes are structured across samples. Restoration provides a separate behavioral test of whether its layer-level predictions identify useful intervention locations. On the 88-question MME subset, Full-selected restoration increases average answer accuracy from 59.09% to 65.15%, averaging three separately applied single-layer restorations per question (Appendix D.5, Tables 26–27).

Together, these analyses distinguish where evidence changes, how those changes relate across layers, and whether the resulting diagnosis can guide an intervention that improves answers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.