- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how layer-wise semantic evidence changes between reference vision-language models and unified checkpoints with mask-prediction capabilities. To characterize these changes, the paper introduces a normalization-aware decomposition of decision margins into input and residual-block contributions that is exact for a fixed forward pass. It then compares representations under checkpoint-sourced readouts to separate representation-side, readout-side, and interaction changes, and fits a calibrated linear transport to map block evidence between checkpoints. Experiments across five model–benchmark comparisons spanning two backbone families show greater representation-side than readout-side drift and improved evidence reconstruction over uniform rescaling.
Critical strengths
-
The paper provides a normalization-aware decomposition of decision margins into input and residual-block contributions that is exact for a fixed forward pass. The checkpoint-sourced comparisons offer a systematic way to distinguish representation-side, readout-side, and interaction changes.
-
The transport analysis covers five model–benchmark comparisons across two backbone families. The group-disjoint reanalysis and affine controls provide additional evidence that the observed predictive structure extends beyond uniform rescaling and layer-wise mean shifts.
-
The evaluation distinguishes evidence reconstruction, sample-level prediction, and behavioral sensitivity. Real-forward interventions recompute downstream states and outputs, while comparisons with simple layer-selection baselines help clarify the scope of the findings.
Critical weaknesses
-
The paper improves evidence reconstruction and sample-level prediction, but how these gains explain specific changes in semantic behavior remains unclear. The additional analyses offer limited support: the transport-selected groups show no greater intervention sensitivity than the last three blocks (Table 20), and the auxiliary controller shows no consistent semantic-retention advantage (Table 26). While these findings do not negate the descriptive contribution, the paper should better establish what new understanding follows from the identified patterns.
-
The main novelty lies in organizing checkpoint differences into a layer-wise evidence analysis that combines representation/readout contrasts, calibrated transport, and behavioral interventions. The underlying technical ingredients are largely established: the fixed-pass decomposition closely follows direct logit attribution, and the transport is regularized linear regression whose summed prediction is equivalent to direct ridge prediction under the same conditions. The contribution therefore appears primarily methodological and empirical, with more limited novelty in the underlying mathematics.
-
The research question is clear, but the motivation for the specific transport analysis and its methodological novelty could be better established. Although the comparisons support a description beyond uniform or independent layer-wise rescaling, it remains unclear which important phenomena this distinction explains beyond existing diagnostic approaches. The fixed-pass decomposition closely follows residual-based logit attribution, while transport is regularized linear regression with structural constraints. Equation (11) explicitly shows that, under matched inputs, loss normalization, and regularization, the summed prediction of full transport is equivalent to direct no-intercept ridge regression. The layer-wise matrix retains information beyond this scalar prediction, but its additional explanatory value remains insufficiently demonstrated, particularly given that the transport-selected layer groups show no greater sensitivity to residual-write rescaling than the last three blocks.
Questions for authors
See weaknesses.
4 comments
The Additional Value of Layer-Resolved Transport
Thank you again for raising the question of what the layer-resolved matrix contributes beyond scalar regression. We agree that the summed prediction alone does not establish that contribution.
The additional object of study is the distribution of evidence changes across layers. Different layerwise changes can produce the same scalar prediction. Our structural comparisons test this distribution directly: Full retains an advantage over Diagonal when both allow layer-specific intercepts, showing that independent layerwise scaling and mean shifts do not capture the checkpoint relation equally well.
The restoration experiment tests a concrete use of these layer-level outputs: selecting intervention locations separately for each question. This differs from ranking fixed layer groups by rescaling sensitivity. The answer-correctness analysis now shows that Full-selected restoration raises average accuracy from 59.09% to 65.15%, whereas last-three restoration gives 57.58% (Appendix D.5, Tables 26–27). These averages cover three separate single-layer restorations per question.
This supports a behavioral use of the layer-resolved predictions beyond scalar fit quality. The additional behavioral advantage of Full over Diagonal remains unresolved; our contribution rests on the source-separated analysis, structural comparisons, and demonstrated restoration use.
Thank you for your rebuttal. The authors’ responses have largely addressed some of my concerns. I will raise my score to 3.
Thank You for the Discussion
We sincerely thank the reviewer for the careful reassessment and constructive feedback throughout the discussion. We are glad that our responses and additional results have addressed the reviewer's concerns.
The Value of Layer-Resolved Transport and Its Empirical Contribution
Thank you for recognizing the source comparisons, group-disjoint evaluation, affine baselines, and real-forward experiments. Your central question is what layer-resolved transport adds beyond existing attribution and regression tools. We have clarified this contribution and added a behavioral experiment testing a concrete use of its layer-level predictions.
1. Why retain the matrix, and where is the contribution? (Weaknesses 2 and 3)
Residual logit attribution and regularized linear regression are established foundations of this work. Our contribution is to compare checkpoints in a common normalization-aware evidence representation, distinguish representation and readout sources, and evaluate the layer-resolved description through structural controls and real-forward interventions. Section 2 and the contribution statement now make this positioning explicit.
The matrix is retained because the question concerns both the total evidence change and its distribution across layers. A scalar predictor cannot distinguish changes with the same total but different layerwise allocations. Section 3.4 formalizes this: when 1⊤K=0, T and T+K give identical summed predictions, although their blockwise predictions can differ. Accurate scalar prediction therefore leaves the layer-resolved description to be tested separately.
We use two tests: reconstruction against matrices with different structural constraints, and whether the layerwise predictions help select intervention locations. The off-diagonal terms describe statistical cross-depth dependence, rather than causal connections between blocks.
2. What do the structural results tell us? (Weaknesses 1 and 3)
First, representation-side drift exceeds readout-side drift in all five settings, with a non-negligible interaction. The example in Section 5.1 illustrates the distinction: the unified trajectory retains a negative margin under Base's readout, while Base's trajectory retains a positive margin under the unified readout. This separates representation and readout changes in a way that an accuracy drop alone cannot.
Second, the transport comparisons test whether simpler descriptions account for the observed changes. Full improves reconstruction over Scalar in all five settings (Table 3). In the four affine controls, the advantage over Diagonal persists when each layer has its own intercept (Appendix C.1, Table 17). Uniform scaling and independent layerwise scaling plus mean shifts therefore do not reach the same reconstruction quality on these evaluation data.
These results turn the broad observation that the model has changed internally into comparisons of evidence sources and layerwise statistical structure. They do not identify the training mechanism that produced those changes.
3. What behavioral evidence has been added? (Weaknesses 1 and 3)
In the original rescaling experiment, the high-score group did not outperform the last three blocks. This remains true after excluding identified links to calibration data (Appendix D.4, Table 24).
The new experiment tests a sample-specific use of the predictions: selecting layers for restoration toward the matched Base write and measuring the resulting complete-margin gain. It uses 88 MME records in 74 groups without identified calibration links. Full and Diagonal each select three layers; we intervene on each layer separately, recompute downstream states and outputs, and average the three margin gains (Appendix D.5, Table 25).Full exceeds last-three selection by +0.420360, with a 95% interval of [0.208057, 0.643320] and Holm-adjusted p=0.000800. Against Diagonal, the difference is +0.065840, with interval [−0.009408, 0.137285] and adjusted p=0.075192.
Thus, sample-specific Full selection has a demonstrated use beyond a fixed last-three rule under this restoration operation. An additional behavioral benefit from cross-depth mixing over Diagonal is not yet established. The experiment is retrospective and label-informed, and matches intervention counts rather than depth or perturbation magnitude.
Restoration gain and rescaling sensitivity are different endpoints, so the new result does not conflict with the earlier last-three comparison. The three-seed OECMT result also remains unchanged: no consistent semantic-retention advantage (Appendix E.3, Table 29). The revised Discussion separates the diagnostic findings and restoration use from the further question of how to turn them into a better training policy.