- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a theoretically-motivated statistical measure for whether transformer representations have changed over training, and shows that it shows a sensitive measure that detects differences between training corpora, and scales appropriately with the magnitude of those differences.
Critical strengths
- The proposed measure makes sense
- The empirical results support the measure detecting what it is intended to
Critical weaknesses
-
My core concern about this work is that it is answering a question that it is unclear if anybody has asked (or wants to). In particular, we are generally concerned about similarities and differences between models for a practical reason — e.g. that we want to know if a model meaningfully improves on another in terms of its accuracy, or if training has had some effect. Those questions seem to be better addressed through explicit behavioral measurements on those domains (or at least, this paper has not validated that the measure tells us anything useful for these kinds of differences). It may often be the case that a detectable difference in representations has no effect on the behaviors we actually care about. The paper does not make a strong case that this measure cleanly detects something we actually care about.
-
The experiments focus on the setting where the probes match the domain(s) that the data changed on (e.g., swapping tinystories for wiki data, and calcuating the metric differences using sequences from both). I struggle to imagine a practical use case, where we would assume we already know which parts of the distribution will have changed (e.g., which data has been swapped out) but this measure would still be useful. It would be thus a much stronger empirical validation to show that estimating the metrics on data from a third non-overlapping domain (e.g., math text, or code, or whatever) would still be able to detect the training differences.
Questions for authors
- Why should we care about this?
- Can you demonstrate that it is selectively useful for some practical use case (i.e., it predicts some downstream measure we actually care about)?
- Can you show that the measure is still sensitive even if you estimate the changes on a different distribution than where the data changed?
3 comments
Thanks to the authors for this response. These additional experiments have mildly improved my impression of the paper, but I think there are still serious issues that would benefit from a more thorough revision and resubmission to a future conference:
i. On layer selection for LoRA, if I understand correctly the full LoRA strictly outperforms your field-based selection on both new task performance (B) and old task retention (A), as shown on task set 1 (since you didn't eval the full lora on 2 and 3)? I don't think you can claim that field-based selection helps if it performs strictly worse than the full LoRA (I guess it uses fewer parameters, but if you want to make that comparison you should compare to stronger baselines like training the lora using only that many parameters / layers, or stronger selection methods like compute matched evals of different layers, etc.)
ii. These experiments are a bit more satisfying, but your response to me here (that evaluation domain doesn't matter) seems to contradict your discussion with the other reviewer above on the two orders of magnitude claim and the domains where it applies? Indeed, the magnitude of the effects on these domains seems much smaller, consistent with the large effects described originally being largely from testing on the changed domain.
Furthermore, these new experiments, as with others in the paper, would benefit from running multiple seeds (e.g., do multiple full loras and multiple selections so we can evaluate the significance of differences w.r.t. the variation within each condition).
We thank the reviewer for the additional assessment and constructive suggestions. We appreciate that the new experiments have improved your assessment, and we hope the following clarifications help address the remaining concerns. We are also conducting additional replications with different random seeds and will report the results if available before the discussion period ends.
(i)
We agree that full LoRA achieves better performance than field-guided adaptation in Task Set 1. However, our objective is not to demonstrate superiority over full LoRA in absolute predictive performance, but to investigate whether the proposed metric field provides useful information for selecting where to adapt under a limited parameter budget.
Importantly, the layer-only baseline already provides a comparison similar to your suggestion. Both the "Layer-only" and "Field Rule" approaches use the same number of trainable parameters. The distinction is that the layer-only baseline applies adaptation uniformly across token positions in the selected layers, whereas the field-guided approach uses the metric field to identify specific layer–token positions for adaptation. The substantial improvements across the three task sets therefore provide evidence that the metric field offers useful information. It can improve adaptation under an identical trainable-parameter budget.
(ii)
We apologize that our previous response did not clearly distinguish between the sensitivity of the measure and the magnitude of the resulting statistic.
Our claim is not that the magnitude of the metric-field difference is invariant across evaluation domains. Rather, the measure remains sensitive to changes in model representations when evaluated on prompts from domains not used for continuation training. While the magnitude varies across evaluation domains, the qualitative conclusion regarding the detection of representational differences remains unchanged in the evaluated settings.
To avoid overstating the generality of our findings, we have already removed the unqualified "two orders of magnitude" statement from the abstract and clarified the scope of the corresponding results. The additional experiments are intended to establish robustness of detection, rather than invariance of effect magnitude, across evaluation domains.
We hope these clarifications better explain the intended contributions
We thank the reviewer for the careful reading and thoughtful questions. We address the critical concerns as well as other questions in two parts: (i) the practical application of the metric field and (ii) the robustness of the metric field on unseen domains.
i
We agree that we did not make a strong case that this measure tells us anything useful for these kinds of differences. To address this critical weakness, we added an application where the metric field difference obtained during training is used to guide model adaptation. This allows us to use the metric to select a subset of parameters to adapt rather than adapting all parameters. One such application is choosing where to adapt a model under a limited parameter budget while retaining performance on an existing task.
The new experiment tests whether the field helps select an effective adaptation (a patch over layer and token position) under a fixed parameter budget. Both the field-guided and layer-only LoReFT rules use the same amount of trainable parameters. Their new-task accuracies are 81.80% versus 56.80%, 83.08% versus 53.00%, and 79.96% versus 29.36%, yielding gains of 25.00, 30.08, and 50.60 percentage points. The results are put in Table 4 of section 5.4 of the revised manuscript.
ii.
The additional experiment keeps continuation training on TinyStories and WikiText unchanged, but evaluates the resulting checkpoints on code and mathematics prompts. The metric field-based measure is still sensitive to the change in the training data on the unseen domains. The results are put in Table 2 of the revised manuscript. The measures behave similarly to the original results, regardless of the evaluation domains. This addresses the concern that the original results might depend on using prompts from the two training domains.