- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies whether a language model can learn information from deployment-time support and retain the corresponding behavior after the support is removed. The authors propose Support-Derived Behavioral Objective (SDBO), which converts a support statement into a canonical question-answer target and performs four SGD updates on the down-projection matrices of the last eight layers of Llama-3.1-8B-Instruct. Experiments on QAEdit compare SDBO with raw-support training and a four-view objective under local and full-model parameter scopes. The authors report that local SDBO improves behavioral success over raw-support training on Llama-3.1-8B-Instruct, while the effect does not reproduce on Qwen3-8B-Base. Through additional projection, radius-matching, and optimization-path experiments, the paper further argues that post-update behavior depends on how an update is formed rather than only on its final parameter support or magnitude.
Critical strengths
- The authors remove the support after the update and evaluate the resulting model instead of relying only on training loss or teacher-forced likelihood. This provides a more meaningful test of whether the information has actually affected the model's behavior. I also appreciate that the negative result on Qwen3-8B-Base is reported.
Critical weaknesses
-
I do not see a clear methodological contribution in SDBO. The method trains on a single question-answer pair derived from the support and computes the loss only on the answer tokens. The experiments are also conducted in a single-fact editing setting on QAEdit, but SDBO is mainly compared with variants designed by the authors themselves. There is no comparison with existing editing methods such as FT-M, ROME, or MEMIT, which have already been evaluated on QAEdit by Yang et al. (2025). I also find the ablations insufficient. Several design choices are bundled together in SDBO, such as converting the support into a canonical QA format and computing the loss only on the answer. The paper does not separate these choices, so it is hard to tell what actually causes the improvement. The authors' four-view objective O1 performs better, but one of its training prompts is identical to the acquisition probe in every released episode, so this comparison is hard to interpret. I think these comparisons and ablations are more important than the extensive statistical analysis currently presented in the paper. Without them, it is difficult for me to understand what SDBO contributes over existing methods, and this is the main reason for my recommendation.
-
I also find the explanation of the main result unclear. Sections 6 and 7 emphasize the importance of the training scope, but Section 7.3 shows that the learning rate has a very large effect. The SDBO learning rates used for the local and full-model settings differ by roughly 300 times after KL calibration. In the same-learning-rate comparison, there is no observed advantage for the local setting over the full-model setting, but both settings perform poorly. Therefore, I do not think this experiment clearly separates the effect of training scope from the effect of learning rate and the resulting optimization path. Table A8 also makes the interpretation less clear: on the replay prompts, the component of the full-model update outside the late layers retains almost all of the original KL, while the projected late-layer component has almost none. Overall, I found it difficult to determine exactly what these experiments establish about training scope. The large number of labels and newly introduced terms, such as S0, S3, OB, and Full-slow, also makes this analysis unnecessarily difficult to follow.
Questions for authors
-
Appendix C.1 shows that the transform T parses a fixed support template with a predefined relation schema, and all 104 supports in the released data follow this format. What happens when the support does not follow this template? For example, can SDBO handle a differently phrased fact or a task rule of the kind mentioned in the introduction?
-
The paper states that evaluation queries never enter target construction. However, the relation label required by T does not appear to be directly provided by the public QAEdit records. How were these relation labels constructed? Did this process use any information from the evaluation questions?
-
I am unclear about what the wrong-target control in Table 2 establishes. The primary scorer checks whether the correct target value or an alias appears in the generated answer, while the wrong-target condition trains the model on a different value. In the released results, the Llama wrong-target states have zero gain even when the corresponding correct-target SDBO states succeed frequently. How often does a wrong-target model actually generate the wrong value that it was trained on? Without this information, how should the correct-vs-wrong contrast be interpreted as evidence of content specificity?
-
I am also unclear about the false-activation test. In the released episode files, the four same-domain negatives are synthetic prompts such as "The unchanged archive lists item Z2 with value quartz. What value is listed for Z2?", which seem largely unrelated to the edited fact. How does success on these prompts demonstrate that the update remains local? In particular, what happens to other facts about the same subject or other facts involving the same relation?
Thank you for your follow-up experiments. I believe that your method is interesting as it is intuitive and does help training in some setups.
However, my core concern remains that the significance is limited as you did not provide enough evidence or intuition that under what conditions should this method hold. Of course it may not work in every scenerio, but I expect a claim like "SDBO has advantage on some certain domain, this is stable across models, data format and some other unimportant factors". In this case the method can be seen as generally usable. However, in current scenerio we can only conclude "SDBO may be useful on some data and model", which greatly limits the potential application of this method. The gain over vanilla SFT is not so surprising to me as SFT on small, unorganized data automatically causes huge performance degrade, especially given your prompt is carefully tuned but the other baselines are not. If you cannot find a method to deal with this problem, others may easily conclude that your core contribution is a prompt tuning experiment on specific model which is not very interesting.
Concerning the problem on restricting and projection, I can see their mathematical difference, but I wonder if there is a intuitive and general understanding on this, for example what model behaviors do restricting scope during training and full finetuning affect respectively.
Due to above concerns my score remains unchanged.