- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SF-DPO to improve multi-turn robustness by teaching language models when to revise or preserve previous answers. The framework constructs four types of preference pairs targeting recovery and fabricated-evidence resistance, and optimizes them using DPO with chosen-response anchoring. Evaluation covers three benchmarks and 33 model configurations.
Critical strengths
-
The paper distinguishes appropriate answer preservation from resistance to legitimate correction. This provides a more meaningful account of multi-turn robustness than answer consistency alone.
-
Experiments span multiple model families, parameter scales, and reasoning configurations. Training exclusively on MMLU and evaluating transfer to CommonsenseQA and TruthfulQA strengthens the empirical contribution.
Critical weaknesses
-
The main comparisons involve activation steering, decoding interventions, history editing, and test-time refinement. Since SF-DPO uses standard DPO with an existing anchoring mechanism, its main contribution lies in preference-data construction.
-
The evidence-filtering procedure does not establish the validity of the evidence itself. Consequently, a false or logically invalid justification supporting the correct answer could still be labeled valid evidence. This weakens the interpretation that SF-DPO learns evidence validity rather than responsiveness to statements supporting correct answers.
Questions for authors
How does SF-DPO compare with supervised fine-tuning on the same preferred responses under matched training data and compute budgets?
What explains the relatively modest fabricated-evidence resistance? Why is the DPOP anchor preferred over the NLL anchor despite its lower FER in Table 4?
4 comments
R3 [Q1]: We conducted an additional experiment comparing SF-DPO with a matched SFT baseline using the same state-forked training instances and preferred responses under matched training data and compute budgets. The results are macro-averaged across four representative models and three benchmarks, as summarized in Rebuttal Table 2 [https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ].
Compared with matched SFT, SF-DPO improves average accuracy across adversarial turns from 0.631 to 0.704 and retention from 0.962 to 0.994, while reducing causal error propagation (ATE$_1$) from 0.038 to 0.013. SF-DPO also achieves higher valid-evidence updating (VEU: 0.898 vs. 0.685), while fabricated-evidence resistance is also higher (FER: 0.477 vs. 0.388).
These results suggest that SF-DPO provides benefits beyond supervised learning from the same preferred responses, particularly in multi-turn robustness, evidence selectivity, and causal error control. We have incorporated this matched SFT comparison into the revised manuscript.
R4 [Q2]: The relatively modest fabricated-evidence resistance (FER) reflects an empirical trade-off between valid-evidence updating (VEU) and fabricated-evidence resistance, as illustrated by our preference-arm ablations in Table 3 of the original manuscript. For example, removing Arm C yields near-perfect FER (0.996) but almost eliminates VEU (0.016), whereas removing Arm D increases VEU to 0.994 but reduces FER to 0.063. These results demonstrate that maximizing resistance to fabricated evidence alone can lead to indiscriminate answer preservation, undermining the ability to update appropriately in response to valid evidence.
This empirical tension is also observed in our additional comparisons with other post-training methods [Rebuttal Table 2: https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ]. ORPO achieves higher VEU than SF-DPO (0.908 vs. 0.898) but lower FER (0.381 vs. 0.477), whereas SimPO and KTO achieve higher FER (0.697 and 0.645, respectively) at the expense of lower VEU (0.705 and 0.829). These results highlight the importance of evaluating both dimensions of evidence selectivity, rather than optimizing either metric in isolation.
This consideration also motivates our choice of the DPOP-inspired anchor over the NLL anchor. Although the NLL anchor achieves higher FER (0.639 vs. 0.477), the DPOP-inspired anchor provides stronger VEU (0.898 vs. 0.741), higher retention (0.994 vs. 0.986), lower causal error propagation (ATE$_1$: 0.013 vs. 0.017), and fewer generated tokens per turn (10.2 vs. 12.2). It also achieves higher average accuracy across adversarial turns (0.704 vs. 0.696). We therefore selected the DPOP-inspired anchor for its more favorable overall balance of multi-turn robustness, evidence selectivity, causal error control, and inference efficiency, while acknowledging the stronger FER achieved by the NLL anchor.
We have further clarified this empirical trade-off and the rationale for our anchoring choice in the revised manuscript.
Reference
[1] Fleiss J L. Measuring nominal scale agreement among many raters[J]. Psychological bulletin, 1971, 76(5): 378.
After reviewing the other reviewers’ comments, which were positive, and noting that the authors’ rebuttal addressed my second concern with supporting references, I have decided to raise my score to 2. However, since my remaining concern is about the claimed contribution (benchmark rather than method), I cannot raise it to 3.
Thank you for reconsidering your assessment and for updating your rating. We sincerely appreciate your thoughtful feedback and understand your remaining concern regarding the methodological contribution. Your comments have helped us better clarify and position our contributions in the revised manuscript.
Thank you for taking the time to review our manuscript carefully and for your constructive feedback. We appreciate your recognition of the importance of distinguishing appropriate answer preservation from legitimate correction, as well as the breadth of our evaluation across models and benchmarks. We have conducted additional experiments and analyses to address your concerns, as detailed below.
R1 [W1]: Our main contributions lie in (i) formulating multi-turn robustness as a state-dependent selective updating problem that jointly considers cascade control and evidence selectivity, and (ii) introducing a state-forked preference construction framework that translates these complementary behaviors into learnable supervision. We agree that SF-DPO builds upon existing preference optimization objectives and chosen-response anchoring, rather than introducing a new optimization algorithm. However, our state-forked preference construction is not inherently tied to DPO and can be combined with alternative post-training objectives, making it a flexible supervision framework rather than a method restricted to a particular optimization objective.
To examine this flexibility empirically, and following the suggestion of Reviewer ZahW, we conducted additional experiments comparing SF-DPO with matched SFT, ORPO, SimPO, and KTO using the same state-forked supervision. As shown in Rebuttal Table 2 [https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ], SF-DPO achieves the highest average accuracy across adversarial turns (Acc1:8 = 0.704) and retention (Ret8 = 0.994), as well as the lowest causal error propagation (ATE$_1$ = 0.013). The alternative post-training methods also achieve improvements over the base models, suggesting that the applicability of state-forked supervision is not limited to DPO.
Together, these results highlight the value of our state-dependent formulation and demonstrate that state-forked supervision can be used across different post-training objectives. We have clarified these contributions and further motivated our choice of DPO in the revised manuscript.
R2 [W2]: Our evidence filtering procedure uses two complementary criteria: FACT assesses whether the target answer is factually correct, while COMMIT evaluates whether the accompanying justification explicitly supports that answer. We acknowledge that these criteria do not independently verify the factual or logical validity of every claim within the justification, as comprehensively assessing the factual accuracy and logical soundness of free-form explanations remains a nontrivial challenge. Independent human evaluation offers one possible approach to addressing this limitation. We therefore appreciate your concern, as well as the related suggestions from Reviewers ZahW and ERJT regarding human validation and independently verified evidence. Motivated by these suggestions, we conducted two additional analyses: (i) an evaluation of evidence selectivity on a human-verified subset and (ii) a human validation study of the FACT--COMMIT filtering procedure.
(i) Evidence selectivity on a human-verified subset. We evaluated 100 evidence instances, comprising 50 valid-evidence and 50 fabricated-evidence instances, with labels independently verified by majority vote from three human annotators. Across four representative models and three benchmarks, the resulting VEU and FER are 0.885 and 0.495, respectively, compared with 0.898 and 0.477 in the original evaluation. The small absolute differences (0.013 for VEU and 0.018 for FER) suggest that our evidence-selectivity findings remain broadly consistent on the human-verified subset.
(ii) Human validation of FACT--COMMIT filtering. We further assessed the reliability of our filtering procedure using 100 evidence instances balanced across the four FACT--COMMIT label combinations (25 per combination). Three human annotators independently evaluated each instance using the same FACT and COMMIT criteria. We measured inter-annotator agreement using Fleiss' κ [1] and agreement between Gemma-4-31B and the human majority vote using Cohen's κ. As shown in Rebuttal Table 1 [https://anonymous.4open.science/r/SFDPO-E9CE/Table1.png ], Gemma-4-31B achieves Cohen's κ values of 0.940 for FACT and 0.760 for COMMIT, corresponding to raw agreement rates of 97.0% and 88.0%, respectively. These results provide additional support for the reliability of FACT--COMMIT filtering under its defined criteria, although they do not establish the factual or logical correctness of every supporting explanation.
We have incorporated both analyses into the revised manuscript, clarified the scope of FACT--COMMIT filtering and explicitly discussed this limitation. More comprehensive human verification of explanation-level factual and logical correctness remains an important yet nontrivial direction for future work.