acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Change Your Mind: SF-DPO for Multi-Turn Language Models

Abstract

Large language models (LLMs) can become trapped by their own mistakes in multi-turn interactions: once an erroneous answer enters the conversation history, it may increase the likelihood of subsequent errors. We refer to this history-dependent propagation as *error self-excitation*. However, suppressing such error propagation alone is insufficient for a robust multi-turn interaction, since models should both recover from erroneous answers while preserving correct ones under adversarial pressure and condition the revision on the validity of newly supplied information. We therefore formulate *multi-turn robustness* as a *state-dependent update problem* and introduce StateFork Direct Preference Optimization (SF-DPO), a lightweight post-training framework that learns when prior answers should be revised and when they should be preserved. SF-DPO holds the conversation history and current user input fixed and forks this state into preferred and rejected continuations through four complementary preference arms targeting *recovery*, *stability*, *valid-evidence updating*, and *fabricated-evidence resistance*. Finally, these state-forked preferences are optimized with DPO and chosen-response anchoring. Across three benchmarks and 33 model configurations from seven model groups spanning approximately 0.5B to 32B parameters, SF-DPO substantially improves *multi-turn robustness* across both *non-reasoning* and *inference-time reasoning* models while reducing *causal error propagation*, maintaining *evidence selectivity*, and requiring low *inference cost*.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Turn Robustness, Error Self-Excitation, Preference Optimization, Causal Intervention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KcbgOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SF-DPO to improve multi-turn robustness by teaching language models when to revise or preserve previous answers. The framework constructs four types of preference pairs targeting recovery and fabricated-evidence resistance, and optimizes them using DPO with chosen-response anchoring. Evaluation covers three benchmarks and 33 model configurations.

Critical strengths
  1. The paper distinguishes appropriate answer preservation from resistance to legitimate correction. This provides a more meaningful account of multi-turn robustness than answer consistency alone.

  2. Experiments span multiple model families, parameter scales, and reasoning configurations. Training exclusively on MMLU and evaluating transfer to CommonsenseQA and TruthfulQA strengthens the empirical contribution.

Critical weaknesses
  1. The main comparisons involve activation steering, decoding interventions, history editing, and test-time refinement. Since SF-DPO uses standard DPO with an existing anchoring mechanism, its main contribution lies in preference-data construction.

  2. The evidence-filtering procedure does not establish the validity of the evidence itself. Consequently, a false or logically invalid justification supporting the correct answer could still be labeled valid evidence. This weakens the interpretation that SF-DPO learns evidence validity rather than responsiveness to statements supporting correct answers.

Questions for authors

How does SF-DPO compare with supervised fine-tuning on the same preferred responses under matched training data and compute budgets?

What explains the relatively modest fabricated-evidence resistance? Why is the DPOP anchor preferred over the NLL anchor despite its lower FER in Table 4?

4 comments

Thank you for taking the time to review our manuscript carefully and for your constructive feedback. We appreciate your recognition of the importance of distinguishing appropriate answer preservation from legitimate correction, as well as the breadth of our evaluation across models and benchmarks. We have conducted additional experiments and analyses to address your concerns, as detailed below.

R1 [W1]: Our main contributions lie in (i) formulating multi-turn robustness as a state-dependent selective updating problem that jointly considers cascade control and evidence selectivity, and (ii) introducing a state-forked preference construction framework that translates these complementary behaviors into learnable supervision. We agree that SF-DPO builds upon existing preference optimization objectives and chosen-response anchoring, rather than introducing a new optimization algorithm. However, our state-forked preference construction is not inherently tied to DPO and can be combined with alternative post-training objectives, making it a flexible supervision framework rather than a method restricted to a particular optimization objective.

To examine this flexibility empirically, and following the suggestion of Reviewer ZahW, we conducted additional experiments comparing SF-DPO with matched SFT, ORPO, SimPO, and KTO using the same state-forked supervision. As shown in Rebuttal Table 2 [https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ], SF-DPO achieves the highest average accuracy across adversarial turns (Acc‾1:8\overline{\text{Acc}}_{1:8} = 0.704) and retention (Ret8\text{Ret}_8 = 0.994), as well as the lowest causal error propagation (ATE$_1$ = 0.013). The alternative post-training methods also achieve improvements over the base models, suggesting that the applicability of state-forked supervision is not limited to DPO.

Together, these results highlight the value of our state-dependent formulation and demonstrate that state-forked supervision can be used across different post-training objectives. We have clarified these contributions and further motivated our choice of DPO in the revised manuscript.

R2 [W2]: Our evidence filtering procedure uses two complementary criteria: FACT assesses whether the target answer is factually correct, while COMMIT evaluates whether the accompanying justification explicitly supports that answer. We acknowledge that these criteria do not independently verify the factual or logical validity of every claim within the justification, as comprehensively assessing the factual accuracy and logical soundness of free-form explanations remains a nontrivial challenge. Independent human evaluation offers one possible approach to addressing this limitation. We therefore appreciate your concern, as well as the related suggestions from Reviewers ZahW and ERJT regarding human validation and independently verified evidence. Motivated by these suggestions, we conducted two additional analyses: (i) an evaluation of evidence selectivity on a human-verified subset and (ii) a human validation study of the FACT--COMMIT filtering procedure.

(i) Evidence selectivity on a human-verified subset. We evaluated 100 evidence instances, comprising 50 valid-evidence and 50 fabricated-evidence instances, with labels independently verified by majority vote from three human annotators. Across four representative models and three benchmarks, the resulting VEU and FER are 0.885 and 0.495, respectively, compared with 0.898 and 0.477 in the original evaluation. The small absolute differences (0.013 for VEU and 0.018 for FER) suggest that our evidence-selectivity findings remain broadly consistent on the human-verified subset.

(ii) Human validation of FACT--COMMIT filtering. We further assessed the reliability of our filtering procedure using 100 evidence instances balanced across the four FACT--COMMIT label combinations (25 per combination). Three human annotators independently evaluated each instance using the same FACT and COMMIT criteria. We measured inter-annotator agreement using Fleiss' κ\kappa [1] and agreement between Gemma-4-31B and the human majority vote using Cohen's κ\kappa. As shown in Rebuttal Table 1 [https://anonymous.4open.science/r/SFDPO-E9CE/Table1.png ], Gemma-4-31B achieves Cohen's κ\kappa values of 0.940 for FACT and 0.760 for COMMIT, corresponding to raw agreement rates of 97.0% and 88.0%, respectively. These results provide additional support for the reliability of FACT--COMMIT filtering under its defined criteria, although they do not establish the factual or logical correctness of every supporting explanation.

We have incorporated both analyses into the revised manuscript, clarified the scope of FACT--COMMIT filtering and explicitly discussed this limitation. More comprehensive human verification of explanation-level factual and logical correctness remains an important yet nontrivial direction for future work.

R3 [Q1]: We conducted an additional experiment comparing SF-DPO with a matched SFT baseline using the same state-forked training instances and preferred responses under matched training data and compute budgets. The results are macro-averaged across four representative models and three benchmarks, as summarized in Rebuttal Table 2 [https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ].

Compared with matched SFT, SF-DPO improves average accuracy across adversarial turns from 0.631 to 0.704 and retention from 0.962 to 0.994, while reducing causal error propagation (ATE$_1$) from 0.038 to 0.013. SF-DPO also achieves higher valid-evidence updating (VEU: 0.898 vs. 0.685), while fabricated-evidence resistance is also higher (FER: 0.477 vs. 0.388).

These results suggest that SF-DPO provides benefits beyond supervised learning from the same preferred responses, particularly in multi-turn robustness, evidence selectivity, and causal error control. We have incorporated this matched SFT comparison into the revised manuscript.

R4 [Q2]: The relatively modest fabricated-evidence resistance (FER) reflects an empirical trade-off between valid-evidence updating (VEU) and fabricated-evidence resistance, as illustrated by our preference-arm ablations in Table 3 of the original manuscript. For example, removing Arm C yields near-perfect FER (0.996) but almost eliminates VEU (0.016), whereas removing Arm D increases VEU to 0.994 but reduces FER to 0.063. These results demonstrate that maximizing resistance to fabricated evidence alone can lead to indiscriminate answer preservation, undermining the ability to update appropriately in response to valid evidence.

This empirical tension is also observed in our additional comparisons with other post-training methods [Rebuttal Table 2: https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ]. ORPO achieves higher VEU than SF-DPO (0.908 vs. 0.898) but lower FER (0.381 vs. 0.477), whereas SimPO and KTO achieve higher FER (0.697 and 0.645, respectively) at the expense of lower VEU (0.705 and 0.829). These results highlight the importance of evaluating both dimensions of evidence selectivity, rather than optimizing either metric in isolation.

This consideration also motivates our choice of the DPOP-inspired anchor over the NLL anchor. Although the NLL anchor achieves higher FER (0.639 vs. 0.477), the DPOP-inspired anchor provides stronger VEU (0.898 vs. 0.741), higher retention (0.994 vs. 0.986), lower causal error propagation (ATE$_1$: 0.013 vs. 0.017), and fewer generated tokens per turn (10.2 vs. 12.2). It also achieves higher average accuracy across adversarial turns (0.704 vs. 0.696). We therefore selected the DPOP-inspired anchor for its more favorable overall balance of multi-turn robustness, evidence selectivity, causal error control, and inference efficiency, while acknowledging the stronger FER achieved by the NLL anchor.

We have further clarified this empirical trade-off and the rationale for our anchoring choice in the revised manuscript.

Reference

[1] Fleiss J L. Measuring nominal scale agreement among many raters[J]. Psychological bulletin, 1971, 76(5): 378.

After reviewing the other reviewers’ comments, which were positive, and noting that the authors’ rebuttal addressed my second concern with supporting references, I have decided to raise my score to 2. However, since my remaining concern is about the claimed contribution (benchmark rather than method), I cannot raise it to 3.

Thank you for reconsidering your assessment and for updating your rating. We sincerely appreciate your thoughtful feedback and understand your remaining concern regarding the methodological contribution. Your comments have helped us better clarify and position our contributions in the revised manuscript.

Reviewer_ERJTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SF-DPO to help language models decide when to revise an answer and when to retain it during multi-turn conversations. It is trained on four types of preference pairs covering error recovery, answer stability, valid corrections, and misleading evidence. Experiments across 33 model configurations on MMLU, CommonsenseQA, and TruthfulQA show improved answer retention and reduced error propagation, with fewer generated tokens.

Critical strengths

The paper makes a useful distinction between being consistent and being correct. The ablations show that a model can retain its answers almost perfectly while becoming unwilling to accept valid corrections.

The evaluation covers many model families and sizes.

Critical weaknesses

The evidence filter checks whether the proposed answer is correct but does not verify the factual or logical validity of its supporting explanation.

The comparisons do not clearly isolate the benefit of SF-DPO over simpler post-training methods. A matched SFT baseline using the same chosen responses is missing, and it is unclear whether the gains hold on unseen follow-up templates.

Questions for authors

Have you independently checked the factual and logical validity of the retained explanations, rather than only the correctness of their target answers? Do the results hold on an independently verified subset?

How does SF-DPO compare with SFT on the same chosen responses under a matched training budget?

2 comments

Thank you for your careful review, constructive feedback, and positive assessment of our work. We appreciate the time and effort you devoted to evaluating our manuscript. We have conducted additional experiments to address your questions and concerns, as detailed below.

R1 [W1&Q1]: Our evidence filtering procedure uses two complementary LLM-as-a-judge criteria: FACT assesses whether the target answer is factually correct, while COMMIT evaluates whether the accompanying justification explicitly supports that answer. We acknowledge that these criteria do not independently verify the factual and logical validity of every claim within the justification, as comprehensively validating the factual accuracy and logical soundness of free-form explanations remains a nontrivial challenge. We therefore particularly appreciate your suggestion to evaluate our findings on an independently verified subset, as well as the related suggestions from Reviewers ZahW and Kcbg regarding human validation. Accordingly, we conducted two additional analyses, focusing first on the independently verified subset and then on the reliability of FACT--COMMIT filtering.

(i) Evidence selectivity on a human-verified subset. We evaluated 100 evidence instances, comprising 50 valid-evidence and 50 fabricated-evidence instances, with labels independently verified by majority vote from three human annotators. Across four representative models and three benchmarks, the resulting VEU and FER are 0.885 and 0.495, respectively, compared with 0.898 and 0.477 in the original evaluation. The small absolute differences (0.013 for VEU and 0.018 for FER) suggest that our evidence-selectivity findings remain broadly consistent on the human-verified subset.

(ii) Human validation of FACT--COMMIT filtering. To further assess the reliability of our filtering procedure, we sampled 100 evidence instances, balanced across the four FACT--COMMIT label combinations (25 per combination). Three human annotators independently evaluated each instance using the same FACT and COMMIT criteria. We assessed inter-annotator agreement using Fleiss' κ\kappa [1] and agreement between Gemma-4-31B and the human majority vote using Cohen's κ\kappa. As shown in Rebuttal Table 1 [https://anonymous.4open.science/r/SFDPO-E9CE/Table1.png ], Gemma-4-31B achieves Cohen's κ\kappa values of 0.940 for FACT and 0.760 for COMMIT, corresponding to raw agreement rates of 97.0% and 88.0%, respectively. These results provide additional support for the reliability of FACT--COMMIT annotations under their defined criteria.

We have clarified the scope of FACT--COMMIT filtering, incorporated both validation analyses, and explicitly discussed this limitation in the revised manuscript.

Reference

[1] Fleiss J L. Measuring nominal scale agreement among many raters[J]. Psychological bulletin, 1971, 76(5): 378.

R2 [W2&Q2]: We conducted two additional experiments to examine the benefits of SF-DPO over simpler post-training methods and its generalization to unseen adversarial follow-up templates.

(i) Comparison with SFT and alternative post-training methods. We compared SF-DPO with a matched SFT baseline using the same state-forked training instances and preferred responses under matched training budgets. Following the suggestion from Reviewer ZahW, we also extended the comparison to ORPO, SimPO, and KTO using the same state-forked supervision. As shown in Rebuttal Table 2 [https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ], the results are macro-averaged across four representative models and three benchmarks. Compared with matched SFT, SF-DPO improves average accuracy across adversarial turns from 0.631 to 0.704 and retention from 0.962 to 0.994, while reducing causal error propagation from 0.038 to 0.013. These results suggest that SF-DPO provides benefits beyond supervised learning from the same preferred responses. Among all the compared methods, SF-DPO also achieves the highest average accuracy and retention, as well as the lowest ATE$_1$, while ORPO achieves slightly higher valid-evidence updating (VEU) and SimPO and KTO exhibit stronger fabricated-evidence resistance (FER).

(ii) Generalization to unseen adversarial follow-up templates. We further evaluated SF-DPO on a held-out follow-up bank comprising eight persuasion strategies adapted from Zeng et al. (2024) [2], with five templates per strategy. The held-out bank is distinct from the original follow-up templates adapted from Li et al. (2025), with no overlapping word 4-grams between the two banks. As shown in Rebuttal Table 4 [https://anonymous.4open.science/r/SFDPO-E9CE/Table4.png ], SF-DPO achieves substantially higher retention than the base models on unseen templates (0.988 vs. 0.450), alongside higher average accuracy across adversarial turns (0.677 vs. 0.409). Its retention on unseen templates is also close to that on seen templates (0.988 vs. 0.994), suggesting that the robustness improvements extend beyond the original follow-up templates. The complete held-out prompt templates are provided in Rebuttal Table 5 [https://anonymous.4open.science/r/SFDPO-E9CE/Table5-1.png and https://anonymous.4open.science/r/SFDPO-E9CE/Table5-2.png ].

Together, these experiments provide further evidence of SF-DPO's multi-turn robustness and generalization to unseen adversarial follow-ups, while demonstrating the applicability of state-forked supervision across different post-training methods. We have incorporated these findings into the revised manuscript.

Reference

[2] Zeng Y, Lin H, Zhang J, et al. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 14322-14350.

Reviewer_ZahWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses an important LLM problem, occurring in case of multi-turn interactions where an initial error in one turn can get amplified in subsequent turns. This problem, named as 'error-self excitation', has been formulated as a state-dependent update problem, and led to a new light-weight post-training framework StateFork Direct Optimization (SF-DPO), which constructs state-forked preferences for recovery, stability, valid-evidence updating, and fabricated-evidence resistance, optimized using DPO and chosen-response anchoring. SF-DPO aims to promote recovery from erroneous answers while preserving correct answers under adversarial follow-ups. Besides, for each multiple-choice questions and target answer, a FACT-COMMIT filtering rule is used to audit the factual status of the target and determine whether there is sufficient support for the target, yielding YES-NO judgment for each task. For this purpose, for each of the FACT and COMMIT tasks, an LLM-as-a-judge strategy is employed
SF-DPO has been evaluated across three benchmarks (MMLU, CommonsenseQA, TruthfulQA), seven model groups with various scales and inference regimes (Gema 2, Gemma 3, Llama 3.x, Qwen2.5, Qwen3 in no-thinking and thinking modes, DeepSeek-R1) ), and 33 model configurations from 0.5B to 32B parameters to assess whether SF-DPO generalize across model families and scale where Qwen3 thinking mode and DeepSeek-R1 serve as inference-time reasoning comparators. Compared against four mitigation baselines (Contrastive Activation Addition, Decoding by Contrasting Layers, Amnesia baseline, Self-Refine) on four complementary dimensions: multi-turn robustness, evidence selectivity, causal error propagation, and inference efficiency, SF-DPO is shown to improve multi-turn robustness while reducing causal error propagation with lower inference cost.

Critical strengths
  • High quality standard: The paper is in overall well written, technically sound, with rigorous problem formulating of multi-turn interaction as state dependent update problem, yielding a novel SateFork-Direct Optimization (SF-DIPO) post-training framework that involves an LLM-as-a-judge strategy to decouple factual auditing from rhetorical commitment in FACT-COMMIT filtering rule, showing a clear novelty in the field.
  • The proposed model SF-DIPO has been extensively tested across three benchmarks, with 7 model families, 33 configurations, cross dataset transfer, together with a comparison with four baseline models and four performance metrics in addition to extensive ablation study as documented in appendix section.
Critical weaknesses

The main weakness of the paper is related to the use of LLM-as-a-judge for factual correctness in FACT-COMMIT filtering rule, which can introduce residual judge hallucination into the training set, despite high Cohen's score (0.83 for FACT and 0.992 for COMMIT) for selecting Gemma-4-31B. A small scale investigation with human validation would highly elucidate this risk.

Questions for authors
  • It will be nice more motivation grounds on why DPO was selected against other alternatives such as ORPO, SimPO, KTO, which have the advantage of not requiring a reference model.
  • It is unclear how the number of follow-up does impact the performance. The current implementation uses 8 distinct adversarial follow-ups, which can also create important latency if not supported by important performance gains. It will be nice some discussion or small light experiments in this regard.
  • The lack of human annotation for LLMs-as-a-judge is weakening the overall quality. It would be interesting to perform a small test on a small sample to test the compatibility of the LLM-as-a-judge in FACT-COMMIT rule with human judgment by using human annotators and computing the Kappa score. This can also be subject to future work
4 comments

Thank you for taking the time to review our manuscript carefully and for your constructive feedback and positive assessment. We appreciate your thoughtful comments and suggestions, which have helped us further strengthen our work. We have conducted additional experiments to address your questions and concerns, as detailed below.

R1 [W1&Q3]: We conducted an additional human validation study to investigate potential errors introduced by LLM-based judging and to assess the agreement between our selected LLM judge, Gemma-4-31B, and human judgments on the FACT--COMMIT filtering criteria. Specifically, we sampled 100 evidence instances, with 25 instances from each of the four FACT--COMMIT label combinations (YES/YES, YES/NO, NO/YES, and NO/NO). Three human annotators independently evaluated each instance for FACT (whether the target answer is factually correct) and COMMIT (whether the justification explicitly supports the target answer). We evaluated annotation reliability using five metrics: Fleiss' κ\kappa [1] measures agreement among the three human annotators; Cohen's κ\kappa measures agreement between Gemma-4-31B and the human majority vote; Agreement reports the percentage of matching labels; Precision (YES) and Recall (YES) measure the precision and recall of Gemma's positive judgments against the human majority vote, respectively.

Rebuttal Table 1 [https://anonymous.4open.science/r/SFDPO-E9CE/Table1.png ] shows strong agreement between Gemma-4-31B and human judgments, with Cohen's κ\kappa values of 0.940 for FACT and 0.760 for COMMIT, corresponding to 97.0% and 88.0% raw agreement, respectively. Notably, Gemma achieves 96.0% FACT precision and 90.0% COMMIT precision on positive judgments, providing additional evidence that the judge's accepted labels are largely consistent with human assessments. The lower agreement on COMMIT, both among human annotators and between Gemma and humans, suggests greater ambiguity in determining whether a justification sufficiently supports a target answer. We acknowledge that this study is still limited by its relatively small sample size, the number of human annotators, and the inherent subjectivity of evaluating evidentiary support. Nevertheless, these findings provide additional empirical support for using Gemma-4-31B in FACT--COMMIT filtering, while residual judgment errors cannot be entirely ruled out.

We have incorporated this human validation study into the revised manuscript, discussed the limitations of human annotation and LLM-as-a-judge approaches, and identified larger-scale human validation as an important direction for future work.

R2 [Q1]: We initially selected DPO because it is a well-established pairwise preference optimization method that naturally accommodates our state-forked preference pairs and integrates with our DPOP-inspired chosen-response anchor. Importantly, our state-forked preference construction is not inherently tied to DPO, and we appreciate your suggestion to consider alternative preference optimization methods. Following your suggestion, we conducted additional experiments comparing SF-DPO with ORPO, SimPO, and KTO using the same state-forked supervision. We also included a matched SFT baseline, as suggested by Reviewers ERJT and Kcbg, to further isolate the benefits of preference optimization. The results are macro-averaged across four representative models and three benchmarks, as summarized in Rebuttal Table 2 [https://anonymous.4open.science/r/SFDPO-E9CE/Table2.png ].

SF-DPO achieves the highest average accuracy across adversarial turns (Acc‾1:8\overline{\text{Acc}}_{1:8} = 0.704), the highest retention (Ret8\text{Ret}_8 = 0.994), and the lowest causal error propagation (ATE1\text{ATE}_1 = 0.013), demonstrating strong multi-turn robustness. The comparison also reveals complementary strengths across optimization objectives: ORPO achieves slightly higher VEU (0.908 vs. 0.898), whereas SimPO and KTO exhibit stronger fabricated-evidence resistance. Overall, these results support our choice of DPO with DPOP-inspired chosen-response anchoring for its favorable balance of multi-turn robustness, causal error control, and inference efficiency, while acknowledging the complementary advantages of alternative objectives.

We have further clarified the motivation for choosing DPO, incorporated these additional comparisons into the Experiments section, and expanded the discussion of alternative preference optimization methods in the revised manuscript.

Reference

[1] Fleiss J L. Measuring nominal scale agreement among many raters[J]. Psychological bulletin, 1971, 76(5): 378.

R3 [Q2]: We have already investigated the impact of adversarial follow-up count through the turn-wise retention trajectories presented in Figure 5 and Appendix C.3. However, we recognize that a more comprehensive summary across different follow-up counts would make these results clearer. Therefore, we provide an additional analysis that covers all 33 model configurations and three benchmarks, as summarized in Rebuttal Table 3 [https://anonymous.4open.science/r/SFDPO-E9CE/Table3.png ].

The results show that the retention improvement of SF-DPO over the base models increases from 0.106 after the first follow-up to 0.371 after eight follow-ups. This growing advantage highlights the importance of evaluating robustness over longer adversarial interactions, where the cumulative degradation of base models becomes more pronounced.

Importantly, the eight adversarial follow-ups are part of our evaluation protocol, adapted from Li et al. (2025), rather than additional inference steps required by SF-DPO. The protocol is designed to systematically evaluate model robustness under diverse and repeated adversarial pressure. At inference time, SF-DPO generates responses directly without requiring additional follow-ups or iterative refinement, and thus incurs no additional inference latency from the eight-turn evaluation protocol.

We have incorporated this expanded analysis and clarified the distinction between the evaluation protocol and inference-time requirements in the revised manuscript.

Reviewer_ZahWOpenReview ↗

Rebuttal

We thanks the authors for addressing the weaknesses and rebuttal request, which I am fully satisfied. It will be nice to add more information about the background of the annotators used in the additional evaluation. My scoring is already on a positive note, so I maintain my score

Thank you for your positive assessment and for confirming that our rebuttal has addressed your concerns. We sincerely appreciate your constructive feedback and continued support. We also appreciate your suggestion regarding the annotators' backgrounds and will include additional details in the final version of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.