acceptodds
Withdrawn from ICLR 2027

DARS: Dynamic Alignment Reward Steering for Robust Adaptive Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

Adaptive reasoning models allocate inference-time computation to tasks of varying difficulty, enabling an efficiency–performance trade-off. Yet two key flaws exist: under-specified rewards that fail to penalize misaligned reasoning budgets, and an “elastic” failure mode in which models tend to revert to non-adaptive reasoning. To this end, we propose DARS (Dynamic Alignment Reward Steering), a novel Reinforcement Learning Finetuning (RLF) framework that combines two synergistic strategies—Minimal Sufficient Reasoning (MSR)-guided optimization and Fisher regularization—to improve the robustness of adaptive reasoning fine-tuning. Concretely, MSR-guided optimization begins with an oracle-calibrated difficulty estimator that assigns each instance to a difficulty tier. Building on this, MSR-driven reward explicitly penalizes both short “arrogant” reasoning and prolonged over-thinking while promoting diverse reasoning paths. To stabilize RLF and mitigate elastic reversion, we further incorporate a Fisher-informed regularizer that constrains updates to high-importance parameters according to the Fisher information matrix. Experiments demonstrate that DARS prevents catastrophic collapse and achieves robust adaptivity across diverse benchmarks. On standard complex reasoning tasks like MATH, it substantially improves the accuracy–efficiency trade-off (+0.6% accuracy with 3.5% fewer tokens on 14B), and on extremely challenging benchmarks like AIME, DARS maintains a much narrower token fluctuation range under adversarial attacks (e.g., reducing the token changes span by over 30% on 14B compared to strong baselines), and effectively resists severe mode reversion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Robust Adaptive Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wUKfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DARS studies the stability of adaptive reasoning during continued fine-tuning. The method first uses supervised fine-tuning to establish multiple reasoning formats, then applies GRPO with an oracle-calibrated length reward, a format-diversity bonus, and Fisher-weighted parameter regularization. The length reward encourages concise correct answers and penalizes incorrect answers that are too short relative to a difficulty-dependent reference. Experiments on Qwen2.5-3B, 7B, and 14B evaluate accuracy, token usage, difficulty–length correlation, and stability after fine-tuning on verbose, answer-only, or creative-writing data. The paper reports improved efficiency and reduced length deviations in several settings, particularly GSM8K.

Critical strengths

A practically relevant problem with encouraging results in some settings. Preserving appropriate reasoning budgets during continued fine-tuning is worthwhile. On 14B GSM8K, DARS reduces the Verbose-induced token increase to +8, compared with +216 for ARM and +89 for AdaCtrl, while maintaining strong accuracy. On clean 14B MATH, it improves accuracy by 0.6 percentage points over ARM with approximately 3.5% fewer tokens.

Useful ablations and a meaningful controlled comparison. The paper evaluates the MSR, oracle, and Fisher components and includes an AdaCtrl+Fisher comparison under matched settings. This control suggests that adding Fisher regularization alone does not reproduce the full benefit. The additional backbone and oracle-noise experiments also help expose the method’s scope.

Critical weaknesses
  1. The methodological advance over existing approaches is limited and insufficiently demonstrated. ARM already addresses format collapse using rewards for less frequent formats and a decaying schedule; AdaCtrl already studies difficulty-aware reasoning budgets. The Fisher-weighted quadratic penalty follows the established EWC mechanism. Applying these ideas to continued-fine-tuning robustness is potentially useful, but the paper needs clearer evidence of what this combination enables beyond existing reward shaping and conservative updates. The proposed elasticity explanation is also heuristic: the stated scaling relationship is not derived or empirically fitted, and behavioral degradation does not by itself establish a return toward the pretrained policy.

  2. The MSR reward does not implement the narrow difficulty-matched budget described in Figure 1. For correct responses, Equation 3 monotonically favors shorter outputs, using the ground-truth solution length rather than an explicit difficulty-tier budget. For incorrect responses, it penalizes only those below a threshold; sufficiently long incorrect responses receive no MSR penalty. Thus, the reward does not generally penalize both under- and over-reasoning around a matched interval. It can also favor extending an incorrect response merely to escape the short-answer penalty. Successful oracle traces provide an average reference length, but do not establish minimal sufficient computation for the student.

  3. The robustness claims exceed the evidence, and the central evaluation protocol is not sufficiently clear. The introduction’s claim of less than 10% length variation does not hold broadly: under the 14B MMLU-STEM Verbose condition, DARS grows from 96 to 300 tokens. In the cross-backbone experiment, the distilled model’s GSM8K length grows from 365 to 1,240 tokens. More importantly, the paper does not adequately specify the attack-stage optimizer, update budget, trainable parameters, or whether DARS retains its protective losses during those updates. Without this information, it is difficult to assess the threat model and comparability of the robustness results.

The mechanism metrics also require more cautious interpretation. Difficulty–length rank correlation does not establish that the allocated budget is appropriate, and a low short/long overlap can arise from consistently choosing one mode. Similarly, low policy KL alone cannot distinguish selective preservation from generally conservative updates. Table 7 also contains an arithmetic inconsistency: 310 to 335 tokens is +25, rather than the reported +75.

Questions for authors
  1. How should Equation 3 be reconciled with Figure 1’s narrow matched-length region? What evidence shows that the reward encourages sufficient reasoning rather than merely shorter correct answers and longer incorrect answers? Please clarify how the reference solution length is selected when multiple reasoning formats are available.

  2. What exactly is the continued-fine-tuning protocol for each attack: training objective, learning rate, number of updates, and trainable parameters? Are Fisher regularization and MSR retained during attack-stage training, and what corresponding protections are used for the baselines?

  3. Does Fisher weighting provide an advantage over ordinary L2 or stronger policy-KL regularization when update magnitude is comparable? Can the model still learn the new fine-tuning tasks while preserving its original reasoning behavior?

  4. How are the difficulty scores for AG and the probabilities and threshold for the Misalignment Band determined? How do these measurements distinguish appropriate budgeting from stable but excessive or single-mode reasoning? Please also explain the substantial failures on MMLU-STEM and the distilled backbone.

Reviewer_hTC7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DARS studies whether adaptive reasoning survives subsequent fine-tuning on demonstrations that mismatch question difficulty and answer length: long answers to easy questions and short answers to hard ones. The method first uses SFT with direct-answer, standard-CoT, and long-CoT formats. An external model then places training questions in three difficulty tiers. During GRPO, DARS combines correctness, format-diversity, and MSR length rewards, while a Fisher-weighted regularizer limits parameter drift. Experiments measure accuracy, token use, AG, and policy KL before and after Verbose, Lazy, and Random perturbation fine-tuning. The clearest contribution is the focus on robustness to later difficulty–length-misaligned fine-tuning, together with the particular combination of fixed oracle difficulty tiers, an outcome-dependent MSR reward, and Fisher regularization. Multiple reasoning formats and diversity rewards, difficulty-aware length control, and Fisher-weighted parameter protection have precedents in ARM, AdaCtrl, and EWC, respectively. The contribution is therefore mainly a combination and an evaluation setting.

Critical strengths
  1. The evaluation targets a genuine failure mode relevant to the research question. Verbose and Lazy expose the model to long answers for easy questions and short answers for hard questions, respectively, while Random tests interference from unrelated tasks. Tracking both accuracy and token use provides a more pertinent test of post-fine-tuning robustness than clean-state token savings alone.
Critical weaknesses
  1. The central cross-task and cross-model robustness claim overstates the evidence. Page 3 claims less than 10% variation in reasoning length after attacks. Yet, in Table 1, Verbose fine-tuning increases the 7B/MMLU-STEM mean from 89 to 285 tokens: +196, or about +220%. The corresponding 3B and 14B increases are 185 and 204 tokens. In Appendix Table 7, DeepSeek-R1-Distilled/GSM8K rises from 365 to 1,240 tokens. These directly contradict an unconditional robustness claim. AG measures correlation between difficulty and length, not whether extra tokens are necessary or correct, so the results alone do not establish that the routing policy has been fully internalized. Appendix Table 7 lists 310→335 tokens as +75 for LLaMA2-7B/GSM8K; the arithmetic gives +25. The reporting problem also extends to the unqualified claim of less than 10% reasoning-length variation under attacks: in Table 1, DARS on MMLU-STEM shows increases of 185, 196, and 204 tokens after Verbose fine-tuning at 3B, 7B, and 14B, from clean means of 85, 89, and 96 tokens, respectively (approximately +218%, +220%, and +213%). These figures do not align with the stated bound under the ordinary interpretation of relative change in mean length; the authors should define its metric and scope and reconcile the main-table results. Together, these issues weaken confidence in precise reporting and causal attribution.

  2. The assumed link between difficulty and solution length is not validated, undermining the interpretation of “necessary” compute allocation. A hard problem may admit a short key insight, while a simpler problem may elicit a long answer because of arithmetic work, output format, or writing style; length distributions also differ across task types. Eq. (2) uses the mean oracle solution length for a difficulty tier, and Eq. (10) defines AG as a Spearman correlation between difficulty and output length, but neither measures the minimum effective computation for an individual problem. In Appendix Table 4, the mean Long-CoT length is 1,635 tokens for CommonsenseQA versus 754 for MATH, illustrating possible task and generation-style confounding. On the same 7B/MMLU-STEM test questions, Table 1 shows that Verbose fine-tuning raises average output length from 89 to 285 tokens while accuracy falls from 72.9% to 71.6%. This does not show that length is useless, but it does show that longer output need not be more useful. MSR treats a short incorrect answer as a length deficit even when the error could be conceptual or logical; a long incorrect answer escapes that length penalty. A high AG therefore cannot by itself establish minimal sufficient reasoning. Task-controlled accuracy-versus-token-budget curves or multiple correct solutions per question would test the claim more directly.

Questions for authors

Please refer to the "Critical Weaknesses" part: 1. Clarification of the robustness claim; 2. Clarification of the link between difficulty and solution length.

Reviewer_pzmuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The problem that the paper studies is robustness of adaptive reasoning models, which choose between direct answers, short CoT and long CoT according to task difficulty. The authors show that brief fine-tuning on data where response length doesn’t match task difficulty causes reasoning mode collapse. The model stops adapting its reasoning length while accuracy appears stable. They attribute this to under-specified rewards and to elastic reversion toward base behavior.vThey propose DARS, a two-stage method. First, Qwen2.5 (3B/7B/14B) is fine-tuned on about 25k multi-format examples from GSM8K, MATH and CommonsenseQA. It is then trained with GRPO using three additions:

  • Oracle-calibrated difficulty tiers: a stronger model labels each question Easy, Medium or Hard, and each tier gets a reference reasoning length.

  • An MSR length reward with a curriculum: correct answers are rewarded for being shorter than the ground-truth solution, and wrong answers that are much shorter than the tier’s reference length are penalized. This is combined with a decaying bonus for format diversity.

  • Fisher-information (EWC-style) regularization: changes to parameters that matter most for the post-SFT behavior are penalized.

Robustness is tested by fine-tuning on three 500-example adversarial sets (Verbose, Lazy, Random) and evaluating on MATH, GSM8K, MMLU-STEM and AIME against Coldstart-SFT, ARM and AdaCtrl. DARS reports small clean gains (+0.6 accuracy with 3.5% fewer tokens on MATH-14B), smaller token shifts under attack (+8 vs. +216 for ARM on GSM8K-14B Verbose), and higher Adaptivity Gain with lower policy KL.

Critical strengths
  • The evaluation of the paper is robust - it uses multiple evaluation matrices other than using just accuracy
  • The question that the paper asks would be very good and significant if it also looks into the existing benefits of long reasoning trace as well
Critical weaknesses
  1. The motivation of the paper treats long COT as cost only. It doesn't discuss the benefits of long reasoning. The paper never discusses the other roles of long CoT: self-verification, error correction, robustness to paraphrase and distribution shift, and out-of-domain generalization.
  2. The method is closely related to existing work on difficulty-aware length rewards for adaptive reasoning, which the paper doesn’t cite or position itself against: LASER-D (https://arxiv.org/abs/2505.15612), DiPO (https://arxiv.org/abs/2601.21418) and TRAAC (https://arxiv.org/abs/2510.01581). TRAAC already penalizes both under- and over-thinking by difficulty tier. The format-diversity bonus is close to ARM’s Ada-GRPO, and the Fisher regularizer is standard EWC.
  3. The paper doesn't completely state how attack fine-tuning is done (loss, steps, learning rate) or whether the Fisher regularizer stays on.
  4. Many results are just written as numbers and not discussed properly like (MATH-14B Coldstart-SFT 75.4 → 80.3, ARM 78.9 → 80.9; GSM8K-3B Coldstart-SFT 84.5 → 90.5; GSM8K-14B DARS 96.0 → 98.4). This contradicts the paper’s framing, but the text never explains it.
Questions for authors
  • How exactly is the attack fine-tuning performed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.