- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DARS studies the stability of adaptive reasoning during continued fine-tuning. The method first uses supervised fine-tuning to establish multiple reasoning formats, then applies GRPO with an oracle-calibrated length reward, a format-diversity bonus, and Fisher-weighted parameter regularization. The length reward encourages concise correct answers and penalizes incorrect answers that are too short relative to a difficulty-dependent reference. Experiments on Qwen2.5-3B, 7B, and 14B evaluate accuracy, token usage, difficulty–length correlation, and stability after fine-tuning on verbose, answer-only, or creative-writing data. The paper reports improved efficiency and reduced length deviations in several settings, particularly GSM8K.
Critical strengths
A practically relevant problem with encouraging results in some settings. Preserving appropriate reasoning budgets during continued fine-tuning is worthwhile. On 14B GSM8K, DARS reduces the Verbose-induced token increase to +8, compared with +216 for ARM and +89 for AdaCtrl, while maintaining strong accuracy. On clean 14B MATH, it improves accuracy by 0.6 percentage points over ARM with approximately 3.5% fewer tokens.
Useful ablations and a meaningful controlled comparison. The paper evaluates the MSR, oracle, and Fisher components and includes an AdaCtrl+Fisher comparison under matched settings. This control suggests that adding Fisher regularization alone does not reproduce the full benefit. The additional backbone and oracle-noise experiments also help expose the method’s scope.
Critical weaknesses
-
The methodological advance over existing approaches is limited and insufficiently demonstrated. ARM already addresses format collapse using rewards for less frequent formats and a decaying schedule; AdaCtrl already studies difficulty-aware reasoning budgets. The Fisher-weighted quadratic penalty follows the established EWC mechanism. Applying these ideas to continued-fine-tuning robustness is potentially useful, but the paper needs clearer evidence of what this combination enables beyond existing reward shaping and conservative updates. The proposed elasticity explanation is also heuristic: the stated scaling relationship is not derived or empirically fitted, and behavioral degradation does not by itself establish a return toward the pretrained policy.
-
The MSR reward does not implement the narrow difficulty-matched budget described in Figure 1. For correct responses, Equation 3 monotonically favors shorter outputs, using the ground-truth solution length rather than an explicit difficulty-tier budget. For incorrect responses, it penalizes only those below a threshold; sufficiently long incorrect responses receive no MSR penalty. Thus, the reward does not generally penalize both under- and over-reasoning around a matched interval. It can also favor extending an incorrect response merely to escape the short-answer penalty. Successful oracle traces provide an average reference length, but do not establish minimal sufficient computation for the student.
-
The robustness claims exceed the evidence, and the central evaluation protocol is not sufficiently clear. The introduction’s claim of less than 10% length variation does not hold broadly: under the 14B MMLU-STEM Verbose condition, DARS grows from 96 to 300 tokens. In the cross-backbone experiment, the distilled model’s GSM8K length grows from 365 to 1,240 tokens. More importantly, the paper does not adequately specify the attack-stage optimizer, update budget, trainable parameters, or whether DARS retains its protective losses during those updates. Without this information, it is difficult to assess the threat model and comparability of the robustness results.
The mechanism metrics also require more cautious interpretation. Difficulty–length rank correlation does not establish that the allocated budget is appropriate, and a low short/long overlap can arise from consistently choosing one mode. Similarly, low policy KL alone cannot distinguish selective preservation from generally conservative updates. Table 7 also contains an arithmetic inconsistency: 310 to 335 tokens is +25, rather than the reported +75.
Questions for authors
-
How should Equation 3 be reconciled with Figure 1’s narrow matched-length region? What evidence shows that the reward encourages sufficient reasoning rather than merely shorter correct answers and longer incorrect answers? Please clarify how the reference solution length is selected when multiple reasoning formats are available.
-
What exactly is the continued-fine-tuning protocol for each attack: training objective, learning rate, number of updates, and trainable parameters? Are Fisher regularization and MSR retained during attack-stage training, and what corresponding protections are used for the baselines?
-
Does Fisher weighting provide an advantage over ordinary L2 or stronger policy-KL regularization when update magnitude is comparable? Can the model still learn the new fine-tuning tasks while preserving its original reasoning behavior?
-
How are the difficulty scores for AG and the probabilities and threshold for the Misalignment Band determined? How do these measurements distinguish appropriate budgeting from stable but excessive or single-mode reasoning? Please also explain the substantial failures on MMLU-STEM and the distilled backbone.