- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the scaffold framework which adaptively adjusts based on model capabilities, named AdaScaffold. The authors first manually defines several high-risk patterns that require intervention. Then, they employ the target model on subset of instances to estimate the relation between the exit reason and the AdaScaffold’s hyperparameters. The hyperparameters are finally obtained by optimizing the empirical covariation. Experiment results shown that the proposed AdaScaffold can improve the resolve rate and reduce inference costs across LLMs with various scales.
Critical strengths
1.The motivation is good. Adaptive scaffold is useful in resource-constrained settings.
2.The proposed method is training-free which is low-cost and easy to follow.
3.Authors use LLMs with various scales to verify their method. And Table 2 shows that AdaScaffold can obtain better performance with lower costs.
Critical weaknesses
1.Authors only employ experiments on SWE benchmark with Qwen and GLM. As a model-specific training-free method, experiments with more LLM architectures and and more datasets are needed for verify the generalizability.
2.AdaScaffold introduces many manually defined rules but there is no ablation study on them. Are all the rules in Table.1 necessary?
3.Though AdaScaffold is effective on Qwen3.5 4B-27B, there is no improvement on much larger GLM-5.2. I think the manually defined and fixed Intervention rules for small LLMs may not be effective on strong/SOTA LLMs. This could be a major limitation of AdaScaffold.
4.Can AdaScaffold be applied to tasks in other domains (e.g., math benchmarks)?
Questions for authors
See Weaknesses
3 comments
Part II
W3: Effectiveness on frontier-scale models
Rebuttal:
Thank you for raising this point. Our objective is to improve the performance–cost trade-off across model capabilities. For an already highly capable model, reducing redundant computation is also a meaningful benefit. Thus, the benefit on GLM-5.2 is primarily improved efficiency.
| GLM-5.2 | Original | AdaScaffold |
|---|---|---|
| Normal exit (%) ↑ | 31.73 | 52.83 |
| Average steps ↓ | 75.24 | 66.09 |
| Average tokens (M) ↓ | 1.44 | 1.19 |
| Net token savings, including adaptation overhead | — | 13.24% |
Even after accounting for trajectory collection and guideline generation, AdaScaffold retains 13.24% net token savings, demonstrating its value beyond smaller models.
W4: Extension to other domains, such as mathematical reasoning
Rebuttal:
Thank you for this suggestion. AdaScaffold can potentially extend to mathematics agents by replacing coding-specific behaviors with domain-appropriate indicators. We propose the following formulation.
Let a reasoning trajectory be . Candidate behaviors, observable from its prefixes, include:
| Behavior | Potential risk | Intervention |
|---|---|---|
| Repeated derivations without progress | Budget exhaustion | Summarize progress and switch strategy |
| Conflicting intermediate results | Unresolved inconsistency | Revisit assumptions and check the conflicting step |
| Consecutive tool errors | Interrupted reasoning | Simplify the computation or change verification method |
| Excessive verification or delayed finalization | No final answer within budget | Perform a targeted check and finalize |
Without using ground-truth answers, define a trajectory-level risk proxy:
For each behavior , let indicate whether its prefix-valid trigger fires under threshold . Following our timing-selection criterion:
=\arg\max_{\theta\in\Theta_k} \left[ \widehat P(I_{k,\theta}=1,Y=1) -\widehat P(I_{k,\theta}=1)\widehat P(Y=1) \right].$$ The agent then generates math-specific guidelines conditioned on the behaviors and selected thresholds, which are activated deterministically during reasoning. We would evaluate this extension against the original scaffold and fixed-schedule baselines, measuring **answer accuracy, final-answer completion rate, and total token cost including calibration overhead**. This is a concrete extension proposal; validation on mathematics benchmarks remains future work. --- We thank the reviewer for these suggestions, which help clarify both the empirical scope of AdaScaffold and its potential extension to other domains. We hope our responses and additional experiments address the reviewer’s concerns. We welcome further questions and would appreciate reconsideration of the score.Dear Reviewer YA5n,
As the author response period is approaching its end, we would like to kindly ask whether our clarifications have addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could increasing your score in light of our responses.
Thank you for your time and consideration.
Best regards, The Authors
Part I
W1: Generalizability across models and datasets
Rebuttal:
Thank you for suggesting broader evaluation. Our experiments cover Qwen3.5 models at 4B, 9B, and 27B, together with the frontier-scale GLM-5.2, examining AdaScaffold across substantially different model capabilities. Given the rebuttal timeline, we prioritize these settings.
We report cross-dataset calibration experiments in the appendix: intervention parameters estimated from R2E-Gym or Rebench trajectories, with the following gains over the original scaffold:
These results support the transferability of the estimated timings across trajectory sources.
W2: Necessity of the predefined intervention rules
Rebuttal:
Thank you for suggesting this ablation. We evaluated grouped ablations over the full evaluation set, removing editing rules (B, C), stopping rules (E, F), or behavior rules (A, D, G, H). These groups cover all interventions in Table 1.
Removing editing, stopping, and behavior rules reduces resolve rate by 8.93, 5.13, and 5.13 percentage points, respectively, relative to full AdaScaffold. Therefore, no single component yields substantial gains on its own. AdaScaffold's improvements stem from the combined effects of its components. This further validates the design of AdaScaffold. Individual components may even degrade performance, while their combination achieves the best results.