acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Adaptation of Scaffolds from Unlabeled Trajectories

Abstract

A scaffold is a workflow that orchestrates how a coding agent observes its environment, uses tools, and produces outputs. Existing scaffolds typically follow a uniform workflow across different models and tasks. In this study, we reveal that such uniform workflows may fail to accommodate diverse model capabilities and varying coding complexities. To address this limitation, we propose *AdaScaffold*, a test-time **ada**ptation framework that adapts **scaffold**s to diverse model capabilities and coding complexities. AdaScaffold uses unlabeled trajectories to estimate when to intervene and inject behavioral guidelines into the original workflow via two core modules: (i) *Intervention Timing Estimation*, which optimizes the timing of each intervention by maximizing the *empirical covariation* between agent mis-behaviors and risks; and (ii) *Guideline Generation*, which prompts the coding agent itself with the estimated timings and predefined mis-behaviors to synthesize model- and task-specific guidelines. Extensive evaluation demonstrates that AdaScaffold improves resolve rates across various model scales while reducing token consumption to 45%–85%. By grounding intervention timing in a closed-form statistical criterion over unlabeled trajectories, AdaScaffold offers an interpretable test-time scaffold-level adaptation mechanism for coding agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Test-Time Adaptation, Agent scaffolding, Coding Agent, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YA5nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the scaffold framework which adaptively adjusts based on model capabilities, named AdaScaffold. The authors first manually defines several high-risk patterns that require intervention. Then, they employ the target model on subset of instances to estimate the relation between the exit reason and the AdaScaffold’s hyperparameters. The hyperparameters are finally obtained by optimizing the empirical covariation. Experiment results shown that the proposed AdaScaffold can improve the resolve rate and reduce inference costs across LLMs with various scales.

Critical strengths

1.The motivation is good. Adaptive scaffold is useful in resource-constrained settings.

2.The proposed method is training-free which is low-cost and easy to follow.

3.Authors use LLMs with various scales to verify their method. And Table 2 shows that AdaScaffold can obtain better performance with lower costs.

Critical weaknesses

1.Authors only employ experiments on SWE benchmark with Qwen and GLM. As a model-specific training-free method, experiments with more LLM architectures and and more datasets are needed for verify the generalizability.

2.AdaScaffold introduces many manually defined rules but there is no ablation study on them. Are all the rules in Table.1 necessary?

3.Though AdaScaffold is effective on Qwen3.5 4B-27B, there is no improvement on much larger GLM-5.2. I think the manually defined and fixed Intervention rules for small LLMs may not be effective on strong/SOTA LLMs. This could be a major limitation of AdaScaffold.

4.Can AdaScaffold be applied to tasks in other domains (e.g., math benchmarks)?

Questions for authors

See Weaknesses

3 comments

W1: Generalizability across models and datasets

Rebuttal:

Thank you for suggesting broader evaluation. Our experiments cover Qwen3.5 models at 4B, 9B, and 27B, together with the frontier-scale GLM-5.2, examining AdaScaffold across substantially different model capabilities. Given the rebuttal timeline, we prioritize these settings.

We report cross-dataset calibration experiments in the appendix: intervention parameters estimated from R2E-Gym or Rebench trajectories, with the following gains over the original scaffold:

Calibration trajectory poolR2E-GymRebench
Resolve-rate gain (pp) ↑+3.7+3.6
Normal-exit gain (pp) ↑+27.4+26.2
Step usage relative to baseline ↓91.45%88.45%
Token usage relative to baseline ↓85.25%80.65%

These results support the transferability of the estimated timings across trajectory sources.


W2: Necessity of the predefined intervention rules

Rebuttal:

Thank you for suggesting this ablation. We evaluated grouped ablations over the full evaluation set, removing editing rules (B, C), stopping rules (E, F), or behavior rules (A, D, G, H). These groups cover all interventions in Table 1.

Originalw/o Editing (BC)w/o Stopping (EF)w/o Behavior (ADGH)AdaScaffold
Resolve rate (%) ↑34.7330.4034.2034.2039.33
Normal exit (%) ↑7.0018.6011.6020.6033.93
Average steps ↓76.4661.1164.1161.6766.81
Average tokens (M) ↓2.1701.6311.7241.6231.690

Removing editing, stopping, and behavior rules reduces resolve rate by 8.93, 5.13, and 5.13 percentage points, respectively, relative to full AdaScaffold. Therefore, no single component yields substantial gains on its own. AdaScaffold's improvements stem from the combined effects of its components. This further validates the design of AdaScaffold. Individual components may even degrade performance, while their combination achieves the best results.

W3: Effectiveness on frontier-scale models

Rebuttal:

Thank you for raising this point. Our objective is to improve the performance–cost trade-off across model capabilities. For an already highly capable model, reducing redundant computation is also a meaningful benefit. Thus, the benefit on GLM-5.2 is primarily improved efficiency.

GLM-5.2OriginalAdaScaffold
Normal exit (%) ↑31.7352.83
Average steps ↓75.2466.09
Average tokens (M) ↓1.441.19
Net token savings, including adaptation overhead—13.24%

Even after accounting for trajectory collection and guideline generation, AdaScaffold retains 13.24% net token savings, demonstrating its value beyond smaller models.


W4: Extension to other domains, such as mathematical reasoning

Rebuttal:

Thank you for this suggestion. AdaScaffold can potentially extend to mathematics agents by replacing coding-specific behaviors with domain-appropriate indicators. We propose the following formulation.

Let a reasoning trajectory be τ=(o0,a0,…,oT)\tau=(o_0,a_0,\ldots,o_T). Candidate behaviors, observable from its prefixes, include:

BehaviorPotential riskIntervention
Repeated derivations without progressBudget exhaustionSummarize progress and switch strategy
Conflicting intermediate resultsUnresolved inconsistencyRevisit assumptions and check the conflicting step
Consecutive tool errorsInterrupted reasoningSimplify the computation or change verification method
Excessive verification or delayed finalizationNo final answer within budgetPerform a targeted check and finalize

Without using ground-truth answers, define a trajectory-level risk proxy:

Y(τ)=1{budget exhaustion, abnormal termination, or missing final answer}.Y(\tau)=\mathbf{1}\{\text{budget exhaustion, abnormal termination, or missing final answer}\}.

For each behavior kk, let Ik,θ(τ)I_{k,\theta}(\tau) indicate whether its prefix-valid trigger fires under threshold θ\theta. Following our timing-selection criterion:

=\arg\max_{\theta\in\Theta_k} \left[ \widehat P(I_{k,\theta}=1,Y=1) -\widehat P(I_{k,\theta}=1)\widehat P(Y=1) \right].$$ The agent then generates math-specific guidelines conditioned on the behaviors and selected thresholds, which are activated deterministically during reasoning. We would evaluate this extension against the original scaffold and fixed-schedule baselines, measuring **answer accuracy, final-answer completion rate, and total token cost including calibration overhead**. This is a concrete extension proposal; validation on mathematics benchmarks remains future work. --- We thank the reviewer for these suggestions, which help clarify both the empirical scope of AdaScaffold and its potential extension to other domains. We hope our responses and additional experiments address the reviewer’s concerns. We welcome further questions and would appreciate reconsideration of the score.

Dear Reviewer YA5n,

As the author response period is approaching its end, we would like to kindly ask whether our clarifications have addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could increasing your score in light of our responses.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_gBX9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AdaScaffold for test-time adaptation of agent scaffolds from unlabeled trajectories. Rather than updating model weights, it adapts the scaffold using information from agent execution. The experiments evaluate changes in Resolve Rate, Normal Exit, and execution cost across three Qwen3.5 model sizes and GLM-5.2, together with component ablations. Overall, the paper studies a reasonable direction for improving coding-agent execution without requiring labeled trajectories.

Critical strengths

The main idea is intuitive and practically relevant. The scaffold affects how an agent uses its execution budget, moves between investigation and editing, and finishes a task. Adapting this behavior from unlabeled trajectories is a reasonable way to improve the agent without changing its underlying model weights.

The experimental coverage is reasonably comprehensive. The paper evaluates three Qwen3.5 model sizes and an additional GLM-5.2 model, rather than relying on a single model configuration. The ablations also help examine the design choices. This provides a useful picture of how the approach behaves across different settings, including its effects on task resolution and execution cost.

Critical weaknesses

My main concern is the attribution of the improvement in Resolve Rate. The authors discuss several possible mechanisms, including reallocating the execution budget, editing earlier, breaking unproductive loops, and submitting in time. However, I would appreciate a more quantitative breakdown of their contributions. In particular, Normal Exit increases substantially, making it important to understand how much of the improvement comes from better termination behavior and how much comes from other changes in the scaffold. This does not imply that improved termination explains all the gains, but a clearer breakdown would make the contribution easier to interpret.

There are also minor presentation issues. The vertical text in Figure 1 is inconvenient to read, and a horizontal layout would help. The text in Figure 2 could be larger. Some parts of the introduction rely heavily on section-by-section signposting; a more connected narrative would make the motivation and contribution easier to follow. These are minor issues rather than substantive objections to the method.

Questions for authors

Could the authors quantify how much of the Resolve Rate improvement is attributable to timely submission versus other changes, such as earlier editing and breaking unproductive loops? A comparison with a simple termination or submission reminder would help isolate these effects. Based on this breakdown, which remaining behaviors appear to offer the most room for further improvement?

3 comments

W1 & Q1: Attribution of resolve-rate improvements

Rebuttal:

Thank you for suggesting a more quantitative breakdown. We evaluated grouped ablations over the full evaluation set, removing editing rules (B, C), stopping rules (E, F), or behavior rules (A, D, G, H), while retaining the remaining components. We also evaluated Only F, which retains only the simple termination. Budgets are relative to Original.

Originalw/o Editing (BC)w/o Stopping (EF)w/o Behavior (ADGH)Only Termination (F)AdaScaffold
Resolve rate (%) ↑34.7330.4034.2034.2027.8039.33
Δ resolve rate (pp) ↑—-4.33-0.53-0.53-6.93+4.60
Normal exit (%) ↑7.0018.6011.6020.604.2033.93
Δ normal exit (pp) ↑—+11.60+4.60+13.60-2.80+26.93
Average steps ↓76.4661.1164.1161.6764.9666.81
Step budget ↓—79.92%83.85%80.66%84.96%87.38%
Average tokens (M) ↓2.1701.6311.7241.6231.7821.690
Token budget ↓—75.16%79.46%74.80%82.10%77.88%

Removing editing rules reduces resolve rate by 8.93 percentage points relative to full AdaScaffold (39.33% to 30.40%); removing stopping or behavior rules reduces it by 5.13 points in each case (to 34.20%). Only F reaches 27.80%, which is 11.53 points below full AdaScaffold and 6.93 points below Original. Therefore, no single component yields substantial gains on its own, including termination alone. AdaScaffold's improvements stem from the combined effects of its components. This further validates the design of AdaScaffold. Individual components may even degrade performance, while their combination achieves the best results.


W2: Presentation

Rebuttal:

Thank you for these helpful suggestions. We have revised the introduction to replace the section-by-section signposting with a more connected narrative linking the motivation, method, and evaluation. We have also changed the vertical text in Figure 1 to a horizontal layout and enlarged the text in Figure 2 to improve readability in the updated manuscript.


We thank the reviewer for these thoughtful comments. We hope our responses and additional experiments address the reviewer’s concerns.

Dear Reviewer gBX9,

As the author response period is approaching its end, we would like to kindly ask whether our clarifications have addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could increasing your score in light of our responses.

Thank you for your time and consideration.

Best regards, The Authors

Thank you for the kind followup, it solves my central decomposition question. I maintain supportive,.

Reviewer_bsEkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaScaffold adapts a coding-agent scaffold without updating model weights. It estimates intervention timings from unlabeled trajectories using behavioral risk proxies, generates natural-language guidelines, and injects them deterministically during execution. On SWE-bench Verified, the method improves resolve rates for three Qwen-3.5 model sizes while reducing reported rollout token consumption.

Critical strengths
  1. The engineering problem is real: unproductive exploration, repeated errors, and failure to submit waste agent budgets. Interpretable interventions with no per-step supervisor calls could provide a useful, inexpensive harness component.
  2. The reported trade-off merits investigation: at 27B, resolve rate increases from 60.00% to 65.40% and average tokens decrease from 2.77M to 1.29M. Three-seed results, cross-pool calibration, and an inspectable anonymous implementation provide a concrete basis for evaluating the idea.
Critical weaknesses
  1. The main incremental contribution is not isolated. Predefined interventions, behavioral risk proxies, and additional phase heuristics may simply implement effective process control missing from the baseline. Covariance with long or abnormal trajectories does not estimate intervention benefit, and random timing is insufficient to establish an advantage over a reasonable fixed schedule. This concern would remain even with a corrected implementation.
  2. A separate reliability concern is verified in the released calibration code: terminal output_patch contributes to every has_patch_prefix, so a read-only prefix can be labeled as having a patch before any edit. These features select checkpoint/force-done candidates, while deployment checks current environment state. Its benchmark impact is unknown and it is not evidence of hidden-label use. Claims also need to account for calibration costs and the GLM-5.2 accuracy decrease (Table 7).
Questions for authors
  1. After replacing terminal patch metadata with prefix-valid patch state, do the selected thresholds and downstream resolve/cost results change? Please provide the corrected calibration and evaluation results, or explain why the released path does not correspond to the reported experiments.
  2. How much benefit remains over a sensible fixed intervention schedule, and what is the total cost including calibration collection and guideline generation?
3 comments

Weakness1 (contribution is not isolated) & Question2:

Rebuttal:

Thank you for suggesting this comparison. To examine adaptive timing against fixed timing, we evaluated three fixed-schedule baselines over the full evaluation set: Loose (fixed_upper), Moderate (fixed_medium), and Strict (fixed_lower). These use the upper, middle, and lower candidates of the calibration grids, respectively. Intervention types, guideline content, and runtime trigger ordering are retained. Budget denotes resource usage relative to the original scaffold.

OriginalFixed (Loose)Fixed (Moderate)Fixed (Strict)AdaScaffold
Resolve rate (%) ↑34.7329.4032.8031.8039.33
Δ resolve rate (pp) ↑—-5.33-1.93-2.93+4.60
Normal exit (%) ↑7.0021.2026.6020.4033.93
Δ normal exit (pp) ↑—+14.20+19.60+13.40+26.93
Average steps ↓76.4664.1259.2555.5066.81
Step budget ↓—83.86%77.49%72.59%87.38%
Average tokens (M) ↓2.1701.7351.5191.3851.690
Token budget ↓—79.96%70.00%63.81%77.88%

All three fixed schedules reduce steps and tokens relative to Original, but their resolve rates fall below its 34.73%: Loose reaches 29.40%, Moderate 32.80%, and Strict 31.80%. AdaScaffold reaches 39.33%, a gain of 4.60 percentage points over Original and 6.53 points over Moderate, the most accurate fixed schedule. Although tight and medium schedules reduce deployment costs and token consumption, they also lead to substantial performance degradation. In contrast, AdaScaffold achieves significant performance gains over fixed scaffolds by determining intervention timing through covariance-based estimation. Besides, we agree that covariance measures association with behavioral risk rather than causal intervention benefit. However, it serves as a label-free timing-selection criterion; these comparisons provide empirical evidence about the resulting performance-cost trade-off.


Weakness2 (reliability concern in the calibration code) & Question1

Rebuttal: Thank you for carefully examining the released code. We checked carefully and confirm that the terminal-patch dependency is a redundant artifact left during code development, rather than an implementation error affecting the reported experiments. We apologize for any confusion caused by this redundant.

Our calibration stage uses historical trajectories collected separately from benchmark evaluation. To ensure that calibration does not use test results or verifier outcomes, the calibration inputs retain only trajectory_steps and exit_reason; output_patch is not included. Therefore, in the reported experiments, record.get("output_patch") or "" always evaluates to an empty string. Consequently, this branch contributed no information: has_patch_prefix[t] reduced to whether an EDIT, CREATE, or INSERT action had occurred by step tt. A read-only prefix was therefore not marked as having a patch through terminal metadata in those experiments.

We will release an updated version of the code with this redundancy removed. Since its removal leaves the calibration computation unchanged for the experimental inputs, it does not change the selected thresholds or the reported downstream resolve and cost results. We have also reviewed the manuscript and tightened the relevant descriptions to eliminate ambiguity and ensure consistency with the implementation.

Questions2 (Total cost including calibration collection and guideline generation):

Rebuttal: Thank you for requesting a complete accounting of adaptation costs. We calculate the overhead using 20 historical unlabeled trajectories for parameter estimation. As shown in the appendix, using only 10–20 trajectories yields timing estimates close to those obtained with larger calibration samples. We include the tokens used for trajectory collection and guideline generation, as well as the runtime of parameter estimation. Parameter estimation requires no additional LLM tokens and takes only 0.75–1.25 seconds. After accounting for all additional token costs, AdaScaffold retains net token savings of 18.02%, 24.54%, 49.37%, and 13.24% for Qwen3.5-4B, 9B, 27B, and GLM-5.2, respectively.

Cost componentQwen3.5-4BQwen3.5-9BQwen3.5-27BGLM-5.2
Parameter estimation time (s)1.061.221.250.75
Unlabeled trajectory collection tokens (M)0.0870.1100.1100.058
Guideline generation tokens (M)0.00270.00260.00260.0013
Total extra tokens (% of baseline)4.10%4.08%4.06%4.12%
Net token savings, including overhead (%)18.02%24.54%49.37%13.24%

Weakness2 (GLM-5.2 Accuracy):

Rebuttal: Thank you for raising this point. Our claim is that AdaScaffold improves the performance–cost trade-off, rather than uniformly increasing accuracy for every model. For an already highly capable frontier model such as GLM-5.2, reducing token consumption is itself a valuable benefit. After accounting for trajectory collection and guideline generation, AdaScaffold still achieves a net token saving of 13.24%, demonstrating the efficiency benefit of adaptive scaffold control.


We thank the reviewer for their comments and hope our responses address their concerns. We welcome further questions and would appreciate reconsideration of the score.

Dear Reviewer bsEk,

As the author response period is approaching its end, we would like to kindly ask whether our clarifications have addressed your concerns. If any questions remain, we would be happy to provide further clarification. Otherwise, we would greatly appreciate it if you could increasing your score in light of our responses.

Thank you for your time and consideration.

Best regards, The Authors

Comments on the paper

1 comment

Dear Reviewers,

The authors have posted their responses and a revised manuscript addressing your questions and concerns. Please review their rebuttal carefully, let them know if any concerns remain, and update your review accordingly.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.