acceptodds
Under review as a conference paper at ICLR 2027

SCAD: Structured Credit Assignment and Distillation for Long-Horizon Agents

Abstract

Training long-horizon agents to solve complex tasks requires effective supervision over extended interaction sequences. However, sparse terminal rewards obscure intermediate contributions, while on-policy distillation can lose informative teacher guidance as student-generated histories grow. To address this problem, we introduce SCAD, which organizes interactions into planning and bounded subtask execution, distills execution in local contexts, and refines planning credit through cross-rollout subtask prefix trees, with planning receiving full terminal credit and execution receiving positive terminal credit and teacher guidance. Across all evaluated benchmarks, SCAD improves macro-average accuracy over the strongest training baseline by 4.48 percentage points for text tasks and 4.19 points for multimodal tasks. These results demonstrate that SCAD effectively combines outcome-based credit assignment with teacher-guided distillation to improve planning and execution in long-horizon agents. Our code is publicly available.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
long-horizon agents, reinforcement learning, on-policy distillation, credit assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wy2YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCAD, a training framework for long-horizon search agents that combines RL with on-policy distillation. A shared policy acts as a Planner that delegates subtasks and as a Worker that executes each subtask in a fresh local context. SCAD distills Worker behavior from a frozen teacher under the same local context, assigns Planner credit by comparing terminal returns of sibling choices in cross-rollout subtask–report prefix trees, and gives Workers only positive terminal credit plus the teacher signal. On 11 text and 8 multimodal search benchmarks, SCAD improves macro-average accuracy over the strongest baselines by 4.48 and 4.19 points, supported by ablations, component analyses, and efficiency measurements.

Critical strengths
  1. The paper is clearly motivated, with three identified challenges mapped one-to-one to three design components, and Table 1 positions the method well against related work.

  2. Scoring Worker tokens under the student's own local context is a simple and practical way to keep teacher supervision informative.

  3. The experiments support the paper's main claims, with a comprehensive benchmark selection covering 19 text and multimodal datasets and a wide range of RL, distillation, and hybrid baselines.

Critical weaknesses
  1. Most of the improvement comes from local OPD, which accounts for 62.2% (text) and 71.9% (multimodal) of the cumulative gain. Tree credit and structured allocation each add 1.1–2.1 points, comparable to the reported seed standard deviations of about ±1.0–1.5.

  2. The ablation adds components in a fixed order on top of local OPD. It does not show whether tree credit or signal allocation still helps without OPD, nor the effect of removing each component from the full model.

  3. Many baselines use flat interaction structures, while SCAD uses the Planner–Worker architecture. Local GRPO within this architecture already reaches 39.88% on text, above flat GRPO (36.34%), OPD (38.06%), and SFT (38.79%), so part of the main-table gain comes from the architecture itself rather than the proposed training signals.

  4. The individual components are closely related to existing work: Planner–Worker context isolation (AgentFlow, Context Folding, CoDA), OPD, prefix or shared-state grouping (GiGPO, Tree-GRPO), and outcome-dependent distillation (VOLD and HyperEyes apply OPD to failed rollouts). The main contribution lies in combining these components.

Questions for authors
  1. Could you report leave-one-out ablations from the full model, as well as results for tree credit and signal allocation without local OPD? This would clarify whether these two components contribute independently of distillation.

  2. Since part of the gain over flat baselines comes from the Planner–Worker architecture, could you compare key baselines (e.g., GRPO, OPD, and ATOD) within the same architecture, as done for ATOD in RQ3?

  3. In the text setting, the teacher (Qwen3-4B-Instruct-2507) has the same size as the student, and the trained student surpasses it (46.10 vs. 45.56). How much of the OPD gain comes from the teacher's knowledge rather than from regularization, and how does SCAD behave with a stronger teacher?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wy2Y (Part 1/2)

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns (Part 1/2). In the revised PDF, changes for you are in purple, changes shared with other reviewers in magenta, and the other reviewers' changes in blue and orange.


W1: "Most of the improvement comes from local OPD ... Tree credit and structured allocation each add 1.1–2.1 points, comparable to the reported seed standard deviations of about ±1.0–1.5."

  • Thank you for this careful reading. We agree that local OPD is the largest contributor; it is the first of our three designs. For the other two, the seed standard deviations describe a single configuration, whereas the ablation stages are paired: they share the checkpoint, seeds, and test questions, so we added a paired bootstrap (10,000 resamples over questions within each dataset and over seeds; Appendix A.4, Table 10):
ModalityComponent addedGain (pp)95% intervalSeeds > 0
TextTree credit+1.24[-0.74, +3.28]3/3
TextSignal allocation+1.11[-0.82, +3.12]3/3
TextBoth+2.35[+0.08, +4.71]3/3
MultimodalTree credit+1.61[-1.49, +4.66]3/3
MultimodalSignal allocation+2.09[-0.63, +4.84]3/3
MultimodalBoth+3.70[+0.77, +6.74]3/3
  1. The joint contribution of the two credit components is significant in both modalities: both intervals exclude zero.
  2. Each single increment is consistent in direction: positive in all 12 seed-level comparisons, with every interval lying predominantly above zero. It is modest, as you note, so Section 4.3 states the claim for the two components jointly, where the evidence is strongest; Table 10 reports all intervals.
  3. The ordering Local OPD + GRPO < + tree credit < + allocation holds in 3/3 seeds for both modalities.
  • Independent controls agree: the ×1.6 scaling control shows that the tree-credit gain is not simply a larger planning advantage (44.61% vs. 46.10%, Figure 7b).

W2 & Q1: "...whether tree credit or signal allocation still helps without OPD, nor the effect of removing each component from the full model."

  • Thank you for this valuable suggestion. We added the full 2×2×2 factorial ablation as Table 4 in Section 4.3 (text, with the same architecture, budget, and 3 seeds as RQ2):
Local OPDTree creditAllocationText Avg. (%)Δ vs. SCAD (pp)
–––39.88-6.22
–✓–40.97-5.13
––✓40.29-5.81
–✓✓41.58-4.52
✓––43.75-2.35
✓–✓44.32-1.78
✓✓–44.99-1.11
✓✓✓46.10–
  1. Leave-one-out (bold): removing any component lowers accuracy, by 4.52 / 1.78 / 1.11 points (paired-bootstrap 95% intervals: [-7.01, -2.08], [-3.81, +0.19], [-3.12, +0.82]).
  2. Tree credit contributes independently of distillation: without any teacher it adds +1.09 points (+1.24 with OPD), as it acts on planning tokens, which never receive the teacher signal.
  3. Signal allocation also helps without OPD, but less (+0.41 vs. +1.11): positive-only credit stops penalizing correct executions in failed trajectories, but without a teacher nothing corrects the wrong ones.
  4. Both components help in either order of addition (tree credit: +1.24 before allocation, +1.78 after; allocation: +0.57 before tree credit, +1.11 after). Together they add 2.35 points, more than the 1.81 from adding each alone.

W3 & Q2: "...part of the main-table gain comes from the architecture itself rather than the proposed training signals." / "Could you compare key baselines ... within the same architecture ...?"

  • Thank you for this fair point. We agree that the architecture contributes, so the fair comparison is the one inside the same architecture. Table 5(a) (new Section 4.8, RQ7) and Appendix A.5 (Table 11) run the key baselines inside SCAD's Planner–Worker (P–W) architecture, each keeping its own supervision rule:
Method (inside P–W)Text Avg. (%)Multimodal Avg. (%)
GRPO (Local GRPO)39.8819.97
OPD41.6827.31
TCOD40.6327.18
ATOD41.2428.41
MemOPD41.8727.92
HyperEyes–28.76
GRPO + local OPD43.7529.43
SCAD (Ours)46.1033.13
  1. With the architecture fixed, SCAD's training signals add 6.22 (text) and 13.16 (multimodal) points over Local GRPO.
  2. The margin is preserved: SCAD leads the best in-architecture baselines by 4.23 and 4.37 points.
  3. The architecture alone does not fix full-history supervision: ATOD stays at 41.24% on text because its teacher still scores long student-generated histories (Figure 6b).
AuthorsOpenReview ↗

Response to Reviewer wy2Y (Part 2/2)


W4: "The individual components are closely related to existing work ... The main contribution lies in combining these components."

Thank you for the thoughtful feedback. We clarify what is new from five perspectives:

a. Motivation Level: Combining RL and on-policy distillation (OPD) for long-horizon agents leaves three open problems that none of the listed works solves on its own: (i) teacher feedback loses discriminative value along long student-generated histories; (ii) a terminal reward cannot tell which planning decision mattered; (iii) uniform mixing lets failures suppress useful teacher guidance. SCAD answers all three with one structure, the subtask and its report: its boundary defines the teacher's context, its prefix the unit of cross-rollout comparison, and its role the allocation of each signal.

b. Technical Level: each design differs from its closest prior work in a testable way:

SCAD designClosest prior workWhat SCAD does differentlyControlled evidence
Subtask-local distillationAgentFlow, Context Folding, CoDA bound the agent's contextThe teacher scores execution under the student's bounded local contextTeacher margin at depth 24: 0.039 → 0.126 nats/token; +2.51 over full-history ATOD in the same architecture (Figure 6)
Prefix-tree planning creditGiGPO needs identical states; Tree-GRPO branches at sampling timeRecovers prefixes post hoc by semantic matching; credits planning onlyGiGPO-style groups in our framework: 44.18% vs. 46.10% (Table 14)
Structured signal allocationVOLD, HyperEyes gate the teacher by outcomeGates the negative terminal credit by role; the teacher stays on+1.11 over signed credit (Figure 8c); HyperEyes in our framework: 28.76% vs. 33.13% (Table 11)

c. Theoretical Level: each design is supported by a proposition with a full proof (Appendix C): a reverse-KL gradient bound for local distillation, exact removal of within-choice return variance for tree credit, and minimality of role-based allocation.

d. Empirical Level: every design matters in the full model: removing local OPD, tree credit, or allocation costs 4.52 / 1.78 / 1.11 points (Table 4). SCAD leads the strongest baselines by 4.48 / 4.19 points in the main tables, by 4.23 / 4.37 points inside the same architecture (Table 5(a)), and by 5.2–9.1 points over ATOD on three agentic action benchmarks (Table 6).

e. Practical Level: SCAD needs no process reward, learned critic, or extra branch sampling, and takes 37.96% less training time than ATOD (Figure 9b).


Q3: "How much of the OPD gain comes from the teacher's knowledge rather than from regularization, and how does SCAD behave with a stronger teacher?"

(1) Teacher knowledge vs. regularization.

  • "Same size" is not "same capability": the teacher is a later, stronger release; under the identical scaffold, the untrained student scores 38.91% vs. 45.56% for the teacher.
  • Self-teacher control (Table 13; text, 3 seeds). We replace the teacher with a frozen copy of the initial student, which keeps the form of the signal but adds no knowledge:
Teacher used for local OPDTeacher acc. (%)Student acc. (%)
None (w/o local OPD)–41.58
Frozen initial student38.9141.93
Qwen3-4B-Instruct-250745.5646.10

Regularization explains 0.35 of the 4.52 points: about 92% of the OPD gain comes from the teacher's knowledge.

(2) Behavior with a stronger teacher (Table 5(b); per-dataset results in Table 12). Larger Qwen3 models are not stronger teachers on these tasks (Qwen3-8B: 43.07%; Qwen3-30B-A3B-Instruct-2507: 43.14%), so we distill Qwen3.5-9B (57.52%) into Qwen3.5-2B, with 4.5× fewer parameters:

MethodID Avg.OOD Avg.All Avg.Gap closed
Teacher (Qwen3.5-9B)59.3754.2857.52–
Student, untrained36.5022.6831.470%
OPD45.0832.5340.5134.7%
ATOD46.8333.8542.1140.8%
SCAD (Ours)52.4339.5947.7662.5%
  • SCAD improves the student by 16.29 points (31.47% → 47.76%) and is the best student on all 11 datasets. Its lead over ATOD grows from 4.48 to 5.65 points, consistent with RQ3: the larger the teacher–student gap, the more full-history supervision degrades relative to local supervision.
  • Action tasks (Table 6). With a 2×-larger Qwen3-8B teacher trained on each environment, the Qwen3-4B student reaches the teacher's level on ALFWorld / WebShop / AppWorld (91.6% vs. 89.4%; 73.6% vs. 72.8%; 40.5% vs. 39.3%) and leads ATOD by 9.1 / 5.2 / 5.4 points.

At last, we sincerely appreciate your valuable feedback and have updated the paper accordingly. If our responses address your concerns, we would be deeply grateful if you could kindly consider raising the score. Thank you very much!

Reviewer_wy2YOpenReview ↗

Response to the authors’ rebuttal

Thank you for the detailed response and the additional experiments, which address my questions about the evidence. Like Reviewer CaeY, I think the contribution comes mainly from combining existing components, and the gains from tree credit and signal allocation are modest. I will keep my positive score of 3.

AuthorsOpenReview ↗

Response to Reviewer wy2Y

Thank you again for your thoughtful follow-up and positive assessment. We are pleased that the additional experiments have addressed your questions about the evidence.

  1. SCAD’s central contribution is to make subtask structure determine how supervision is constructed and assigned: local contexts keep teacher feedback informative for execution, semantic subtask–report histories enable comparisons between planning decisions, and role-based allocation preserves execution guidance while retaining failure feedback for planning. These designs address the distinct learning needs of planning and execution, with three formal results grounding their behavior.

  2. Empirically, tree credit and signal allocation jointly add 2.35 points on text and 3.70 points on multimodal tasks, beyond local OPD within the same architecture, with both paired 95% confidence intervals above zero (Table 10). These gains require no learned critic or additional branch sampling, making the approach practical as well as effective.

We sincerely appreciate your careful review and constructive feedback throughout the discussion, which have helped us present SCAD’s contribution and supporting evidence more clearly.

Reviewer_dWufOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structured Credit Assignment and Distillation (SCAD), a training framework combining On-Policy Distillation and structured credit assignment. SCAD proposes three key designs: 1. subtask-local distillation, which only uses teacher guidance within isolated local contexts; 2. cross-rollout planning credit, which constructs the planning advantage by matching subtask-report prefixes in different rollouts and comparing the outcome of different planning decisions; 3. structured signal allocation, which assigns full GRPO advantages and prefix-tree credit to planning decisions, but only non-negative GRPO advantages and teacher feedback to execution decisions. Experiments show that SCAD outperforms RL and OPD baselines across 11 text-based and 8 multi-modal environments.

Critical strengths

Strong Empirical Evaluations. The paper evaluated SCAD on 19 different environments and compared with RL, OPD and hybrid baselines. The ablation studies and analysis of the three key components local distillation, prefix-tree credit and structured signal allocation provides empirical evidence to justify the effectiveness of SCAD.

Critical weaknesses
  1. SCAD mainly combines existing techniques such as Plan-and-Execute, OPD and cross-rollout credit assignment [1, 2], therefore the novelty is relatively limited. The choice of blocking negative execution advantage also appears heuristic and needs more justification. Currently it is unclear why the negative advantage needs to be blocked, or why not only use the teacher signal for execution.

  2. Although existing experiments cover a wide range of QA tasks, it is unclear how SCAD generalize to agentic action tasks such as ALFWorld, WebShop, AppWorld, etc.

[1] Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408. [2] He, Shuo, et al. "Hierarchy-of-groups policy optimization for long-horizon agentic tasks." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer dWuf (Part 1/2)

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns (Part 1/2). In the revised PDF, changes for you are in orange, changes shared with other reviewers in magenta, and the other reviewers' changes in blue and purple.


W1a: "SCAD mainly combines existing techniques such as Plan-and-Execute, OPD and cross-rollout credit assignment [1, 2], therefore the novelty is relatively limited."

Thank you for the thoughtful feedback. To clarify our contributions, we summarize them from five perspectives:

a. Motivation Level:

  • Combining RL and on-policy distillation (OPD) for long-horizon agents leaves three open problems that none of the listed techniques solves on its own: (i) teacher feedback loses discriminative value along long student-generated histories; (ii) a terminal reward cannot tell which planning decision mattered, and states rarely recur in open-ended tasks; (iii) uniform mixing lets failures suppress useful teacher guidance on correctly executed steps.
  • SCAD is not a stack of independent modules. One structure, the subtask and its report, answers all three: the subtask boundary defines the teacher's context, the subtask–report prefix the unit of cross-rollout comparison, and the planning/execution role the allocation of each signal.

b. Technical Level:

  • Subtask-local distillation is not Plan-and-Execute plus OPD. The key is where the teacher looks. Inside the identical Planner–Worker architecture, full-history distillation (ATOD) reaches 41.24% and local distillation 43.75% (Figure 6c); the teacher's preference margin for correct execution at depth 24 rises from 0.039 to 0.126 nats/token (Figure 6b).
  • Cross-rollout planning credit differs from GiGPO [1] and HGPO [2] in what is compared and how it is matched (both are now discussed in Related Work and listed in Table 1):
GiGPO [1]HGPO [2]SCAD (Ours)
Unit of comparisonSingle environment stepStep with its k-step historyPlanning decision (subtask)
Matching ruleExact state identityExact state and historySemantic subtask–report prefix
Needs recurring statesYesYesNo
Sparse groupsNo step-level signalAdaptive weightingPrior shrinkage (Eq. 7)
Credited tokensAll stepsAll stepsPlanning only
Teacher / distillationNoneNoneSubtask-local OPD
  • GiGPO and HGPO are designed for environments where states recur (ALFWorld, WebShop). In open-ended search, 98.2% of the planning decisions after the first one have no identical state in another rollout of the same question, so step-level groups reduce to singletons; semantic subtask–report prefixes give multi-candidate comparisons in 72.66% of trajectories.
  • We also ran both grouping rules inside our framework, changing only the planning credit (Appendix A.8, Table 14(a), summarized in Section 4.5; text, 3 seeds):
No cross-rollout creditPlanning advantage ×1.6GiGPO-style groupsHGPO-style groupsSCAD prefix tree
44.3244.6144.1844.8646.10
  • Structured signal allocation assigns terminal credit by role, which neither Plan-and-Execute, OPD, nor group-based credit assignment considers (see W1b).

c. Theoretical Level (full proofs in Appendix C):

  • Proposition 1: at fixed local contexts, the clipped teacher signal is a reverse-KL gradient approximation whose bias is bounded by the clipped log-ratio tails.
  • Proposition 2: tree credit removes exactly the within-choice return variance, and prior shrinkage shifts the mean without changing sibling contrasts.
  • Proposition 3: role-based allocation is the smallest change to uniform fusion under which negative terminal credit never overrides teacher feedback, and it equals a teacher-only update on failed trajectories.

d. Empirical Level:

  • SCAD improves over the strongest trained baselines by 4.48 points on 11 text benchmarks and 4.19 points on 8 multimodal benchmarks, by 4.26 points for a smaller 1.7B student, and by 5.2–9.1 points on three agentic action benchmarks (see W2).

e. Practical Level:

  • SCAD needs no process reward, no learned critic, and no extra branch sampling; all signals come from the rollouts that GRPO already collects.
  • Local contexts allow parallel teacher scoring: 60 steps take 4.92 hours vs. 7.93 for ATOD (37.96% less time, Figure 9b).

In summary, SCAD identifies why RL and OPD interfere on long horizons and shows that a single subtask–report structure resolves it: where the teacher looks, what is compared across rollouts, and who receives which credit. We hope this clarification helps highlight the significance of our contributions.

AuthorsOpenReview ↗

Response to Reviewer dWuf (Part 2/2)


W1b: "The choice of blocking negative execution advantage also appears heuristic ... it is unclear why the negative advantage needs to be blocked, or why not only use the teacher signal for execution."

Thank you for raising this point. The rule follows from a measured property of long-horizon rollouts; we expanded its justification in Section 3.3, Section 4.6 (RQ5), and Appendix A.8 (Table 14(b)).

Why block the negative terminal advantage for execution?

  • Terminal failure is an unreliable label for execution, while terminal success is a reliable one (Figure 8(a)):
Terminal outcomeSubtask correctSubtask wrongLabel precision
Pass (positive credit)30.79%4.97%86.1%
Fail (negative credit)21.52%42.72%66.5%

41.1% of all correctly executed subtasks lie in failed trajectories. A signed advantage would penalize each for a failure caused elsewhere (a poor decomposition, a premature stop, or another subtask).

  • Blocking routes the failure signal instead of discarding it. Planning keeps the signed terminal and tree credit, and execution errors are corrected by the signed teacher gap, which is local and per-token.
  • Direct comparison. Only the terminal credit of execution changes; planning is fixed (Figure 8(c) and Table 10; 3 seeds):
SettingSigned (+ and −)Positive onlyGain
Text44.9946.10+1.11
Multimodal31.0433.13+2.09

Positive-only credit is better in both modalities and in all three seeds of each.

  • It is principled. By Proposition 3, SCAD's execution advantage is the projection AS=max⁡(AU,δ)A_S=\max(A_U,\delta) of the uniform-fusion advantage AUA_U onto the set where the teacher signal δ\delta is never overridden, in the spirit of self-imitation learning (Oh et al., ICML 2018).

Why not use only the teacher signal for execution?

  • The teacher gap measures local preference, not whether execution helped solve the task. The environment reward does, and its positive part is a high-precision label (86.1%).
  • Controlled evidence (planning fixed, Figure 8(c); text, 3 seeds):
Execution signalAcc. (%)Gain
Teacher only44.73–
+ signed terminal credit44.99+0.26
+ positive terminal credit (SCAD)46.10+1.37

W2: "...it is unclear how SCAD generalize to agentic action tasks such as ALFWorld, WebShop, AppWorld, etc."

Thank you for this valuable suggestion. We evaluated SCAD on all three benchmarks you mention and added them as a new research question (Section 4.9, RQ8, Table 6; setup in Appendix A.7).

  • Setup. We keep the three designs and adapt only tools and prompts: the Planner delegates subgoals via branch, and each Worker acts in a local context and returns a report. The student is Qwen3-4B; each teacher is a Qwen3-8B model trained on that environment. OPD and ATOD share the student, teacher, interface, and rollout budget. All numbers use held-out test splits.

ALFWorld (success rate, %)

MethodSeen (140 games)Unseen (134 games)OverallΔ vs. SCAD
Teacher (Qwen3-8B)85.793.389.4-2.2
Student, untrained42.941.042.0-49.6
OPD82.977.680.3-11.3
ATOD83.681.382.5-9.1
SCAD (Ours)92.990.391.6–

WebShop (500 test tasks)

MethodSuccess rate (%)Task scoreΔ vs. SCAD
Teacher (Qwen3-8B)72.886.4-0.8
Student, untrained36.270.9-37.4
OPD66.281.6-7.4
ATOD68.483.7-5.2
SCAD (Ours)73.687.2–

AppWorld (168 test-normal tasks)

MethodTask goal completion (%)Tasks solvedΔ vs. SCAD
Teacher (Qwen3-8B)39.366 / 168-1.2
Student, untrained0.00 / 168-40.5
OPD34.558 / 168-6.0
ATOD35.159 / 168-5.4
SCAD (Ours)40.568 / 168–
  1. SCAD transfers to action tasks: success rises from 42.0% to 91.6% on ALFWorld, from 36.2% to 73.6% on WebShop, and from 0.0% to 40.5% on AppWorld.
  2. SCAD outperforms both baselines on all three benchmarks, by 9.1 / 5.2 / 5.4 points over ATOD and 11.3 / 7.4 / 6.0 over OPD, at least as large as on the QA benchmarks (4.48 / 4.19).
  3. The 4B student reaches the level of its 2×-larger teacher on all three (on ALFWorld, not significantly different under a paired McNemar test), while OPD and ATOD remain 4.2–9.1 points below it.

SCAD's three designs are thus not specific to question answering.


At last, we sincerely appreciate your valuable feedback and have updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Thank you for the detailed response. My concerns are largely resolved. I do appreciate the explanation on the contributions and the effort for additional experiments. I have raised my score to 3.

AuthorsOpenReview ↗

Response to Reviewer dWuf

Thank you very much for your thoughtful reassessment and positive feedback. Your constructive feedback has greatly helped us improve the quality of our paper. We sincerely appreciate your time and careful consideration of our responses and additional experiments.

Reviewer_CaeYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the training of small language-model agents (2–4B parameters) that answer hard factual questions by multi-step web search, where the only reliable feedback is whether the final answer is correct and a teacher model’s step-level guidance degrades over long student-generated histories. SCAD trains a shared Planner–Worker policy with three ingredients: on-policy distillation from a frozen teacher applied only to Worker tokens inside bounded subtask contexts, planning credit estimated from a prefix tree that pools terminal returns of rollouts sharing the same subtask–report history, and an allocation rule that gives planning tokens signed terminal plus tree credit while execution tokens receive only positive terminal credit plus the teacher signal. On 11 text and 8 multimodal search QA datasets, with Qwen3-4B and Qwen3-VL-2B students trained for 60 steps on 400/347 examples, SCAD reaches 46.1% and 33.1% macro accuracy, 4.5 and 4.2 points above the strongest reimplemented baselines (ATOD, HyperEyes), with ablations attributing most of the gain to the local distillation and about 2.4/3.7 points to the two credit components.

Critical strengths

A critical strength of this paper is the finding that teacher supervision for a search agent should be given inside bounded subtask contexts rather than over the full student history: the teacher’s preference margin rises from 0.039 to 0.126 nats/token at depth 24 when the context is local (Figure 6b), and local OPD beats ATOD’s full-history distillation by 2.5 points inside the same Planner–Worker framework, which isolates the effect from the architecture. This is a simple, transferable design rule for anyone distilling agents, and it accounts for most of the paper’s gain. It is established with great care: controlled comparisons within a fixed architecture, a ×1.6 advantage-scaling control showing that tree credit is not just a bigger planning advantage, a Full-E control isolating the allocation rule, a decomposition of tree credit into shrinkage and sibling terms (Table 5), three seeds with standard deviations in both modalities, a 1.7B replication, sensitivity analyses, teacher-token and wall-clock efficiency, and the full prompts, algorithms, baseline settings and anonymous code. Few agent-training papers separate architecture from training signal this cleanly, and the diagnosis in Section 4.4 (26% disagreement between subtask-level and terminal labels) gives a mechanistic reason for the design rather than only a leaderboard result.

Critical weaknesses

The local-distillation finding is very convincing, less so the two further components the paper presents as its credit-assignment contributions. Prefix-tree planning credit and role-based allocation contribute 1.2 and 1.1 points on text and 1.6 and 2.1 on multimodal, with seed standard deviations of 1–2.6 points, test sets of 814 and 510 questions whose macro-average sampling error is near 2 points, and no paired test, so the controls that isolate them are well designed but the effects they isolate are within the noise. The practical gain is also modest: the text student ends at 46.1% against 45.6% for its untrained teacher, an off-the-shelf model of the same size, and the baselines it beats are the authors’ reimplementations of recent methods under a 60-step budget at which SCAD is still improving while ATOD has plateaued (Figure 4).

Questions for authors

(1) Please report paired bootstrap intervals over test questions and seeds for the tree-credit and allocation increments in both modalities, and state in how many of the three seeds the ordering local OPD < +tree credit < +allocation holds. (2) The text student ends level with its untrained same-size teacher. With a stronger teacher (Qwen3-8B or 14B Instruct) at the same budget, does the student exceed the teacher and does the gap to ATOD grow? And does the gap persist at 120–200 training steps, where Figure 4 suggests ATOD has plateaued while SCAD has not? (3) Under plain GRPO the Planner–Worker architecture scores 20.0% on multimodal tasks, eight points below flat VSearcher (28.4%); why does the architecture hurt there, and does the local-OPD gain partly consist of recovering from it? Running TCOD and MemOPD inside the framework, as you did for ATOD, would help. (4) Are the validation questions used for checkpoint selection (including Bamboogle, PopQA and FVQA, which are labelled OOD) and the ID training questions disjoint from the test questions? (5) How do you position your work against the tree-based methods ARPO (Dong et al., ICLR 2026) and TreePO (Li et al., ICML 2026)?

Suggestion: state the teacher’s accuracy next to the student’s in the text and discuss what matching a same-size teacher means in practice.

4 comments
AuthorsOpenReview ↗

Response to Reviewer CaeY (Part 1/2)

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns (Part 1/2). In the revised PDF, changes for you are in blue, changes shared with other reviewers in magenta, and the other reviewers' changes in orange and purple.


W1 & Q1: "...the effects they isolate are within the noise." / "Please report paired bootstrap intervals ... and state in how many of the three seeds the ordering local OPD < +tree credit < +allocation holds."

  • Thank you for this constructive suggestion. We added the analysis as Appendix A.4 (Table 10) and summarize it in Section 4.3. All ablation stages share the checkpoint, training pool, seeds, and test questions, so we run a paired bootstrap (10,000 resamples over questions within each dataset and over seeds):
ModalityComponent addedGain (pp)95% intervalSeeds > 0
TextTree credit+1.24[-0.74, +3.28]3/3
TextSignal allocation+1.11[-0.82, +3.12]3/3
TextBoth+2.35[+0.08, +4.71]3/3
MultimodalTree credit+1.61[-1.49, +4.66]3/3
MultimodalSignal allocation+2.09[-0.63, +4.84]3/3
MultimodalBoth+3.70[+0.77, +6.74]3/3
  1. The joint contribution is significant in both modalities: both intervals exclude zero.
  2. Each single increment is consistent in direction: positive in all 12 seed-level comparisons, with every interval lying predominantly above zero. A single 1–2-point increment is modest relative to sampling noise with three seeds, as you note, so Section 4.3 states the claim for the two components jointly, where the evidence is strongest; Table 10 reports all intervals.
  3. The ordering Local OPD + GRPO < + tree credit < + allocation holds in 3/3 seeds for text and 3/3 seeds for multimodal tasks (per-seed accuracies in Table 10(b)).
  • Independent controls agree: the ×1.6 scaling control (44.61% vs. 46.10%, Figure 7b), and the new factorial ablation (Table 4), where removing tree credit / allocation from SCAD costs 1.78 / 1.11 points and tree credit helps without any distillation (+1.09).

W2 & Q2: "...the text student ends at 46.1% against 45.6% for its untrained teacher ... under a 60-step budget ..." / "With a stronger teacher ... does the student exceed the teacher and does the gap to ATOD grow? And does the gap persist at 120–200 training steps?"

a. "Same size" is not "same capability". The teacher is a later, stronger release: under the identical scaffold, the untrained student scores 38.91% vs. 45.56% for the teacher. With the same teacher, data, and budget, ATOD gains 2.38 points (39.24 → 41.62) and SCAD 7.19 (38.91 → 46.10).

b. Stronger teacher (Section 4.8, Table 5(b); per-dataset results in Table 12). Larger Qwen3 models are not stronger teachers on these tasks (Qwen3-8B: 43.07%; Qwen3-30B-A3B-Instruct-2507: 43.14%; both below 45.56%). We therefore distill Qwen3.5-9B (57.52%) into Qwen3.5-2B, with 4.5× fewer parameters and the same data, tools, and protocol:

MethodID Avg.OOD Avg.All Avg.Gap closed
Teacher (Qwen3.5-9B)59.3754.2857.52–
Student, untrained36.5022.6831.470%
OPD45.0832.5340.5134.7%
ATOD46.8333.8542.1140.8%
SCAD (Ours)52.4339.5947.7662.5%
  • Does the student exceed the teacher? No. With a 4.5×-larger teacher, the student rises from 31.47% to 47.76% (teacher: 57.52%). SCAD closes 62.5% of the initial gap (ATOD: 40.8%) and is the best student on all 11 datasets.
  • Does the gap to ATOD grow? Yes, from 4.48 to 5.65 points. This matches RQ3: the stronger the teacher relative to the student, the more full-history supervision loses to subtask-local supervision.

c. Longer training (Table 5(c)). Main setting, validation-selected checkpoint within each budget, same 3 seeds:

Steps60120160200
ATOD41.6242.0341.7942.15
SCAD (Ours)46.1046.4746.7146.58
  • Both methods have largely converged by 60 steps (largest change 0.53 / 0.61 points, within one seed standard deviation), and the gap persists at 4.43–4.92 points, so the 60-step budget favors neither method.

d. Action tasks with a larger teacher (Section 4.9, Table 6). On ALFWorld, WebShop, and AppWorld, with a 2×-larger Qwen3-8B teacher trained on each environment, the Qwen3-4B student reaches the teacher's level (e.g., 91.6% vs. 89.4% on ALFWorld) and leads ATOD by 9.1 / 5.2 / 5.4 points.

e. On baseline reimplementation. All online methods share the student, teacher, training pool, tools, token budget, judge, and rollout budget; each baseline keeps its published objective and hyperparameters (Appendix F).

AuthorsOpenReview ↗

Response to Reviewer CaeY (Part 2/2)


Q3: "Under plain GRPO the Planner–Worker architecture scores 20.0% on multimodal tasks ... why does the architecture hurt there, and does the local-OPD gain partly consist of recovering from it? Running TCOD and MemOPD inside the framework ... would help."

  • Following your suggestion, we ran TCOD and MemOPD, together with OPD, ATOD, and HyperEyes, inside the Planner–Worker (P–W) framework (Table 5(a); Appendix A.5, Table 11). Multimodal overall accuracy (%):
GRPOTCODOPDATODMemOPDHyperEyesSCAD
19.9727.1827.3128.4127.9228.7633.13

All rows share the P–W interface, tools, budgets, student, and teacher, and keep their own supervision rules.

  • Why does the architecture hurt under plain GRPO? The starting point is not the cause: the untrained student scores 16.28% in the P–W scaffold vs. 17.18% in HyperEyes's single-agent scaffold. The cause is reward sparsity acting on two roles at once: 71.4% of multimodal rollout groups in the first 20 updates are all-failure and carry zero GRPO advantage (46.2% on text), so neither the Planner nor the Worker receives a signal. Once a teacher signal is available, every baseline reaches 27.18–28.76% in the same architecture.
  • Does the local-OPD gain partly consist of recovering from it? Yes, partly, and by design: local OPD supplies a dense execution signal exactly where terminal reward is silent; subtask boundaries make teacher feedback informative, and teacher feedback makes bounded execution trainable. But the gain is not only recovery: with the same teacher in the same framework, the five baselines stay below Local OPD + GRPO (29.43%) and 4.37–5.95 points below SCAD; tree credit and allocation account for 3.70 points of this margin (29.43% → 33.13%, Table 10).

Q4: "Are the validation questions ... and the ID training questions disjoint from the test questions?"

  • Yes. The training, validation, and test pools are mutually disjoint (now stated in Section 4.1 and Appendix A.9): they are drawn without replacement and partitioned by instance ID before any experiment (text: 400 training / 130 validation / 814 test questions; multimodal: 347 training and 510 test questions plus a separate validation pool), and after normalizing question strings no question appears in more than one pool.
  • On the OOD label. Bamboogle, PopQA, and FVQA contribute a few validation questions for checkpoint selection, disjoint from their test questions. We agree this is weaker than strict OOD: Tables 2, 3, and 7 now mark them separately (‡, validation-only), and Table 15 reports the strict-OOD average:
Strict OODTeacherBest baselineSCAD
Text: GAIA + xbench24.5523.49 (HiPER)26.42
Multimodal: InfoSeek + LiveVQA33.0231.33 (HyperEyes)37.20

The conclusions are unchanged: SCAD leads the best baseline by 2.93 (text) and 5.87 (multimodal) points.


Q5: "How do you position your work against the tree-based methods ARPO ... and TreePO ...?"

  • We added both to Related Work and Table 1. ARPO and TreePO, like Tree-GRPO, build trees on the sampling side; SCAD builds its tree on the estimation side:
ARPOTreePOSCAD (Ours)
Tree sourceEntropy-triggered branching after tool callsSegment-wise tree sampling with pruningPost hoc, from G independent rollouts
Extra branchesYesYesNo
MatchingExact token prefixExact segment prefixSemantic subtask–report prefix
Credited tokensTool-use stepsAll segmentsPlanning decisions only
TeacherNoneNoneSubtask-local distillation
  • In free-form search, independent rollouts almost never share an exact token prefix, so exact-prefix trees must be created by branching; SCAD recovers comparable prefixes from completed rollouts and handles sparsity with prior shrinkage. ARPO and TreePO are RL-only and credit every step or segment, whereas SCAD applies tree credit only to planning tokens. The two directions are complementary: sampling-side branching would give SCAD's tree more sibling comparisons (Tree-GRPO: 40.81% vs. 46.10% for SCAD, Table 2).

Suggestion: "State the teacher's accuracy next to the student's in the text and discuss what matching a same-size teacher means in practice."

  • Section 4.2 now reads "SCAD reaches 46.10% text (teacher: 45.56%) and 33.13% multimodal (teacher: 30.90%) overall macro accuracy", and discusses the same-size setting as a controlled test of how much teacher competence each training signal transfers (2.38 points for ATOD vs. 7.19 for SCAD).

At last, we sincerely appreciate your valuable feedback and have updated the paper accordingly. If our responses address your concerns, we would be deeply grateful if you could kindly consider raising the score. Thank you very much!

Reviewer_CaeYOpenReview ↗

Reply to the rebuttal

Thank you for the careful revision. It largely answers my questions about the evidence, and I appreciate the candid statement that tree credit and allocation are resolved jointly rather than individually. I still don’t feel comfortable raising my score though, because like the other reviewers I felt the novelty is limited. I agree with Reviewer dWuf that the paper mainly combines existing techniques. Looking closer, I find that each of its three components has a close precedent, so its significance relies on how compelling the specific combination you tested is. To me the following two questions seem most useful to address this point, but feel free to make the best case for this combination from your own perspective:

  1. Your reply to Reviewer dWuf separates the tree from GiGPO and HGPO by semantic matching and prior shrinkage. RTMC (Wang et al., 2026), which you cite, already matches rollouts on abstracted state signatures instead of identical states and smooths sparse values toward the group success rate. Is it correct that what is specific to SCAD is embedding-based matching at subtask level and planning-only credit? If so, which result shows that this part is what makes the tree work? (RTMC is an arXiv-only preprint, so I ask this to clarify the positioning but do not count it against the paper.)

  2. Why is a separate planning credit needed? What share of failed tasks fail through planning rather than execution, and why does the teacher not supervise planning, whose context is short?

In summary, I think the score is partly a matter of taste regarding significance. I think it’s neither a 2 nor a 4, but your revision definitely improved the paper and raised my score a bit after the decimal point; I am not sure I can make this any more objective. I probably feel similarly to Reviewer wy2Y, who also mentioned the novelty concern but placed less emphasis on it than Reviewer dWuf and chose a score of 3.

For the final version, I would also suggest stating the number of seeds for Tables 5, 6, 11 and 12, mentioning that larger Qwen3 models were not stronger teachers, reporting the paired interval for local OPD + GRPO versus ATOD inside the architecture, and adding an SFT baseline to the stronger-teacher setting.

AuthorsOpenReview ↗

Response to Reviewer CaeY

Thank you again for your thoughtful follow-up and careful consideration of our revisions. We sincerely appreciate the opportunity to explain the contribution and motivation of SCAD.


Q1: What distinguishes SCAD from RTMC, and which results support its tree design?

SCAD makes subtask intent and returned evidence the basis for learning planning decisions. RTMC estimates action-step advantages from state–action signatures of tool interactions. SCAD reconstructs semantic subtask–report histories: embedding matching aligns subtask intents, while report matching preserves the evidence available to the Planner. The tree compares alternative subtask choices under these histories and assigns credit to the planning decisions. The Worker learns to execute each chosen subtask through local teacher feedback. The subtask structure thus determines both what is compared for planning and where the teacher supervises execution.

This structure produces effective planning comparisons and measurable gains. Matched prefixes enable comparisons between planning choices in 72.66% of trajectories. Planner and Worker settings are explicit below (Figs. 7 and 8c; Tables 8 and 14a). All Planners retain signed GRPO; positive Worker GRPO retains only positive terminal advantages.

Planner creditWorker supervisionAccuracy (%)
GRPO only (no tree)Positive GRPO + local OPD44.32
Uniform planning-advantage scalingPositive GRPO + local OPD44.61
Exact-state groupingPositive GRPO + local OPD44.18
Exact-history groupingPositive GRPO + local OPD44.86
Sibling comparisons onlyPositive GRPO + local OPD45.21
GRPO + full treeSigned GRPO + local OPD44.99
GRPO + full tree (SCAD)Positive GRPO + local OPD46.10

44.32 → 46.10 adds tree credit with the Worker rule fixed; 44.99 → 46.10 changes Worker allocation with Planner credit fixed.

The design also has theoretical and practical value: Proposition 2 establishes removal of within-choice return variance before scaling and clipping, and the tree reuses completed rollouts with no learned critic or extra branch sampling.


Q2: Why separate planning credit, and why use outcome supervision for planning despite its short context?

Planning learns which subtask contributes to task success; execution learns how to carry it out correctly. With Worker = positive GRPO + local OPD, GRPO already trains planning effectively (44.32% accuracy); tree credit raises accuracy to 46.10% (Fig. 7).

The failure analysis explains why these roles benefit from different signals. Within failed trajectories, 33.5% of subtasks are correctly executed (Fig. 8a). A negative terminal advantage can suppress teacher guidance for these correct local executions. SCAD preserves signed outcome credit for planning, while execution receives local teacher corrections and positive terminal credit. This preserves useful execution guidance while retaining failure feedback for planning; Proposition 3 establishes that the execution allocation prevents negative terminal credit from overriding the teacher signal.

For planning, a short context does not remove the potential conflict between teacher imitation and task outcomes. On a failed trajectory with negative terminal advantage, teacher imitation may still encourage the planning tokens and offset the negative outcome signal. SCAD therefore learns planning directly from task outcomes. With Worker = signed GRPO + local OPD fixed, GRPO + tree planning achieves 44.99%, compared with 44.78% after adding Planner OPD (Fig. 8c). These results support training planning with task rewards and execution with local teacher guidance.

We have also incorporated your reporting suggestions: 3 seeds in Tables 5/11/12 and 1 in Table 6.

Main setting: Qwen3-4B student. Larger Qwen3 models were not stronger teachers:

TeacherAccuracy (%)
Qwen3-4B-Instruct-250745.56
Qwen3-8B43.07
Qwen3-30B-A3B-Instruct-250743.14

Stronger-teacher setting: Qwen3.5-9B → Qwen3.5-2B.

Model / methodAccuracy (%)
Teacher (Qwen3.5-9B)57.52
Student (Qwen3.5-2B), untrained31.47
Student + SFT32.64
Student + SCAD47.76

For Local OPD + GRPO versus ATOD within the same architecture (Appendix A.5):

SettingGain (pp)Paired 95% interval
Text+2.51[+0.17, +4.91]
Multimodal+1.02[-2.30, +4.32]

Finally, we sincerely appreciate your time, thoughtful insights, and constructive feedback again. Your comments have helped us communicate the motivation and contribution of SCAD more clearly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you very much for your time and constructive feedback. We appreciate your recognition of the "simple, transferable design rule" for local distillation (CaeY), the "Strong Empirical Evaluations" (dWuf), and the "clearly motivated" approach (wy2Y).

We have clarified the method and expanded the evaluation, and kindly invite you to consult the revised PDF. Changes are marked in blue for CaeY, orange for dWuf, purple for wy2Y, and magenta for shared concerns.

Summary of our revisions, with each concern, the change, and where to find it:

Concern / changeWhere in the revised paperReviewer(s)
Contribution and positioning: clarified how subtask structure defines local supervision, semantic planning comparisons, and role-based credit.Secs. 2-3; Table 1; App. A.8All
Component contributions: added full factorial ablations and paired analyses of tree credit and signal allocation.Sec. 4.3; Tables 4, 10CaeY, wy2Y
Architecture control: compared key baselines within the same Planner-Worker architecture.Sec. 4.8; Tables 5(a), 11CaeY, wy2Y
Teacher and training budget: added stronger-teacher and SFT comparisons, a self-teacher control, and training up to 200 steps.Sec. 4.8; Tables 5(b-c), 12-13CaeY, wy2Y
Signal allocation: expanded the negative-credit justification and teacher-only execution / Planner OPD comparisons.Secs. 3.3, 4.6; Fig. 8; App. A.8CaeY, dWuf
Action tasks: added ALFWorld, WebShop, and AppWorld evaluations.Sec. 4.9; Table 6dWuf
Evaluation protocol: clarified seed counts, disjoint data splits, and strict-OOD evaluation.Sec. 4.1; table captions; App. A.9CaeY

The individual response threads provide the detailed evidence. Your constructive feedback has greatly improved our paper. Thank you again for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.