- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SCAD, a training framework for long-horizon search agents that combines RL with on-policy distillation. A shared policy acts as a Planner that delegates subtasks and as a Worker that executes each subtask in a fresh local context. SCAD distills Worker behavior from a frozen teacher under the same local context, assigns Planner credit by comparing terminal returns of sibling choices in cross-rollout subtask–report prefix trees, and gives Workers only positive terminal credit plus the teacher signal. On 11 text and 8 multimodal search benchmarks, SCAD improves macro-average accuracy over the strongest baselines by 4.48 and 4.19 points, supported by ablations, component analyses, and efficiency measurements.
Critical strengths
-
The paper is clearly motivated, with three identified challenges mapped one-to-one to three design components, and Table 1 positions the method well against related work.
-
Scoring Worker tokens under the student's own local context is a simple and practical way to keep teacher supervision informative.
-
The experiments support the paper's main claims, with a comprehensive benchmark selection covering 19 text and multimodal datasets and a wide range of RL, distillation, and hybrid baselines.
Critical weaknesses
-
Most of the improvement comes from local OPD, which accounts for 62.2% (text) and 71.9% (multimodal) of the cumulative gain. Tree credit and structured allocation each add 1.1–2.1 points, comparable to the reported seed standard deviations of about ±1.0–1.5.
-
The ablation adds components in a fixed order on top of local OPD. It does not show whether tree credit or signal allocation still helps without OPD, nor the effect of removing each component from the full model.
-
Many baselines use flat interaction structures, while SCAD uses the Planner–Worker architecture. Local GRPO within this architecture already reaches 39.88% on text, above flat GRPO (36.34%), OPD (38.06%), and SFT (38.79%), so part of the main-table gain comes from the architecture itself rather than the proposed training signals.
-
The individual components are closely related to existing work: Planner–Worker context isolation (AgentFlow, Context Folding, CoDA), OPD, prefix or shared-state grouping (GiGPO, Tree-GRPO), and outcome-dependent distillation (VOLD and HyperEyes apply OPD to failed rollouts). The main contribution lies in combining these components.
Questions for authors
-
Could you report leave-one-out ablations from the full model, as well as results for tree credit and signal allocation without local OPD? This would clarify whether these two components contribute independently of distillation.
-
Since part of the gain over flat baselines comes from the Planner–Worker architecture, could you compare key baselines (e.g., GRPO, OPD, and ATOD) within the same architecture, as done for ATOD in RQ3?
-
In the text setting, the teacher (Qwen3-4B-Instruct-2507) has the same size as the student, and the trained student surpasses it (46.10 vs. 45.56). How much of the OPD gain comes from the teacher's knowledge rather than from regularization, and how does SCAD behave with a stronger teacher?
4 comments
Response to Reviewer wy2Y (Part 2/2)
W4: "The individual components are closely related to existing work ... The main contribution lies in combining these components."
Thank you for the thoughtful feedback. We clarify what is new from five perspectives:
a. Motivation Level: Combining RL and on-policy distillation (OPD) for long-horizon agents leaves three open problems that none of the listed works solves on its own: (i) teacher feedback loses discriminative value along long student-generated histories; (ii) a terminal reward cannot tell which planning decision mattered; (iii) uniform mixing lets failures suppress useful teacher guidance. SCAD answers all three with one structure, the subtask and its report: its boundary defines the teacher's context, its prefix the unit of cross-rollout comparison, and its role the allocation of each signal.
b. Technical Level: each design differs from its closest prior work in a testable way:
| SCAD design | Closest prior work | What SCAD does differently | Controlled evidence |
|---|---|---|---|
| Subtask-local distillation | AgentFlow, Context Folding, CoDA bound the agent's context | The teacher scores execution under the student's bounded local context | Teacher margin at depth 24: 0.039 → 0.126 nats/token; +2.51 over full-history ATOD in the same architecture (Figure 6) |
| Prefix-tree planning credit | GiGPO needs identical states; Tree-GRPO branches at sampling time | Recovers prefixes post hoc by semantic matching; credits planning only | GiGPO-style groups in our framework: 44.18% vs. 46.10% (Table 14) |
| Structured signal allocation | VOLD, HyperEyes gate the teacher by outcome | Gates the negative terminal credit by role; the teacher stays on | +1.11 over signed credit (Figure 8c); HyperEyes in our framework: 28.76% vs. 33.13% (Table 11) |
c. Theoretical Level: each design is supported by a proposition with a full proof (Appendix C): a reverse-KL gradient bound for local distillation, exact removal of within-choice return variance for tree credit, and minimality of role-based allocation.
d. Empirical Level: every design matters in the full model: removing local OPD, tree credit, or allocation costs 4.52 / 1.78 / 1.11 points (Table 4). SCAD leads the strongest baselines by 4.48 / 4.19 points in the main tables, by 4.23 / 4.37 points inside the same architecture (Table 5(a)), and by 5.2–9.1 points over ATOD on three agentic action benchmarks (Table 6).
e. Practical Level: SCAD needs no process reward, learned critic, or extra branch sampling, and takes 37.96% less training time than ATOD (Figure 9b).
Q3: "How much of the OPD gain comes from the teacher's knowledge rather than from regularization, and how does SCAD behave with a stronger teacher?"
(1) Teacher knowledge vs. regularization.
- "Same size" is not "same capability": the teacher is a later, stronger release; under the identical scaffold, the untrained student scores 38.91% vs. 45.56% for the teacher.
- Self-teacher control (Table 13; text, 3 seeds). We replace the teacher with a frozen copy of the initial student, which keeps the form of the signal but adds no knowledge:
| Teacher used for local OPD | Teacher acc. (%) | Student acc. (%) |
|---|---|---|
| None (w/o local OPD) | – | 41.58 |
| Frozen initial student | 38.91 | 41.93 |
| Qwen3-4B-Instruct-2507 | 45.56 | 46.10 |
Regularization explains 0.35 of the 4.52 points: about 92% of the OPD gain comes from the teacher's knowledge.
(2) Behavior with a stronger teacher (Table 5(b); per-dataset results in Table 12). Larger Qwen3 models are not stronger teachers on these tasks (Qwen3-8B: 43.07%; Qwen3-30B-A3B-Instruct-2507: 43.14%), so we distill Qwen3.5-9B (57.52%) into Qwen3.5-2B, with 4.5× fewer parameters:
| Method | ID Avg. | OOD Avg. | All Avg. | Gap closed |
|---|---|---|---|---|
| Teacher (Qwen3.5-9B) | 59.37 | 54.28 | 57.52 | – |
| Student, untrained | 36.50 | 22.68 | 31.47 | 0% |
| OPD | 45.08 | 32.53 | 40.51 | 34.7% |
| ATOD | 46.83 | 33.85 | 42.11 | 40.8% |
| SCAD (Ours) | 52.43 | 39.59 | 47.76 | 62.5% |
- SCAD improves the student by 16.29 points (31.47% → 47.76%) and is the best student on all 11 datasets. Its lead over ATOD grows from 4.48 to 5.65 points, consistent with RQ3: the larger the teacher–student gap, the more full-history supervision degrades relative to local supervision.
- Action tasks (Table 6). With a 2×-larger Qwen3-8B teacher trained on each environment, the Qwen3-4B student reaches the teacher's level on ALFWorld / WebShop / AppWorld (91.6% vs. 89.4%; 73.6% vs. 72.8%; 40.5% vs. 39.3%) and leads ATOD by 9.1 / 5.2 / 5.4 points.
At last, we sincerely appreciate your valuable feedback and have updated the paper accordingly. If our responses address your concerns, we would be deeply grateful if you could kindly consider raising the score. Thank you very much!
Response to the authors’ rebuttal
Thank you for the detailed response and the additional experiments, which address my questions about the evidence. Like Reviewer CaeY, I think the contribution comes mainly from combining existing components, and the gains from tree credit and signal allocation are modest. I will keep my positive score of 3.
Response to Reviewer wy2Y
Thank you again for your thoughtful follow-up and positive assessment. We are pleased that the additional experiments have addressed your questions about the evidence.
-
SCAD’s central contribution is to make subtask structure determine how supervision is constructed and assigned: local contexts keep teacher feedback informative for execution, semantic subtask–report histories enable comparisons between planning decisions, and role-based allocation preserves execution guidance while retaining failure feedback for planning. These designs address the distinct learning needs of planning and execution, with three formal results grounding their behavior.
-
Empirically, tree credit and signal allocation jointly add 2.35 points on text and 3.70 points on multimodal tasks, beyond local OPD within the same architecture, with both paired 95% confidence intervals above zero (Table 10). These gains require no learned critic or additional branch sampling, making the approach practical as well as effective.
We sincerely appreciate your careful review and constructive feedback throughout the discussion, which have helped us present SCAD’s contribution and supporting evidence more clearly.
Response to Reviewer wy2Y (Part 1/2)
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns (Part 1/2). In the revised PDF, changes for you are in purple, changes shared with other reviewers in magenta, and the other reviewers' changes in blue and orange.
W1: "Most of the improvement comes from local OPD ... Tree credit and structured allocation each add 1.1–2.1 points, comparable to the reported seed standard deviations of about ±1.0–1.5."
W2 & Q1: "...whether tree credit or signal allocation still helps without OPD, nor the effect of removing each component from the full model."
W3 & Q2: "...part of the main-table gain comes from the architecture itself rather than the proposed training signals." / "Could you compare key baselines ... within the same architecture ...?"