- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Graph-Conditioned On-Policy Agent Distillation (GC-OPD) for multi-turn language agents. Repeated teacher executions are indexed through shared state locators while retaining each execution's complete provenance. After a student episode, the method retrieves successful or failed records from the current state or a prior state anchor and gives these records, together with student hindsight, to a frozen teacher that scores the student's original tokens. On ScienceWorld, ALFWorld, and WebShop, the reported means substantially improve over vanilla OPD; the teacher is not task-specifically optimized.
Critical strengths
-
Substantial practical gains without task-specific teacher optimization. Using the same frozen, off-the-shelf teacher, GC-OPD improves success from 24.70% to 48.78% on ScienceWorld (4B), 53.36% to 85.26% on ALFWorld Unseen, and 29.10% to 37.65% on WebShop. The ScienceWorld cost comparison also includes execution-library collection, making the reported advantage over GRPO teacher preparation meaningful. These results support a useful alternative to optimizing a large teacher before distillation.
-
Thoughtful treatment of execution history, supported by informative ablations. The method preserves complete source histories rather than assuming that matching physical states makes trajectory segments interchangeable. Historical anchors also provide references after students diverge from recorded successful paths. On ScienceWorld 1.7B, full GC-OPD reaches 46.18%, versus 31.64% for hindsight-only supervision and 43.68% without external failed records. These controls substantiate the value of external execution evidence beyond simply exposing the student’s outcome.
Critical weaknesses
The experiments do not sufficiently establish the distinctive contribution of graph-conditioned retrieval.** The paper discusses closely related context-conditioned and hindsight-based distillation approaches, including OPCD, OPID, Skill-SD, and DART-SD, but does not directly compare against them. Moreover, hindsight-only and minimum-prefix controls leave open whether simpler retrieval using the same execution library and matched context budgets would perform similarly. This gap materially weakens the attribution of the gains to the proposed state/history retrieval design and the assessment of its methodological novelty.
Questions for authors
- Could you provide a method-by-method comparison of the training information available to each baseline: external execution records, student hindsight, success/failure labels, privileged state descriptors, and planner/oracle data? Which comparisons isolate the proposed retrieval mechanism, and which compare systems with different supervision resources?
- GC-OPD uses an additional K16 execution library and longer teacher scoring contexts. Does its advantage persist when competing methods receive the same total budget for teacher inference and environment interaction, including preparation? Equal student epochs alone do not establish equal training resources.
- With the same K16 library, student hindsight, and teacher context budget, how does GC-OPD compare against task-level reference selection, longest-prefix matching, or observation-based semantic retrieval? The minimum-prefix control appears insufficient to establish that the gains require the proposed state/history retrieval design.
- Why are context-conditioned and hindsight-based approaches such as OPCD, Skill-SD, OPID, and DART-SD discussed but absent from the main comparisons? Could the closest applicable approach be evaluated using the same backbone, execution sources, and resource budget? Such a comparison would materially affect my assessment of the contribution beyond existing privileged-context distillation.
- How were baseline implementations validated after changing history construction, disabling native thinking mode, and imposing the shared parser? Have the reproductions recovered their expected performance under their original protocols, and were method-specific hyperparameters retuned under the adapted protocol? A common protocol improves consistency but may affect different methods unequally.
3 comments
Response to Reviewer 23dP (2/2)
Q4 — Closest applicable approach, under the same backbone, sources and budget
Why these methods are not in the main table. They target different settings and do not run unchanged in ours, which distills from a frozen off-the-shelf teacher in multi-turn environments. OPCD conditions a teacher on static task-level context for single-turn distillation. OPID and Skill-SD condition a self-teacher on hindsight skills, and OPID combines this with RL. DART-SD generates recovery responses for offline supervised training.
What we evaluated. OPID is the closest to our multi-turn setting. A full ScienceWorld reproduction of OPID, including its RL objective, was not feasible within the rebuttal period, so we reproduced its skill-extraction procedure and evaluated it under our setting: same student, same frozen 32B teacher, same K=16 library, same student hindsight and context cap. Following OPID, each trajectory is analyzed on its own and converted into one episode-level skill (a reusable workflow for a success, a mistake to avoid for a failure) and at most five step-level skills at critical steps chosen by the analyzer; a critical step uses its step-level skill and every other step the episode-level skill. OPID's RL objective is not used.
Result. OPID-style skills reach 39.42 ± 1.07% (14.52 rounds), versus 46.18 ± 1.13% (13.04 rounds) for GC-OPD: 6.76 points lower. ScienceWorld tasks are long and multi-step, and whether an action is right depends on concrete details that a summary drops; GC-OPD lets the teacher read the complete state-matched record instead.
On the analyzer. OPID extracts skills with a stronger external model (GLM-5.2 by default). We use the student itself as the analyzer, so that no method in this comparison uses a model beyond the student and the frozen 32B teacher. Giving the skill route a stronger model would not be like-for-like: GC-OPD could use the same model to collect more executions and enlarge the library. Base GC-OPD involves only the student and one off-the-shelf teacher: its library consists of that teacher's own executions, and it uses no stronger model, planner or oracle (only the GA variant adds planner records). This matches practice, where no planner is available and one often has to start from a relatively weak teacher.
Q5 — Validation of baseline implementations
Released implementations and hyperparameters. TCOD and FTB are run from their released code and ScienceWorld configurations, each keeping its original curriculum and loss; hyperparameters were not retuned. The planning-based variants use ScienceWorld's official gold-path planner. TurnOPD has no released implementation; we implement its adaptive rollout depth and progressive turn-normalized loss from the paper. Each curriculum baseline is trained for 190 student updates.
Protocol. Every ScienceWorld student receives five recent observation–action pairs with native thinking disabled; malformed or rejected actions consume a decision. FTB also disables native thinking in its main experiments; TCOD originally uses a two-step history and explicit thought tags. The protocol is identical for all methods, including vanilla OPD and GC-OPD.
Do the reproductions recover the expected behaviour? Absolute numbers depend on protocol and evaluation split (the same Qwen3-32B teacher scores 12.0% in FTB's setup and 29.74% in ours), so we compare each method with vanilla OPD within the same setting:
| Published source / our reproduction | Vanilla OPD | TCOD-F2B | TCOD-B2F | FTB |
|---|---|---|---|---|
| TCOD: ALFWorld 7B→3B, Unseen SR | 60.45 | 79.19 | 70.90 | – |
| FTB: ALFWorld 8B→4B, SR | 67.4 | 66.5 | 68.6 | 74.1 |
| Our ALFWorld GRPO 32B→1.7B, Unseen SR | 79.48 | 83.21 | 85.07 | 81.90 |
| FTB: ScienceWorld 32B→4B, SR | 7.3 | 11.3 | 9.5 | 14.0 |
| TCOD: ScienceWorld Qwen3-30B-A3B→4B, SR | 15.95 | 17.12–17.85 | 15.88–17.46 | – |
| Our ScienceWorld original 32B→4B, SR | 24.70 | 23.60 | 23.13 | 23.21 |
| Our ScienceWorld original 32B→1.7B, SR | 22.31 | 23.34 | 21.06 | 22.70 |
ALFWorld. The curriculum methods mostly improve over vanilla OPD in the original papers, and with the GRPO teacher the same implementations all improve in ours (+2.4 to +5.6 Unseen points).
After the table. The published papers themselves already report small or mixed effects on ScienceWorld: TCOD's own text calls its variants comparable to vanilla OPD (15.88–17.85 vs. 15.95 in their Table 3), and our results agree — with the original teacher, every baseline stays within 1.6 SR points of vanilla OPD.
Thank you for the additional experiments. The new retrieval controls are helpful, but two important concerns remain unresolved:
-
Why not use the same frozen 32B teacher to extract skills and include the extraction cost in the comparison? This would introduce no additional model and help distinguish the contribution of the retrieval mechanism from skill-extraction quality.
-
The response does not establish whether the baseline implementations recover their expected performance under their original protocols, or whether the protocol changes affect their effectiveness.
Given the late timing of the response and based on the evidence I have been able to assess and the unresolved concerns above, I will maintain my current score.
Response to Reviewer 23dP (1/2)
Thank you for the detailed questions.
Weakness — Distinctive contribution of graph-conditioned retrieval
Under the same K=16 library, student hindsight, frozen teacher and context cap, GC-OPD (46.18%) is above every simpler selection rule that we evaluated (Q3). It is also above the closest related approach, OPID-style skills, where the student model summarizes each trajectory into episode-level and step-level skills that are given to the same teacher (Q4). Details follow.
Q1 — Which information does each method receive?
✓ = available; * = planner/oracle records. All scoring teachers are frozen during student training.
The lower block isolates the retrieval mechanism: library, hindsight, outcome labels, teacher and context cap are identical (results in Q3 and Table R1 of our summary post). The upper block compares systems with different supervision resources. Base GC-OPD uses no planner or oracle records; TCOD-B2F* and FTB* also use privileged simulator access (replay, forking).
Q2 — Teacher inference, interaction and preparation cost
Same teacher rollouts. The K=16 library and GRPO teacher training use the same number of teacher rollouts: 48,272 (3,017 training tasks × 16). All five GRPO-teacher baselines in the main table (vanilla OPD, TCOD-B2F, TCOD-F2B, FTB, TurnOPD) receive this budget through their teacher; the best of them reaches 38.70% on the 1.7B student, versus 46.18% for GC-OPD.
GPU-hours. Including preparation, GC-OPD costs 281 GPU-hours versus 1,217 for the GRPO-teacher route, whose student training is the same vanilla OPD (47.31) preceded by 1,169 GPU-hours of teacher optimization. With a K=4 library GC-OPD still exceeds the GRPO-teacher route (40.56% vs. 38.70%) at under 10% of its cost.
Teacher input length. GC-OPD's teacher reads 3,485 tokens per scored decision versus 665 for vanilla OPD; this is included in the GPU-hours above. Length does not explain the gain: random same-task selection has the longest input (3,821 tokens) and is 8.70 points lower.
Q3 — Alternatives using the same sources and hindsight
These are the three alternatives named in the question. Task-level selection draws one successful and one failed complete record from the task at random, independently of the current state. Longest-prefix selects the records whose action sequence shares the longest prefix with the actions already taken by the student. Observation-based semantic retrieval keeps the graph and the retrieval procedure of GC-OPD but matches observation text by TF-IDF similarity instead of simulator state. A prefix breaks after one extra inspection even when both executions later reach the same state, and similar text can hide a closed versus an open drawer. Exact state matching handles both cases.
GC-OPD adds 8.70, 5.80 and 3.36 points over these three alternatives.