acceptodds
Under review as a conference paper at ICLR 2027

GRAPH-CONDITIONED ON-POLICY AGENT DISTILLATION FROM OFF-THE-SHELF TEACHERS

Abstract

On-policy distillation (OPD) trains compact language agents with teacher feedback on student-generated trajectories. In multi-turn tasks, compounding errors can move students beyond the teacher's effective supervision. We introduce Graph-Conditioned On-Policy Agent Distillation (GC-OPD), which enriches an off-the-shelf teacher's scoring context with execution evidence. A graph indexes repeated teacher executions by shared states while preserving complete successful and failed histories. After each student episode, GC-OPD retrieves current-state references or historical alternatives and combines them with student hindsight to score the original thought–action tokens. Using the same original teachers, GC-OPD improves mean success over vanilla OPD from 24.70% to 48.78% on ScienceWorld (4B student), from 53.36% to 85.26% on ALFWorld Unseen, and from 29.10% to 37.65% on WebShop. At matched student sizes, it also achieves higher mean success than every evaluated OPD baseline using GRPO-trained teachers on ScienceWorld and ALFWorld; the strongest such ScienceWorld 4B baseline reaches 46.66%. GC-OPD requires no task-specific teacher optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, LLM agents, multi-turn decision making, credit assignment, knowledge distillation, imitation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_23dPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Graph-Conditioned On-Policy Agent Distillation (GC-OPD) for multi-turn language agents. Repeated teacher executions are indexed through shared state locators while retaining each execution's complete provenance. After a student episode, the method retrieves successful or failed records from the current state or a prior state anchor and gives these records, together with student hindsight, to a frozen teacher that scores the student's original tokens. On ScienceWorld, ALFWorld, and WebShop, the reported means substantially improve over vanilla OPD; the teacher is not task-specifically optimized.

Critical strengths
  1. Substantial practical gains without task-specific teacher optimization. Using the same frozen, off-the-shelf teacher, GC-OPD improves success from 24.70% to 48.78% on ScienceWorld (4B), 53.36% to 85.26% on ALFWorld Unseen, and 29.10% to 37.65% on WebShop. The ScienceWorld cost comparison also includes execution-library collection, making the reported advantage over GRPO teacher preparation meaningful. These results support a useful alternative to optimizing a large teacher before distillation.

  2. Thoughtful treatment of execution history, supported by informative ablations. The method preserves complete source histories rather than assuming that matching physical states makes trajectory segments interchangeable. Historical anchors also provide references after students diverge from recorded successful paths. On ScienceWorld 1.7B, full GC-OPD reaches 46.18%, versus 31.64% for hindsight-only supervision and 43.68% without external failed records. These controls substantiate the value of external execution evidence beyond simply exposing the student’s outcome.

Critical weaknesses

The experiments do not sufficiently establish the distinctive contribution of graph-conditioned retrieval.** The paper discusses closely related context-conditioned and hindsight-based distillation approaches, including OPCD, OPID, Skill-SD, and DART-SD, but does not directly compare against them. Moreover, hindsight-only and minimum-prefix controls leave open whether simpler retrieval using the same execution library and matched context budgets would perform similarly. This gap materially weakens the attribution of the gains to the proposed state/history retrieval design and the assessment of its methodological novelty.

Questions for authors
  1. Could you provide a method-by-method comparison of the training information available to each baseline: external execution records, student hindsight, success/failure labels, privileged state descriptors, and planner/oracle data? Which comparisons isolate the proposed retrieval mechanism, and which compare systems with different supervision resources? 
  2. GC-OPD uses an additional K16 execution library and longer teacher scoring contexts. Does its advantage persist when competing methods receive the same total budget for teacher inference and environment interaction, including preparation? Equal student epochs alone do not establish equal training resources. 
  3. With the same K16 library, student hindsight, and teacher context budget, how does GC-OPD compare against task-level reference selection, longest-prefix matching, or observation-based semantic retrieval? The minimum-prefix control appears insufficient to establish that the gains require the proposed state/history retrieval design. 
  4. Why are context-conditioned and hindsight-based approaches such as OPCD, Skill-SD, OPID, and DART-SD discussed but absent from the main comparisons? Could the closest applicable approach be evaluated using the same backbone, execution sources, and resource budget? Such a comparison would materially affect my assessment of the contribution beyond existing privileged-context distillation. 
  5. How were baseline implementations validated after changing history construction, disabling native thinking mode, and imposing the shared parser? Have the reproductions recovered their expected performance under their original protocols, and were method-specific hyperparameters retuned under the adapted protocol? A common protocol improves consistency but may affect different methods unequally.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 23dP (1/2)

Thank you for the detailed questions.

Weakness — Distinctive contribution of graph-conditioned retrieval

Under the same K=16 library, student hindsight, frozen teacher and context cap, GC-OPD (46.18%) is above every simpler selection rule that we evaluated (Q3). It is also above the closest related approach, OPID-style skills, where the student model summarizes each trajectory into episode-level and step-level skills that are given to the same teacher (Q4). Details follow.

Q1 — Which information does each method receive?

✓ = available; * = planner/oracle records. All scoring teachers are frozen during student training.

MethodScoring teacherStudent hindsightExternal recordsOutcome labelsState descriptorEarlier-state anchorsPlanner/oracle
Vanilla OPD, TCOD-F2Boriginal / GRPO––––––
TurnOPDoriginal / GRPO––––––
TCOD-B2F*original / GRPO–planner prefixes✓replay to intermediate states–✓
FTB*original / GRPO–planner prefix + teacher bridge✓environment fork/reset–✓
GRPO teacher + OPDGRPO–48,272 rollouts, used for teacher RLtask reward–––
Hindsight onlyoriginal✓–✓–––
Random / min-prefix / longest-prefixoriginal✓K=16✓–––
OPID-style skillsoriginal✓K=16, as skills✓–––
Observation-based retrievaloriginal✓K=16✓public text only✓ (text)–
Current-state onlyoriginal✓K=16✓✓––
GC-OPDoriginal✓K=16 (48,272 executions)✓✓✓–
GC-OPD + GA*original✓K=16 + planner records✓✓✓✓

The lower block isolates the retrieval mechanism: library, hindsight, outcome labels, teacher and context cap are identical (results in Q3 and Table R1 of our summary post). The upper block compares systems with different supervision resources. Base GC-OPD uses no planner or oracle records; TCOD-B2F* and FTB* also use privileged simulator access (replay, forking).

Q2 — Teacher inference, interaction and preparation cost

RoutePreparationStudent trainingTotal GPU-hSR (%)
GC-OPD (K=16)212.4868.79281.2746.18
GC-OPD (K=4)53.1264.20117.3240.56
GRPO teacher + vanilla OPD1,169.2847.311,216.5938.70
Vanilla OPD047.3147.3122.31

Same teacher rollouts. The K=16 library and GRPO teacher training use the same number of teacher rollouts: 48,272 (3,017 training tasks × 16). All five GRPO-teacher baselines in the main table (vanilla OPD, TCOD-B2F, TCOD-F2B, FTB, TurnOPD) receive this budget through their teacher; the best of them reaches 38.70% on the 1.7B student, versus 46.18% for GC-OPD.

GPU-hours. Including preparation, GC-OPD costs 281 GPU-hours versus 1,217 for the GRPO-teacher route, whose student training is the same vanilla OPD (47.31) preceded by 1,169 GPU-hours of teacher optimization. With a K=4 library GC-OPD still exceeds the GRPO-teacher route (40.56% vs. 38.70%) at under 10% of its cost.

Teacher input length. GC-OPD's teacher reads 3,485 tokens per scored decision versus 665 for vanilla OPD; this is included in the GPU-hours above. Length does not explain the gain: random same-task selection has the longest input (3,821 tokens) and is 8.70 points lower.

Q3 — Alternatives using the same sources and hindsight

Source selectionTokensSR (%)Rounds
Random same-task pair3,82137.48 ± 0.7115.23
Longest shared action-prefix pair3,57440.38 ± 0.8513.76
Observation-based semantic retrieval3,34142.82 ± 0.6413.68
GC-OPD (graph + exact state matching)3,48546.18 ± 1.1313.04

These are the three alternatives named in the question. Task-level selection draws one successful and one failed complete record from the task at random, independently of the current state. Longest-prefix selects the records whose action sequence shares the longest prefix with the actions already taken by the student. Observation-based semantic retrieval keeps the graph and the retrieval procedure of GC-OPD but matches observation text by TF-IDF similarity instead of simulator state. A prefix breaks after one extra inspection even when both executions later reach the same state, and similar text can hide a closed versus an open drawer. Exact state matching handles both cases.

GC-OPD adds 8.70, 5.80 and 3.36 points over these three alternatives.

AuthorsOpenReview ↗

Response to Reviewer 23dP (2/2)

Q4 — Closest applicable approach, under the same backbone, sources and budget

Why these methods are not in the main table. They target different settings and do not run unchanged in ours, which distills from a frozen off-the-shelf teacher in multi-turn environments. OPCD conditions a teacher on static task-level context for single-turn distillation. OPID and Skill-SD condition a self-teacher on hindsight skills, and OPID combines this with RL. DART-SD generates recovery responses for offline supervised training.

What we evaluated. OPID is the closest to our multi-turn setting. A full ScienceWorld reproduction of OPID, including its RL objective, was not feasible within the rebuttal period, so we reproduced its skill-extraction procedure and evaluated it under our setting: same student, same frozen 32B teacher, same K=16 library, same student hindsight and context cap. Following OPID, each trajectory is analyzed on its own and converted into one episode-level skill (a reusable workflow for a success, a mistake to avoid for a failure) and at most five step-level skills at critical steps chosen by the analyzer; a critical step uses its step-level skill and every other step the episode-level skill. OPID's RL objective is not used.

Result. OPID-style skills reach 39.42 ± 1.07% (14.52 rounds), versus 46.18 ± 1.13% (13.04 rounds) for GC-OPD: 6.76 points lower. ScienceWorld tasks are long and multi-step, and whether an action is right depends on concrete details that a summary drops; GC-OPD lets the teacher read the complete state-matched record instead.

On the analyzer. OPID extracts skills with a stronger external model (GLM-5.2 by default). We use the student itself as the analyzer, so that no method in this comparison uses a model beyond the student and the frozen 32B teacher. Giving the skill route a stronger model would not be like-for-like: GC-OPD could use the same model to collect more executions and enlarge the library. Base GC-OPD involves only the student and one off-the-shelf teacher: its library consists of that teacher's own executions, and it uses no stronger model, planner or oracle (only the GA variant adds planner records). This matches practice, where no planner is available and one often has to start from a relatively weak teacher.

Q5 — Validation of baseline implementations

Released implementations and hyperparameters. TCOD and FTB are run from their released code and ScienceWorld configurations, each keeping its original curriculum and loss; hyperparameters were not retuned. The planning-based variants use ScienceWorld's official gold-path planner. TurnOPD has no released implementation; we implement its adaptive rollout depth and progressive turn-normalized loss from the paper. Each curriculum baseline is trained for 190 student updates.

Protocol. Every ScienceWorld student receives five recent observation–action pairs with native thinking disabled; malformed or rejected actions consume a decision. FTB also disables native thinking in its main experiments; TCOD originally uses a two-step history and explicit thought tags. The protocol is identical for all methods, including vanilla OPD and GC-OPD.

Do the reproductions recover the expected behaviour? Absolute numbers depend on protocol and evaluation split (the same Qwen3-32B teacher scores 12.0% in FTB's setup and 29.74% in ours), so we compare each method with vanilla OPD within the same setting:

Published source / our reproductionVanilla OPDTCOD-F2BTCOD-B2FFTB
TCOD: ALFWorld 7B→3B, Unseen SR60.4579.1970.90–
FTB: ALFWorld 8B→4B, SR67.466.568.674.1
Our ALFWorld GRPO 32B→1.7B, Unseen SR79.4883.2185.0781.90
FTB: ScienceWorld 32B→4B, SR7.311.39.514.0
TCOD: ScienceWorld Qwen3-30B-A3B→4B, SR15.9517.12–17.8515.88–17.46–
Our ScienceWorld original 32B→4B, SR24.7023.6023.1323.21
Our ScienceWorld original 32B→1.7B, SR22.3123.3421.0622.70

ALFWorld. The curriculum methods mostly improve over vanilla OPD in the original papers, and with the GRPO teacher the same implementations all improve in ours (+2.4 to +5.6 Unseen points).

After the table. The published papers themselves already report small or mixed effects on ScienceWorld: TCOD's own text calls its variants comparable to vanilla OPD (15.88–17.85 vs. 15.95 in their Table 3), and our results agree — with the original teacher, every baseline stays within 1.6 SR points of vanilla OPD.

Thank you for the additional experiments. The new retrieval controls are helpful, but two important concerns remain unresolved:

  1. Why not use the same frozen 32B teacher to extract skills and include the extraction cost in the comparison? This would introduce no additional model and help distinguish the contribution of the retrieval mechanism from skill-extraction quality.

  2. The response does not establish whether the baseline implementations recover their expected performance under their original protocols, or whether the protocol changes affect their effectiveness.

Given the late timing of the response and based on the evidence I have been able to assess and the unresolved concerns above, I will maintain my current score.

Reviewer_kTjuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Graph-Conditioned On-Policy Agent Distillation (GC-OPD), a method for improving agent distillation using execution experience collected from an off-the-shelf teacher without further optimizing the teacher's parameters.

The method constructs an execution graph that links state visits across recorded trajectories while preserving each trajectory's identity, history, and outcome. During student training, the graph retrieves relevant successful execution records at the student's current state or, when eligible successful support is unavailable, at earlier visited states. Remaining interaction cost is used to rank associated successful continuations. The selected records and the student's hindsight are then provided to a frozen teacher, which scores the student's original thought–action tokens. Only the student is updated. Training first uses vanilla OPD and then continues with graph-conditioned feedback.

The authors evaluate GC-OPD in ScienceWorld, ALFWorld, and WebShop. They report improvements over vanilla OPD in all three environments and higher mean success than the evaluated GRPO-teacher OPD baselines on ScienceWorld and ALFWorld at matched student sizes. The paper also studies the contribution of execution evidence and reports computational costs for selected training configurations.

Critical strengths
  1. A practical approach to teacher preparation. GC-OPD uses an off-the-shelf teacher and keeps it frozen during student training. This avoids the additional teacher-side optimization stage and its associated memory and computation requirements.

  2. Retrieval preserves trajectory context. The execution graph links shared states across recorded executions while retaining source identities and complete histories. The method can retrieve historical alternatives when no eligible successful record matches the student's current state, addressing an important challenge in multi-turn interaction.

  3. The authors report improvements over vanilla OPD in ScienceWorld, ALFWorld, and WebShop. For example, the reported success rate for the 4B student in ScienceWorld increases from 24.70% with vanilla OPD to 48.78% with GC-OPD. The ablation comparing successful records alone with successful and failed records also provides evidence that execution evidence can affect student performance.

Critical weaknesses
  1. The contribution of individual components is not fully isolated. GC-OPD combines state-based graph associations, source-preserving history retrieval, successful and failed execution evidence, and student hindsight. The paper includes useful ablations, including a comparison between success-only records and success-plus-failure records. However, the reported results do not fully establish how much of the overall improvement comes specifically from graph-based retrieval rather than from adding relevant execution context and hindsight to teacher scoring. More controlled comparisons among retrieval strategies would strengthen the attribution of gains.

  2. The reported variability does not establish robustness across training runs. The main results report means and standard deviations over four inference seeds for each trained model. This measures evaluation variability for a fixed trained model, not variability across independent training runs. Consequently, the results do not establish how consistently GC-OPD improves performance under different training seeds. This is particularly relevant when interpreting relatively small differences between methods.

  3. The use of environment-specific state descriptors limits evidence for generality. The retrieval mechanism depends on state locators tailored to each environment. The paper describes different state information for ScienceWorld, ALFWorld, and WebShop. These choices may be appropriate for the evaluated tasks, but it remains unclear how much engineering is required to construct reliable locators in a new environment and how sensitive the method is to imperfect state matching. The experiments demonstrate effectiveness in the selected environments, but do not by themselves establish broad applicability to arbitrary agentic tasks.

  4. Comparisons involving additional execution resources need clearer interpretation. The main-results table marks certain configurations in which planner/oracle records are used during training. These resources may provide useful supervision, but they make it important to distinguish gains from the GC-OPD procedure itself from gains enabled by the additional source of execution information. The paper reports a computational-cost comparison for a particular ScienceWorld setup; further clarification of resource accounting and the exact information available to each compared method would help readers assess the fairness and practical implications of the comparisons.

Questions for authors
  1. Contribution of graph-based retrieval: Can the authors provide controlled comparisons between the proposed retrieval procedure and simpler alternatives, such as selecting a successful record from the same task without graph-based state matching? How much of the gain depends on retrieving records from earlier student states when no eligible successful reference is available at the current state?

  2. Robustness across training runs: Can the authors report results from multiple independent student-training seeds, with variability across training runs in addition to the four inference seeds? This would help establish whether the improvements are robust to training randomness.

  3. Sensitivity to state matching: How sensitive is GC-OPD to the design and precision of the environment-specific state descriptors? In particular, can approximate or imperfect state matches retrieve misleading records, and how does this affect student performance?

  4. Resource fairness and reproducibility: For comparisons that use planner/oracle records, could the authors clarify exactly how these records are generated and which methods receive access to them? Could the authors also provide a more detailed breakdown of source collection, teacher scoring, and student-training costs for the compared configurations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer kTju

Thank you for the constructive questions. We respond with the requested selection controls, independent training runs and resource accounting.

W1 / Q1 — Contribution of graph-based retrieval

Common resources are the K=16 library, original frozen scorer, K=16 execution-index structure, full student hindsight and context cap. Below we vary what is used for matching and which sources are supplied. Hindsight records the student's own completed episode; complete sources include full action/observation histories and outcomes.

Scoring evidenceTokensSR (%)Rounds
Hindsight only2,10031.64 ± 0.5713.83
Random same-task success/failure pair3,82137.48 ± 0.7115.23
Longest-action-prefix pair3,57440.38 ± 0.8513.76
Observation-based retrieval (graph + TF-IDF text matching)3,34142.82 ± 0.6413.68
GC-OPD (graph + exact state matching)3,48546.18 ± 1.1313.04

Observation-based retrieval keeps the execution graph but replaces simulator descriptors with public observation/action windows. It finds exact matches or TF-IDF similarity ≥0.65, first at the current visit, then at previous student visits with short unanchored fallback if necessary. Thus the 42.82% result shows that graph structure remains useful without privileged descriptors, while richer state matching is better.

Current-state-only, without historical anchors or unaligned fallback, reaches 38.50%, 7.68 points below the full method. Among 42,537 decisions, 16.23% receive a current-state successful reference and a further 25.82% receive one through an earlier student-state anchor.

W2 / Q2 — Different training seeds

The purpose is to test training variability, not repeat inference on one checkpoint. Each cell below is one independently trained ScienceWorld 1.7B student evaluated with four inference seeds on 1,661 test tasks; the last column is the sample SD across the four trained students.

MethodPaper (42)434445Mean ± SD
GC-OPD (original 32B)46.1847.4545.6147.2346.62 ± 0.87
Vanilla OPD (GRPO 32B)38.7036.2639.4335.7337.53 ± 1.81
Vanilla OPD (original 32B)22.3121.4926.2222.7623.20 ± 2.08

The lowest GC-OPD student (45.61) exceeds every baseline student; its training-run SD is lower than either baseline.

W3 / Q3 — Descriptor precision and imperfect matches

Replacing the exact state locator with approximate text matching costs only 3.36 points. We keep the full graph and the current-then-earlier retrieval procedure, replacing only the matching signal (simulator state → public observation text), and still reach 42.82% — 11.18 points above hindsight-only and 4.12 points above vanilla OPD with a GRPO-trained teacher (38.70%). The gain therefore comes from associations on the graph, not from access to the simulator's full internal state: when internal state is unavailable in a new environment, public observations come close to exact matching.

Do imperfect matches mislead? Yes in principle, but the teacher only re-scores the student's original tokens; it never executes a retrieved action nor rewrites the response. Full source history, outcome, match label and the student's own episode let the teacher judge relevance rather than imitate. Our ScienceWorld case study shows this: for the rejected look in drawer, adding a record that opened the drawer first drops the teacher's probability on look from 97.69% to 26.81%.

Implementation needs object facts, focus/goal flags and parser state in ScienceWorld; ALFWorld supplies grounded PDDL facts and WebShop requires page/item/option fields. Observation-based retrieval reaches 42.82% without internal state. No locator, graph or teacher is needed at deployment.

W4 / Q4 — Sources, planner use and cost

MethodPlanner/oracleSR (%)Student training GPU-h
Vanilla OPDNo22.3147.31
TCOD-F2BNo prefix23.3429.94
TCOD-B2FYes21.0645.58
FTBYes22.7058.18
TurnOPDNo23.8434.81
GC-OPDNo46.1868.79
GC-OPD + GAYes53.6166.52

Without planner data GC-OPD is strongest; GA also leads the planner-based methods. ScienceWorld uses its gold-path planner; ALFWorld uses replay-verified TextWorld plans; WebShop uses its training-product oracle. GA executes and indexes these sources.

For base GC-OPD, K=16 collection is 212.48 GPU-h and student training including scoring is 68.79, totaling 281.27. Teacher scoring reads 3,485 versus 665 tokens per decision for vanilla. Scoring and optimization were timed together; total GRPO-teacher plus OPD cost is 1,216.59 GPU-h. Preparation and training are reported separately above rather than allocating the training total artificially.

Reviewer_pfPrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GC-OPD is a two-stage method for distilling small LLM agents from a frozen, off-the-shelf teacher. It first runs one epoch of vanilla OPD. Then, after each student episode, the teacher re-scores the student's tokens with extra context. This context holds the full student episode with its feedback and outcome. It also holds up to one successful and one failed teacher run of the same task. These runs come from repeated teacher attempts on training tasks. They are matched to the student by simulator state, at the current or an earlier step. Against vanilla OPD with the same teacher, ScienceWorld success rises from 22.31 to 46.18 at 1.7B and from 24.70 to 48.78 at 4B. ALFWorld Unseen goes from 53.36 to 85.26, and WebShop from 29.10 to 37.65.

Critical strengths
  1. Holding the teacher fixed, GC-OPD beats vanilla OPD by a wide margin in every environment. On ScienceWorld the gain is about 24 points at both sizes (Table 1), far larger than the spread across inference seeds. At deployment, the student runs without the teacher or the graph. I also checked the ScienceWorld code. Its splits are disjoint, and checkpoints are picked on dev only.

  2. The paper tests a practical question: can a frozen teacher's own repeated runs replace RL-training the teacher? On ScienceWorld 1.7B, GC-OPD gets 46.18 using 281 GPU-hours. OPD with a GRPO-trained teacher gets 38.70 using 1,217. More teacher runs per task help steadily (K=4/8/16 give 40.56/42.20/46.18), and all of these beat hindsight alone (31.64).

Critical weaknesses

Choosing teacher runs by current and past simulator state is what separates GC-OPD from earlier context-conditioned OPD. Few experiments test this choice. The only control, minimum-prefix (33.70), is specified in neither the appendix nor the code. It even scores below the K=1 run (38.06), so it may be a weak control. No run removes the matches at earlier steps, though they supply more successful references than current-state matches. A run with random same-task teacher runs plus hindsight would show whether state matching matters.

Questions for authors
  1. In 13 of 16 ScienceWorld SR cells, the re-implemented baselines fall below vanilla OPD with the same teacher. What tuning did they get?
  2. How were the GRPO teachers trained? Without a dev split, how were ALFWorld checkpoints chosen?
1 comment
AuthorsOpenReview ↗

Response to Reviewer pfPr

Thank you for the constructive questions. The table below contains the requested controls; the full set is Table R1 in our summary post.

W1 — What is minimum-prefix, and does selection matter?

For each student episode, minimum-prefix selects the successful and failed complete records from that task's K=16 pool that share the shortest action prefix with the student. It uses this pair throughout the episode alongside hindsight. It is deliberately least-aligned, not random. K=1 still uses state-based selection on its only candidate, which may align with the student; minimum-prefix instead chooses the least related candidate among 16. Hence 33.70% below K=1's 38.06% is not contradictory. The appendix will define the selector and include its code.

Requested comparisonSR (%)Rounds
Hindsight only31.64 ± 0.5713.83
Minimum-prefix33.70 ± 0.6316.76
Random same-task pair37.48 ± 0.7115.23
Current-state only38.50 ± 0.6313.05
Full GC-OPD46.18 ± 1.1313.04

Random same-task records. For each episode, one successful and one failed complete record are uniformly sampled from the same task's K=16 pool when available; neither uses state matching. The fixed pair accompanies the student's complete hindsight for all decisions. Evidence comes from the right task, but is not selected for the student's current situation. Its 37.48% improves on hindsight alone, while GC-OPD adds 8.70 points and uses shorter teacher inputs (3,485 vs. 3,821 tokens on 3,383 common decisions).

Removing earlier-state matches. Current-state-only disables both earlier-state anchors and unaligned fallbacks. It reaches 38.50%, 7.68 points below the full method. Earlier-state anchors supply aligned successful references in 25.82% of decisions, beyond the 16.23% from current states.

Q1 — How were the baselines tuned?

TCOD and FTB use the released ScienceWorld configurations and original losses. Each curriculum baseline receives 190 student updates under the shared protocol. TurnOPD is implemented from its published adaptive-depth and turn-normalized objective.

Two caveats before reading the numbers. First, the published ScienceWorld numbers are not directly comparable with ours: the teachers there reach only 18.42% (TCOD) and 12.0% (FTB), even though FTB's teacher is the same Qwen3-32B that scores 29.74% under our protocol, and the evaluation splits differ. We therefore compare each method with vanilla OPD within one setting. Second, ScienceWorld is much harder than ALFWorld for planner-based curricula: even the gold-path planner solves only half of the ScienceWorld training tasks within 30 decisions (1,510/3,017), versus nearly all in ALFWorld (3,548/3,553), so on half the tasks there is no complete solution for near-goal curriculum starts.

Source and metricVanillaTCOD-F2BTCOD-B2FFTB
Their ALFWorld SR (FTB, 8B→4B)67.466.568.674.1
Our ALFWorld Unseen SR (GRPO 32B→1.7B)79.4883.2185.0781.90
Their ScienceWorld SR (TCOD, Qwen3-30B-A3B→4B)15.9517.12–17.8515.88–17.46–
Their ScienceWorld SR (FTB, 32B→4B)7.311.39.514.0
Our ScienceWorld SR (original 32B→4B)24.7023.6023.1323.21

ALFWorld. The curriculum methods mostly improve over vanilla OPD in the original papers, and with the GRPO teacher the same implementations all improve in ours (+2.4 to +5.6 Unseen points).

After the table. The published papers themselves already report small or mixed effects on ScienceWorld: TCOD's own text calls its variants comparable to vanilla OPD (15.88–17.85 vs. 15.95 in their Table 3), and our results agree — with the original teacher, every baseline stays within 1.6 SR points of vanilla OPD. Neither paper evaluates ScienceWorld with an RL-trained teacher, so the GRPO-teacher block of our main table, where the curriculum baselines fall below vanilla OPD, has no published counterpart; in that block the baselines use the same released configurations, GRPO teacher and 190-update budget as vanilla OPD.

Q2 — Teachers and checkpoint selection

Both GRPO teachers start from Qwen3-32B and are trained for about one epoch, with 16 rollouts per task, 32 tasks per update, learning rate 1e-6 and KL regularization. On ScienceWorld, checkpoints are selected on the 305 held-out development tasks. ALFWorld has no separate development split, so ALFWorld checkpoints are selected by Seen success, and Unseen is used only for testing. The GRPO teacher (update 92) reaches 83.57 Seen / 80.22 Unseen over four inference seeds; it is used by the GRPO-teacher baselines, not by GC-OPD.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary: graph-based state matching drives the gain of GC-OPD

We thank all reviewers for their careful and constructive feedback.

All three reviewers ask whether the gain of GC-OPD requires selecting records by state, or whether any same-task record plus student hindsight would suffice. Table R1 answers this with controlled runs: same-task records help, and selecting them by state through the graph adds a further 8.70 points.

Table R1. Controls on how external records are chosen. ScienceWorld, Qwen3-1.7B student, off-the-shelf Qwen3-32B scoring teacher. All rows below vanilla OPD continue training from the same vanilla-OPD model and share the K=16 library, the complete student hindsight and the teacher-context cap; only the way external records are chosen differs. SR: mean ± SD over four inference seeds on the 1,661 test tasks. Tokens: mean teacher input per scored decision, measured on the same 3,383 student decisions. Rules that pair records supply one successful and one failed record when available; a missing outcome is omitted.

RuleHow external records are chosenTokensSR (%)Rounds
Vanilla OPDno hindsight, no external record66522.31 ± 0.4019.66
Hindsight onlyno external record2,10031.64 ± 0.5713.83
Minimum-prefixone successful + one failed record sharing the shortest action prefix with the student3,73033.70 ± 0.6316.76
Random same-taskone successful + one failed record drawn at random from the same task3,82137.48 ± 0.7115.23
Current-state onlygraph, exact state matching at the current state only (no earlier states)2,58938.50 ± 0.6313.05
OPID-style skillsrecords summarized into episode-level and step-level skills2,30939.42 ± 1.0714.52
Longest-prefixone successful + one failed record sharing the longest action prefix with the student3,57440.38 ± 0.8513.76
Observation-based retrievalsame graph and retrieval procedure as GC-OPD; states matched approximately by TF-IDF similarity of observation text (≥ 0.65)3,34142.82 ± 0.6413.68
GC-OPDgraph, exact state matching at the current state, then at earlier student states3,48546.18 ± 1.1313.04

Does any same-task record plus hindsight suffice? No. Random same-task records do help (31.64% → 37.48%), but selecting the records by state adds a further 8.70 points (46.18%). Random supplies a successful and a failed record at every decision and gives the teacher the longest input (3,821 tokens); GC-OPD scores higher with a shorter input (3,485 tokens). The gain therefore comes from which record is selected, not from the presence of same-task records or from a longer prompt.

Does the graph help without exact simulator state? Yes. Observation-based retrieval keeps the graph and the retrieval procedure of GC-OPD and only replaces exact state matching by approximate text matching. It reaches 42.82%, above every rule that does not use the graph (at most 40.38%). Exact state matching adds a further 3.36 points, because similar text can describe different states (e.g., a closed vs. an open drawer).

Do earlier states matter? Yes. Restricting graph matching to the current state gives 38.50%, 7.68 points below the full method. Of the 42,537 training decisions, 16.23% receive a current-state successful reference and a further 25.82% receive one only through an earlier student state.

Complete records vs. summaries. OPID-style skills built from the same K=16 library reach 39.42%, 6.76 points below GC-OPD: in long multi-step tasks, a summary drops the step-level details that the teacher needs to judge an action.

The deployed student uses neither the graph nor the teacher.

Robustness and cost. Across four independent student trainings, GC-OPD reaches 46.62 ± 0.87%, versus 37.53 ± 1.81% for vanilla OPD with a GRPO-trained teacher and 23.20 ± 2.08% for vanilla OPD with the same original teacher (per-seed results in our response to kTju). GC-OPD and the GRPO-teacher route use the same number of teacher rollouts (48,272); GC-OPD costs 281 GPU-hours in total versus 1,217 (breakdown, and the information available to each method, in our response to 23dP).

Takeaway. On ScienceWorld, the original teacher's average success is only 29.74%, yet GC-OPD uses its repeated executions to train a 46.18% student. GRPO raises the same teacher's success to 48.43%, but vanilla OPD with that stronger teacher reaches only 38.70%. In this setting, improving the evidence available to the teacher outperforms improving the teacher itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.