acceptodds
Under review as a conference paper at ICLR 2027

VTD: Distilling Symbolic Task Structure for Long-Horizon Agent Learning

Abstract

Solving long-horizon tasks requires agents to make a sequence of interdependent decisions before receiving a reliable outcome signal. On-policy distillation supplements this sparse feedback with token-level guidance from a stronger teacher on trajectories generated by the student agent. However, the teacher conditions each prediction on the student's interaction history, so earlier mistakes can distort all subsequent guidance. Our analysis further shows that even teachers that can solve a task on their own increasingly favor the failing student's actions when conditioned on this history. We therefore propose Verifiable Task Decomposition (VTD), which separates task decomposition from trajectory execution using a symbolic progress graph constructed by the teacher before interaction. During rollout, the environment turns completed graph milestones into verified progress that complements the terminal outcome in policy optimization. Graph distillation transfers this decomposition capability to the student, allowing the student to generate progress graphs without teacher access at test time. Experiments on ALFWorld, WebShop, and TextCraft show that VTD accelerates policy learning and outperforms the strongest Qwen2.5-7B baseline by 1.8, 3.4, and 8.0 points, respectively. VTD also exhibits strong structural generalization beyond the training distribution, reaching 92.8% success on held-out ALFWorld task families compared with 68.0% for the strongest baseline and extending to deeper TextCraft recipe hierarchies.

Primary area:
reinforcement learning
Keywords:
Long-horizon language agents, Symbolic task decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kkNfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Verifiable Task Decomposition (VTD) to address the limited corrective value of online policy distillation when learning from failed student trajectories. Rather than distilling teacher actions, VTD uses a symbolic graph of verifiable milestones and dependencies constructed before interaction. Environment-verified milestone completion supplements final task success during training, while graph distillation enables students to generate graphs without teacher access at inference. Experiments with Qwen2.5 models on ALFWorld, WebShop, and TextCraft show that VTD outperforms existing baselines, while ablation studies confirm the contributions of verified progress and graph distillation.

Critical strengths
  1. The paper effectively reformulates teacher supervision for long-horizon agents by replacing error-prone, trajectory-conditioned token guidance with task-level symbolic representations, particularly benefiting settings where teachers are accessible only through black-box APIs.

  2. The paper demonstrates strong empirical performance across three environments and two model scales, supported by informative ablations of progress verification and compelling evidence of compositional generalization on held-out ALFWorld tasks.

Critical weaknesses
  1. The contribution of the symbolic graph structure is insufficiently isolated. Eq. (5) rewards milestone completion but ignores dependency edges, and existing ablations do not compare against flat milestone lists or ordered plans. Structural acceptance checks formatting rather than semantic correctness, so it remains unclear whether the gains come from graph distillation or milestone-based reward shaping.
  2. The evidence for the proposed OPD failure mechanism is incomplete. Decreases in KL divergence and increases in late-stage action agreement are consistent with weakened corrective supervision, but alternative explanations exist (e.g., hard-to-recover states, fewer meaningful actions, repetition). No controlled teacher-based OPD comparison is provided, and SDAR’s self-distillation design does not test VTD against the external-teacher supervision that motivates the paper.
Questions for authors
  1. Does the dependency structure and semantic quality of the generated graph materially affect performance?**Could the authors compare the full graph against the same milestones with removed or randomized edges, or against a flat milestone list? Please also clarify how often teacher- and student-generated graphs are semantically complete and correct, rather than merely structurally accepted.
  2. Can the authors provide more direct evidence for the claimed degradation of teacher supervision?**What is the teacher’s task-completion rate when taking over from different prefixes of failed student trajectories, compared with matched successful prefixes? A teacher-based OPD comparison under a comparable teacher or compute budget would also clarify whether VTD specifically addresses the proposed failure mode.
4 comments

We thank the reviewer for recognizing that VTD reformulates teacher supervision for black-box teachers through task-level symbolic structure and achieves compelling compositional generalization. We address each concern below.

[Cons 1a & Q1a. Contribution of the dependency structure?] We emphasize that Eq. (5) scores milestones without edges by design, so every control below receives exactly the same reward. However, to directly address the reviewer's question, we keep the teacher milestones and verified progress fixed on TextCraft and vary only how the dependencies reach the student.

Qwen2.5-1.5B, TextCraftSuccess, all itemsΔSuccess, 3-level itemsΔ
Full graph (VTD)83.565.2
Ordered plan83.50.062.1−3.1
Flat milestone list75.5−8.051.5−13.7
Randomized edges63.7−19.836.4−28.8
Full graph, no graph distillation39.7−43.818.2−47.0

Under the same reward, success follows the dependency content. The ordered plan keeps every prerequisite as a topological order of the same graph, so it matches the full graph in overall success. The flat milestone list and randomized edges both lower success. Without graph distillation, the student writes a valid graph for only 3.4% of items, and success collapses. VTD's gains come from the dependency structure and its distillation, which milestone-based reward shaping alone cannot provide.

[Cons 1b & Q1b. Semantic quality of the graphs?] Our results show that teacher and student graphs are semantically correct at high rates, and that their semantic content materially affects performance. A graph is semantically complete and correct when its milestones and dependencies match the task's ground truth.

Semantically complete and correctALFWorldWebShopTextCraft
Teacher graphs100.099.2100.0
Student graphs at test time (Qwen2.5-7B)100.0100.077.6

To measure whether this quality matters, we keep the trained student fixed and replace the graph it writes at test time with the ground-truth graph, another item's graph, or no graph.

Graph in context at test time (Qwen2.5-7B, TextCraft)Success, all itemsSuccess, 3-level items
Student's own graph (VTD)97.097.0
Ground-truth graph98.798.5
Another item's graph90.383.3
No graph89.975.8

A correct graph keeps success high, while another item's graph or no graph lowers it. The student's own graphs perform almost as well as the ground-truth graph. On ALFWorld, a graph from another task family also lowers success (87.2 vs. 99.2). The semantic quality of the graph materially drives VTD's performance.

[Cons 2a & Q2a. Direct evidence for the degraded supervision?] We want to make clear that the degradation comes from conditioning on the failed history, and direct measurements rule out each alternative explanation. We let each teacher take over a failed student trajectory at turn t and act for 30 more steps, with either the failed history or only the task and the environment's description of the current state. We use the 128 tasks and both teachers of Figure 1.

TeacherContext at takeovert = 0t = 10t = 20t = 30t = 40
Qwen2.5-7B (pretrained)Failed history66.826.817.411.96.3
Qwen2.5-7B (pretrained)Task and current state66.448.154.150.051.2
Qwen2.5-1.5B (GRPO-adapted)Failed history92.261.752.241.435.9
Qwen2.5-1.5B (GRPO-adapted)Task and current state92.875.280.578.377.3

For the 25 tasks where the student has both a successful and a failed trajectory, we also compare takeovers after the first 10 turns of each.

Teacher, takeover at t = 10 (25 tasks)From the startAfter a successful prefixAfter a failed prefix
Qwen2.5-7B (pretrained)81.078.055.0
Qwen2.5-1.5B (GRPO-adapted)96.094.077.0

Both teachers continue a successful prefix almost as well as they solve the task from the start, while a failed prefix sharply lowers their success. Given only the task and the current state, the teachers succeed far more often from the same states, so these states remain recoverable.

On the failed trajectories of Figure 1, we also count the admissible actions at each turn. We then recompute the teacher's action agreement on turns where the student neither repeats one of its previous five actions nor takes an invalid action.

Failed student trajectories of Figure 1Turns 1–10Turns 41–50
Admissible actions per turn28.028.5
Action agreement of Qwen2.5-7B, without repeated or invalid actions53.6%80.9%

Late turns offer as many admissible actions as early turns, so fewer choices cannot explain the late agreement. Removing repeated and invalid actions leaves the rise in agreement intact. Conditioning on the failed history removes the teacher's corrective ability, which is the failure mode VTD avoids.

[Cons 2b & Q2b. Teacher-based OPD under a comparable budget?] We emphasize that VTD outperforms on-policy distillation (OPD) from the strongest available teacher while using far fewer teacher tokens. OPD needs the teacher's token distribution on the student's outputs, so its teacher must be an open-weight model with the student's tokenizer. On ALFWorld, we train the same student with GRPO and a reverse-KL term toward Qwen2.5-72B-Instruct, the strongest such model, under VTD's training budget and evaluation.

Qwen2.5-1.5B, ALFWorldTeacherTokens per teacher queryTotal teacher tokensSuccess
GRPO———72.8
OPDQwen2.5-72B-Instruct7,795200M84.6
VTDGPT-5.51,2364.4M92.2

Both methods train for 200 steps with 128 student trajectories per step. OPD runs the teacher over all 25,600 of these trajectories, whereas VTD only asks the teacher to write one graph for each of the 3,553 training tasks. OPD thus uses 45 times as many teacher tokens yet stays well below VTD. Each OPD signal is computed on the student's own trajectory, so it inherits the loss of corrective value under a failed history. VTD surpasses OPD from the strongest open teacher with 2% of OPD's teacher tokens, and it can learn from black-box teachers that OPD cannot use.

Dear Reviewer kkNf,

Thank you for your review. We posted our response a few days ago, reporting the comparisons you asked for against removed or randomized edges and a flat milestone list, together with how often teacher and student graphs are semantically complete and correct. To give direct evidence for the degraded supervision, we measured teacher takeover from failed and matched successful prefixes and checked the alternative explanations you listed. We also compared VTD with teacher-based OPD, including the teacher tokens each method uses.

While the discussion phase is open, we would be glad to know whether the response answers your questions.

Sincerely,

The Authors

Dear Reviewer kkNf,

Thank you again for your review. As the discussion phase closes tomorrow, we would be glad to know whether our response answers your questions.

Sincerely,

Reviewer_WWzuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VTD transfers task decompositions rather than next-token predictions from a teacher to a long-horizon agent. Before interaction, the teacher generates a symbolic progress graph using an environment-specific predicate vocabulary. Deterministic verifiers record completed milestones, and the fraction completed is added to terminal success for group-relative policy optimization. A graph-generation loss trains the student to produce its own graph at evaluation. Experiments cover ALFWorld, WebShop, and TextCraft with Qwen2.5-1.5B and 7B. The paper reports improvements on all three benchmarks at 7B, substantial gains from progress rewards, and generalization to held-out task families and deeper recipes.

Critical strengths
  1. The progress signal addresses a concrete exploration problem, with a meaningful controlled ablation. Under otherwise matched training, removing verified progress reduces 1.5B success from 92.2% to 66.4% on ALFWorld and from 79.4% to 62.2% on WebShop (Table 3). This is stronger evidence than the aggregate baseline comparison that intermediate verification helps when terminal successes are scarce.

  2. The framework combines practical teacher access with promising transfer results. It needs neither teacher logits nor teacher queries during interaction, and evaluation uses student-generated graphs. The 7B results of 99.2%, 86.7%, and 97.0%, together with 92.8% on held-out ALFWorld families, make the approach worth investigating. The predicate definitions and construction prompts also make the intended implementation reasonably concrete.

Critical weaknesses
  1. The experiments do not isolate the contribution of symbolic dependency structure from milestone rewards and additional supervision. Appendix B states that verification ignores dependency edges. Appendix E also supplies substantial decomposition rules, including recursive recipe selection and demand aggregation for TextCraft. The current ablations establish the usefulness of progress rewards and graph-token supervision, but do not show whether dependency edges improve over the same milestone set without edges, or whether a simpler planner using the supplied rules would suffice. Figure 4's structural acceptance metric does not establish semantic correctness, and its no-distillation control also removes training for the required output format. These distinctions matter to the claimed contribution of learning task structure, beyond supplying a verified checklist and a supervised planning prefix.

  2. The proposed explanation for OPD's failure is stronger than the diagnostic supports. Figure 1 measures reduced divergence and increased action agreement on fixed unsuccessful student trajectories. An unsuccessful trajectory can contain appropriate recovery actions; agreement does not establish that the teacher has lost corrective ability. The experiment does not measure teacher continuation success from those prefixes or compare contexts that preserve the current state while removing the earlier failed history. Moreover, the downstream comparison uses SDAR self-distillation, rather than a matched strong-teacher OPD intervention. The black-box teacher restriction is understandable, but these results do not yet establish that avoiding history-conditioned supervision explains VTD's gains.

Questions for authors
  1. Contribution isolation: With the same teacher milestones, progress rewards, and supervision budget, how much performance is lost when dependency edges are removed? A rule-generated graph control using Appendix E's decomposition rules would also help determine how much teacher-generated structure contributes. These comparisons could change my assessment of the method's added value.

  2. Clarification of the OPD diagnosis: Can the teachers recover when allowed to continue from late student states, with a fixed remaining action budget? How does this change when the context is reset to the task and an equivalent description of the current state? This would distinguish impaired correction from agreement on reasonable actions in already difficult states.

  3. Generalization controls: Were all baseline policies given equivalent information about the held-out task semantics? Appendix E explicitly exposes clean/hot/cool in the shared prompt. What are the semantic correctness and success rates of student graphs on CLEAN/HEAT with and without graph distillation? This would help separate learned decomposition from supplied task semantics and pretrained capabilities.

  4. Evaluation clarification: Do the three reported seeds represent independent training runs, evaluation subsets/sampling, or both? Appendix C describes subsets and sampling seeds, while Table 1 mixes reproduced and previously reported results. Please specify whether those entries use the same ALFWorld split, task subsets, and checkpoint-selection rule. This is particularly relevant to the 1.8-point 7B ALFWorld margin.

7 comments

We appreciate the reviewer recognizing that verified progress solves a central exploration problem, and that VTD achieves strong transfer with only black-box teacher access. We address each concern below.

[Cons 1a & Q1. Contribution of the dependency structure?] We want to clarify that VTD uses dependency edges to guide and train the student, not to score it. Because verified progress credits milestones in any valid order, every control below receives exactly the same reward. However, to directly address the reviewer's question, we keep the teacher milestones, verified progress, and graph-distillation budget fixed on TextCraft and change only the dependencies.

Qwen2.5-1.5B, TextCraftSuccess, all itemsΔSuccess, 3-level itemsΔ
VTD83.565.2
Same milestones, no edges75.5−8.051.5−13.7
Same milestones, shuffled edges63.7−19.836.4−28.8
Rule-compiled graph (Appendix E rules)81.4−2.159.1−6.1

With the reward unchanged, removing or shuffling the edges lowers success. The reward ignores the order of completion, so the edges are the only source of dependency information. A planner built from the rules in Appendix E does not suffice (81.4 vs. 83.5). VTD learns task structure beyond a verified checklist.

[Cons 1b. Semantic correctness beyond output format?] We want to make clear that VTD's graphs are semantically correct, not only structurally accepted. We count a graph as correct when its milestones and dependencies match the task's ground truth. However, to directly separate the decomposition from the output format, we force every graph of the no-distillation student into the required format with constrained decoding.

Qwen2.5-1.5B, ALFWorld (Figure 4 setting)Graphs in the required formatCorrect graphs
VTD100.0100.0
No graph distillation0.00.0
No graph distillation, constrained decoding100.00.4

Constrained decoding fixes the format but not the content. The resulting graphs miss the final milestone, repeat milestones, or depend on milestones that do not exist. Graph distillation transfers the teacher's decomposition capability, which format training alone cannot provide.

[Cons 2a & Q2. Teacher recovery from late student states?] We want to clarify that the decline in Figure 1 comes from the failed history, not from the states the student reaches. We let each teacher take over a failed student trajectory at turn t and act for 30 more steps. The teacher sees either the failed history or only the task and the environment's description of the current state. We use the 128 tasks and both teachers of Figure 1.

TeacherContext at takeovert = 0t = 10t = 20t = 30t = 40
Qwen2.5-7B (pretrained)Failed history66.826.817.411.96.3
Qwen2.5-7B (pretrained)Task and current state66.448.154.150.051.2
Qwen2.5-1.5B (GRPO-adapted)Failed history92.261.752.241.435.9
Qwen2.5-1.5B (GRPO-adapted)Task and current state92.875.280.578.377.3

With the failed history, the success of both teachers declines steadily from t = 0 to t = 40. With only the task and the current state, their success stays far higher and does not decline at later turns. Supervision conditioned on a failed history loses its corrective value, and VTD avoids this failure by giving the teacher no trajectory to condition on.

[Cons 2b. Comparison with strong-teacher OPD?] We emphasize that VTD outperforms on-policy distillation (OPD) from the strongest teacher whose token distributions are available. OPD needs the teacher's token distribution on every student output, so its teacher must be an open-weight model with the student's tokenizer. On ALFWorld, we train the same student with GRPO and a reverse-KL term toward Qwen2.5-72B-Instruct, the strongest such model, under VTD's training budget and evaluation.

Qwen2.5-1.5B, ALFWorldTeacherSuccess
GRPO—72.8
OPDQwen2.5-72B-Instruct84.6
VTDGPT-5.592.2

OPD improves over GRPO with this teacher but remains well below VTD. Each OPD signal is computed on the student's own trajectory, so it inherits the loss of corrective value under a failed history. Supervision through task structure surpasses token-level supervision from the strongest open teacher, and VTD can also learn from black-box teachers that OPD cannot use.

[Q3. Equal task semantics and generalization without graph distillation?] We want to clarify that every method receives the same task semantics, because the instruction itself names the target state. All methods see the same instruction, such as "clean some apple and put it in sidetable", and the same admissible actions. The clean, hot, and cool values in Appendix E are arguments of the environment-level TRANSFORM predicate. No training graph contains the clean or hot state. However, to directly separate learned decomposition from the supplied semantics, we train Qwen2.5-7B without graph distillation under the protocol of Table 2 and evaluate its own graphs on Clean and Heat. We also force its graphs into the required format with constrained decoding.

Qwen2.5-7B, Clean and Heat held outSuccess, held-outSuccess, CleanSuccess, HeatCorrect graphs
VTD92.891.494.9100.0
No graph distillation83.579.389.7100.0
No graph distillation, constrained decoding85.682.889.799.0

Both students write correct graphs from the same semantics and the same pretrained model, yet the student without graph distillation succeeds far less. Constrained decoding does not close this gap. Graph distillation, not the supplied semantics or pretrained capability, accounts for VTD's advantage on the held-out families.

[Q4. Seeds, splits, and checkpoint selection?] We want to clarify that the 3 random seeds in Table 1 are evaluation seeds, as Appendix C describes. On ALFWorld and WebShop they select the 128-task evaluation subsets, and on TextCraft they set the sampling. The cited ALFWorld and WebShop baselines use the same seen split and 128-task subsets. The baselines we reproduce share VTD's evaluation, and VTD always reports its final checkpoint. However, to address training randomness directly, we also train VTD with three training seeds in every 7B setting.

SettingStrongest baselineVTD (Table 1)VTD, 3 training seedsΔ
ALFWorld, 7B97.499.298.5 ± 0.8+1.1
WebShop, 7B83.386.786.3 ± 1.0+3.0
TextCraft, 7B89.097.096.2 ± 0.7+7.2

On the complete seen and unseen ALFWorld splits, VTD-7B reaches 98.6 and 99.0. VTD's 7B advantage holds across training seeds, and its ALFWorld performance does not depend on the evaluation subset.

Dear Reviewer WWzu,

Thank you for your review. We posted our response a few days ago, reporting the two controls you suggested for contribution isolation: the same milestones without dependency edges and a rule-generated graph from Appendix E. To clarify the OPD diagnosis, we let each teacher continue from late student states with either the failed history or a reset context, and we compared VTD with a matched strong-teacher OPD. The response also reports the semantic correctness of graphs under constrained decoding, Clean and Heat results without graph distillation, and VTD trained with three training seeds.

While the discussion phase is open, we would be glad to know whether the response answers your questions, and welcome any further ones.

Sincerely,

The Authors

Reviewer_5RCDOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes VTD: before interaction, the teacher generates a symbolic progress graph for the task; the environment verifies milestones completed by the student trajectory; and verified progress is combined with terminal success for policy optimization. Graph distillation allows the student to generate its own graph at test time. The paper evaluates Qwen2.5-1.5B/7B on ALFWorld, WebShop, and TextCraft, and reports task-family holdout results, a deeper recipe, and conditions without verified progress or graph distillation.

Critical strengths
  • VTD avoids having the teacher continue to provide degraded supervision on failed histories; instead, it first supplies verifiable task structure and then lets the environment score each student trajectory.
  • Held-out task families and deeper TextCraft levels suggest that the structured representation may support compositional generalization beyond training trajectories; graph-generation acceptance rates and verified progress also provide richer analysis than a single terminal-success metric.
Critical weaknesses
  • Each environment requires manually defined predicates, parameter schemas, termination conditions, and verifiers; the current experiments include neither new predicates nor unknown environments or learned verifiers. The results therefore cannot be extrapolated to general symbolic task-structure distillation.
  • There is no orthogonal separation of graph context, verified reward, graph distillation, and teacher-graph quality; meanwhile, some baselines come from different implementations, temperatures, training steps, or reporting protocols. The observed performance gaps alone are insufficient to support an conclusion that “VTD outperforms the strongest baseline.”
Questions for authors
  1. What are the accuracy, retry rate, node/edge statistics, cost, and consistency of GPT-5.5 graphs? If the teacher graph itself has no independent correctness evaluation, how should the assess “structure learning” rather than “strong prompt input”?
  2. Can the authors add, under the same training and evaluation protocol, four orthogonal controls: fixed graph context without q; q without graph context; randomized/shuffled dependency edges; and no graph distillation? These results would directly determine acceptance.
  3. In at least one environment, can the authors introduce a new predicate or verifier absent during training and test whether the student can still form an effective graph? If not, should the contribution be positioned as environment-specific structured-reward engineering?
5 comments

We thank the reviewer for recognizing that VTD overcomes a fundamental limitation of teacher supervision through verifiable task structure and achieves compositional generalization. We address each concern below.

[Cons 1 & Q3. Generalization to unseen predicates and verifiers?] We want to clarify that the predicate vocabulary is an environment-level interface for verifying progress, not task-level reward design. All tasks in an environment share four or five predicates defined once. However, to directly address the reviewer's question, we evaluate VTD on a predicate and two verification conditions that never appear in training. Holding out every Look task removes EXAMINE_UNDER_LIGHT from all training graphs. Holding out every Clean and Heat task removes the clean and hot states. We compare with GRPO under the protocol of Table 2.

Absent during training (Qwen2.5-7B)GRPO successVTD successΔCorrect VTD graphs
New predicate: EXAMINE_UNDER_LIGHT (Look held out)77.480.6+3.296.8
New verification conditions: clean, hot (Clean and Heat held out, Table 2)46.492.8+46.4100.0

VTD solves far more held-out tasks than GRPO. No training task rewards these milestones, so the reward cannot have taught them. The student composes verifiable structure from units it never saw in training, which structured-reward engineering cannot provide. Adding a predicate only requires its verification rule.

[Cons 2a & Q2. Orthogonal controls for each component?] Our results show that every component of VTD contributes to its gains. We add the four requested controls on TextCraft with Qwen2.5-7B, and each changes exactly one component.

Qwen2.5-7B, TextCraftSuccess, all itemsΔSuccess, 3-level itemsΔ
VTD97.097.0
Fixed graph context, no verified progress94.1−2.992.4−4.6
Verified progress, no graph context91.1−5.987.9−9.1
Shuffled dependency edges94.5−2.590.9−6.1
No graph distillation90.3−6.778.8−18.2

Success drops under every control, and the drops are largest on three-level items, whose dependencies are deepest. Verified progress provides a learning signal before the first success. The graph specifies which milestones to pursue, and its edges fix their order. Graph distillation teaches the student to construct the graph by itself. VTD's gains require all four components and grow with the depth of the task.

[Cons 2b. Comparability with the strongest baselines?] We emphasize that we take each cited baseline result directly from its original paper, which uses the evaluation protocol VTD follows. Each baseline we reproduce uses its public implementation and our evaluation. On TextCraft, GRPO and GiGPO are taken at their best checkpoint, whereas VTD is always taken at its final checkpoint. However, to directly address training randomness, we train VTD with three training seeds on WebShop and TextCraft at both scales.

SettingStrongest baselineVTD (Table 1)VTD, 3 training seedsΔ
WebShop, 1.5B75.679.479.1 ± 1.3+3.5
WebShop, 7B83.386.786.3 ± 1.0+3.0
TextCraft, 1.5B58.383.582.7 ± 1.1+24.4
TextCraft, 7B89.097.096.2 ± 0.7+7.2

VTD's advantage holds across training seeds and does not depend on checkpoint selection.

[Q1. Quality of teacher graphs and evidence of structure learning?] We want to make clear that every teacher graph is verified against the environment's ground truth, and that the student never sees a teacher graph at test time. VTD does not rely on retries, and each graph is used exactly as the teacher writes it. We apply the same check to the graphs the student writes at test time.

Graph quality and costALFWorldWebShopTextCraft
Correct teacher graphs100.099.2100.0
Correct when regenerated100.0100.0100.0
Correct student graphs at test time (Qwen2.5-7B)100.0100.077.6
Nodes, mean (max)4.1 (6)8.1 (18)4.9 (13)
Edges, mean (max)3.1 (5)12.2 (32)3.9 (12)
Cost per graph$0.009$0.024$0.014

The teacher's graphs are correct and stay correct when regenerated, at a cost of at most $0.024 per graph. The student writes correct graphs without any teacher input, so the structure it uses is learned rather than supplied in the prompt. To test whether the student acts on this structure, we keep the trained TextCraft student fixed and replace the graph it writes at test time with the ground-truth graph, the graph of another item, or no graph.

Graph in context at test time (Qwen2.5-7B, TextCraft)Success, all itemsSuccess, 3-level items
Student's own graph (VTD)97.097.0
Ground-truth graph98.798.5
Another item's graph90.383.3
No graph89.975.8

A correct graph keeps success high, while another item's graph or no graph lowers it. The student's own graphs perform almost as well as the ground-truth graph. On ALFWorld, a graph from another task family also lowers success (87.2 vs. 99.2). The student has internalized the teacher's task structure and relies on it to act.

Dear Reviewer 5RCD,

Thank you for your review. We posted our response a few days ago, reporting the four orthogonal controls you asked for under the same training and evaluation protocol, together with a new predicate and new verification conditions absent during training. The response also reports the accuracy, statistics, cost, and consistency of the GPT-5.5 graphs and uses a test-time graph replacement to separate structure learning from strong prompt input. To address training randomness, we further trained VTD with three training seeds.

While the discussion phase is open, we would be glad to know whether the response answers your questions, and welcome any further ones.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.