acceptodds
Under review as a conference paper at ICLR 2027

Environment-Free Learning from Agent Trajectories with Bottleneck Upsampling

Abstract

Language agents are increasingly deployed in complex environments, and every deployment leaves behind trajectories. Rebuilding the environment for post-training, as reinforcement learning requires, is often impractical: it may be proprietary, involve real users, or be too expensive to run again. Environment-free learning, which improves an agent from these trajectories alone, is therefore an important problem. A natural approach is to relabel the trajectories: at every step, a teacher shows what should have been done, and the agent is trained to follow. In context distillation (CD) and on-policy context distillation (OPCD), the teacher is the same model conditioned on experience summarized from the trajectories. We find that this recipe works well on some benchmarks but delivers limited gains on others, and we trace the shortfall to the trajectories. The deployed agent that collected them is weak, so few episodes reach the later stages of a task, and the training data hold few steps from them. For example, on a shopping task, few episodes reach a product page, so the trained agent does not learn what to do there, such as choosing options and placing the order. We call the first stage that few episodes reach the bottleneck. We propose CDUP (Context Distillation with bottleneck UPsampling), a simple and effective recipe that has an LM agent segment the trajectories into task stages, locates the bottleneck from the share of episodes reaching each stage, and upsamples the steps from it onward at a fixed ratio. On WebShop, CDUP improves success over CD by 16.4 percentage points with Llama-3.2-3B and 3.1 with Qwen2.5-3B, and on ScienceWorld by 16.9 and 9.9. Bottleneck upsampling also raises WebShop success under OPCD and with an external expert teacher. Code is available at https://anonymous.4open.science/r/cdup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training, self-distillation, agent

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_VATDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper investigates how to improve a language agent using previously collected trajectories without further interaction with the environment during training. It points out that Uniform Context Distillation (OPCD) underestimates the decision-making power of task stages rarely reached by weakly deployed policies. CDUP uses a more powerful Language Model (LM) agent to generate task stages and a classifier, identifying the first stage reached less than a fixed proportion, and upsampling the retained states of that stage and subsequent stages while keeping the number of training visits constant. Experiments were conducted using WebShop and seven ScienceWorld task types, employing Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct agents, respectively. CDUP is compared to CD and OPCD, incorporating stochastic states, loss weighting, and stage balancing controls, while the upsampling effects of OPCD and external teacher models are tested on WebShop. In the reported experiments, CDUP outperforms both CD and OPCD.

Critical strengths
  1. An effective diagnostic approach supplemented with a controlled intervention. The distinction between teacher objectives and allocating sufficient training to corresponding states is crucial for learning from a fixed log. In the primary experiment, CD and CDUP shared a maintained library of objectives, training access, and updates, while the matched randomized state control group maintained the allocation pattern while varying which states received additional access. Combined with stage achievement analysis, these comparative results demonstrate that, in the test protocol, state-specific focused training is more valuable than general CD training.

  2. A simple intervention applicable to different training objectives and teachers. Upsampling alters the allocation of existing training budgets without requiring new actions during training. Table 3 shows the effectiveness of the OPCD and external teacher SFT approaches on WebShop, demonstrating the intervention's effectiveness outside the primary CD setup.

Critical weaknesses
  1. Limited Evidence on Generalization and Robustness. Experiments conducted solely on the 3B model cannot determine whether the method's advantages are limited to small, weak models. Despite limited computational resources, experiments on the 7B model are crucial for validating this possibility. Furthermore, supervised learning results rely on a single training seed, requiring repeated training with an error range to assess its variability and reliability.

  2. Limited Baseline Coverage. This paper attributes the inadequacy of learning to the fact that the recorded trajectories rarely reach the later stages of the task. Experience-enhanced ReAct and Reflexion-style cues may also help the agent correct unsuccessful behaviors and reach these stages. Although the paper includes ReAct-style cues and contextual teachers, the advantages of CDUP over dedicated reflection-based methods are unclear due to limited comparisons. Explicit comparisons with these cuing methods will help determine the method's advantages.

Questions for authors
  1. Comparison with Hint Methods Given that the contextual teacher outperforms CDUP in Table 2, could you provide a clear comparison with ReAct and Reflexion using the same model?

  2. Generalization Ability Beyond 3B Models Although the paper acknowledges limitations in computational resources, validation on larger models remains crucial to determine if these improvements are limited to small, weak models. Could you evaluate CDUP on a 7B model for validation?

  3. Comparison with Fine-Grained Trajectory Scoring Given that the paper discusses teacher-based supervision, could you compare CDUP with training guided by fine-grained scoring assigned by another model based on trajectory steps, using the same log data and training budget?

  4. Variability of Training Runs Could you provide the average performance and error range across multiple runs?

  5. Stage Assignment How are stages assigned when the agent follows an alternative path, returns to an earlier stage, or accesses a state outside the defined stage sequence (e.g., WebShop description and review pages)?

8 comments

Thank you for the careful review and the constructive questions. We ran the requested experiments and added them to the revised paper, where new text is in blue (Appendix F). We address each point below.


W1 / Q2: Experiments only on 3B models leave open whether the method's benefits are specific to small, weak models. […] Could you evaluate CDUP on a 7B model to test this?

Response: We ran the full pipeline with two larger models, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, on both WebShop and ScienceWorld. Success rate (%) on the 384 test episodes:

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.1-8BQwen2.5-7BLlama-3.1-8BQwen2.5-7B
Base policy0.00.514.616.9
In-context teacher29.461.788.863.3
CD28.126.866.931.3
CDUP (ours)32.031.580.734.9

Larger models are stronger before and after training. Uniform CD is better than at 3B: on WebShop its success rises from 2.3% and 7.8% to 28.1% and 26.8%, and on ScienceWorld from 21.4% and 15.4% to 66.9% and 31.3%. CDUP still improves over CD in all four settings, by 3.9, 4.7, 13.8 and 3.6 points. The benefit of bottleneck upsampling is therefore not specific to small, weak models. These experiments are in the revised paper (Appendix F.1).


W1 / Q4: Supervised results also rely on a single training seed […] Could you provide the mean performance and error bars across multiple runs?

Response: We retrained CD, OPCD and CDUP with two more training seeds in all four settings of Table 2, so each number below is a mean ± sd over three seeds. On WebShop with Qwen2.5-3B, where the difference between CD and CDUP is the smallest, CD and CDUP have five seeds. Success rate (%):

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
CD1.6 ± 0.79.4 ± 1.022.5 ± 1.113.7 ± 1.5
OPCD1.6 ± 1.16.5 ± 0.921.4 ± 3.315.7 ± 2.7
CDUP (ours)18.1 ± 0.810.1 ± 2.436.1 ± 2.723.5 ± 2.0

Task score (WebShop) or progress (ScienceWorld), in %:

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
CD3.2 ± 1.323.6 ± 1.153.6 ± 0.746.5 ± 0.4
OPCD5.0 ± 3.131.6 ± 5.047.9 ± 1.647.3 ± 1.1
CDUP (ours)41.6 ± 4.838.8 ± 3.058.9 ± 2.252.0 ± 1.2

The results of the submission are reproduced. CDUP is above CD and OPCD in every setting, in success and in task score or progress. In three settings CDUP is above CD on every seed, by 16.5, 13.6 and 9.8 points of success on average. On WebShop with Qwen2.5-3B success also rises (9.4 to 10.1), and the gain is larger in task score (23.6 to 38.8, higher on every seed). The revised paper reports these means and standard deviations (Appendix F.2).

W2 / Q1: Experience-augmented ReAct and Reflexion-style prompting may also help agents correct unsuccessful behavior and reach these stages. […] could you provide explicit comparisons against ReAct and Reflexion using the same model?

Response: We first clarify how these methods relate to our setting, and then report a new Reflexion experiment.

Reflexion addresses a different setup. Reflexion retries the same task several times and needs feedback from the environment after each trial. Our setting allows one trial per task and no environment access afterwards: the agent is improved only from the trajectories it has already logged. This is the practical case when the environment is proprietary, involves real users, or is too expensive to run again.

What the paper already contains. Every policy in the paper acts with a ReAct-style prompt, so the "Base policy" row of Table 2 is ReAct with the same model. The "In-context teacher" row is experience-augmented ReAct: the same model and prompt plus an experience context summarized from the logs.

Comparison with Reflexion (new). We ran Reflexion with the same models. The base policy gets up to three trials on each test task; after a failed trial it writes a reflection on its own trajectory, and the reflections are added to the prompt of the next trial. As a control, we also retry up to three times without reflections. CDUP is evaluated as in the paper, with one trial per task. Success rate (%) on the 384 test tasks:

WebShopWebShopScienceWorldScienceWorld
PolicyTrials per taskLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
Base policy (ReAct)10.00.09.66.5
In-context teacher (experience from the logs)140.421.956.345.8
Reflexionup to 30.00.015.612.2
Retry without reflectionup to 30.00.018.015.4
CDUP (ours)118.810.938.325.3
  • With these models, Reflexion raises success on ScienceWorld by 6.0 and 5.7 points and does not raise it on WebShop. Retrying without reflections does as well on ScienceWorld (18.0% and 15.4%). On WebShop no trial of the base policy ends in a successful purchase, so there is no success to reflect toward.
  • CDUP is above Reflexion in all four settings, with one trial per task and no feedback from the environment: 18.8% and 10.9% against 0.0% on WebShop, and 38.3% and 25.3% against 15.6% and 12.2% on ScienceWorld.
  • Experience summarized from the logs is more informative than the model's own reflections. With the experience context, the same model reaches 21.9% to 56.3% success in a single attempt.

CDUP trains this experience into the model, so the agent needs neither the context nor the retries at deployment. This comparison is in the revised paper (Appendix F.3).

Q3: […] could you compare CDUP with training guided by fine-grained scores assigned to trajectory steps by an additional model, using the same logged data and training budget?

Response: We are not aware of an established method of this form for our setting, but we find it an interesting idea. We see two ways to instantiate it: the first is already in the paper, and we implemented the second.

Interpretation 1: the additional model scores the student's own responses. This is OPCD (Section 2.2). The teacher scores every token of the student's responses at the logged states, with the same states and visit budget as CD. In Table 2 it stays close to CD: 2.6% against 2.3% on WebShop and 21.1% against 21.4% on ScienceWorld with Llama-3.2-3B. Fine-grained scores change the training signal at each state, but not which states are trained on, so the rarely logged states still receive few visits. When we add bottleneck upsampling to OPCD, WebShop success rises from 2.6% to 6.5% (Table 3a).

Interpretation 2: the additional model scores the logged steps, and training keeps the best ones (new). Here the additional model provides no target actions.

  • The scorer is the in-context teacher, that is, the same model with the experience context.
  • Each logged step gets a score from 1 to 5. The scorer reads the state and the action the agent took, and rates the action (5: the action the guidance calls for in this situation; 3: harmless but not useful; 1: invalid, or against the guidance). The full scoring prompt is in the revised paper (Appendix F.4).
  • Training keeps the steps rated 4 or 5, with the same number of training visits as CD and CDUP.

Success rate (%):

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
Base policy0.00.09.66.5
Step-score filtering0.00.014.19.1
CD2.37.821.415.4
CDUP (ours)18.810.938.325.3

Step-score filtering improves over the base policy on ScienceWorld and stays below CD on both benchmarks. Scores can only select among the actions that the logs contain, and the logs contain few good actions at the rarely reached stages.

In both forms, fine-grained scores from the teacher do not provide gains comparable to CDUP.


Q5: How are stages assigned when an agent follows alternative paths, returns to an earlier stage, or visits states outside the defined stage sequence, such as WebShop description and review pages?

Response: Stages have one use in CDUP: they decide which logged states receive extra training visits. Two rules implement this.

  1. Each state gets its own stage label. The classifier reads the label from what the state shows (the task, the current observation, the admissible actions and the earlier steps), not from the position of the state in the episode.
  2. Each episode gets a reach. The reach is the furthest stage the episode attains (Eq. 4). The bottleneck is the first stage that few episodes reach, and every state labeled with that stage or a later one is upsampled.

The three cases you ask about follow from these two rules.

  • Returning to an earlier stage. A state visited after going back carries the label of the earlier stage, because that is what it shows. The reach of the episode does not drop, so loops and long stays in one stage neither inflate nor deflate it.
  • States outside the sequence. WebShop description, features and review subpages belong to no stage (Appendix B). They count toward no stage's reach and are not upsampled.
  • Alternative paths. Stage discovery may return several ordered sequences when tasks follow different procedures (Appendix C), as it does on ScienceWorld. The two rules are then applied within each sequence.

This description is in the revised paper (Appendix F.7).


We hope these results and clarifications address your concerns. If they do, we would appreciate it if you would consider raising your score. We are happy to discuss any remaining question.

Reviewer_5HGGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve language agents from stored trajectories without further environment interaction. It finds that weak agents rarely reach later task stages, leaving few training examples for these decisions. The proposed method, CDUP, uses an LM agent to identify task stages and upsamples states from the first low-reach stage onward. Experiments compare CDUP with uniform context distillation on WebShop and ScienceWorld, with additional tests using on-policy distillation and an external teacher.

Critical strengths
  1. The paper addresses a practical problem: learning from deployment logs when the environment cannot be used again. The proposed change is simple and requires no additional environment interaction during training.

  2. The comparisons are carefully controlled. CD and CDUP share the states, targets, total training visits, and optimizer updates. Random-state, loss-weighting, and balanced-sampling controls help assess whether selecting bottleneck states matters.

  3. The gains are substantial in several settings. With Llama-3.2-3B, success improves from 2.3% to 18.8% on WebShop and from 21.4% to 38.3% on ScienceWorld. Upsampling also improves results with OPCD and an external teacher, providing evidence beyond the main CD setting.

Critical weaknesses
  1. All supervised experiments use one training seed (Appendix E.2). The paired evaluation tests do not measure variation across independently trained models. This is particularly relevant to Qwen2.5-3B on WebShop, where the success gain is not significant under the reported test, although task score improves clearly.

  2. The method uses the stages and classifier from the first LM-agent analysis (Appendix C). The paper studies threshold and upsampling-ratio sensitivity, but does not test different stage definitions or classification errors. These checks would help assess the reliability of bottleneck selection.

Questions for authors
  1. Are the main improvements stable across independent training seeds, particularly the success gain on WebShop with Qwen2.5-3B?

  2. How consistent are the stages and selected bottleneck across repeated LM-agent analyses? How do alternative stage definitions or classification errors affect final performance?

  3. Does CDUP remain effective on tasks with branching or repeated stages, beyond the relatively clear task sequences evaluated here?

3 comments

Thank you for the careful review and the positive assessment of our controlled comparisons. We address your three questions in order. The first two led to new experiments, and for the third we analyzed the test episodes of the submission.


W1 / Q1: All supervised experiments use one training seed (Appendix E.2). […] Are the main improvements stable across independent training seeds, particularly the success gain on WebShop with Qwen2.5-3B?

Response: Yes. We retrained CD, OPCD and CDUP with two more training seeds in all four settings of Table 2, so each number below is a mean ± sd over three seeds. On WebShop with Qwen2.5-3B, where the difference between CD and CDUP is the smallest, CD and CDUP have five seeds. Success rate (%):

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
CD1.6 ± 0.79.4 ± 1.022.5 ± 1.113.7 ± 1.5
OPCD1.6 ± 1.16.5 ± 0.921.4 ± 3.315.7 ± 2.7
CDUP (ours)18.1 ± 0.810.1 ± 2.436.1 ± 2.723.5 ± 2.0

Task score (WebShop) or progress (ScienceWorld), in %:

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
CD3.2 ± 1.323.6 ± 1.153.6 ± 0.746.5 ± 0.4
OPCD5.0 ± 3.131.6 ± 5.047.9 ± 1.647.3 ± 1.1
CDUP (ours)41.6 ± 4.838.8 ± 3.058.9 ± 2.252.0 ± 1.2

The main improvements are stable. CDUP is above CD and OPCD in every setting, in success and in task score or progress. In three settings CDUP is above CD on every seed, by 16.5, 13.6 and 9.8 points of success on average. On WebShop with Qwen2.5-3B success also rises (9.4 to 10.1), and the gain is larger in task score (23.6 to 38.8, higher on every seed). The revised paper reports these means and standard deviations (Appendix F.2).


W2 / Q2: The method uses the stages and classifier from the first LM-agent analysis (Appendix C). […] How consistent are the stages and selected bottleneck across repeated LM-agent analyses? How do alternative stage definitions or classification errors affect final performance?

Response: We repeated the stage discovery of Appendix C on the same prompt and trajectories: once more with GPT-5.6 Sol (run 2; run 1 is the analysis used in the paper), and with two other models, GPT-6 Luna and Claude Haiku 5.5. Each analysis writes its own stage names, boundaries and classifier, so the comparison covers repeated analyses, alternative stage definitions and the classification differences that come with them. We then trained CDUP on the states that each analysis selects. Success rate (%):

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
CD2.37.821.415.4
CDUP, stages from GPT-5.6 Sol, run 118.810.938.325.3
CDUP, stages from GPT-5.6 Sol, run 218.810.926.023.7
CDUP, stages from GPT-6 Luna18.810.933.624.5
CDUP, stages from Claude Haiku 5.518.810.926.623.7
  • WebShop: the bottleneck is identical. The three other analyses place it at the same stage as run 1 and select exactly the same states, so the training run and its result are the same.
  • ScienceWorld with Qwen2.5-3B: the bottleneck is consistent. All analyses select the same boundary, and success stays within 1.6 points.
  • ScienceWorld with Llama-3.2-3B: every selection helps. One stage in this log lies close to the threshold, so the analyses differ in how many states they select (between 16 and 198, against 157 in run 1). All three other selections improve over CD, by 4.6 to 12.2 points.

In short, the four analyses agree on the bottleneck on WebShop and on ScienceWorld with Qwen2.5-3B. A different stage definition can change the size of the gain, and in no case does it put CDUP below CD. This study is in the revised paper (Appendix F.5).

Q3: Does CDUP remain effective on tasks with branching or repeated stages, beyond the relatively clear task sequences evaluated here?

Response: Yes. Our two benchmarks already contain repeated stages, skipped stages and branching, and we checked how CDUP does on exactly those tasks. We grouped the test tasks of the submission by the shape of their path and compared CD and CDUP within each group. If CDUP only worked on clean linear sequences, its gain would vanish in the groups with repeated stages or with a different procedure. It does not.

Repeated and skipped stages (WebShop). A WebShop task states which options the product must have. With no option, the option stage is skipped; with two or more, it is repeated on the same page. Success rate (%) with Llama-3.2-3B, by the number of options the task requires:

Options required by the taskTest episodesCDCDUP (ours)
None (option stage skipped)933.366.7
One4513.331.1
Two (option stage repeated)2430.020.2
Three or more (option stage repeated)870.03.4

CD solves none of the 330 tasks that need the option stage more than once, and CDUP solves 52 of them. Episodes are also not straight lines through the stages: under the CDUP policy, 190 of the 384 test episodes return to an earlier stage at least once.

Branching (ScienceWorld). The seven ScienceWorld task types follow three different procedures. Success rate (%) with Llama-3.2-3B, by procedure:

ProcedureTest episodesCDCDUP (ours)
Find an object and deliver it (4 task types)2407.131.7
Focus on one animal (2 task types)9667.771.9
Focus on two animals in order (1 task type)480.04.2

The gain is largest on the procedure that has the bottleneck, and the other procedures do not lose.

Why the method handles these cases. A state is labeled by what it shows, so a state reached after a loop or a repeat gets the stage it is in now, and it is upsampled whenever that stage is at or after the bottleneck. When tasks follow different procedures, stage discovery returns a separate ordered stage sequence for each group of tasks that share a procedure, and the bottleneck is found within each sequence separately.


Thank you again for the constructive review. We are happy to discuss any further question.

Thank you to the authors for the clarifications and additional experiments. After considering the response, I will maintain my original rating.

Reviewer_WxUDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes CDUP, which improves language-model agents from historical trajectories without environment access. Since weak agents rarely reach late task stages, standard training underrepresents them. CDUP identifies this "bottleneck" stage and upsamples its states during training, showing that emphasizing rare late-stage states is key to effective offline agent learning.

Critical strengths

The identification of the "bottleneck" is meaningful. This points out the shortcomings of standard offline learning.

Critical weaknesses
  1. Missing benchmark results: ALFWorld is introduced as a core benchmark, yet the main experiments and ablations focus almost exclusively on WebShop and ScienceWorld, leaving its evaluation incomplete.
  2. Limited model scale: Experiments rely solely on 3B models (Llama-3.2-3B, Qwen2.5-3B), lacking evidence that the findings generalize to larger or smaller scales.
  3. Arbitrary hyperparameters: The threshold and upsampling ratio are fixed across all settings without theoretical justification or extensive tuning analysis.
  4. Dependency on strong models for stage segmentation without ablation: The method relies on a powerful external LLM (GPT-5.6) to segment trajectories into stages, yet lacks experiments using weaker or alternative models. This omission leaves it unclear whether the approach is robust to imperfect segmentation or strictly dependent on top-tier capabilities, while also introducing hidden API costs and complexity.
  5. Limited applicability to heterogeneous tasks: CDUP assumes trajectories share a uniform, linear stage progression. In real-world environments with diverse tasks, global bottleneck detection becomes meaningless as distinct task flows interfere. Without prior task clustering, this approach cannot generalize beyond highly homogeneous benchmarks like WebShop.
Questions for authors

see weaknesses.

3 comments

Thank you for the review and for recognizing the bottleneck diagnosis. We address the five weaknesses in order. Two of them led to new experiments: larger models (W2) and other segmenters (W4).


W1: Missing benchmark results: ALFWorld is introduced as a core benchmark, yet the main experiments and ablations focus almost exclusively on WebShop and ScienceWorld, leaving its evaluation incomplete.

Response: ALFWorld has a different role in the paper from WebShop and ScienceWorld. It is the benchmark where context distillation already works, and the contrast with the other two is what motivates CDUP.

  • The paper starts from an uneven result. With the same recipe, CD reaches its in-context teacher on ALFWorld (16.4% against 14.8% with Llama-3.2-3B, and 23.8% against 25.5% with Qwen2.5-3B), and falls 14 to 38 points short of it on WebShop and ScienceWorld (Table 1).
  • CDUP is designed for this shortfall. We trace it to stages that the logs rarely reach, and bottleneck upsampling recovers it (Table 2). The main experiments and ablations therefore focus on the two benchmarks where the shortfall exists.
  • On ALFWorld there is no shortfall to recover. The student already matches the teacher that produces its training targets. We therefore report CD and OPCD there (Table 4, Table 14) and add no upsampling arm.

The paper states this in Section 2.4, in Section 3.2 and in Appendix A.


W2: Limited model scale: Experiments rely solely on 3B models (Llama-3.2-3B, Qwen2.5-3B), lacking evidence that the findings generalize to larger or smaller scales.

Response: We ran the full pipeline with two larger models, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, on both WebShop and ScienceWorld. Success rate (%) on the 384 test episodes:

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.1-8BQwen2.5-7BLlama-3.1-8BQwen2.5-7B
Base policy0.00.514.616.9
In-context teacher29.461.788.863.3
CD28.126.866.931.3
CDUP (ours)32.031.580.734.9

At more than twice the model size, uniform CD is better than at 3B (for example, 66.9% success against 21.4% on ScienceWorld with Llama), and CDUP still improves over CD in all four settings, by 3.9, 4.7, 13.8 and 3.6 points. These experiments are in the revised paper (Appendix F.1).


W3: Arbitrary hyperparameters: The threshold and upsampling ratio are fixed across all settings without theoretical justification or extensive tuning analysis.

Response: The results are robust to both values (Appendix D.2), and each value has a simple reading.

  • Threshold η\eta: which stage counts as rarely reached. With η=10%\eta = 10\%, a stage is a bottleneck when nine in ten logged episodes never reach it. The threshold only decides which stage is the first one below it, so the result changes only when η\eta crosses the reach of a stage. Any η\eta between 6.25% and 11.88% gives exactly the same training run in all four settings (Figure 4). At η=20%\eta = 20\% only one setting changes (ScienceWorld with Llama-3.2-3B), and CDUP is still above CD there: 31.8% against 21.4%.
  • Ratio κ\kappa: how much of the training budget moves to the bottleneck states. Under uniform training these states receive 0.6% to 3.9% of the training visits. With κ=8\kappa = 8 they receive 4.3% to 24.4%, and the other states keep most of the budget. Raising κ\kappa from 8 to 32 with Qwen2.5-3B gives 27.9% against 25.3% on ScienceWorld and 10.2% against 10.9% on WebShop (Table 9), both above uniform training (15.4% and 7.8%). Matched random-state controls at κ=8\kappa = 8 and κ=32\kappa = 32 stay at 17.2% and 16.7%, so what matters is which states are upsampled, not the exact ratio.

W4: Dependency on strong models for stage segmentation without ablation: The method relies on a powerful external LLM (GPT-5.6) to segment trajectories into stages, yet lacks experiments using weaker or alternative models. […] while also introducing hidden API costs and complexity.

Response: We ran this ablation. We repeated the stage discovery of Appendix C with two other models on the same prompt and trajectories: GPT-6 Luna and Claude Haiku 5.5, a smaller and cheaper model. We then trained CDUP on the states that each model selects. The table also includes a second run with GPT-5.6 Sol (run 1 is the analysis used in the paper). Success rate (%):

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.2-3BQwen2.5-3BLlama-3.2-3BQwen2.5-3B
CD2.37.821.415.4
CDUP, stages from GPT-5.6 Sol, run 118.810.938.325.3
CDUP, stages from GPT-5.6 Sol, run 218.810.926.023.7
CDUP, stages from GPT-6 Luna18.810.933.624.5
CDUP, stages from Claude Haiku 5.518.810.926.623.7
  • A top-tier segmenter is not required. On WebShop both models, including the smaller one, select exactly the states that GPT-5.6 Sol selects, so the result is the same. On ScienceWorld both improve over CD: with Qwen2.5-3B they stay within 1.6 points of GPT-5.6 Sol, and with Llama-3.2-3B the gain over CD is 12.2 and 5.2 points.
  • The cost is small. Segmentation is one offline LM-agent run per benchmark, before training. Its output is a stage list and a classifier written as code, so labeling the training states needs no further model call. One run with GPT-5.6 Sol reads under 0.8M input tokens, about 95% of them cached, and writes 9k to 12k tokens. One run with Claude Haiku 5.5 took under five minutes and cost about $0.06 at list price.

W5: Limited applicability to heterogeneous tasks: CDUP assumes trajectories share a uniform, linear stage progression. […] Without prior task clustering, this approach cannot generalize beyond highly homogeneous benchmarks like WebShop.

Response: CDUP does not assume one global stage sequence, and it needs no separate clustering step. ScienceWorld is a concrete case of heterogeneous tasks in the paper.

The seven ScienceWorld task types follow different procedures. Stage discovery receives the pooled trajectories of all seven and returns separate ordered stage sequences for the tasks that find and deliver an object and for the tasks that focus on animals by lifespan. The bottleneck rule is then applied within each sequence, so one task flow does not interfere with the bottleneck of another. Success rate (%) with Llama-3.2-3B on the test episodes, by procedure:

ProcedureTest episodesCDCDUP (ours)
Find an object and deliver it (4 task types)2407.131.7
Focus on one animal (2 task types)9667.771.9
Focus on two animals in order (1 task type)480.04.2
  • CDUP improves the procedure that has a bottleneck and does not hurt the others. The find-and-deliver tasks, whose later stages the logs rarely reach, improve from 7.1% to 31.7%. The two other procedures stay at or above CD.
  • WebShop tasks also follow different stage paths. They require between zero and three or more options, so one stage is skipped in some tasks and repeated in others, and almost every logged episode returns to an earlier stage at least once. The bottleneck rule applies unchanged.
  • Where no stage is rarely reached, CDUP is the same as CD. Nothing is selected for such a sequence, so upsampling cannot hurt it.

We hope these results and clarifications address your concerns. If they do, we would appreciate it if you would consider raising your score. We are happy to discuss any remaining question.

Reviewer_WxUDOpenReview ↗

Response to authors' rebuttal

Thank you to the authors for their rebuttal, which has addressed some of my concerns. I will raise my score to 3.

Reviewer_K9AUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how to improve an LLM agent from trajectories when the environment can no longer be run. The authors start from context distillation, and they find large gaps to this in-context teacher on WebShop and ScienceWorld. They attribute the gaps to coverage: a weak agent rarely reaches the later stages of a task, so the trained student gets there but fails at the decisions made in those rarely logged states. Their method, CDUP, has an LM agent split the trajectories into ordered stages, marks the first stage reached by fewer than 10% of episodes and everything after it as the bottleneck, and trains on those states 8 times as often at a fixed budget. With two 3B models, CDUP improves over uniform context distillation, beats a matched random-states control, and also helps on-policy context distillation and an external expert teacher.

Critical strengths
  1. I found the diagnosis are clear and easy to check. Stage-reach curves on held-out set show that the distilled student does reach the stages its data rarely contains, but then fails at the decisions there, while its teacher keeps going.
  2. The control experiments are carefully designed and budget matched, hence the result is trustworthy.
Critical weaknesses

My main concern is that the setting makes the bottleneck extreme by construction, so I am not sure how far the result reaches. The deployed agents are very weak to start with, and because the logs barely show what to do past the bottleneck, that knowledge comes mostly from the guide gpt-5.6 sol writes. I suspect the guide can supply much that neither the agent nor its logs contain, then what the paper shows, is that a much stronger model's guidance can be distilled into the rarely visited states of a weak 3B agent. That is useful, but it does not tell us whether bottlenecks this sharp exist for the capable agents that motivate the paper.

Questions for authors

All WebShop experiments use the Spark prompt. Under it, base Qwen2.5-3B clicks buy now in 0.0% of episodes, against 9.6% under WebShop’s original prompt. If the deployment trajectories were collected with the original prompt instead, would the product page still fall below the 10% bottleneck threshold, and how large would CDUP’s gain over CD be? This would show whether the method still helps when the deployed agent is less weak.

2 comments

Thank you for the precise summary and the thoughtful questions. We ran the experiment you proposed, and we address each point below.


Q: All WebShop experiments use the Spark prompt. Under it, base Qwen2.5-3B clicks buy now in 0.0% of episodes, against 9.6% under WebShop's original prompt. If the deployment trajectories were collected with the original prompt instead, would the product page still fall below the 10% bottleneck threshold, and how large would CDUP's gain over CD be?

Response: Thank you for suggesting this experiment. We ran it as proposed: we collected the Qwen2.5-3B deployment trajectories again with WebShop's original prompt, and repeated the stage analysis, relabeling, training and evaluation under that prompt.

CDUP still improves over CD under the original prompt, by 13.3 points of success. The deployed agent is less weak under this prompt, as you expected: 43.9% of its logged episodes reach the product page. Uniform CD still falls short of its in-context teacher (18.5% against 30.7%). As in the submission, CDUP upsamples the states from the product page onward, with the same ratio κ=8\kappa = 8 and the same visit budget. Results in % on the 384 test episodes:

PolicySuccessScore
Base policy0.54.3
In-context teacher30.753.3
CD18.549.5
CDUP (ours)31.857.8

CDUP reaches the level of the teacher. Upsampling therefore still helps when the deployed agent is less weak. The submission uses the Spark prompt because it is the prompt of the released Spark teacher, so that the external-teacher experiment (Section 4.3) and the self-teacher experiments share one prompt. This experiment is in the revised paper (Appendix F.6).


W1-a: My main concern is that the setting makes the bottleneck extreme by construction, so I am not sure how far the result reaches. The deployed agents are very weak to start with […] it does not tell us whether bottlenecks this sharp exist for the capable agents that motivate the paper.

Response: Thank you for raising this. We tested it with more capable agents: we ran the full pipeline with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct on both benchmarks. Success rate (%) on the 384 test episodes:

WebShopWebShopScienceWorldScienceWorld
PolicyLlama-3.1-8BQwen2.5-7BLlama-3.1-8BQwen2.5-7B
Base policy0.00.514.616.9
In-context teacher29.461.788.863.3
CD28.126.866.931.3
CDUP (ours)32.031.580.734.9

These agents are no longer very weak: with Llama-3.1-8B, uniform CD alone reaches 66.9% success on ScienceWorld, against 21.4% at 3B. CDUP still improves over CD in all four settings, by up to 13.8 points. Uniform distillation therefore still undertrains the rarely reached stages with more capable agents, and upsampling them still helps.

W1-b: […] because the logs barely show what to do past the bottleneck, that knowledge comes mostly from the guide gpt-5.6 sol writes. I suspect the guide can supply much that neither the agent nor its logs contain, then what the paper shows, is that a much stronger model's guidance can be distilled into the rarely visited states of a weak 3B agent.

Response: Thank you for this point, which lets us clarify what the paper compares. Our comparison holds the guide fixed and varies only how the student is trained on it. GPT-5.6 Sol only writes the guide, once and before training. The teacher that produces every training target is the self-teacher: the same 3B model as the student, with the guide in its prompt.

  • Every method reads the same guide. The in-context teacher, CD, OPCD and CDUP all use the same experience context, and CD and CDUP also share the teacher responses, the visits and the updates. Whatever the guide contains, every baseline has it under the same conditions. The comparison asks one question: given the same guidance, which training procedure gets it into the student?
  • With the same guide, uniform training does not learn the decisions at the bottleneck. With Llama-3.2-3B on WebShop, CD's student buys in 7.0% of episodes, against 99.2% for its self-teacher, and CDUP raises this to 57.6% (Table 7). This difference cannot come from the guide, because the guide is identical. Giving the same extra visits to randomly chosen states does not help either: 3.4% success against 18.8% for CDUP in this setting (Table 12).

The guide by itself does not solve the tasks. With the guide in its prompt, the same 3B model reaches 21.9% to 56.3% success in the four settings (Table 2). The guide gives general guidance. For example, its part for the WebShop search and results pages reads (Qwen2.5-3B):

A search page. Your action is one search[...] holding four to eight words: the product type plus two or three describing words. A results page, where actions shaped click[b...] are listed. Your action is one click[b...]. Titles are short and leave out most attributes, so a product page is the place where an attribute is confirmed.

It does not say what to search for or which product to open for any task; the 3B model decides this at every state.

The guide determines what the self-teacher can show, and it is the same for every method. Our contribution is the allocation of training visits: with the guide fixed, the student learns the self-teacher's decisions at rarely logged states only when training gives these states enough visits.


Thank you again for the thoughtful review. We are happy to discuss any further question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.