- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper investigates how to improve a language agent using previously collected trajectories without further interaction with the environment during training. It points out that Uniform Context Distillation (OPCD) underestimates the decision-making power of task stages rarely reached by weakly deployed policies. CDUP uses a more powerful Language Model (LM) agent to generate task stages and a classifier, identifying the first stage reached less than a fixed proportion, and upsampling the retained states of that stage and subsequent stages while keeping the number of training visits constant. Experiments were conducted using WebShop and seven ScienceWorld task types, employing Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct agents, respectively. CDUP is compared to CD and OPCD, incorporating stochastic states, loss weighting, and stage balancing controls, while the upsampling effects of OPCD and external teacher models are tested on WebShop. In the reported experiments, CDUP outperforms both CD and OPCD.
Critical strengths
-
An effective diagnostic approach supplemented with a controlled intervention. The distinction between teacher objectives and allocating sufficient training to corresponding states is crucial for learning from a fixed log. In the primary experiment, CD and CDUP shared a maintained library of objectives, training access, and updates, while the matched randomized state control group maintained the allocation pattern while varying which states received additional access. Combined with stage achievement analysis, these comparative results demonstrate that, in the test protocol, state-specific focused training is more valuable than general CD training.
-
A simple intervention applicable to different training objectives and teachers. Upsampling alters the allocation of existing training budgets without requiring new actions during training. Table 3 shows the effectiveness of the OPCD and external teacher SFT approaches on WebShop, demonstrating the intervention's effectiveness outside the primary CD setup.
Critical weaknesses
-
Limited Evidence on Generalization and Robustness. Experiments conducted solely on the 3B model cannot determine whether the method's advantages are limited to small, weak models. Despite limited computational resources, experiments on the 7B model are crucial for validating this possibility. Furthermore, supervised learning results rely on a single training seed, requiring repeated training with an error range to assess its variability and reliability.
-
Limited Baseline Coverage. This paper attributes the inadequacy of learning to the fact that the recorded trajectories rarely reach the later stages of the task. Experience-enhanced ReAct and Reflexion-style cues may also help the agent correct unsuccessful behaviors and reach these stages. Although the paper includes ReAct-style cues and contextual teachers, the advantages of CDUP over dedicated reflection-based methods are unclear due to limited comparisons. Explicit comparisons with these cuing methods will help determine the method's advantages.
Questions for authors
-
Comparison with Hint Methods Given that the contextual teacher outperforms CDUP in Table 2, could you provide a clear comparison with ReAct and Reflexion using the same model?
-
Generalization Ability Beyond 3B Models Although the paper acknowledges limitations in computational resources, validation on larger models remains crucial to determine if these improvements are limited to small, weak models. Could you evaluate CDUP on a 7B model for validation?
-
Comparison with Fine-Grained Trajectory Scoring Given that the paper discusses teacher-based supervision, could you compare CDUP with training guided by fine-grained scoring assigned by another model based on trajectory steps, using the same log data and training budget?
-
Variability of Training Runs Could you provide the average performance and error range across multiple runs?
-
Stage Assignment How are stages assigned when the agent follows an alternative path, returns to an earlier stage, or accesses a state outside the defined stage sequence (e.g., WebShop description and review pages)?
8 comments
W2 / Q1: Experience-augmented ReAct and Reflexion-style prompting may also help agents correct unsuccessful behavior and reach these stages. […] could you provide explicit comparisons against ReAct and Reflexion using the same model?
Response: We first clarify how these methods relate to our setting, and then report a new Reflexion experiment.
Reflexion addresses a different setup. Reflexion retries the same task several times and needs feedback from the environment after each trial. Our setting allows one trial per task and no environment access afterwards: the agent is improved only from the trajectories it has already logged. This is the practical case when the environment is proprietary, involves real users, or is too expensive to run again.
What the paper already contains. Every policy in the paper acts with a ReAct-style prompt, so the "Base policy" row of Table 2 is ReAct with the same model. The "In-context teacher" row is experience-augmented ReAct: the same model and prompt plus an experience context summarized from the logs.
Comparison with Reflexion (new). We ran Reflexion with the same models. The base policy gets up to three trials on each test task; after a failed trial it writes a reflection on its own trajectory, and the reflections are added to the prompt of the next trial. As a control, we also retry up to three times without reflections. CDUP is evaluated as in the paper, with one trial per task. Success rate (%) on the 384 test tasks:
| WebShop | WebShop | ScienceWorld | ScienceWorld | ||
|---|---|---|---|---|---|
| Policy | Trials per task | Llama-3.2-3B | Qwen2.5-3B | Llama-3.2-3B | Qwen2.5-3B |
| Base policy (ReAct) | 1 | 0.0 | 0.0 | 9.6 | 6.5 |
| In-context teacher (experience from the logs) | 1 | 40.4 | 21.9 | 56.3 | 45.8 |
| Reflexion | up to 3 | 0.0 | 0.0 | 15.6 | 12.2 |
| Retry without reflection | up to 3 | 0.0 | 0.0 | 18.0 | 15.4 |
| CDUP (ours) | 1 | 18.8 | 10.9 | 38.3 | 25.3 |
- With these models, Reflexion raises success on ScienceWorld by 6.0 and 5.7 points and does not raise it on WebShop. Retrying without reflections does as well on ScienceWorld (18.0% and 15.4%). On WebShop no trial of the base policy ends in a successful purchase, so there is no success to reflect toward.
- CDUP is above Reflexion in all four settings, with one trial per task and no feedback from the environment: 18.8% and 10.9% against 0.0% on WebShop, and 38.3% and 25.3% against 15.6% and 12.2% on ScienceWorld.
- Experience summarized from the logs is more informative than the model's own reflections. With the experience context, the same model reaches 21.9% to 56.3% success in a single attempt.
CDUP trains this experience into the model, so the agent needs neither the context nor the retries at deployment. This comparison is in the revised paper (Appendix F.3).
Q3: […] could you compare CDUP with training guided by fine-grained scores assigned to trajectory steps by an additional model, using the same logged data and training budget?
Response: We are not aware of an established method of this form for our setting, but we find it an interesting idea. We see two ways to instantiate it: the first is already in the paper, and we implemented the second.
Interpretation 1: the additional model scores the student's own responses. This is OPCD (Section 2.2). The teacher scores every token of the student's responses at the logged states, with the same states and visit budget as CD. In Table 2 it stays close to CD: 2.6% against 2.3% on WebShop and 21.1% against 21.4% on ScienceWorld with Llama-3.2-3B. Fine-grained scores change the training signal at each state, but not which states are trained on, so the rarely logged states still receive few visits. When we add bottleneck upsampling to OPCD, WebShop success rises from 2.6% to 6.5% (Table 3a).
Interpretation 2: the additional model scores the logged steps, and training keeps the best ones (new). Here the additional model provides no target actions.
- The scorer is the in-context teacher, that is, the same model with the experience context.
- Each logged step gets a score from 1 to 5. The scorer reads the state and the action the agent took, and rates the action (5: the action the guidance calls for in this situation; 3: harmless but not useful; 1: invalid, or against the guidance). The full scoring prompt is in the revised paper (Appendix F.4).
- Training keeps the steps rated 4 or 5, with the same number of training visits as CD and CDUP.
Success rate (%):
| WebShop | WebShop | ScienceWorld | ScienceWorld | |
|---|---|---|---|---|
| Policy | Llama-3.2-3B | Qwen2.5-3B | Llama-3.2-3B | Qwen2.5-3B |
| Base policy | 0.0 | 0.0 | 9.6 | 6.5 |
| Step-score filtering | 0.0 | 0.0 | 14.1 | 9.1 |
| CD | 2.3 | 7.8 | 21.4 | 15.4 |
| CDUP (ours) | 18.8 | 10.9 | 38.3 | 25.3 |
Step-score filtering improves over the base policy on ScienceWorld and stays below CD on both benchmarks. Scores can only select among the actions that the logs contain, and the logs contain few good actions at the rarely reached stages.
In both forms, fine-grained scores from the teacher do not provide gains comparable to CDUP.
Q5: How are stages assigned when an agent follows alternative paths, returns to an earlier stage, or visits states outside the defined stage sequence, such as WebShop description and review pages?
Response: Stages have one use in CDUP: they decide which logged states receive extra training visits. Two rules implement this.
- Each state gets its own stage label. The classifier reads the label from what the state shows (the task, the current observation, the admissible actions and the earlier steps), not from the position of the state in the episode.
- Each episode gets a reach. The reach is the furthest stage the episode attains (Eq. 4). The bottleneck is the first stage that few episodes reach, and every state labeled with that stage or a later one is upsampled.
The three cases you ask about follow from these two rules.
- Returning to an earlier stage. A state visited after going back carries the label of the earlier stage, because that is what it shows. The reach of the episode does not drop, so loops and long stays in one stage neither inflate nor deflate it.
- States outside the sequence. WebShop description, features and review subpages belong to no stage (Appendix B). They count toward no stage's reach and are not upsampled.
- Alternative paths. Stage discovery may return several ordered sequences when tasks follow different procedures (Appendix C), as it does on ScienceWorld. The two rules are then applied within each sequence.
This description is in the revised paper (Appendix F.7).
We hope these results and clarifications address your concerns. If they do, we would appreciate it if you would consider raising your score. We are happy to discuss any remaining question.
Thank you for the careful review and the constructive questions. We ran the requested experiments and added them to the revised paper, where new text is in blue (Appendix F). We address each point below.
Response: We ran the full pipeline with two larger models, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, on both WebShop and ScienceWorld. Success rate (%) on the 384 test episodes:
Larger models are stronger before and after training. Uniform CD is better than at 3B: on WebShop its success rises from 2.3% and 7.8% to 28.1% and 26.8%, and on ScienceWorld from 21.4% and 15.4% to 66.9% and 31.3%. CDUP still improves over CD in all four settings, by 3.9, 4.7, 13.8 and 3.6 points. The benefit of bottleneck upsampling is therefore not specific to small, weak models. These experiments are in the revised paper (Appendix F.1).
Response: We retrained CD, OPCD and CDUP with two more training seeds in all four settings of Table 2, so each number below is a mean ± sd over three seeds. On WebShop with Qwen2.5-3B, where the difference between CD and CDUP is the smallest, CD and CDUP have five seeds. Success rate (%):
Task score (WebShop) or progress (ScienceWorld), in %:
The results of the submission are reproduced. CDUP is above CD and OPCD in every setting, in success and in task score or progress. In three settings CDUP is above CD on every seed, by 16.5, 13.6 and 9.8 points of success on average. On WebShop with Qwen2.5-3B success also rises (9.4 to 10.1), and the gain is larger in task score (23.6 to 38.8, higher on every seed). The revised paper reports these means and standard deviations (Appendix F.2).