- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PACE trains an assistant to decide when to intervene and what advice to give while a player controls the actions. During training, it restores the same game state and compares silence, the original advice, and alternative messages. Task progress and intervention costs provide separate feedback for the decision to speak and the advice content. The model first learns from teacher examples, then improves through reinforcement learning. Separate policies are trained for ALFWorld and Minecraft.
Critical strengths
- Separating feedback for intervention timing and advice content addresses a real issue: good advice can still be unnecessary. Comparing against silence gives a direct way to assess whether speaking helped.
Critical weaknesses
-
Broader transfer is unclear. Separate policies are trained for the two environments, and training needs replayable states, task milestones, and reward definitions. Transfer to new players is useful, but does not establish transfer to new domains. It is unclear how much setup and retraining a new application would need.
-
Training-data size and total cost are not fully reported. Appendix B explains the data sources, generation process, and RL schedule, but I could not find the total numbers of assistance demonstrations and visual captions, or a complete cost breakdown including branch rollouts. This makes it difficult to judge data efficiency and the practical cost of adapting the method.
Questions for authors
-
Please report the numbers of demonstrations and captions, the training task coverage, and total environment steps and compute, including branch rollouts. Are the human-study task goals excluded from training?
-
What other domains do you think this method could realistically support? What would be needed to obtain replayable states, milestones, and rewards there, and how much new data or training would be required?
-
Can the trained policy handle held-out tasks without further fine-tuning? Is there a reason to train separate policies rather than one policy across environments?
4 comments
Dear Reviewer,
As the discussion period is nearing its end, we hope our responses have addressed your concerns. We would greatly appreciate any further feedback and would be happy to clarify any remaining questions.
Thank you again for your time and consideration.
Thanks to the authors for the clear answers and new experiments. The data and compute numbers answer my cost question and the held-out Minecraft goals show the method works on unseen tasks. A test in a new domain would still help but my main concerns are addressed. I will raise my score to weak accept.
Thank you very much for your reconsideration. We are glad that our responses have addressed your main concerns. We sincerely appreciate your constructive suggestions, which have helped us further strengthen our work.
Q1. Training data, task coverage, and compute
ALFWorld. We used 6,622 step-level assistance demonstrations from 3,311 of the 3,553 training task instances (about 93%), covering all six task categories. We used 3,128 rollout records to initialize the critic. RL episodes were sampled from the same ALFWorld training split.
Minecraft. We used 1,017 visual captions from a human gameplay corpus. The Minecraft RL training set covered 20 distinct task goals.
The table reports environment steps and compute for the PACE RL runs on two NVIDIA A100 80 GB GPUs.
Of the three human-study goals, Craft a Stone Pickaxe and Obtain an Iron Ingot were included in the Minecraft training goal set. Make Bed was held out from that set.
Q2. Other domains, porting requirements, and cost
Adapting PACE to new domains requires restorable states, measurable task progress, demonstrations, while the credit assignment method remains unchanged. For example, it could be applied to web tasks using restorable browser sessions and to coding tasks using repository checkpoints and tests. For reference, ALFWorld used 6,622 demonstrations and 90 RL updates, while Minecraft used 1,017 visual captions and 60 RL updates as mentioned above.
Q3. Can the trained policy handle held-out tasks without further fine-tuning? Is there a reason to train separate policies rather than one policy across environments?
A1: Yes. We additionally evaluate PACE on held-out Minecraft tasks without further training, using the same frozen player. PACE improves success rates across all evaluated tasks, demonstrating generalization to goals excluded from RL training.
A2: We train separate companions for ALFWorld and Minecraft to accommodate their different environment dynamics, task structures, and observation modalities, while keeping the nested credit assignment method unchanged. Since the two environments have different interaction and task execution mechanisms, each companion needs to learn when to intervene and what advice to provide based on its own environment feedback. Joint training could introduce gradient interference, as updates from one environment may conflict with those from the other. Separate training avoids this additional complexity. GiGPO [1] and GraphGPO [2] also adopt benchmark-specific training.
[1] Group-in-Group Policy Optimization for LLM Agent Training. Feng et al. NeurIPS 2025.
[2] Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning. Cheng et al. ICML 2026.