acceptodds
Under review as a conference paper at ICLR 2027

The Right Advice at the Right Time: Learning Proactive Assistance through Nested Credit Assignment

Abstract

In complex interactive tasks, users may spend considerable time pursuing ineffective strategies before pausing to seek external help. Proactive assistance can address this gap by offering contextual guidance without waiting for explicit requests or taking over task execution. Learning such assistance involves two coupled decisions: when to intervene and what advice to provide. Whether intervention is worthwhile depends on the advice provided, while evaluating that advice must account for the user’s ability to progress without assistance. Downstream task outcomes therefore provide ambiguous feedback for learning intervention timing and advice content. In this work, we propose PACE Proactive Assistance through nested Credit Estimation, a reinforcement learning framework that jointly trains a generative vision–language companion through nested credit assignment. By comparing estimated returns under silence and alternative advice from the same interaction state, PACE derives distinct learning signals for the intervention decision and conditional advice generation. These signals guide joint optimization of timing and content while balancing task progress against intervention cost. Experiments with multiple LLM-controlled players in ALFWorld and Minecraft show that PACE achieves higher overall task success rates with lower intervention rates than general-purpose LLMs and specialized game-playing models used as companions. A human study further demonstrates the effectiveness of PACE.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Companion Agents, Reinforcement Learning, Large Language Model Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_h9raOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

PACE trains an assistant to decide when to intervene and what advice to give while a player controls the actions. During training, it restores the same game state and compares silence, the original advice, and alternative messages. Task progress and intervention costs provide separate feedback for the decision to speak and the advice content. The model first learns from teacher examples, then improves through reinforcement learning. Separate policies are trained for ALFWorld and Minecraft.

Critical strengths
  1. Separating feedback for intervention timing and advice content addresses a real issue: good advice can still be unnecessary. Comparing against silence gives a direct way to assess whether speaking helped.
Critical weaknesses
  1. Broader transfer is unclear. Separate policies are trained for the two environments, and training needs replayable states, task milestones, and reward definitions. Transfer to new players is useful, but does not establish transfer to new domains. It is unclear how much setup and retraining a new application would need.

  2. Training-data size and total cost are not fully reported. Appendix B explains the data sources, generation process, and RL schedule, but I could not find the total numbers of assistance demonstrations and visual captions, or a complete cost breakdown including branch rollouts. This makes it difficult to judge data efficiency and the practical cost of adapting the method.

Questions for authors
  1. Please report the numbers of demonstrations and captions, the training task coverage, and total environment steps and compute, including branch rollouts. Are the human-study task goals excluded from training?

  2. What other domains do you think this method could realistically support? What would be needed to obtain replayable states, milestones, and rewards there, and how much new data or training would be required?

  3. Can the trained policy handle held-out tasks without further fine-tuning? Is there a reason to train separate policies rather than one policy across environments?

4 comments

Q1. Training data, task coverage, and compute

ALFWorld. We used 6,622 step-level assistance demonstrations from 3,311 of the 3,553 training task instances (about 93%), covering all six task categories. We used 3,128 rollout records to initialize the critic. RL episodes were sampled from the same ALFWorld training split.

Minecraft. We used 1,017 visual captions from a human gameplay corpus. The Minecraft RL training set covered 20 distinct task goals.

The table reports environment steps and compute for the PACE RL runs on two NVIDIA A100 80 GB GPUs.

EnvironmentRL updatesMain-trajectory stepsBranch-rollout stepsTotal environment stepsGPU hours
ALFWorld90~12.4k~35k~47.4k~27h
Minecraft60~14k~32k~46k~21h

Of the three human-study goals, Craft a Stone Pickaxe and Obtain an Iron Ingot were included in the Minecraft training goal set. Make Bed was held out from that set.

Q2. Other domains, porting requirements, and cost

Adapting PACE to new domains requires restorable states, measurable task progress, demonstrations, while the credit assignment method remains unchanged. For example, it could be applied to web tasks using restorable browser sessions and to coding tasks using repository checkpoints and tests. For reference, ALFWorld used 6,622 demonstrations and 90 RL updates, while Minecraft used 1,017 visual captions and 60 RL updates as mentioned above.

Q3. Can the trained policy handle held-out tasks without further fine-tuning? Is there a reason to train separate policies rather than one policy across environments?

A1: Yes. We additionally evaluate PACE on held-out Minecraft tasks without further training, using the same frozen player. PACE improves success rates across all evaluated tasks, demonstrating generalization to goals excluded from RL training.

Held-out taskSolo (SR / MS)+ Companion (SR / MS)
Craft Boat80% / 0.90100% / 1.00
Craft Torch70% / 0.8990% / 0.94
Smelt Stone10% / 0.5170% / 0.91

A2: We train separate companions for ALFWorld and Minecraft to accommodate their different environment dynamics, task structures, and observation modalities, while keeping the nested credit assignment method unchanged. Since the two environments have different interaction and task execution mechanisms, each companion needs to learn when to intervene and what advice to provide based on its own environment feedback. Joint training could introduce gradient interference, as updates from one environment may conflict with those from the other. Separate training avoids this additional complexity. GiGPO [1] and GraphGPO [2] also adopt benchmark-specific training.

[1] Group-in-Group Policy Optimization for LLM Agent Training. Feng et al. NeurIPS 2025.

[2] Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning. Cheng et al. ICML 2026.

Dear Reviewer,

As the discussion period is nearing its end, we hope our responses have addressed your concerns. We would greatly appreciate any further feedback and would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Thanks to the authors for the clear answers and new experiments. The data and compute numbers answer my cost question and the held-out Minecraft goals show the method works on unseen tasks. A test in a new domain would still help but my main concerns are addressed. I will raise my score to weak accept.

Thank you very much for your reconsideration. We are glad that our responses have addressed your main concerns. We sincerely appreciate your constructive suggestions, which have helped us further strengthen our work.

Reviewer_uS2bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PACE, a reinforcement learning framework for training a proactive companion that provides advice without taking over the user’s actions. To learn the coupled decisions of when to intervene and what advice to provide, PACE constructs matched branches from the same interaction state, including silence, the current advice, and alternative advice. Subsequent task progress and intervention costs provide distinct learning signals for timing and content. Its central contribution is learning the value of advice relative to silence, rather than merely generating correct advice. Experiments in ALFWorld, Minecraft, and a human study show improved task performance, reduced unnecessary intervention, and transfer to player models unseen during training.

Critical strengths

The method directly addresses the identified problem. The silence branch accounts for the player’s progress without the current advice, while separate credit signals for intervention and content address attribution in proactive assistance.

The experiments target the core design choices. Shared-credit PPO/GRPO comparisons, timing comparisons with fixed advice, and cross-player and human evaluations support the method beyond aggregate performance improvements.

Critical weaknesses

Limited direct validation of long-term credit estimates. Branches are evaluated over a finite horizon, with subsequent returns estimated by a critic. The paper does not sufficiently establish whether the estimated HELP-versus-SILENCE differences agree with full-episode returns, particularly for advice with delayed benefits.

Limited isolation of proactive assistance benefits in the human study. The study includes only 20 participants and three tasks, and allows participant-initiated questions. This limits attribution of the overall benefits specifically to the proactive intervention policy.

Questions for authors

How often do finite-horizon branch estimates agree with full-episode returns on whether HELP outperforms SILENCE? Does changing the branch rollout horizon affect the main conclusions?

Can the human study distinguish benefits from proactive guidance versus responses to participant-initiated questions? How much of the improvement in completion time and user experience comes specifically from the intervention timing policy?

3 comments

Q1. Do finite-horizon estimates agree with full-episode returns, and does the horizon affect the main conclusions?

On the states where helping and staying silent lead to a clearly different outcome (∣ΔQfull∣≥0.1|\Delta Q_{\mathrm{full}}| \geq 0.1; 84 pairs, 42% of 200), the sign of the finite-horizon return difference agrees with the full-episode comparison in 64%, 83%, and 90% of cases for N=4,8,16N=4,8,16. Across all pairs, including near-ties where the sign is sensitive to estimation noise, agreement is 56%, 73%, and 82%.

Evaluation setN=4N=8N=16
Absolute full-episode return gap ≥ 0.164%83%90%

Our ALFWorld training uses N=8N=8, achieving 83% sign agreement with full-episode returns for substantial return gaps. We are conducting additional training runs with N=16N=16 and N=fullN=\mathrm{full} to further assess the robustness of PACE's performance across different rollout horizons.

Q2. Can the human study distinguish benefits from proactive guidance versus responses to participant-initiated questions? How much of the improvement in completion time and user experience comes specifically from the intervention timing policy?

A1: We separate the human-study results based on whether participants asked questions. Among participants who did not ask questions, PACE reduces completion time compared with Solo across all three tasks, demonstrating the benefits of proactive guidance independently of participant-initiated questions.

ConditionStone PickaxeIron IngotMake Bed
Solo112.3202.5120.4
GPT, proactive only89.8158.0101.1
PACE, proactive only90.2137.292.5
GPT, with questions–183.9110.3
PACE, with questions114.7116.596.1

A2: A key contribution of PACE is learning the when and the what together. Intervention at different moments means responding to different player states and needs, which in turn changes what advice is appropriate. The advice provided can also influence the player's subsequent actions and need for further guidance. Timing and content are therefore closely interdependent, and evaluating either in isolation would not fully capture the effectiveness of the given advice. Our human study therefore evaluates the complete assistance policy during natural interaction.

To examine timing more directly, Fig. 2(a) compares the learned policy with fixed-frequency and random intervention schedules using the same advice generator. In the automated Minecraft evaluation, learned timing achieves higher success rates, including over the Every-5-Steps schedule at a similar intervention rate. This suggests that choosing when to intervene matters, rather than simply providing more assistance. In the human study, participants also rated PACE higher on timing and intervention appropriateness (Fig. 2(b)). We will clarify how these two evaluations provide complementary evidence for the effectiveness of PACE in our updated paper.

For clarity, we provide the numerical results underlying Fig. 2(a) and Fig. 2(b) below.

Fig. 2(a): Intervention timing analysis

MethodSRHELP
Solo56.7%0.0%
Every Step90.0%100.0%
Every 2 Steps73.3%35.2%
Every 5 Steps86.7%18.6%
Random83.0%32.4%
PACE93.3%16.6%

Fig. 2(b): Subjective evaluation (7-point Likert scale)

MethodHelpfulnessTimingAppropriatenessClarity
GPT4.404.354.454.55
PACE6.105.805.705.30

Follow-up on Q1: Full-Episode Rollout Experiment

We further evaluated PACE using full Monte Carlo returns, keeping all other training settings unchanged.

Return EstimationSR ↑HELP ↓Training Time
PACE (N=8 + Critic)56.7%44.5%27 GPU-hours
PACE (Full MC)60.9%47.3%46 GPU-hours

Full MC improves success by 4.2 percentage points but requires approximately 70% more training time. This suggests that N=8 already provides effective credit estimates at substantially lower computational cost, offering a practical balance between performance and efficiency.

Dear Reviewer,

As the discussion period is nearing its end, we hope our responses have addressed your concerns. We would greatly appreciate any further feedback and would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Reviewer_FRF6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies proactive assistance, where a companion watches an advisee act in a multi-step task and decides at each step whether to intervene and what advice to give. Since the companion affects outcomes only through the advisee, the paper frames learning these two decisions as a nested credit-assignment problem. PACE trains a vision-language companion by restoring states where it chose to help and rolling out silence, the realized advice and fresh alternative messages with a frozen advisee. The gate is credited with the return difference between the realized advice and silence, and each message with its return relative to all branches, under a reward of task progress minus an intervention cost. In ALFWorld and Minecraft, PACE reaches higher success with fewer interventions than general-purpose and game-playing models used as companions, including with players not seen in training, and a human study in Minecraft rates it above GPT-4o-mini.

Critical strengths
  1. The nested formulation of proactive assistance is relatively new. Although the method builds on standard components, the paper treats when to intervene and what to say as two nested decisions whose effect reaches the task only through the advisee. After supervised initialization, PACE learns both decisions from downstream task outcomes rather than from labeled intervention points or judgments. Comparing the realized advice with silence from the same restored state also makes staying silent an explicit decision that can be credited and measured.

  2. The gains are clear in both environments, including with new players. In each environment, a single companion trained with one advisee reaches the highest overall success with every player, while intervening less than every companion baseline. A human study in Minecraft also rates it above GPT-4o-mini and shows faster task completion than playing alone.

Critical weaknesses
  1. The experiments do not fully show why nested credit works better than shared credit. The main tables compare PACE only with companions that are not trained for the companion role, so only the comparison with Shared-Credit PPO and GRPO in Figure 3 tests nested credit against shared credit. These two baselines are described in one sentence, and the paper does not say whether they start from the same supervised initialization, use matched branches or get the same rollout budget, although Table 2 shows that initialization matters. The variant without fresh branches also shows the largest degradation in success, with HELP falling from 44.5% to 3.5%, but the paper does not explain this drop or test any value of K other than 0 and the default of 2. So the ablations do not yet show clearly which part of the nested credit drives the gains over shared credit.

  2. The progress-based cost reset does not check that the progress relates to the advice. Eq. 4 clears the recency-dependent extra cost whenever any new milestone is completed after the previous intervention, so the next intervention pays only the base cost. The paper motivates this by noting that a new task stage may make further guidance relevant, but the completed milestone need not be the one the advice addressed, for example when the advisee makes progress on its own. An intervention can then become cheap again after progress that the earlier advice did not cause, and repeated advice that is ignored is penalized less when unrelated progress happens in between. The only related ablation removes the extra cost entirely, which raises HELP from 44.5% to 56.5%, and no variant that ties the reset to the advised milestone is tested. The design also depends on the predefined milestones, which are part of the companion's context and define the milestone rewards and the prefixes for fresh advice, while the ALFWorld milestones are not described. So it remains unclear how much of the learned selectivity depends on this reset rule and on the milestone definitions.

Questions for authors
  1. Clarification on W1. Please describe how Shared-Credit PPO and GRPO are implemented, including their initialization, whether they use matched branches and their rollout budget.

  2. Requested comparison for W1. Please explain the large drop without fresh branches, and report results for other values of K, such as K=1 and K=4, with the same settings. This would show how the benefit of nested credit depends on the number of alternative messages per state.

  3. Suggestion on W1. Each branch return comes from a single rollout. Please consider averaging several rollouts per branch, including the silence branch, or using the main trajectory as a second continuation of the realized advice. This could make the gate and advice credits less noisy.

  4. Clarification on W1. Please discuss how the matched branch rollouts relate to VinePPO [1], which also estimates credit by rolling out from restored intermediate states.

  5. Requested comparison for W2. Please test a variant that clears the extra cost only when the milestone addressed by the previous advice is completed, and describe how the ALFWorld milestones are defined.

[1] Kazemnejad, Amirhossein, et al. "VinePPO: Refining Credit Assignment in RL Training of LLMs." Forty-second International Conference on Machine Learning. 2025.

5 comments

Thank you for the careful assessment and constructive suggestions. We provide the requested implementation details and additional experiments below.

Q1. Shared-credit implementations and rollout budgets.

All three methods start from the same SFT-C checkpoint with the same critic initialization and use 90 updates with eight main episodes per update, totaling 720 main episodes per run. All branch rollouts use a horizon of N=8N=8.

MethodBranches/samples per expanded stateCredit assignmentTotal environment steps
Shared-Credit PPO2 branches: SILENCE + replayMain-trajectory advantage shared by gate and advice~39.7k
Shared-Credit GRPO4 freely sampled complete outputsOne group-relative advantage per output, shared by gate and advice~43.6k
PACE4 matched branches: SILENCE + replay + 2 fresh adviceReplay–SILENCE comparison for gate; group-relative credit for advice~47.4k

Following traditional PPO, Shared-Credit PPO computes advantages from main-trajectory returns using a critic. For GRPO, all four outputs start from the same restored state and receive identical milestone information.

The methods therefore share initialization and the main-episode schedule, while branch construction and total interaction costs differ. PACE uses approximately 19.4% more environment steps than Shared-Credit PPO and 8.7% more than Shared-Credit GRPO. These comparisons alone do not completely isolate credit decomposition from branch sampling and additional interactions.

Q2. Why does HELP drop at K=0K=0, and how sensitive is PACE to KK?

At K=0K=0, HELP dropped to 10–15% during content-only training and declined slightly further during joint training. In this variant, content learning relies only on the SILENCE–replay comparison, without alternative advice samples, which may contribute to the decline. The reported training HELP rate includes 10% gate exploration.

Using population standard deviation, the two-branch normalization in Eq. (6) becomes σ=max⁡(0.15,∣D∣/2)\sigma=\max(0.15,|D|/2), where D=Q1−Q0D=Q_1-Q_0. Consequently:

  • Content credit (Eq. 8): A^t,1=D/(2σ)\hat A_{t,1}=D/(2\sigma), saturating at ±1\pm 1 for ∣D∣≥0.30|D|\geq 0.30.
  • Gate credit (Eq. 7, joint phase): Δt=clip⁡(D/σ,−2,2)\Delta_t=\operatorname{clip}(D/\sigma,-2,2), saturating at ±2\pm 2 under the same condition.

This saturation reduces the sensitivity of credit estimates to return-gap magnitudes, potentially affecting subsequent optimization. However, saturation is symmetric and does not itself favor SILENCE. We therefore regard the absence of alternative-advice comparisons and the reduced sensitivity to return magnitudes as possible contributing factors, rather than an established explanation for the decline.

Sensitivity to K.

We evaluated the variants with the Qwen3.5-4B advisee in ALFWorld.

KSRHELP
153.6%51.1%
256.7%44.5%
457.1%46.2%

Increasing KK from 1 to 2 improves success while reducing HELP. Further increasing KK to 4 provides only a 0.4-point success gain at higher rollout cost. Overall, PACE performs consistently across K=1,2,4K=1,2,4, with K=2K=2 offering a practical balance between performance and rollout cost.

Q3. Sampling noise in branch returns

We followed your suggestions and you are right, thank you. We repeated the HELP–SILENCE comparison eight times at each of 20 HELP states, keeping the state, advice, policy, and critic fixed. The return differences had an average standard deviation of approximately 0.2 across states. Under independent repetitions, averaging MM estimates would theoretically reduce this to 0.2/M0.2/\sqrt{M}, or approximately 0.07 at M=8M=8.

For the second suggestion, we use the main trajectory as a second continuation of the realized advice. Across 20 states with eight repetitions each, the average standard deviation of gate-credit estimates decreased from 0.220 to 0.211 (4.0%). This suggests a modest reduction in sampling noise.

Q4. How do PACE's matched rollouts relate to VinePPO?

VinePPO [1] provides a valuable approach to credit assignment in mathematical reasoning, while PACE focuses on credit assignment for a companion advising a separate player. Both use intermediate-state rollouts, with some differences in the following aspects:

AspectVinePPOPACE
Return estimationFull Monte Carlo continuationsN-step rollouts with critic bootstrapping
Credit assignmentOne advantage per reasoning step, shared across its tokensSeparate gate and advice credits from branch comparisons
InteractionThe trained LLM generates the solution itselfThe companion advises a separate player
Auxiliary continuationsUsed only for value estimation, not direct policy updatesUsed for branch comparisons and advice-policy updates

We are adding this distinction and relevance to VinePPO in our Related Work section.

Q5. Cost reset variant and ALFWorld milestone definitions

The original reset rule responds to a change in task stage rather than assuming that the previous advice caused it. Completing a milestone may introduce new decisions requiring guidance and the reset clears only the additional recency penalty, while the base intervention cost remains.

We evaluated the requested variant using the following setting: When providing advice, the companion specifies the milestone(s) it targets. The intervention cost resets when any of these milestones is completed. If no target is specified, we use the next uncompleted milestone in the task plan. Both variants were trained from the same initialization with otherwise identical training settings.

Reset ruleSRHELP
Any-milestone (original)56.7%44.5%
Advised-milestone (requested)54.0%40.3%

The stricter rule reduces intervention frequency by 4.2 percentage points, with 2.7 points lower success, showing an observed trade-off between intervention frequency and task completion.

ALFWorld milestones are automatically derived from ALFRED expert plans (plan.high_pddl), without additional manual annotation. All methods use the same milestone definitions. PACE uses them for progress rewards, cost resets, and advice context and prefix conditioning.

[1] VinePPO: Refining Credit Assignment in RL Training of LLMs. Kazemnejad et al. ICML 2025.

Follow-up on Q4: Full Monte Carlo Experiment

Inspired by VinePPO, we also tested full Monte Carlo rollouts without critic bootstrapping, keeping all other training settings unchanged.

Return EstimationSR ↑HELP ↓Training Time
PACE (N=8 + Critic)56.7%44.5%27 GPU-hours
PACE (Full MC)60.9%47.3%46 GPU-hours

Full MC increases observed success by 4.2 percentage points, while increasing HELP by 2.8 points and training compute by 70.4%. These results show that PACE’s nested credit formulation remains effective without bootstrapping branch returns. They also clarify the practical trade-off: full MC achieves higher success, whereas the default achieves lower training cost and intervention frequency.

Reviewer_CxYsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACE, a reinforcement learning framework for proactive assistance that jointly learns when to intervene and what advice to provide through nested credit assignment. The method evaluates alternative advice and silence branches to derive separate learning signals for intervention timing and advice generation. Experiments in ALFWorld and Minecraft, together with a human study, demonstrate improvements in task success rates while reducing unnecessary interventions.

Critical strengths
  • The proposed method is logically consistent, with each component serving a clear purpose rather than being an unnecessary addition.

  • The ablation experiments are reasonably comprehensive and provide convincing empirical evidence supporting the effectiveness and rationale of the proposed design.

Critical weaknesses
  • The experiments are conducted in only two environments, ALFWorld and Minecraft, which limits the evidence for the method's generalizability to broader application scenarios.

  • Although the Introduction discusses several closely related approaches, the paper lacks direct comparisons with these methods in terms of both task performance and computational efficiency. This makes it difficult to fully assess the advantages of the proposed approach over existing methods.

Questions for authors
  • Could the authors provide direct comparisons with relevant existing methods under comparable training computational budgets, considering both task performance and computational efficiency?

  • Could the authors extend the evaluation to a broader range of task scenarios to further demonstrate the generalizability of the proposed method?

3 comments

Thank you for the constructive suggestions. We address the comparison and generalization concerns below.

Q1. Comparisons under comparable computational budgets.

Our general-purpose LLM baselines are prompted as companions without additional task-specific training. The related methods discussed in the Introduction require adaptation to our setting. Proactive Agent [1] learns from acceptance-based supervision, while CoLabScience [2] uses positive–unlabeled scientific-discussion data and reference-based objectives within an Observer–Presenter architecture. Our benchmarks provide task-outcome feedback but no annotated intervention decisions or reference advice. GiGPO [3], meanwhile, trains an agent that directly executes environment actions, whereas our companion influences outcomes through a separate, frozen player.

To compare their learning approaches within this setting, we introduce two baselines:

  • Adapted Proactive Agent baseline: We use an LLM teacher to generate intervention labels and reference advice, supplying the supervision unavailable in the original benchmarks. This enables comparison with Proactive Agent’s supervised approach, with the change from human to LLM-generated annotations explicitly identified.
  • GiGPO-7B as companion: We use the publicly released GiGPO-7B checkpoint, prompted to provide guidance to the frozen player without additional fine-tuning, to assess how an agent-trained policy performs in the companion role.

These comparisons evaluate supervised intervention learning and a pretrained RL policy in our assistance setting. Adapting CoLabScience's full pipeline would require additional dialogue-specific supervision and reference-based rewards unavailable in our benchmarks. We evaluate all baselines on the same ALFWorld tasks with the same frozen player, available information, and evaluation protocol. We report training costs separately, as the methods differ in their training setups.

MethodSRHELPGPU-hours
Proactive Agent42.1%54.7%~19h
GiGPO-7B52.9%56.7%released checkpoint
PACE56.7%44.5%~27h

GiGPO does not report the training cost of its 7B model. For reference, the estimated cost of GiGPO-1.5B is approximately 30 H100 GPU-hours on two H100 GPUs.

Q2. Broader task evaluation.

ALFWorld and Minecraft are established testbeds for interactive agents. Examples include ReAct (ICLR 2023) [4] on ALFWorld, and ADAM and MrSteve (ICLR 2025) [5,6] and Voyager (TMLR 2024) [7] on Minecraft. They provide complementary household-task and open-ended gameplay settings.

Minecraft contains goals with different resource requirements, prerequisite dependencies, and action sequences. We train the Qwen3.5-9B companion using 1,017 visual captions for initialization and RL on Minecraft. To broaden task coverage, for the time being, we evaluate the same checkpoint on three additional new goals excluded from RL training, without further fine-tuning and using the same frozen Qwen3.5-4B player.

Additional goalSolo SR / MSPACE SR / MS
Craft Boat80% / 0.90100% / 1.00
Craft Torch70% / 0.8990% / 0.94
Smelt Stone10% / 0.5170% / 0.91

PACE improves both success and milestone progress on all three goals. These results complement our cross-player and human evaluations and strengthen the evidence for transfer across goals within Minecraft.

[1] Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance. Lu et al. ICLR 2025.

[2] “Excuse me, may I say something...” CoLabScience, A Proactive AI Assistant for Biomedical Discovery and LLM-Expert Collaborations. Wu et al. ACL 2026.

[3] Group-in-Group Policy Optimization for LLM Agent Training. Feng et al. NeurIPS 2025.

[4] ReAct: Synergizing Reasoning and Acting in Language Models. Yao et al. ICLR 2023.

[5] ADAM: An Embodied Causal Agent in Open-World Environments. Yu and Lu. ICLR 2025.

[6] MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory. Park et al. ICLR 2025.

[7] Voyager: An Open-Ended Embodied Agent with Large Language Models. Wang et al. TMLR 2024.

Thank you for your response and additional experiments. The new comparisons are informative and address my original concerns. I also appreciate that these results have been incorporated into the revised manuscript.

While the computational comparisons are not strictly budget-matched and the additional generalization experiments remain within Minecraft, I acknowledge the practical challenges involved. Overall, these additions substantially strengthen the empirical evaluation and help clarify the advantages and limitations of the proposed method. I will raise my score to 3.

Thank you for your reconsideration and encouraging feedback. We sincerely appreciate your thoughtful comments, from which we have learned a great deal. They have also given us valuable insights into how to approach our future work with greater rigor and care.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely appreciate all the constructive feedbacks, and greatly welcome any opportunities for further discussion. Below, we briefly summaries the key improvements we have made:

  • Additional baselines. We evaluated an adapted Proactive Agent baseline and GiGPO-7B as a companion.
  • Generalization to unseen tasks. We evaluated three additional held-out Minecraft goals without further training to demonstrate generalizability.
  • Nested credit assignment. We investigated and explained the HELP collapse at K=0, evaluated sensitivity to K, and tested an alternative milestone reset rule.
  • Sampling noise and VinePPO. We quantified branch-return noise, evaluated main-trajectory reuse, and clarified the methodological differences from VinePPO.
  • Finite-horizon estimation. We compared N-step estimates against full-episode returns across different horizons
  • Human study. We analyzed proactive assistance without participant-initiated questions and clarified the relationship between intervention timing and advice content.
  • Training costs and applicability. We detailed training data, rollout budgets, and computational costs, and discussed requirements for extending PACE to other domains.

We have incorporated additional experiments and clarifications into the revised manuscript, with all changes highlighted in blue for ease of reference:

  • The revisions include the VinePPO discussion in Section 2, training and implementation details in Appendix B, additional ablations and return estimation experiments, including Full MC, in Appendix C, human study analysis in Appendix D, baseline comparisons in Appendix F, and generalization to unseen Minecraft goals in Appendix G.

We sincerely appreciate all the comments and suggestions received, they have substantially improved the completeness of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.