acceptodds
Under review as a conference paper at ICLR 2027

PROSE: Learning Progress from Cross-Trajectory Outcomes for Long-Horizon Embodied Tasks

Abstract

Long-horizon embodied tasks are difficult to learn from sparse terminal rewards, because an agent may need to act for hundreds of steps before receiving any feedback about its intermediate behavior. Existing approaches provide intermediate guidance by scoring individual observations based on their relevance to the task goal or by relying on predefined subgoals or dense progress supervision. However, goal relevance alone may be weakly predictive of eventual task success, and the effectiveness of subgoal-based guidance depends on whether they remain appropriate under the current environment dynamics and execution capabilities. Instead of requiring explicit intermediate targets, we identify recurring behavior patterns associated with successful outcomes across trajectories and use them to supervise promising behaviors before terminal feedback arrives. We introduce PROSE (Progress Recognition from Outcome Statistics across Experience), which abstracts long trajectories into visually coherent periods, aligns similar periods across episodes into recurring situations and transitions, estimates their progress from cross-trajectory outcome statistics, and refines it along individual trajectories by penalizing subsequently lost progress and accounting for how each transition is visually and behaviorally executed. When progress persistently stalls, a targeted exploration controller uses the agent's previous observations to redirect exploration away from unpromising regions. Across five long-horizon MineDojo tasks, integrating PROSE with two world-model agents improves success rate by 9.1 points and reduces steps to first success by 14.4% on average. These results show that PROSE complements different world-model agents by turning sparse outcomes into intermediate supervision for long-horizon embodied tasks.

Primary area:
reinforcement learning
Keywords:
Long-Horizon Tasks, Sparse Rewards, Progress Recognition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_w9LqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROSE derives intermediate supervision for sparse-reward embodied tasks from completed trajectories. It segments observations into periods, groups recurring situations using visual and semantic descriptors, and scores transitions by their association with terminal success. A degradation penalty and learned visual/action residuals refine these scores, which are transferred to a reward predictor for world-model rollouts. A spatial-memory controller redirects exploration when progress stalls. Across five MineDojo tasks and two reproduced world-model baselines, the combined method improves all ten success-rate point estimates, with a reported mean gain of 9.1 percentage points. Incremental ablations on Cut a tree and Obtain water support contributions from several components.

Critical strengths
  1. The method connects sparse terminal outcomes to reusable transition-level supervision without manually specifying intermediate goals. The separation between a shared transition prior and execution-specific residuals is technically meaningful: visually similar situations can require different credit depending on how the agent acts (Section 2.2; Appendix A.2).

  2. The controlled comparisons show consistent improvements across two different world-model agents and five tasks, rather than relying on comparisons with published numbers alone. The additional Obtain water ablation in Appendix B.3 supports the Cut a tree trend, and the failure analysis explicitly recognises limitations in perception and fine-grained control. These results make a credible case for the system's practical usefulness.

Critical weaknesses
  1. The training-experience accounting is insufficiently explicit for the central efficiency claim. Appendix C reports sizeable archived graph-fitting pools, while Appendix B describes online graph and residual updates. The provenance and timing of those pools, their inclusion in the interaction budget, and the corresponding baseline budgets are not clear. Similarly, Tables 1–2 report three evaluation seeds, but do not establish the number of independent training runs; the treatment of failed episodes in average steps to first success is unspecified. These details determine whether the gains reflect comparable learning resources and what the efficiency numbers mean.

  2. The evidence for the progress signal itself is limited relative to its role in optimisation. The residual is trained from trajectory outcomes and the deployed reward mix eventually gives the progress predictor weight 0.95 (Appendices A.2–A.3, B.1). The incremental ablations show aggregate benefit, but do not establish whether success-associated transitions remain useful under policy shift or whether repeated visits can receive misleading credit. This matters because the actor is trained against this learned proxy after warm-up, although the paper appropriately describes the edge statistics as associations rather than causal effects.

Questions for authors
  1. Clarification: were all graph-fitting trajectories collected within each reported training run and included in the same environment-interaction budget as its baseline? Please specify independent training versus evaluation seeds, evaluation episode counts, and how unsuccessful episodes enter Table 2.

  2. Targeted validation of the existing mechanism: do archived trajectories show repeated rewarded transitions without eventual completion, or degradation in held-out reward prediction as the policy changes? A small diagnostic using existing traces, alongside the rule for unsupported/unmapped transitions, would help assess the proxy concern; a new benchmark suite is not requested.

3 comments

[Weakness 1 / Question 1] Training-experience accounting and evaluation details

  1. PROSE reuses baseline training trajectories to initialize its graph and replay buffer. These trajectories are collected by the unmodified baseline before PROSE online training. Online graph and residual updates subsequently incorporate trajectories collected during the PROSE online training run.

  2. PROSE still improves over the baseline when both use the same archived replay data. The baselines reported in Table 1–2 (main paper) follow their original training setup and start with an empty replay buffer. To address the concern that PROSE may benefit from additional offline training experience unavailable to the baseline, we additionally evaluate LS-Imagine initialized with the same archived offline replay trajectories used by PROSE. PROSE achieves a much higher success rate of 92.33%, compared with 83.67% for the baseline with offline replay on "Cut a tree" task (Table A). Even with only 700 trajectories, including 100 successful ones, PROSE achieves 86.67%, exceeding the baseline (LS-Imagine) initialized with all 2975 archived trajectories.

  3. The reported mean and standard deviation reflect variation across evaluation seeds. For each method–task setting trained in our original study, we used one independent training run.

  4. For unsuccessful episodes, we assign the maximum episode length (1,000 steps for Cut a tree, Obtain water, Obtain sand, and Obtain wool, and 2,000 steps for Mine iron) when computing the average steps to first success, following CPP and LS-Imagine. For example, an unsuccessful episode on "Cut a tree" contributes 1,000 steps to this metric.


Table A: Effects of offline replay and initial data availability on Cut a tree. Successful and Total denote the numbers of successful and total trajectories in the initial offline pool.

SettingSuccessfulTotalSuccess rate (%)
LS-Imagine*——81.33 ± 2.73
LS-Imagine* + 100% offline pool2,2762,97583.67 ± 4.04
PROSE (100% offline pool)2,2762,97592.33 ± 4.16
PROSE (75% offline pool)1,7072,23191.67 ± 4.04
PROSE (50% offline pool)1,1381,48890.00 ± 1.85
PROSE (25% offline pool)56974493.67 ± 1.58
PROSE (100 successful trajectories)10070086.67 ± 4.73
PROSE (25 successful trajectories)2570059.67 ± 2.08

[Weakness 2 / Question 2] Progress rewards under repeated visits and policy shift

  1. Archived trajectories do contain repeated rewarded transitions without eventual task completion but the execution-specific refinement design helps to mitigate the misleading credit. 11 of 160 failed trajectories receive positive net progress rewards from graph cycles, and this number drops to 1 after incorporating the execution-specific refinement.

  2. Progress rewards continue to distinguish successful trajectories from failed ones under policy shift. We apply the same graph to 400 baseline trajectories and newly collected 400 trajectories from PROSE training runs (excluded in graph construction). Each trajectory is scored by summing its progress rewards. AUROC is used to measure how often successful trajectories receive higher scores than failed trajectories, with 0.5 indicating random ranking. With 250 successful and 150 failed trajectories in each group, AUROC is 0.954 for the baseline and 0.910 for PROSE.

    Beyond evaluating progress-reward discrimination under policy change, we further examine cross-backbone transfer. Specifically, we construct the offline graph using trajectories collected by LS-Imagine and use it to provide progress supervision during DreamerV3+PROSE training. Unlike the previous experiment, which evaluates reward discrimination under a policy change within the same backbone, this experiment transfers the graph across different world-model agents. On Cut a tree, the resulting policy achieves 89.33 ± 1.53% success and 370.45 ± 6.97 steps to first success. For reference, the original DreamerV3+PROSE results in the main paper are 83.00 ± 4.69% and 405.01 ± 39.69 steps. This result provides evidence that progress supervision learned from LS-Imagine trajectories can transfer to a different world-model policy.

  3. For unsupported/unmapped transitions in collected trajectories, we set the progress reward to zero when node mapping fails. For edges supported by fewer than five distinct trajectories, we set the frequency reward to zero while retaining the execution-specific visual-action residual.

Thank you for the additional analyses. The equal-replay comparisons across all five tasks, independent training runs on two tasks, and clarification of unsuccessful episodes substantially address my evaluation concerns. The cycle and policy-shift diagnostics also strengthen the evidence for the progress signal, although I don't feel the trajectory-level discrimination establishes accurate local credit. I remain at weak acceptance.

Thank you and we are glad that the additional analyses substantially address your evaluation concerns and strengthen the evidence for the progress signal. We agree with your suggestions and comments. The evidence for local credit is indirect, coming from its effect on policy learning. We will state this explicitly in the revised paper.

Reviewer_A7qYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses sparse supervision in long-horizon embodied tasks by learning intermediate progress signals from recurring situations and transitions across trajectories. The method first divides trajectories into visually coherent periods and aligns similar periods using semantic and visual-temporal information. Their progress is estimated from cross-trajectory outcome statistics and further refined by considering whether progress is maintained and how each transition is executed. It also includes an exploration strategy when progress stalls. Experiments with two world-model agents on five MineDojo tasks show improvements in success rate and completion efficiency.

Critical strengths
  • It makes sense to mine intermediate supervision across successful and failed trajectories. This can provide useful guidance based on accumulated experience.

+The results across five MineDojo tasks show consistent improvements over the reproduced baselines with both world-model agents.

Critical weaknesses

-The description of existing work in the abstract is not accurate. It characterizes subgoal-based guidance as relying on predefined subgoals, but the subgoals in [1] and [2] are learned or dynamically generated. The authors should revise this description and explain more clearly what their outcome-based progress supervision offers over learned subgoal guidance.

-The application scope of the method seems limited, and its generality has not been sufficiently tested. The method starts from visually coherent periods and depends on identifying recurring situations across trajectories. However, a single subtask can involve substantial visual changes, while different subtasks can have very similar observations. Although semantic descriptions may help, it is unclear whether the abstraction and matching strategy can reliably capture subtle differences that matter for task progress. All experiments are within MineDojo. Results on other long-horizon embodied tasks would be needed to support broader applicability.

-Several key parts of the method are heuristic, including the segmentation and matching rules, the construction of progress rewards, and the penalty for lost progress. The ablations show improvements from adding these components, but give limited explanation of why these particular choices are appropriate or how much the results depend on them. This makes it difficult to draw more general conclusions about the design of intermediate supervision for long-horizon tasks.

-The comparison with other intermediate supervision strategies is insufficient. The paper reports results for LS-Imagine and CPP, but does not compare different ways of constructing intermediate guidance under the same backbone and comparable data conditions. Subgoal-based methods such as [2], and the visual subgoal guidance used in [1], are relevant alternatives to consider. A controlled comparison with an appropriate subgoal-based baseline would help show whether the proposed supervision strategy is actually better than other ways of providing intermediate guidance.

-The novelty of the cross-trajectory supervision idea needs clearer discussion. Related ideas have already been explored in language-agent training. For example, GiGPO [3] groups recurring states across trajectories to obtain finer-grained credit signals. Although the paper cites this work, the discussion does not make the differences sufficiently clear. I would like a more concrete explanation of what is new in PROSE’s cross-trajectory alignment and progress estimation.

[1] Physical Intelligence et al. π₀.₇: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities. 2026. [2] Huang et al. Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidance. CoRL 2024. [3] Feng et al. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025.

Questions for authors

Please refer to the weakness part.

9 comments

[Weaknesses 1 & 4] Learned subgoal guidance and controlled comparison

We thank the reviewer for pointing this out. Our related work (Sec. 4) discusses learned and dynamically generated subgoals, but the abstract's reference to "predefined subgoals" does not accurately reflect this literature. We will revise the abstract and introduction accordingly.

  1. PROSE learns intermediate progress from task outcomes without requiring explicit target states for the policy to reach. The subgoal-image generator in π0.7\pi_{0.7} [1] is trained using end frames of segments annotated with subtask instructions. For example, in a task to obtain an item from a fridge, "open the fridge door" is a subtask. The final frame of the segment annotated with "open the fridge door" serves as the training target for subgoal-image generation. EGR-PO [2] generate subgoals conditioned on the current state and a specified final goal, such as a target location provided by the environment in AntMaze [4]. Our setting instead provides a task objective and terminal success/failure labels, without a goal-state input. For "Cut a tree", success means collecting a log, not reaching a specified final observation. Successful episodes can therefore involve different trees and final observations. The task can be checked for completion without providing a goal-state input (e.g., the tree location), however, using the goal-conditioned subgoal generator in [2] requires us to provide or construct one. We study how to learn intermediate supervision from collected trajectories and task outcomes without this additional goal-state input.

  2. To compare with learned-subgoal guidance, we adapt EGR-PO [2] to our setting. PROSE achieves higher mean success rates and fewer mean steps to first success than this adapted baseline (Table A). Among the two baselines [1,2], annotated subtask segments is required to reproduce π0.7\pi_{0.7}'s subgoal-image supervision [1], thus, we adapt EGR-PO [2] by constructing final goals from archived successful trajectories. To provide the goal-state input missing in our setting, we sample a terminal latent state from a successful replay episode as the final goal gg for environment interaction and keep it fixed within the episode. We use LS-Imagine as the base method for both PROSE and learned subgoal guidance adapted from EGR-PO [2]. Following EGR-PO [2], we train the subgoal generator using future replay states as targets, with the current latent state and final goal gg as inputs. The actor in this adapted baseline uses the generated subgoal to guide action selection, and its critic is conditioned on the same subgoal. PROSE achieves higher mean success rates and fewer mean steps to first success without constructing a final-goal input, instead learning progress from collected trajectories and terminal outcomes.


Table A: Comparison with adapted learned subgoal guidance.

TaskMethodSuccess rate (%) ↑Steps to first success ↓
Cut a treelearned subgoal baseline86.00 ± 1.73399.52 ± 16.86
PROSE92.33 ± 4.16323.42 ± 24.14
Obtain waterlearned subgoal baseline80.00 ± 11.36456.92 ± 90.20
PROSE83.50 ± 3.32401.74 ± 17.47

[Weakness 2] Abstraction and matching strategy, and broader applicability

Our contribution is to learn intermediate progress from recurring situations and transitions across successful and failed trajectories. The representations used for cross-trajectory alignment can be adapted to each environment. As suggested, we provide results on additional benchmark (ALFWorld [5]) to show that our approach is applicable to different datasets and tasks.

Regarding the comment of "a single subtask can involve substantial visual changes, while different subtasks can have very similar observations," here is our clarification.

  1. A single subtask with substantial visual changes can be represented by multiple periods and PROSE does not require the entire subtask to form one visually coherent period. Each period is matched separately across trajectories, and progress is estimated for transitions between the matched situations using terminal outcomes.

  2. Periods with similar observations do not mean they're going to receive the same progress estimate. Matching requires both a shared semantic descriptor and sufficient visual-temporal similarity. Even for occurrences of the same recurring transition, the execution-specific refinement uses local action dynamics to adjust the progress estimate. For example, two periods may show similar views of a tree, while one records the agent approaching it and the other records the agent moving away. Their temporal changes and action sequences provide information for distinguishing these executions despite their similar views. Removing semantic buckets or visual-temporal matching also lowers mean success shown in Table B. The execution-specific refinement increases mean success on "Cut a tree" from 87.33% to 91.25% and reduces mean steps to first success from 438.81 to 340.43 (main-paper Table 3).


Table B: Effects of semantic buckets and visual-temporal matching on policy success rates (%).

MethodCut a treeObtain water
LS-Imagine* (no graph)81.33 ± 2.7367.00 ± 4.97
PROSE92.33 ± 4.1683.50 ± 3.32
PROSE w/o semantic buckets87.00 ± 2.6581.67 ± 2.08
PROSE w/o visual-temporal matching87.67 ± 2.8982.00 ± 2.65

  1. We apply PROSE's intermediate supervision to household tasks with textual observations on another benchmark dataset (ALFWorld), where GiGPO [3] is used as the base method. The adaptation keeps PROSE's key idea of estimating progress from cross-trajectory terminal outcomes, while adapting period construction and representations to textual observations. We use text features and semantic descriptors to construct periods and identify recurring situations. For policy learning, we use the resulting graph nodes for GiGPO's anchor-state grouping and progress rewards to adjust its advantages. Adding PROSE to GiGPO's \textit{w/ std} configuration yields 91.9±1.2%91.9 \pm 1.2\% overall success across six household task categories, compared with 86.7±1.7%86.7 \pm 1.7\% reported for GiGPO \textit{w/ std} (Table E).

Table C: Success rates (%) on ALFWorld with Qwen2.5-1.5B-Instruct.

MethodPickLookCleanHeatCoolPick2All
GiGPO w/ std94.4 ± 5.967.5 ± 4.694.8 ± 3.894.4 ± 7.879.8 ± 4.776.4 ± 5.486.7 ± 1.7
GiGPO w/ std + PROSE97.0 ± 1.180.0 ± 2.099.3 ± 0.797.4 ± 4.494.5 ± 2.577.3 ± 2.391.9 ± 1.2

[Weakness 3] Segmentation and matching rules, progress rewards, and the penalty for lost progress

  1. The segmentation rules identify changes between visually coherent portions of a trajectory rather than treating every frame-level change as a boundary. We therefore compare visual consistency within adjacent windows with cross-window similarity and select local peaks as period boundaries.

  2. The matching rules use semantic and visual-temporal representations to identify recurring situations across trajectories. The coarse semantic descriptor summarizes the task situation, and the visual-temporal descriptor retains finer visual and temporal information. A period is assigned to the nearest node prototype when their cosine similarity exceeds δ=0.75\delta = 0.75. We keep a node only if at least five distinct trajectories are assigned to it.

    To examine how the matching threshold affects alignment coverage and progress estimation, we vary δ\delta and evaluate on trajectories excluded from graph construction. A higher threshold can leave more periods unmatched, so we measure this effect using coverage, the percentage of periods assigned to retained nodes. We also compute Area Under the Receiver Operating Characteristic Curve (AUROC) using summed progress rewards as trajectory scores to assess how the threshold affects the distinction between successful and failed trajectories. We use δ=0.75\delta = 0.75 in our experiments, as it achieves balanced results in matched nodes, coverage, and AUROC (Table D).


Table D: Sensitivity to the matching threshold δ\delta.

δ\deltaCut a tree: NodesCoverageAUROCObtain water: NodesCoverageAUROC
0.651199.6%0.913998.9%0.904
0.75 (Ours)1499.6%0.9541898.7%0.928
0.8510184.7%0.7445764.1%0.773

  1. The construction of progress rewards measures each recurring transition's association with success relative to the population success rate. A transition receives positive credit when trajectories containing it succeed more frequently than average, and negative credit otherwise. The prior count shrinks estimates with limited support toward the population success rate, while the execution-specific visual-action residual accounts for different realizations of the same transition.

    To examine how task performance depends on the progress reward weight, (1−α)=0.95(1-\alpha) = 0.95 in Eq. (26), we compare weights of 0.95, 0.8, and 0.5 on "Cut a tree". The corresponding mean success rates are 92.33% (the original Table 1 result), 76.33%, and 85.33%. The weight of 0.95 yields the highest mean success rate among the tested settings, which is used in our experiments.

  2. The penalty for lost progress reduces over-crediting when previously achieved progress is subsequently lost. For example, an agent may approach a tree, a transition associated with successful log collection, but subsequently move away without collecting a log. We therefore penalize decreases in estimated success probability along the graph path.

    We vary the weight applied to the degradation penalty rkdegr^{deg}_k across 0.5, 1 (the value used in the paper), and 2. Each test trajectory is scored by summing its progress rewards, and AUROC measures how well these scores distinguish terminal success from failure. On Cut a tree, AUROC remains between 0.954 and 0.955 across the tested weights, indicating stable trajectory-level success/failure discrimination under different penalty scales.

Reviewer_QC4EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to build a graph of recurring situations from completed trajectories to provide progress indicators that serve as denser learning signals in sparse-reward settings. They use visual coherency from MineCLIP descriptors to segment trajectories and combine with semantic descriptors from VLM to construct a transition graph with estimated progress statistics of recurring transitions from trajectory outcomes. For supervising MBRL agents, the paper proposes degradation penalty for losing progress, execution-specific action residual for variations inside scenarios, and targeted exploration when the agent stops making progress. The results demonstrate improvements in long-horizon MineDojo tasks.

Critical strengths
  • The paper presents an interesting approach to leverage foundational models to automatically annotate coherent transitions for policy supervision. The results also demonstrate consistent gains and the ablations provide evidence supporting the proposed design choices.
Critical weaknesses
  • The proposed approach relies on foundation models (MineCLIP, VLM) to provide task-relevant representations for progress estimation. This introduces semantic knowledge beyond the agent’s collected experience. So I believe the paper should compare against simpler baselines (e.g., infusing foundation embeddings into learning reward models ) that can truly isolate the need for the proposed transition graph construction and supervision.
    • I’m also curious about how much more is gained from the explicit VLM semantic annotation representing a period's event stage, target visibility etc., which seems to be highly tailored to the task. Additionally, potential labeling biases/inaccuracies for such automated semantic annotation is not discussed.
  • The paper does not characterize the reliability and costs (computational/data) of the proposed pipeline.
    • There are several heuristic elements (subsampling, event boundary detection, alignment thresholds etc.,) for cross-trajectory alignment of segments across long-horizon trajectories. It is unclear how this can scale with increasing data and potentially expanded on to multi-task scenarios.
      • Additionally, the paper should provide more intuition for some of the proposed thresholds / design choices of the cross-trajectory alignment procedure. And provide more insights as to why other clustering alternatives aren’t preferable here.
    • The approach also seems to rely on quite a few successful trajectories to learn useful estimates in the progress graph (Table 5) which might hinder scalability. It is unclear how the method fairs when there are limited successful experiences.
  • The procedure appears somewhat tailored to the MineDojo tasks (with evaluation restricted to this environment). It is unclear how readily it transfers to other scenarios.
Questions for authors

See weaknesses. Some additional questions below:

  1. Could PROSE be compared to an auxiliary learned reward model (over the same MineCLIP embeddings) that directly predicts the trajectory outcome and uses that for policy supervision? This will help demonstrate the need for the progress graph construction.
  2. Can the authors clarify how the offline graph-fitting trajectories are obtained? Also, is this accounted for with the compared baselines (do they also start with this offline dataset?)
  3. Can the authors explain how sensitive the approach is to the number of successful trajectories available for progress graph construction? It would be interesting to see a controlled study along this axis.
5 comments

[Weakness 1 & Question 1] Simple reward baseline and VLM semantic annotation gain

  1. PROSE provides more effective intermediate supervision than an auxiliary learned reward model using the same MineCLIP period embeddings, showing the benefit of the proposed progress graph construction. Following the reviewer's suggestion, we use LS-Imagine as the policy-learning backbone and add an auxiliary learned reward model that directly predicts trajectory outcome without constructing a progress graph. Specifically, the model predicts the probability of terminal success, PkP_k, from the first kk periods and uses Pk−Pk−1P_k - P_{k-1} as the intermediate reward for policy learning. We train the auxiliary reward model using the same archived trajectories and terminal-outcome labels as PROSE.

    We first evaluate both reward signals on trajectories excluded from reward-model training and graph construction, scoring each trajectory by summing its period rewards. Area Under the Receiver Operating Characteristic Curve (AUROC) measures how often successful trajectories receive higher scores than failed trajectories, with 0.5 indicating random ranking. PROSE achieves an AUROC of 0.954, compared with 0.885 for the auxiliary learned reward model. We further evaluate their effectiveness for policy learning using LS-Imagine. The auxiliary reward model achieves 88.67% success, compared with 92.33% for PROSE (Table A). These results support the benefit of the proposed transition graph construction and supervision over the baseline directly predicting trajectory outcomes from the same MineCLIP embeddings.


Table A: Comparison to learned reward model on Cut a tree.

MethodSuccess rate (%)AUROC
PROSE92.33 ± 4.160.954
Auxiliary learned reward model88.67 ± 3.510.885

  1. The explicit VLM semantic annotation uses the same field definitions across the evaluated MineDojo tasks, with the task goal specifying the relevant target. These descriptions guide cross-trajectory matching, while progress rewards are estimated from terminal outcomes. Removing VLM semantic buckets while keeping visual-temporal matching reduces mean success from 92.33% to 87.00% on Cut a tree and from 83.50% to 81.67% on Obtain water. To test whether VLM descriptions alone is enough to define graph nodes, we remove finer visual-temporal matching and assign periods sharing the same VLM semantic descriptor to one node. This reduces mean success to 87.67% and 82.00%, respectively (Table B). The comparable performance drops indicate that VLM semantic annotations contribute to cross-trajectory alignment, but do not alone account for PROSE's gains, as visual-temporal matching provides a similarly important contribution.

Table B: Online task success rates (mean ± standard deviation, %).

Cross-trajectory alignmentCut a treeObtain water
PROSE92.33 ± 4.1683.50 ± 3.32
Without VLM semantic buckets87.00 ± 2.6581.67 ± 2.08
Only VLM semantic bucket87.67 ± 2.8982.00 ± 2.65
K-means within each semantic bucket (k=2)80.33 ± 1.5375.00 ± 4.36

  1. Some failed cases reveal task-conditioned target misidentification, where visually misleading objects are assigned the task-target role without sufficient visual evidence. Examples include treating lava as the water target and interpreting interactions with animals as interactions with the water target. Such annotation errors may place periods in inappropriate semantic buckets and affect transition-outcome statistics, which can be further enhanced in future work. We'll follow the suggestion and add this in the updated paper as part of the future work.

[Weakness 2 & Question 3] Costs and design choices

  1. Cross-trajectory alignment handles increasing data by matching new periods to shared graph nodes rather than all previously collected periods. The computational cost of each new match therefore depends on the number of candidate graph nodes, rather than directly on the number of collected trajectories. After the initial offline graph is constructed, the number of retained nodes remains at 14 for Cut a tree after incorporating 1,737 new online trajectories. For Obtain water, it increases from 18 to 19 after incorporating 2,102 new online trajectories. Annotation with Qwen3-VL-8B takes approximately 8–9 seconds per period on a machine using one H100 GPU. For multi-task scenarios, the current implementation maintains a separate graph for each task. Joint multi-task learning has not been evaluated and is left for future work.

  2. (a) Design intuition. Observation subsampling reduces the number of frames compared during segmentation. Boundary detection favors visually coherent observations within adjacent windows and dissimilar observations across them, identifying changes between coherent periods. The alignment threshold controls how much visual-temporal difference is allowed among periods assigned to the same node. Within the same coarse semantic bucket, a period is assigned to the nearest node prototype when their cosine similarity is at least δ=0.75\delta = 0.75. Requiring support from at least five distinct trajectories restricts the nodes being added appear across multiple episodes. We vary δ\delta and evaluate on trajectories that were not used for graph construction (Table C). Coverage measures the percentage of periods assigned to retained nodes. Each trajectory is scored by summing its progress rewards and AUROC measures how often successful trajectories receive higher scores than failed trajectories, with 0.5 indicating random ranking. Both δ=0.65\delta = 0.65 and 0.750.75 maintain high coverage and AUROC. Increasing δ\delta to 0.85 produces more nodes but leaves more periods unmapped and reduces AUROC on both tasks. These results support δ=0.75\delta = 0.75 as a setting that preserves high coverage and outcome discrimination.


Table C: Sensitivity to the matching threshold δ\delta.

δ\deltaCut a tree: NodesCoverageAUROCObtain water: NodesCoverageAUROC
0.651199.6%0.913998.9%0.904
0.75 (Ours)1499.6%0.9541898.7%0.928
0.8510184.7%0.7445764.1%0.773

  1. (b) Other clustering alternatives. PROSE achieves higher mean task success than the tested alternatives (Table B). To examine the contributions of semantic and visual-temporal matching, we consider two ablations. Without semantic buckets, we retain visual-temporal matching but remove coarse semantic grouping. With only semantic buckets, we assign one node to each bucket without further visual-temporal matching. Both variants achieve lower mean success rates than PROSE on both tasks. We also compare prototype matching with K-means within each semantic bucket. The selected K=2K = 2 produces 18 nodes for Cut a tree and 18 for Obtain water, comparable to PROSE's 14 and 18 nodes for the two tasks. Despite the comparable graph sizes, K-means achieves lower mean success rates: 80.33% versus 92.33% on Cut a tree, and 75.00% versus 83.50% on Obtain water. This suggests that the performance difference is not simply due to the number of graph nodes, but also reflects how periods are grouped and matched. Unlike K-means clustering with fixed KK, our prototype matching allows the number of nodes to vary across semantic buckets according to their visual-temporal diversity.

  2. PROSE maintains high task success with a smaller initial offline pool, although performance degrades when successful trajectories become extremely scarce (Table D). We subsample successful and failed trajectories proportionally at 100%, 75%, 50%, and 25%, maintaining a successful fraction of approximately 76.5%. Each subset is used for graph construction, execution-specific residual fitting, and initial replay. Across the four proportional subsampling settings, PROSE maintains comparable success rates (90.00%–93.67%) despite reducing the initial pool to 25% of its original size. All four settings also outperform LS-Imagine initialized with the 100% offline pool.

    We further fix the initial pool at 700 trajectories and reduce the number of successful trajectories. With 100 successful trajectories (14.3% successful fraction), PROSE achieves 86.67% success, exceeding LS-Imagine initialized with all 2,975 archived trajectories. With only 25 successful trajectories (3.6% successful fraction), PROSE still achieves 59.67% success. This shows that PROSE can work in the condition of extremely limited initial successful experience, while additional successful trajectories help improve performance.


Table D: Effects of offline replay and initial data availability on Cut a tree. Successful and Total denote the numbers of successful and total trajectories in the initial offline pool. Successful fraction is calculated as Successful / Total.

SettingSuccessfulTotalSuccessful fraction (%)Success rate (%)
LS-Imagine*———81.33 ± 2.73
LS-Imagine* + 100% offline pool2,2762,97576.583.67 ± 4.04
PROSE (100% offline pool)2,2762,97576.592.33 ± 4.16
PROSE (75% offline pool)1,7072,23176.591.67 ± 4.04
PROSE (50% offline pool)1,1381,48876.590.00 ± 1.85
PROSE (25% offline pool)56974476.593.67 ± 1.58
PROSE (100 successful trajectories)10070014.386.67 ± 4.73
PROSE (25 successful trajectories)257003.659.67 ± 2.08

[Weakness 3] Applicability beyond MineDojo

We apply PROSE's intermediate supervision to household tasks with textual observations on another benchmark dataset (ALFWorld [a]), where GiGPO [b] is used as the base method. The adaptation keeps PROSE's key idea of estimating progress from cross-trajectory terminal outcomes, while adapting period construction and representations to textual observations. Specifically, we use text features and semantic descriptors to construct periods and identify recurring situations. The resulting graph nodes are used for GiGPO's anchor-state grouping, and progress rewards adjust its advantages. Adding PROSE to GiGPO's \textit{w/ std} configuration yields 91.9±1.2%91.9 \pm 1.2\% overall success across six household task categories, compared with 86.7±1.7%86.7 \pm 1.7\% reported for GiGPO \textit{w/ std} (Table E).


Table E: Success rates (%) on ALFWorld with Qwen2.5-1.5B-Instruct.

MethodPickLookCleanHeatCoolPick2All
GiGPO w/ std94.4 ± 5.967.5 ± 4.694.8 ± 3.894.4 ± 7.879.8 ± 4.776.4 ± 5.486.7 ± 1.7
GiGPO w/ std + PROSE97.0 ± 1.180.0 ± 2.099.3 ± 0.797.4 ± 4.494.5 ± 2.577.3 ± 2.391.9 ± 1.2

[Question 2] Offline graph-fitting trajectories

  1. (How the offline graph-fitting trajectories are obtained?) The offline graph-fitting trajectories were collected by the unmodified baseline before PROSE online training and reused for graph construction and replay initialization.

  2. (Is this accounted for with the compared baselines?) Following the training setup in baseline original papers, our reproduced baselines in Tables 1–2 of the main paper start with an empty replay buffer rather than being initialized with this offline dataset.

  3. PROSE achieves a higher mean success rate than LS-Imagine when both are initialized with the same offline replay dataset. We additionally initialize LS-Imagine with the same 2,975 archived trajectories used by PROSE on Cut a tree. PROSE achieves 92.33% success, compared with 83.67% for this baseline (Table D). PROSE also achieves higher mean success rates with less initial offline data. Using 25% of the full pool (744 trajectories), it achieves 93.67% success, and using 700 trajectories with only 100 successful ones, it achieves 86.67%. Both exceed the 83.67% achieved by LS-Imagine initialized with all 2,975 archived trajectories.

References:

[a] Alfworld: Aligning text and embodied environments for interactive learning. ICLR 2021.

[b] Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025.

Comments on the paper

2 comments

We thank the reviewers for their insightful comments and constructive feedback, where the reviewers recognize PROSE’s strengths: an interesting approach (QC4E), useful outcome-based guidance (A7qY, w9Lq), and consistent empirical gains (QC4E, A7qY, w9Lq).

We address the questions from each reviewer below. And we'll update the paper to incorporate the reviewers' suggestions, the added experiments, and our clarifications.

Thank you for taking the time to review our clarifications and additional experiments. We look forward to further discussion.

AuthorsOpenReview ↗

Summary of the rebuttal discussion and the paper revision.

As the discussion period draws to a close, we briefly summarize the main changes and clarifications made in response to the reviewers' feedback, which has helped us strengthen the paper.

Improvements to the paper

  1. Comparable training conditions (QC4E, A7qY, w9Lq). We state how the archived trajectories are obtained and used (Sec. 3), add a baseline initialized with the same archived trajectories on all five tasks (Tables 4 and 9), report results over independent training runs (Table 8), and define how failed episodes enter the steps metric (App. B.1).
  2. Other forms of intermediate supervision (QC4E, A7qY). We add a comparison with an auxiliary learned reward model (App. B.5, Table 11) and with a learned-subgoal baseline adapted from EGR-PO (App. B.6, Table 12), and revise the description of subgoal-based guidance in the abstract and introduction.
  3. Design choices and VLM annotation (QC4E, A7qY). We add ablations of the semantic buckets, the visual-temporal matching and a K-means alternative (Table 7), the sensitivity to the matching threshold, progress-reward weight and degradation penalty (App. B.3), the computational cost (App. B.1), and the labeling errors of the VLM (App. C).
  4. Applicability beyond MineDojo (QC4E, A7qY). We apply PROSE to textual observations on ALFWorld, built on GiGPO (Table 5).
  5. Reliability of the progress rewards (w9Lq). We add diagnostics for repeated visits, policy shift and cross-backbone transfer (App. B.7).

Reviewers' responses

  • QC4E: the rebuttal and experiments "address most of my concerns" and "the paper presents interesting ideas" (rating revised to weak acceptance).
  • A7qY: the concerns about "unequal access to archived trajectories and insufficient independent training runs have been resolved."
  • w9Lq: the additional analyses "substantially address my evaluation concerns" and the diagnostics "strengthen the evidence for the progress signal" (maintain weak acceptance).

Remaining concern

  • Robustness of the visual matching (A7qY). Our latest response and App. B.4 (Table 10) analyze visually similar and visually different pairs of periods.

We sincerely thank the reviewers and the Area Chair for their time, thoughtful feedback, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.