acceptodds
Under review as a conference paper at ICLR 2027

Seeing Farther to Act Now: Long-Horizon Future Modeling for Robot Manipulation

Abstract

World Action Models (WAMs) provide predictive context for robot manipulation by conditioning action generation on imagined future visual states. However, existing WAMs commonly couple the visual prediction horizon to the action horizon, restricting future information to the immediate interaction and potentially obscuring longer-term task consequences. We show that prediction horizon introduces a fundamental trade-off: short-horizon futures preserve fine-grained motion and interaction details, whereas longer-horizon futures capture broader task progression but with greater visual uncertainty, making the most useful temporal scale task dependent. Building on this insight, we propose Seeing Farther to Act Now (SFAN), a multi-timescale world-action model that decouples visual prediction from the executable action horizon and assigns complementary roles to different temporal scales. SFAN uses sparsely sampled long-horizon futures to provide broader task context and generate the primary action prediction, while a complementary short-horizon pathway densely samples nearby futures to predict a residual correction conditioned on the primary action. This asymmetric design allows longer-range futures to guide interaction selection while preserving the fine-grained local information needed to refine how the selected interaction is executed. Extensive experiments on simulated and real-world robot manipulation demonstrate that SFAN consistently outperforms existing baselines, with particularly strong gains on tasks that benefit from longer-horizon future context.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Policy, World Action Models, Long-Horizon Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_q5JmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Seeing Farther to Act Now (SFAN), a world-action model that decouples the visual prediction horizon from the action horizon for robot manipulation. Motivated by the trade-off between broader task context and fine-grained local detail, SFAN uses sparsely sampled long-horizon futures to generate a primary action chunk and densely sampled short-horizon futures to predict residual corrections conditioned on that chunk. An auxiliary waypoint objective further supervises long-range task progression. Experiments on RoboCasa, composite RoboCasa365 tasks, and two real-world manipulation tasks report improved overall success rates over the evaluated baselines, while ablations support the complementary roles of long-horizon guidance and short-horizon refinement.

Critical strengths

A compelling idea supported by coherent method design and empirical evidence. I find the paper’s central argument convincing: the future information needed for a good action decision can extend beyond the current action horizon. SFAN implements this idea through a complementary design, using sparse long-horizon predictions to guide primary action generation and dense short-horizon predictions to provide residual corrections for precise execution. The experiments support this design choice: combining both pathways achieves 69% success on RoboCasa, compared with 66% and 63% for the short- and long-horizon variants alone, with further improvements reported on composite and real-world tasks. This alignment between motivation, method design, and empirical results is a key strength supporting my positive assessment of the paper.

Critical weaknesses

1. Insufficient isolation of the contribution of temporal complementarity. Although the horizon and fusion ablations provide useful evidence, SFAN also introduces an additional action model, staged residual training, and auxiliary waypoint supervision. Section 3.4 further mentions additional trajectories for residual learning without clarifying their quantity or whether the comparison methods receive equivalent data. A matched control retaining the full architecture, supervision, and training budget while using short-horizon futures in both pathways would more directly establish the benefit of combining temporal scales. Without such a comparison, the relative contributions of longer future context and the additional refinement machinery remain unclear, limiting confidence in the paper’s central methodological claim.

2. Unclear control implications of the inference overhead. Table 4 reports 1.77 seconds per action prediction for SFAN, compared with 0.70 seconds for Cosmos-Policy. However, the paper does not specify the control frequency, the number of actions executed before replanning, or whether inference overlaps with execution. These details determine the effective feedback rate and the delay between observation and action. Although the authors acknowledge the computational overhead, the reported success rates do not establish how the methods compare under the same timing constraints. This limits the evidence for SFAN’s practical advantage in responsive closed-loop manipulation.

Questions for authors

Q1. Fairness of training data and supervision. Section 3.4 mentions additional trajectories used during residual training. Could the authors clarify the amount of additional data and whether the baselines and ablation variants use equivalent training data? This is important for determining whether the reported improvements can be attributed to the proposed method rather than additional supervision.

Q2. Isolating the benefit of multi-timescale modeling. Could the authors provide a capacity-matched comparison in which both pathways use short-horizon predictions, while retaining the same architecture, training strategy, and auxiliary supervision? Although the existing ablations are informative, such a comparison would more directly establish whether the improvement comes from complementary temporal horizons rather than the additional refinement module.

Q3. Inference latency and closed-loop execution. Table 4 reports an inference latency of 1.77 seconds per action prediction, substantially higher than Cosmos-Policy (0.70 seconds). Could the authors clarify the actual control frequency, execution horizon, and whether inference is performed asynchronously? How does this latency affect closed-loop performance in the real-world experiments?

Reviewer_d6C7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SFAN studies visual prediction horizons beyond the executable action chunk. A sparse long-horizon stream conditions a primary action policy, while a dense short-horizon stream predicts a residual correction. Future end-effector waypoints provide auxiliary supervision for the long-horizon policy. The two-stage model is evaluated on standard and composite RoboCasa tasks and two real-world manipulation tasks.

Critical strengths

The horizon analysis motivates a useful design choice. Longer prediction helps some task categories and hurts others, making a single fixed temporal scale inadequate for the tested suite. SFAN gives the two scales different control roles rather than merely concatenating their features. The comparisons with direct fusion, a multi-scale representation, and a short-primary design are relevant evidence for this choice (Fig. 6). Appendix C also varies prediction density at a fixed horizon, so the analysis goes beyond changing temporal stride alone.

The composite-task results provide the strongest support. Average full-task success reaches 20.75%, compared with 9.25% for Cosmos-Policy and 11.75% for GR00T-N1.5, with stage-wise results showing better completion across most tasks (Sec. 4.1; Fig. 3). The standard benchmark and physical tasks add breadth. The paper also reports the inference cost explicitly, allowing the performance benefit to be weighed against the additional latency.

Critical weaknesses

The residual-stage data need clearer accounting. Section 3.4 says training uses the original data together with additional trajectories, but does not identify their number or source, or say whether the stated demonstration counts include them. It is also unclear which baselines and ablations receive them. The fusion variants are useful, but their waypoint supervision, action-network capacity, and training budgets are not specified precisely enough to tell which factors they hold fixed. As a result, the experiments support the full package more strongly than the particular explanation of long-range guidance versus local refinement. This is a reporting uncertainty, rather than a finding that the comparisons are unfair.

The smaller gains deserve a more cautious interpretation. Standard RoboCasa improves from a reported 66% to 69%, while two-pen retrieval improves from 33% to 40% over Cosmos-Policy with 15 trials, corresponding to one additional successful rollout. The main comparisons do not show variation across training runs or paired outcome uncertainty. These modest differences matter when assessing the latency increase from 0.70 to 1.77 seconds per action prediction (Table 4). The composite results remain more persuasive, and the disclosed cost does not by itself undermine them.

Questions for authors
  1. How many additional trajectories enter residual training, where do they come from, and which comparisons use them? For Figure 6, please specify what is held fixed across data, waypoint supervision, action-network capacity, and training budget.
  2. How stable are the composite gains across training runs? Paired rollout outcomes or uncertainty estimates would also help interpret the smaller standard-benchmark and physical-task differences.
Reviewer_Ja3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the prediction horizon of world-action models affects robot manipulation performance. The authors observe an important factor, which is that short-horizon predictions tend to preserve fine-grained interaction details, while longer-horizon predictions can provide more information about task progression. Based on this observation, they propose SFAN, which uses long-horizon information to determine the primary action prediction, while short-horizon information provides residual refinement. The long-horizon branch is also trained with additional waypoint supervision. Experiments on RoboCasa/RoboCasa365 and real-world manipulation tasks show improvements over existing baselines, especially on multi-stage tasks.

Critical strengths

S1. This paper studes an interestsing problem regarding the prediction horizon which is an important but underexplored design choice in WAMs, highlighting the trade-off between capturing long-term task progression and preserving fine-grained details for precise manipulation.

S2. I think the motivation and overall architecture are reasonable. Instead of simply extending the prediction horizon, the authors use long-horizon predictions to generate the primary action and short-horizon predictions to refine it. I think this is a reasonable design choice for combining information from different time scales. The two-stage training strategy and additional waypoint supervision also make intuitive sense in this setting.

Critical weaknesses

I think the motivation and overall architecture are reasonable. However, I am not fully convinced that the experimental results directly support the paper's main claim that long-horizon predictions improve interaction selection, while short-horizon predictions enhance local execution precision. The observed performance gains could potentially be attributed to the additional supervision rather than the proposed division of roles between long- and short-horizon predictions. The current experiments do not seem to provide sufficient evidence to establish this distinction.

Questions for authors

Q1. I think the author will need more Evidence for the main claim. How do the experiment results support the claim that long-horizon predictions improve interaction selection, while short-horizon predictions improve local execution accuracy? The current results mainly show improvements in overall success rates, but do not seem to directly demonstrate this division of roles. How much of the performance improvement comes from the additional waypoint supervision? Could the authors provide an ablation without the waypoint loss to better isolate the contribution of long-horizon prediction?

Q2. The paper mentions using additional trajectories for residual training. Is this additional data also used to train the baselines? Could the performance gains partly come from the additional training data or increased model capacity?

Q3. Inference efficiency: SFAN has a much higher inference latency than Cosmos-Policy (1.77s vs. 0.70s). Real-world robot control requires fast execution and quick interaction with the environment. How does this affect real-world robot control, and is the performance improvement worth the additional inference cost?

Comments on the paper

1 comment

Dear Reviewers,

Please take time to review the responses from the authors and engage with them for further questions and comments if any.

Time left is not much and please do shortly.

Thanks

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.