acceptodds
Under review as a conference paper at ICLR 2027

FORECASTEDIT: LEARNING TO EDIT WORLD FORECASTS FOR 4D OCCUPANCY PREDICTION

Abstract

Future-world models for autonomous driving typically improve prediction by learning to generate increasingly accurate future states from observations. We study a complementary problem: once a future world has already been predicted, can a model learn what should be edited rather than regenerate the entire forecast? We introduce ForecastEdit, which treats a frozen world-model prediction as an explicit reference and learns selective, evidence-grounded revisions. ForecastEdit combines evidence-guided boundary editing to retract unsupported occupancy with executable semantic editing that converts ontology-structured object dynamics into localized, utility-aware edits, followed by lightweight semantic commitment. On Occ3D-nuScenes, ForecastEdit achieves state-of-the-art semantic forecasting performance with 14.82 mIoU, improving the SparseWorld anchor by 1.62 points while also improving occupied IoU. The edited future representation further improves downstream planning, reducing average trajectory error from 0.27 to 0.23m and collision rate from 0.29% to 0.16%. Controlled comparisons show that these gains cannot be explained by generic post-hoc correction alone. Our results suggest that stronger world models may require not only better forecasting, but also principled mechanisms for editing the futures they already predict.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, 4D Occupancy Forecasting, Autonomous Driving, Forecast Editing, Semantic Occupancy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_skckOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

For autonomous driving, predicting how the surrounding environment will evolve is crucial. This paper treats a frozen world-model prediction as an explicit reference and learns to apply selective, evidence-grounded revisions. The effectiveness of the proposed method is validated on real-world autonomous driving datasets.

Critical strengths

Unlike existing paradigms that rely on redundant predictions, this approach introduces a reference-based editing strategy, which improves both the efficiency and performance of occupancy prediction. The method incorporates "evidence-guided boundary editing" and "executable semantic editing" to refine the initial reference prediction from multiple perspectives. It specifically addresses the core challenges of learning how to edit a reference prediction and when to retain its original content, with its efficacy demonstrated through extensive experimentation.

Critical weaknesses
  1. In the EGBE component, while provenance information helps refine boundaries, identifying the specific "contributing refine points" is non-trivial. The validity of the identified "direct contributors" and "class-consistent supporting points" requires further verification.
  2. Regarding semantic editing, the authors utilize an ontology to extract information concerning existence, kinematics, interaction, and visibility. The rationale for selecting these four ontology-defined states should be clarified, and it should be explained whether there is a specific supervisory strategy for extracting these states.
  3. Relying solely on SparseWorld as a forecasting anchor is insufficient to demonstrate the method's generalizability or model-agnostic applicability; the approach should be evaluated by integrating it with a broader range of model architectures.
  4. In the planning performance evaluation, the proposed method exhibits a significantly higher collision rate at the 1-second horizon compared to models like UniAD. This raises concerns regarding whether the method is suitable for short-term occupancy prediction tasks.
Questions for authors

Refer to Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer skck (Part 1/2)

W1: EGBE Contributor Identification and Validity

A: We thank the reviewer for raising this concern. We clarify that EGBE contributors are automatically constructed from Point-to-Voxel Provenance retained by the frozen Renderer, rather than manually selected. Direct Contributors participate in Occupancy rendering, while Class-consistent Supporting Points are identified through the Renderer's same-class spatial support.

Instead of manually annotated point-level weights, Candidate-conditioned Attention learns each contributor's relative importance, while the Utility-aware Retraction Objective optimizes preserve/retract decisions. Thus, contributor identification is grounded in Renderer provenance, with its effectiveness learned through the editing objective.

To validate both contributor types, we conduct additional ablations comparing Direct-only, Direct + Random Local Points, and Full Provenance, achieving mIoU scores of 13.91, 13.86, and 14.17, respectively. The full approach improves mIoU by 0.26 and 0.31 points, while adding random local points provides no gain. These results suggest that class-consistent supporting points provide meaningful evidence beyond direct provenance and generic local aggregation, supporting the validity of our provenance construction.

EGBE Contributor ConfigurationmIoU
Direct Contributors Only13.91
Direct Contributors + Random Local Points13.86
Full Provenance (Ours)14.17


W2: Ontology Design and Supervision

A: We thank the reviewer for this suggestion. We select Existence, Kinematics, Interaction, and Visibility to capture four complementary aspects of object dynamics: existence continuity, motion evolution, interaction constraints, and observation uncertainty.

Specifically, Existence distinguishes object persistence, temporary disappearance, and scene exit. Kinematics describes motion states such as stationary, accelerating, decelerating, and turning. Interaction captures relationships such as following, overtaking, and cutting in, providing constraints for multi-agent forecasting. Visibility distinguishes actual existence changes from observation loss due to occlusion, as temporary invisibility does not imply scene exit.

Regarding supervision, Existence, Kinematics, and Interaction labels are deterministically derived from matched ground-truth trajectories, motion changes, and pairwise trajectory relationships, respectively. Visibility labels use historical observation ratios and detection confidence. Unreliably matched objects are excluded to avoid noisy supervision. The four states are trained with cross-entropy and Brier calibration losses, while pairwise relations use multi-label BCE and Brier losses, together with soft logical consistency constraints.

Importantly, these states are not merely auxiliary predictions. Their predicted probabilities are projected back into object representations to directly condition multimodal future dynamics decoding, influencing future motion, existence, and local Occupancy. Thus, they provide complementary structured semantics while contributing directly to future scene modeling.

W3: Cross-Backbone Generalization

A: We thank the reviewer for this suggestion. We agree that evaluation solely on SparseWorld is insufficient to fully establish ForecastEdit's applicability across different architectures.

We therefore extend ForecastEdit to SparseWorld-TC[1] and COME[2], two occupancy forecasting models with different architectures, while preserving their original forecasting settings. As shown in the table below, ForecastEdit consistently improves both backbones.

These results suggest that ForecastEdit's effectiveness is not limited to SparseWorld-specific properties but complements different forecasting mechanisms. Together with our original SparseWorld results, these experiments provide further evidence of cross-backbone generalization. We will include the results and analyses in the revised manuscript.

MethodmIoU 1s↑2s↑3s↑Avg.↑IoU 1s↑2s↑3s↑Avg.↑
COME (pred. traj.)25.5718.3513.4119.1145.3637.0630.4637.63
ForecastEdit (COME, pred.)28.1920.8415.4921.5149.7640.0032.8940.89
COME (GT traj.)26.5621.7318.4922.2648.0843.8440.2844.07
ForecastEdit (COME, GT)28.5522.9020.3923.9550.8244.8643.2646.31
SparseWorld-TC-Small27.9525.5123.3525.6050.6949.1547.2349.02
ForecastEdit (SparseWorld-TC-Small)30.4427.6125.0527.7053.8352.4550.6852.32


[1] SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model, Du et al., CVPR 2026

[2] COME: Adding Scene-Centric Forecasting Control to Occupancy World Model, Shi et al., NeurIPS 2025

AuthorsOpenReview ↗

Response to Reviewer skck (Part 2/2)

W4: Short-Horizon Planning Safety

A: We thank the reviewer for raising this concern. We acknowledge that ForecastEdit's 1s collision rate (0.14%) is higher than UniAD's (0.05%), indicating room for improvement in short-horizon planning safety. However, this does not imply that ForecastEdit is unsuitable for short-horizon Occupancy Prediction. At 1s, ForecastEdit improves mIoU from 14.93 to 16.70 and occupied IoU from 22.96 to 23.89, demonstrating improved short-horizon forecasting quality. Importantly, Occupancy accuracy and downstream planning safety are not equivalent optimization objectives.

UniAD is a separate planning-oriented model trained with auxiliary supervision from Maps, Boxes, and Occupancy. In contrast, ForecastEdit retains the frozen SparseWorld Planner and improves planning solely through refined scene representations, without explicitly optimizing for collision risk. Thus, collision rates depend not only on Occupancy quality but also on the Planner's decision policy and its response to different prediction errors.

One possible explanation is the trade-off between Occupancy accuracy and planning safety. ForecastEdit primarily improves predictions by retracting unsupported occupancy, which may also introduce some recall loss. In Occupancy-conditioned Planning, false-positive occupancy may induce conservative behavior. Therefore, better Occupancy predictions do not necessarily translate into lower collision rates at every horizon.

Although ForecastEdit improves average collision rates and longer-horizon planning performance, we acknowledge that short-horizon safety remains an area for improvement. We will explicitly discuss this distinction between Occupancy forecasting quality and short-horizon planning safety in the revised Limitations section.

Reviewer_2phMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies occupancy forecasting in the context of autonomous driving. It proposes to directly edit the output of SparseWorld, an existing occupancy forecaster. It introduces (1) EGBE to post-process occupied voxels to free and (2) ESE to turn object tracks into occupancy. Overall, the proposed method improves mIoU from 13.20 (SparseWorld) to 14.82. In particular, EGBE helps with cars and stuff classes such and drivable surface or sidewalk and ESE helps with smaller objects. However, the SOTA claim does not hold.

Critical strengths

None

Critical weaknesses

The claim that ForecastEdit is SOTA on Occ3D-nuScenes does not hold. Two important baselines are missing from Table 1. ForecastEdit scores 14.82 mIoU on Occ3D-nuScenes, while SparseWorld-TC-Small (Table 3 of [1]) achieves 21.57 mIoU and COME (Table 1 of [2]) achieves 19.11 both with camera input & predicted ego trajectory. [1] SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model, Du et al., CVPR 2026 [2] COME: Adding Scene-Centric Forecasting Control to Occupancy World Model, Shi et al., NeurIPS 2025

Questions for authors
  1. How was the motion planning experiment done? The paper has no description of this, except “planning components (of SparseWorld) remain unchanged” (Line 315-317). However, the architecture of SparseWorld (Figure 2 of SparseWorld) suggests planning decoding does not depend on decoded future occupancy. Since this paper freezes SparseWorld and directly edits the output future occupancy, it is unclear to me why the planning numbers ForecastEdit differ from SparseWorld’s, considering the L2 and collision rate metrics only depend on the trajectory. Am I missing something?
  2. Figure 1(a) is much blurrier than Figure 1(b). Plus, the zoomed in view does not seem to match the overall view.
  3. Eq. (4) and Eq. (13) are the same equation repeated twice.
  4. Figure 4 seems low-quality and looks like a scanned figure with background imperfectly removed.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2phM (Part 1/2)

W: SOTA Claim and Cross-Backbone Generalization

A: We first clarify that ForecastEdit does not aim to introduce a new forecasting backbone, but to improve existing future predictions through structured editing on top of a strong, frozen world model. We will revise the SOTA-related wording in the original manuscript to make it more precise and avoid ambiguity.

Regarding baseline selection, SparseWorld-TC and COME differ from our setting by introducing additional future conditioning. Specifically, SparseWorld-TC conditions occupancy forecasting on predicted ego trajectories, while COME introduces a separate scene-centric forecasting branch and injects its predictions through ControlNet. Our main experiments instead follow the original SparseWorld setting, using it as a unified editing anchor with a fixed backbone, renderer, and evaluation protocol. This controlled setup isolates the contribution of forecast editing rather than conflating it with differences in forecasting architectures or conditioning.

Regarding generalization, we agree that evaluating different backbones is important. We therefore further extend ForecastEdit to SparseWorld-TC and COME, comparing edited and original predictions under their respective forecasting settings.

For COME, we evaluate both predicted-trajectory and ground-truth-trajectory settings. For SparseWorld-TC, the reviewer-cited 21.57 mIoU (Table 3) uses BEVPlanner-predicted trajectories. However, the public repository does not provide the corresponding trajectory files or the complete pose transformation and integration pipeline, making exact reproduction difficult. To avoid additional trajectory prediction and coordinate conversion errors, we instead adopt the authors' official SparseWorld-TC-Small GT-trajectory setting (25.60 mIoU) and evaluate performance before and after editing.

As shown in the table below, ForecastEdit consistently improves performance across different forecasting architectures and conditioning settings. These results suggest that ForecastEdit is not a SparseWorld-specific post-processing module, but a structured editing method complementary to different forecasting mechanisms.

We will include these additional baseline comparisons and cross-backbone results in the revised manuscript.

MethodmIoU 1s↑2s↑3s↑Avg.↑IoU 1s↑2s↑3s↑Avg.↑
COME (pred. traj.)25.5718.3513.4119.1145.3637.0630.4637.63
ForecastEdit (COME, pred.)28.1920.8415.4921.5149.7640.0032.8940.89
COME (GT traj.)26.5621.7318.4922.2648.0843.8440.2844.07
ForecastEdit (COME, GT)28.5522.9020.3923.9550.8244.8643.2646.31
SparseWorld-TC-Small27.9525.5123.3525.6050.6949.1547.2349.02
ForecastEdit (SparseWorld-TC-Small)30.4427.6125.0527.7053.8352.4550.6852.32


Q1: Planning with a Frozen Decoder

A: We thank the reviewer for raising this question. As already stated in Sec. 5.1 (Lines 315–317), the perception, renderer, and planning components of SparseWorld remain unchanged throughout our experiments. We further clarify that freezing refers to the Planner's architecture and parameters, not its inputs.

Specifically, we convert the Future Occupancy edited by ForecastEdit into Scene Queries and feed them into the frozen SparseWorld Planning Decoder. Both ForecastEdit and the original SparseWorld baseline use exactly the same Planner architecture and parameters.

Thus, although the original SparseWorld Planning Decoder does not directly depend on the decoded Occupancy output, ForecastEdit passes the edited scene information to the Planner through an Occupancy-to-Scene-Query conversion, thereby affecting trajectory predictions. This explains why L2 error and collision rate can change even with a fully frozen Planner.

Table 2 aims to verify whether the edited future scene representation provides more useful information for downstream planning. The results show that improved Occupancy forecasting can also translate into better trajectory planning. We will clarify this input conversion process in the revised manuscript to avoid ambiguity.

AuthorsOpenReview ↗

Response to Reviewer 2phM (Part 2/2)

Q2–Q4: Figure Quality and Equation Redundancy

A: We thank the reviewer for pointing out these issues.

Figure 1. Figures 1(a) and 1(b) were created together on the same canvas. We will re-export the figure at a higher resolution to ensure visual consistency and readability.

Redundancy of Eqs. (4) and (13). We agree that these equations are redundant. Eq. (4) first defines the final forecast composition rule, while Eq. (13) was intended to explain semantic commitment but repeats the same formulation. We will remove Eq. (13) and directly refer to Eq. (4) in the revised manuscript.

Figure 4. Figure 4 is not a scanned image but was generated and assembled using Python from experimental visualizations. We will re-export it at a higher resolution to improve its visual quality.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 2phM,

Thank you again for your valuable feedback. We would like to follow up on our previous response.

To address your main concerns, we have clarified ForecastEdit's methodological positioning, SOTA-related claims, and the rationale behind our baseline selection, and conducted additional cross-backbone generalization experiments on COME and SparseWorld-TC. The results demonstrate that ForecastEdit consistently improves performance across different forecasting architectures and trajectory-conditioning settings, further validating its effectiveness and generalizability. In addition, we have provided a detailed explanation of how the edited Occupancy is converted into Scene Queries and fed into the fully frozen Planner, and outlined specific revisions to address figure quality and equation redundancy.

We hope these clarifications help resolve previous misunderstandings and adequately address your main concerns. Meanwhile, the additional experimental evidence further supports the core contributions and conclusions of our work. If you have any remaining questions, we would be more than happy to provide further clarification and discussion. If you believe that these clarifications and additional experiments have sufficiently addressed your previous concerns, we sincerely ask that you reconsider the technical contributions of our work in light of the more comprehensive evidence and consider revising your overall assessment and score accordingly.

Thank you again for taking the time to review our work. We look forward to hearing your further feedback.

Best regards,

The Authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer Z68Z (Part 1/3)

W1: Dataset Selection and Cross-Backbone Generalization

A: We thank the reviewer for the constructive feedback. We agree that evaluating ForecastEdit requires controlling for confounding effects from generic post-hoc correction. To further assess reliability and generalizability, we supplement our original generic correction baselines and component-level analyses with cross-backbone experiments.

Specifically, we extend ForecastEdit to two different occupancy forecasting backbones, SparseWorld-TC[1] and COME[2], comparing performance before and after editing under their original forecasting settings. As shown in the table below, ForecastEdit consistently improves both models, suggesting that its gains stem from structured editing rather than backbone-specific properties.

Regarding dataset selection, we adopt Occ3D-nuScenes due to its widespread use in occupancy prediction and forecasting. Originally introduced as a standardized 3D occupancy benchmark[3], it has been adopted by representative methods, including OccWorld[4], PreWorld[5], COME[2], and SparseWorld[6], spanning different forecasting architectures and paradigms. This provides a consistent evaluation basis and facilitates comparisons with existing work.

We will include these additional experiments and analyses in the revision to strengthen the evidence for ForecastEdit's effectiveness and generalizability.

MethodmIoU 1s↑2s↑3s↑Avg.↑IoU 1s↑2s↑3s↑Avg.↑
COME (pred. traj.)25.5718.3513.4119.1145.3637.0630.4637.63
ForecastEdit (COME, pred.)28.1920.8415.4921.5149.7640.0032.8940.89
COME (GT traj.)26.5621.7318.4922.2648.0843.8440.2844.07
ForecastEdit (COME, GT)28.5522.9020.3923.9550.8244.8643.2646.31
SparseWorld-TC-Small27.9525.5123.3525.6050.6949.1547.2349.02
ForecastEdit (SparseWorld-TC-Small)30.4427.6125.0527.7053.8352.4550.6852.32

[1] SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model, Du et al., CVPR 2026

[2] COME: Adding Scene-Centric Forecasting Control to Occupancy World Model, Shi et al., NeurIPS 2025

[3] Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving, Tian et al., NeurIPS 2023

[4] OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving, Zheng et al., ECCV 2024

[5] PreWorld: Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving, Li et al., ICLR 2025

[6] SparseWorld: A Flexible, Adaptive, and Efficient 4D Occupancy World Model Powered by Sparse and Dynamic Queries, Dang et al., AAAI 2026



W2: Source-Side Calibration and Manual Design

A: We would like to clarify that the class-dependent quantities in ForecastEdit are not independently hand-tuned parameters, but are automatically estimated or derived from source-domain data through predefined procedures.

Specifically, EGBE's class-specific thresholds are determined using independent calibration records, while class-specific routes are selected from three predefined policies based on scene-disjoint source-side reliability evidence. Table 7 in Appendix A.2 reports these automatically selected routes rather than manual assignments. In ESE, class-conditional utility weights are derived from editing outcomes, while Semantic Commitment's class coefficients are computed from prediction-to-ground-truth transition statistics. Group-level weights are dynamically updated based on excess risk and used only during training.

These mechanisms therefore differ from manually specified hyperparameters. All statistics are fixed before formal evaluation. We will further clarify their in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Z68Z (Part 2/3)

Q1: Inference Efficiency and Resource Overhead

A: We thank the reviewer for this question. We conduct additional paired evaluations comparing ForecastEdit, the frozen forward pass, and full regeneration, with short-budget continued fine-tuning as an additional baseline.

Under FP32 and batch size 1, we evaluate 30 fixed scenes with identical inputs, one forecasting anchor per scene, and three repeated measurements per method. The fine-tuning baseline uses 128 training scenes and 64 parameter updates.

ForecastEdit achieves better prediction quality than full regeneration on 96.7% of scenes and higher mIoU than short-budget fine-tuning on 93.3%. Compared with full regeneration, it adds only 3.39 MiB (0.189%) peak GPU memory and 29.15 ms (+20.4%) average inference latency.

These results suggest that ForecastEdit effectively refines predictions with modest additional resource overhead, without retraining the full forecasting model.

MetricFrozen BackboneShort-Budget Fine-TuningForecastEditΔ vs. RegenerationΔ vs. Fine-Tuning
mIoU (%) ↑13.06413.14114.473+1.409 pp+1.332 pp
Peak GPU Memory (MiB) ↓1797.381797.381800.77+3.39 (+0.189%)+3.39 (+0.189%)
Avg. Inference Latency (ms) ↓142.64140.56171.79+29.15 (+20.4%)+31.23 (+22.2%)


Q2: Tracker Misses and ESE Recovery

A: We thank the reviewer for this question. We agree that upstream tracking misses can limit track-based editing. To quantify this effect, we further analyze tracker coverage and recovery under simultaneous tracking and forecasting misses.

Tracker Configuration and Miss Rate. We use Sparse4D v3 (ResNet-50) to generate 3D object tracks from multi-view camera inputs. Among pedestrian instances supported by GT Occupancy, the historical Tracker cache achieves 76.53% coverage.

Recovery under Simultaneous Misses. We further investigate a more challenging case: can future Occupancy still be recovered when both the Tracker and the base world model miss the same pedestrian? Unlike object tracking, Occupancy Forecasting predicts future geometric and semantic states without necessarily requiring an established object track. Thus, tracking failure does not imply that all relevant scene information is lost.

We identify 1,027 pedestrian instances missed by the Tracker and completely absent from the base model's future Occupancy predictions. On this subset, full ESE still recovers correct Occupancy support for 237 instances (23.08%) at one or more future horizons.

Mechanism. ESE includes not only a trajectory-based object editing pathway but also a Semantic Commitment branch that directly operates on frozen Query/Refine representations and semantic predictions. This branch does not require a corresponding track and can recover missing Occupancy from spatial and semantic evidence retained by the world model.

Therefore, tracking misses constrain object-level editing coverage but do not eliminate the Fill capability of full ESE. We will include the Tracker configuration and these coverage and recovery analyses in the revised manuscript.

MetricResult
Camera-based trackerSparse4D v3 (ResNet-50)
Pedestrian track coverage / miss rate76.53% / 23.47%
Recovery under tracker-uncovered and baseline-missed cases237/1,027 (23.07%)
AuthorsOpenReview ↗

Response to Reviewer Z68Z (Part 3/3)

Q3: Frozen Planner and Input Distribution Shift

A: We thank the reviewer for raising this concern. We clarify that freezing the Planner refers to keeping its architecture and all parameters unchanged, rather than fixing its input representation.

Specifically, we do not retrain or fine-tune the Planner on edited representations. At inference, we convert the Future Occupancy edited by ForecastEdit into Scene Queries and feed them into the frozen SparseWorld Planning Decoder. Thus, SparseWorld and ForecastEdit use exactly the same Planner, differing only in the scene representations provided as input.

Regarding potential input distribution shift, we conduct an additional paired analysis. Using GT Occupancy from 128 training scenes as a reference set, we compute the mean 5-NN distance before and after editing in a predefined spatial-semantic representation. The results show that editing reduces the mean distance by 13.14%, with all evaluated scenes moving closer to the reference set. Moreover, 98.86% of occupied locations after editing remain within the spatial support of the original prediction.

These results suggest that ForecastEdit primarily removes unreliable occupancy and corrects local semantics while preserving the original spatial structure, rather than introducing arbitrary scene content. The edited representations are therefore closer to the GT Occupancy distribution, reducing the risk of Planner input distribution shift.

Thus, the planning gains arise from more reliable Occupancy-derived Scene Queries rather than Planner parameter updates or additional training. We will clarify this evaluation protocol and distribution-shift analysis in the revised manuscript.

MetricResult
Reduction in mean 5-NN distance to training GT occupancy13.14%
Edited occupancy positions within original prediction support98.86%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.