- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GeoState constructs fixed-length scene tokens from frozen VGGT-Ω features using source geometry and camera cues, then decodes depth with query-camera rays. A temporal model predicts future scene tokens while a separate model predicts camera motion; metric scale is calibrated once from an observed LiDAR frame. The authors argue that this factorization reduces camera-induced variation in the predictive state. Results on Waymo, KITTI, and nuScenes support useful depth forecasting, and the cross-query experiment supports consistency among nearby views decoded from one state.
Critical strengths
- The method provides a useful operational separation between forecasting a scene state and choosing a camera for decoding it. Equations (7)–(10) permit the same predicted state to be queried from different cameras, and Figure 3 shows geometric agreement across nearby query views. This makes the predicted representation more flexible than a forecast tied to one output view.
- The forecasting gains are consistent across the reported datasets and horizons. Tables 1–2 show improvements over camera-matched Copy-Last on Waymo, KITTI, and nuScenes, while Table 10 shows larger gains on dynamic Waymo windows. These results support the value of predicting scene evolution beyond re-querying the last observed state, with metric scale fixed from the observed context.
Critical weaknesses
The mechanism used to explain the representation ablation is not isolated. Fixed scene tokens with ray-conditioned decoding have a close precedent in 3DRAE, while FR3D already separates ego-motion and scene prediction; GeoState’s distinctive claim is that this representation makes physical state evolution less sensitive to input-camera variation. However, source-camera poses and rays enter the encoder in Equations (4)–(5), Figure 3 appropriately measures local consistency across queries of one state; it does not measure how the state changes with its input views. Section 4.5 interprets Table 4 as suggesting that the view-decoupled state is easier to predict: the two methods have similar reconstruction quality, while GeoState improves predicted-depth AbsRel from 0.144 to 0.133 at 2.0 s. However, the compressed-feature baseline has a 2.11M-parameter compressor, whereas GeoState’s tokenizer and decoder have about 206M parameters together and use a different, ray-conditioned decoding path. The reported gains establish the utility of the full system, but do not determine how much comes from suppressing input-view variation.
Questions for authors
- Equations (4)–(5) provide source-camera poses and rays to the tokenizer; what in the method encourages it to use these cues to align views rather than retain camera-specific variation in the predictive state?
- The paper argues that GeoState reduces changes in the predictive state caused by camera motion. Could the authors test this directly? For the same scene and coordinate frame, encode the scene from different input views with similar coverage, then decode both resulting states from the same output camera. How much do the decoded depths and subsequent forecasts differ?
- Can the Table 4 comparison be repeated with a compressed-feature baseline whose encoder and camera-conditioned decoder have comparable capacity, while retaining the same state size and world model? If this is infeasible, what evidence rules out encoder or decoder design as the main cause of the forecasting gap?
4 comments
Response to Reviewer a82h (Part 2/2)
(c) Subsequent forecast consistency. To test whether this robustness also holds during temporal prediction, we conduct a separate diagnostic on 100 static Waymo validation windows. Within each model, conditions A/B share the same final observed state to maintain a common forecasting origin, together with the same coordinate anchor and metric scale, while the preceding three observations are shifted:
H_B=[S_{t-0.6}^{B},S_{t-0.4}^{B},S_{t-0.2}^{B},S_t].$$ Each timestamp supplies three synchronized views. We compare GeoState with the View-coupled baseline introduced in Table R4. Both methods use GT source-camera conditioning and undergo the same history perturbation. Each model autoregressively predicts future states from its A/B histories. At each horizon, GeoState is decoded with the same GT FRONT query camera for both conditions; the baseline produces its native FRONT depth. We measure A/B depth discrepancy on identical LiDAR-verified common regions. This static-scene diagnostic tests sensitivity to shifted historical observations, including their temporal sampling phase, while keeping the final observed state fixed. **Table R3. A/B forecast depth discrepancy (AbsRel↓).** | Representation | +0.6 s | +1.0 s | +2.0 s | |---|---:|---:|---:| | View-coupled | 0.01096 | 0.01254 | 0.01527 | | GeoState | **0.00711** | **0.00864** | **0.01311** | | Relative reduction | **35.1%** | **31.1%** | **14.2%** | GeoState has lower mean forecast discrepancy at all reported horizons, indicating more stable subsequent predictions under the tested history perturbations. These results complement the direct source-view test by examining the effect of changing observations on autoregressive prediction. ### **To Q3:** Our original feature-compression baseline was designed under the consideration that the Scene Tokenizer–Decoder is trained solely with depth supervision, so comparable reconstruction accuracy suggested sufficient capacity for this task. We agree that a controlled comparison with matched architectures and model capacity more directly tests the role of view decoupling, and therefore add a view-coupled counterpart with the same encoder–decoder architectures, parameter count, latent-state size, inputs, and world-model architecture, using comparable training budgets. The key difference lies in the representation's reference frame: GeoState encodes scene states in a shared coordinate system and decodes depth using predicted camera-conditioned queries, whereas the view-coupled variant represents each state in its corresponding FRONT-camera coordinate frame and decodes depth using an identity-pose query. Both retain the same ray-conditioned decoder architecture; the query pose is expressed according to each representation's coordinate frame. **Table R4. Capacity-matched comparison on Waymo (AbsRel ↓ / $\delta_1$ ↑).** | Representation | Recon. (Avg. 3 Steps) | t+0.6s | t+1.0s | t+2.0s | |---|---|---|---|---| | View-coupled | 0.073 / 0.930 | 0.094 / 0.902 | 0.112 / 0.875 | 0.149 / 0.814 | | GeoState(Pred. camera) | 0.072 / 0.931 | **0.079 / 0.921** | **0.091 / 0.902** | **0.126 / 0.848** | Under comparable reconstruction performance, GeoState consistently outperforms the view-coupled variant across all reported forecasting horizons. With encoder–decoder capacity and decoder structure controlled, these results support our central hypothesis that separating camera-induced variation from scene dynamics facilitates more effective world modeling. We will update the corresponding table in the revised manuscript with this controlled ablation. ### **To Weaknesses:** We thank the reviewer for identifying these limitations. We address the weaknesses in our responses to Q1–Q3. Regarding sensitivity to input-camera variation and the insufficiency of the original evidence, we provide **a mechanism-level explanation in Q1** and **experimental validation in Q2 (Tables R1–R3)**. Q1 explains how the training objectives encourage states to preserve geometrically meaningful information rather than reproduce camera-dependent feature variations. Q2 shows smaller normalized representation changes and lower common-query depth discrepancies than VGGT-Ω under changing source observations, as well as more stable subsequent forecasts than the view-coupled baseline under history perturbations. Regarding the capacity and decoder differences in the original Table 4, **Q3 (Table R4)** adds a controlled ablation. The results suggest that with architectures and capacity matched, GeoState retains its forecasting advantage at comparable reconstruction quality. This supports our hypothesis that separating camera-induced variation from scene dynamics facilitates temporal prediction.Response to Rebuttal
Thank you for the additional experiments. The response substantially addresses my main concern, particularly through the capacity-matched comparison in Table R4. Matching the encoder–decoder architecture, parameter count, latent-state size, and world model, while obtaining similar reconstruction accuracy, makes the forecasting improvements considerably more informative than the original compressor comparison. As described, this experiment substantially alleviates my concern about capacity and decoder-architecture differences. Table R1 also addresses an important distinction in my review: it changes the source observations while holding the output camera fixed, rather than querying a single state from different cameras.
Overall, I find the new evidence sufficient to substantially alleviate my original concern and am inclined to raise my recommendation to Weak Acceptance.
Thank You for Your Constructive Feedback
We sincerely thank the reviewer a82h for the thoughtful follow-up and positive assessment of the additional evidence. Your suggestions helped us strengthen the controlled comparison and more directly evaluate sensitivity to source-view changes, making the evidence for our central claim clearer. We have incorporated these additional experiments and clarifications into the revised manuscript and uploaded the updated PDF. We greatly appreciate your time and constructive feedback throughout the review process.
Response to Reviewer a82h (Part 1/2)
We thank the reviewer a82h for the detailed and constructive feedback. We address these concerns below with additional controlled experiments and analyses.
To Q1:
View decoupling is encouraged by two complementary training mechanisms:
To Q2:
We appreciate the reviewer's suggestion. To directly evaluate source-view sensitivity, we construct conditions that share the same static scene, reference frame, and output camera, but differ in their source observations. We examine both the resulting representations and decoded depths, and additionally test how changes in historical observations affect subsequent forecasts. We will incorporate these additional experiments and analyses into the revised manuscript.
Specifically, we select 100 static Waymo validation windows (seed 42). Three shared anchor images at time t establish the coordinate frame. Condition A uses three source views at t+0.2 s, while condition B uses three source views at t+0.6 s. Under each condition, VGGT-Ω jointly processes the shared anchors and the corresponding source views, and the Scene Tokenizer reads only the source FRONT tokens. Both conditions are evaluated at the same GT FRONT camera at t+0.4 s, without query-time RGB inputs. Metric scale is calibrated from the shared anchors, without output-depth alignment.
(a) Common-query depth consistency. GeoState directly decodes depth at the common query camera, while VGGT-Ω reprojects its calibrated source FRONT depth into that view. We measure the symmetric relative depth discrepancy between A/B and accuracy against GT LiDAR on identical LiDAR-verified commonly visible pixels. This shared evaluation region focuses the comparison on geometry visible under both source conditions.
Table R1. Common-query depth discrepancy and accuracy (↓).
GeoState reduces depth discrepancy by 9.8% while maintaining comparable accuracy against GT LiDAR. This indicates that GeoState produces more consistent geometry under changing source observations without sacrificing geometric fidelity, providing output-level evidence of reduced source-view sensitivity.
(b) Representation stability. We further measure relative L2 and cosine distances between representations under the same A/B conditions:
E_{\cos} = 1-\cos(\operatorname{vec}X_A,\operatorname{vec}X_B).$$ For VGGT-$\Omega$, we directly compare the complete source FRONT patch-token grids at matching spatial indices. For GeoState, we compare corresponding latent slots. Both distances are computed on flattened representations and averaged over the same static windows. **Table R2. Representation variation under source-view changes (↓).** | Representation | Relative L2 | Cosine distance | |---|---:|---:| | VGGT-$\Omega$, layer 4 | 0.58516 | 0.17326 | | VGGT-$\Omega$, layer 11 | 0.73070 | 0.26903 | | VGGT-$\Omega$, layer 17 | 1.05183 | 0.55474 | | VGGT-$\Omega$, layer 23 | 1.04999 | 0.55316 | | GeoState latent state | **0.20729** | **0.02250** | GeoState exhibits substantially smaller normalized representation changes under different source observations. This suggests reduced camera-induced variation in the latent states used for temporal prediction, complementing the decoded-depth consistency results in Table R1 and supporting the motivation for view-decoupled world modeling.