- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CoView-WAM adds epipolar-restricted retrieval to ImageWAM’s action-conditioning cache and uses training-only bidirectional future-view interaction and depth supervision. The authors argue that these components improve cross-view information transfer and encourage 3D-consistent future representations. Matched comparisons show better manipulation performance on LIBERO, CALVIN, and five real-robot tasks; the evidence for the proposed geometric mechanisms is less direct.
Critical strengths
The current-view module places a valid geometric prior directly in the action-conditioning path. Equations 1–3 restrict retrieval using camera calibration without requiring correspondence labels, and the current-only ablation in Table 4 improves LIBERO success from 97.55% to 98.65% over matched ImageWAM. This is evidence that structured cross-view retrieval can be useful for control.
Critical weaknesses
- The methodological novelty of the current-view operator appears limited. Epipolar candidate selection followed by similarity-weighted fusion has precedents [1, 2]. CoView-WAM’s use in an action-conditioning cache is distinct, but Eqs. 2–5 always retrieve candidates and use one gate across target patches; the gate cannot explicitly reject a bad local match while retaining good ones. Moreover, the detached residual in Equation 4 gives the matching scores no direct gradient through this retrieval path. Table 5 isolates a 0.45-point advantage for geometric over global candidate selection on near-saturated LIBERO, without variation across training runs.
- The future branch does not yet establish its proposed cross-view geometric effect. Training-only future-depth supervision has a close precedent in [3]. CoView-WAM’s depth readout also has a local feature path (Appendix A.3), and Eq. 9 applies depth and spatial-difference losses within each view. Accurate per-view depth could therefore be learned without a cross-view-aligned representation, although the bidirectional interaction may encourage one. The joint future-interaction-plus-depth gain over the current-only model is 0.55 points on LIBERO (Table 4), with no direct cross-view consistency measurement or variation across training runs. [1] He, Y., Yan, R., Fragkiadaki, K., & Yu, S.-I. “Epipolar Transformers.” arXiv:2005.04551, 2020. [2] Wang, J., et al. “MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulation.” arXiv:2602.09878, 2026. [3] Li, Y., et al. “WAM4D: Fast 4D World Action Model via Spatial Register Tokens.” arXiv:2606.14048, 2026.
Questions for authors
- For occluded target patches or regions without view overlap, how often does the Top-32 set contain a valid visible correspondence? Under matched capacity, does a per-patch gate or null-match option improve the current shared-gate design on those cases?
- Relative to a capacity-matched local-only depth head and the current-plus-depth model, does the full future branch improve held-out cross-view depth agreement after calibrated reprojection? For example, using the known camera calibration, please report cross-view reprojection error for predicted future depths on mutually visible regions (such as colmap or worldscore[4] ), and compare the full model with Current+Depth and a local-only depth head.
[1] Duan, H., et al. “WorldScore: A Unified Evaluation Benchmark for World Generation.” ICCV, 2025.
3 comments
Response to Reviewer vj77 (Part2)
Q1. Occlusion and regions without view overlap
Occluded regions provide limited reliable cross-view correspondence, but the gate and the cosine weights mitigate the effect. When the retrieval statistics indicate unreliable matches, the gate lowers the overall injection into the wrist representation, and the cosine weights lower the contribution of the candidates that agree poorly with the query. Out-of-view regions are of less practical concern, since manipulation occurs within the external camera's field of view.
A per-patch gate or an explicit null match would be hard to learn and would add inference cost. Which candidate is correct is defined geometrically, so a learned gate has no supervision and tends to degenerate. Instead, our method uses the extrinsics to narrow the search and the cosine weights to rank the candidates; as Figure 2 shows, this reduces irrelevant cross-view matches.
Q2. Cross-view agreement of the future predictions
The two views' predicted future depths agree more closely once the future interaction is added. We measure agreement by reprojecting one view's predicted depth into the other view and comparing it with the depth predicted there; Full gives the lowest error in both directions, roughly halved relative to the two baselines, whereas the baselines differ from each other by only about 3%, so the gap is attributable to the future branch rather than to depth supervision or to the current interaction.
E: external view, W: wrist view
| Configuration | E→W (mm) | W→E (mm) |
|---|---|---|
| ImageWAM + Depth | 136.65 | 125.82 |
| Current + Depth | 132.64 | 125.62 |
| Full model | 65.21 | 77.05 |
Both views' ground-truth depths are rendered from the same 3D scene, so lower depth error in both views is indirect evidence of closer agreement between the two views. Relative to Current + Depth, the patch-level MAE falls from 9.39 to 8.96 mm (external 9.96 → 9.54, wrist 8.82 → 8.38), so both predictions move closer to the same underlying geometry. The agreement measure above tests this directly and also places Full lowest.
| Configuration | Overall (mm) | External (mm) | Wrist (mm) |
|---|---|---|---|
| ImageWAM + Depth | 9.42 | 10.04 | 8.80 |
| Current + Depth | 9.39 | 9.96 | 8.82 |
| Full model | 8.96 | 9.54 | 8.38 |
Follow-up to Reviewer vj77
We thank the reviewer again for the careful review. We now measure cross-view agreement directly by reprojecting predicted future depth: relative to the depth-matched baseline the error falls from 132.64 to 65.21 mm (E-to-W) and from 125.62 to 77.05 mm (W-to-E), while per-view depth MAE moves only from 9.39 to 8.96 mm, so the future interaction improves agreement rather than per-view accuracy. We also add the CALVIN candidate-selection comparison (3.779 to 3.875), and we now state how our retrieval differs from prior epipolar-based approaches: candidates are constructed analytically from calibration rather than learned as sampling offsets, and the same operator updates the action-conditioning cache read by the action expert. As the discussion period closes soon, we would be grateful to hear whether this addresses the reviewer's concerns, and we would be glad to clarify any remaining point.
Response to Reviewer vj77 (Part1)
Thank you for your careful review and detailed questions. We have added two evaluations: a candidate-selection comparison on CALVIN, and a cross-view depth-agreement measurement against the matched-depth baselines. We respond to the four points below.
W1. Differences from prior work, the shared gate
Our key differences from prior epipolar-based methods lie in the retrieval strategy and interaction target. Epipolar Transformers focuses on multi-view pose estimation, while MVISTA-4D uses learnable offsets for epipolar sampling in RGB-D generation. In contrast:
The shared gate modulates global feature injection strength, while cosine weights provide local match modulation. Although the gate cannot explicitly reject individual mismatches, cosine weighting reduces the contribution of less relevant candidates.
Stop-gradient prevents retrieval scores from drifting away from appearance-based correspondence. It keeps the retrieval criterion independent of the optimization objective while preserving gradients through the direct feature path and learnable gates. The stop-gradient is needed because the gate's injection strength is estimated from the current features: letting the matching scores receive gradients would make that estimate track the optimization objective instead of the appearance correspondence, whereas the gradient on the original path is passed back through the subsequent interaction branch.
We further validate our geometric retrieval's (Geometry Top-32) advantage on CALVIN. The 0.45-point LIBERO improvement alone may be insufficient to validate our method, we further add the experiments on CALVIN. Geometry Top-32 improves CALVIN's average completed sequence length from 3.779 to 3.875 over Global Semantic Top-32.
W2. Cross-view effect of the future branch
We added the evaluation for cross-view consistency, and the results support that future geometric interaction gives better cross-view generation consistency. Adding the bidirectional interaction to the future branch of Current + Depth reduces the reprojection error in both reprojection directions, and per-view depth MAE also reduces from 9.39 to 8.96 mm.
The gain from future geometric interaction also appears on CALVIN. Since LIBERO is saturated with 0.55 points' improvement, we perform the corresponding ablation on CALVIN. The future branch's bidirectional interaction also shows a gain from 3.920 to 3.987.
Our approach to cross-view interaction differs from that of WAM4D. Unlike WAM4D, which aggregates cross-view context into spatial register tokens, we perform per-location feature updates at each Transformer block under epipolar constraints, with shared depth queries preserving geometric correspondence in depth prediction. This enables fine-grained, geometry-aware cross-view interaction. WAM4D is belongs to the concurrent work and will be sited in the revision.