- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Scene-Grounded Test-Time Training (SG-TTT), a method for adapting world-action models to deployment-time distribution shifts using unlabeled robot interactions. The authors argue that existing future-prediction-based test-time training methods typically supervise predicted visual observations and proprioceptive states independently, without explicitly exploiting their spatial relationships. To address this limitation, SG-TTT introduces two components. First, a factual scene anchor is constructed using realized multi-view observations and end-effector poses, together with calibrated camera geometry. Second, component-wise scene consistency evaluates the spatial inconsistency introduced by each predicted future-state component while keeping the remaining components fixed to their observed values. The resulting scene-consistency objectives are1. A well-motivated approach to exploiting geometric relationships for test-time supervision. A major strength of this work is its formulation of test-time supervision through relationships among multiple observed signals rather than independent prediction targets. The use of multi-view images and end-effector poses to construct a shared factual scene anchor is intuitive and well aligned with the physical structure of robotic manipulation. The proposed component-wise routing mechanism is also interesting, as it allows individual future-state predictions to be evaluated in the context of other realized scene components. This provides a potentially useful framework for leveraging heterogeneous sensor observations during deployment without requiring additional task demonstrations or reward labels. 2. Meaningful validation under distribution shifts, including real-world experiments. The paper evaluates SG-TTT across seven distribution shifts in LIBERO-Plus, providing a relatively broad assessment of robustness to visual and robot-state variations. The experiments also include comparisons against different future-prediction TTT objectives and ablations of the shared scene reference and component-wise routing mechanisms. Importantly, the authors evaluate the method on a physical robot across three manipulation tasks under nominal conditions and two types of deployment shifts. The real-world results show meaningful improvements over the frozen Cosmos Policy, particularly under camera and robot-initialization variations. These experiments strengthen the practical relevance of the proposed approach and demonstrate its potential usefulness beyond simulation. used to adapt a pretrained Cosmos Policy through LoRA updates. The method is evaluated on LIBERO-10, LIBERO-Plus under seven distribution shifts, and three real-world manipulation tasks. SG-TTT improves the average LIBERO-Plus success rate from 80.7% to 82.5% compared with the frozen policy and achieves improvements in real-world manipulation under camera and robot-initialization shifts.
Critical strengths
- A well-motivated approach to exploiting geometric relationships for test-time supervision. A major strength of this work is its formulation of test-time supervision through relationships among multiple observed signals rather than independent prediction targets. The use of multi-view images and end-effector poses to construct a shared factual scene anchor is intuitive and well aligned with the physical structure of robotic manipulation.
- Meaningful validation under distribution shifts, including real-world experiments. The paper evaluates SG-TTT across seven distribution shifts in LIBERO-Plus, providing a relatively broad assessment of robustness to visual and robot-state variations. The experiments also include comparisons against different future-prediction TTT objectives and ablations of the shared scene reference and component-wise routing mechanisms. Importantly, the authors evaluate the method on a physical robot across three manipulation tasks under nominal conditions and two types of deployment shifts. The real-world results show meaningful improvements over the frozen Cosmos Policy, particularly under camera and robot-initialization variations.
Critical weaknesses
- Limited improvements over existing TTT approaches and insufficient real-world comparisons. Although SG-TTT consistently improves average performance over the compared methods, the improvements over existing test-time adaptation approaches are relatively modest in simulation. On LIBERO-Plus, SG-TTT achieves an average success rate of 82.5%, compared with 81.1% for Direct-future TTT and 81.4% for the strongest single-signal baseline. Similarly, the ablation study shows that introducing a shared factual scene reference improves average success from 65.6% to 66.4% on three selected shifts, while component-wise routing further increases it to 67.3%. These improvements provide evidence supporting the proposed design, but their practical significance remains unclear given the additional requirements for a pretrained geometry encoder, camera calibration, and multi-view geometric consistency computation. Furthermore, although the real-world results demonstrate meaningful improvements over the frozen Cosmos Policy, the paper does not compare SG-TTT against the existing future-prediction TTT baselines on physical robots. Consequently, it is difficult to determine whether the real-world improvements specifically arise from the proposed relational supervision or from test-time adaptation more generally. Additional real-world comparisons against matched TTT baselines, together with uncertainty estimates and computational-cost measurements, would substantially strengthen the evidence for the proposed method. Also, reporting standard error for each experiment would help identify the significance of the proposed method.
- Potential inconsistency between collected future observations and the updated policy during test-time adaptation. A methodological concern arises from the use of a fixed adaptation dataset collected using the frozen initial policy. During data collection, the initial policy generates an action sequence, which is executed to obtain realized future observations and end-effector states. These observations subsequently serve as supervision targets throughout adaptation. However, during optimization, the WAM regenerates its future predictions using updated LoRA parameters. Since the WAM jointly predicts actions and future states, adapting its parameters may also change the predicted action sequences. This creates a potential inconsistency: the realized future observations correspond to actions executed by the original policy, whereas the updated model may predict different actions whose actual future outcomes could differ. The paper does not explicitly analyze this issue or quantify how action predictions change during test-time adaptation. Although the approach empirically improves task performance, it is unclear whether this supervision remains consistent as the policy deviates from the behavior that generated the adaptation data.
Questions for authors
What is the computational cost of test-time adaptation? SG-TTT introduces a pretrained geometry encoder and computes multiple scene-consistency objectives during adaptation. Could the authors report the wall-clock time and GPU memory required for collecting the adaptation data and performing the 64 optimization updates? How do these costs compare with the direct future-prediction TTT baselines?
9 comments
9Abe 2/9 · W1 (A1)
Across the nine matched task–condition pairs, SG-TTT improves task Score in 9/9 and SR in 5/9, with SR tied in the other four. Score measures completion of ordered task stages, capturing execution progress even when binary success is unchanged. The three-task mean SR follows Frozen → Direct-future → SG-TTT: 40.0% → 50.0% → 56.7% under Robot Init and 30.0% → 43.3% → 46.7% under Camera, corresponding to +6.7% and +3.3% beyond Direct-future (differences calculated from task-level results before rounding). Under Nominal, SG-TTT reaches 73.3% vs. 63.3% for Direct-future. The mean task-score gains are +9.2%/+8.3%/+7.5% under Nominal/Robot Init/Camera. In particular, Cube → Bowl yields +10.0%/+20.0%/+10.0% SR across these conditions, while Pen → Box and Cube → Top Drawer gain in task Score under both shifts even when SR ties. These matched hardware comparisons provide direct evidence of gains beyond Direct-future TTT: higher task-stage Scores in all nine settings and higher full-task SR in five.
Uncertainty estimates. Table R-H1 reports each task's SR with its approximate binomial standard error (SE) from 10 evaluation rollouts per task–condition–method cell, computed as for observed success fraction . Three-task averages are reported separately as descriptive cross-task summaries. These task-level estimates complement the matched hardware comparisons by making the uncertainty of the observed success rates explicit.
Three-seed robustness and the spatial-relational insight
The submitted LIBERO-Plus protocol fixes benchmark evaluation seed 195 and reports three independent adaptation repetitions. We additionally evaluated benchmark seeds 196 and 197. Table R-S1 summarizes results across evaluation seeds 195–197 as mean ± SD [SE]; here the SD and SE reflect variation across the three seed-level results, distinct from the submitted within-seed adaptation repetitions. SE = SD/√3, calculated from the displayed SD. For Avg. (7), we first compute the equal-weight seven-shift average within each evaluation seed, then summarize those three averages. SG-TTT achieves 82.79 ± 0.35% SD [0.20 SE], versus 81.20 ± 0.07% for Direct-future, 81.49 ± 0.28% for Visual-only, and 80.84 ± 0.29% for Frozen Cosmos: +1.59%/+1.30%/+1.95%, respectively.
Table R-S1. Seven-shift LIBERO-Plus comparison across seeds 195–197 (%, mean ± SD [SE]). SE = SD/√3; Avg. (7) is formed within each seed before summarizing across seeds.
| Method | Layout | Camera | Robot Init | Light | Background | Noise | Language | Avg. (7) |
|---|---|---|---|---|---|---|---|---|
| Cosmos Policy | 92.74 ± 0.74 [0.43] | 59.51 ± 0.77 [0.44] | 62.68 ± 0.64 [0.37] | 95.99 ± 0.36 [0.21] | 72.66 ± 1.83 [1.06] | 85.97 ± 0.67 [0.39] | 96.34 ± 0.94 [0.54] | 80.84 ± 0.29 [0.17] |
| Visual-only TTT | 93.70 ± 0.74 [0.43] | 61.73 ± 0.90 [0.52] | 64.72 ± 0.39 [0.23] | 97.45 ± 0.36 [0.21] | 70.47 ± 1.40 [0.81] | 87.16 ± 0.68 [0.39] | 95.21 ± 0.66 [0.38] | 81.49 ± 0.28 [0.16] |
| Proprio-only TTT | 93.38 ± 1.03 [0.59] | 61.81 ± 0.41 [0.24] | 64.55 ± 0.39 [0.23] | 93.07 ± 0.36 [0.21] | 72.43 ± 0.87 [0.50] | 85.82 ± 0.64 [0.37] | 96.17 ± 0.54 [0.31] | 81.03 ± 0.26 [0.15] |
| Direct-future TTT | 91.99 ± 0.85 [0.49] | 60.46 ± 0.60 [0.35] | 64.38 ± 0.25 [0.14] | 97.08 ± 0.00 [0.00] | 71.97 ± 1.25 [0.72] | 86.93 ± 0.26 [0.15] | 95.56 ± 0.52 [0.30] | 81.20 ± 0.07 [0.04] |
| SG-TTT | 94.34 ± 0.81 [0.47] | 62.21 ± 0.90 [0.52] | 66.75 ± 0.39 [0.23] | 97.08 ± 0.36 [0.21] | 74.97 ± 1.22 [0.70] | 87.75 ± 0.45 [0.26] | 96.43 ± 0.40 [0.23] | 82.79 ± 0.35 [0.20] |
Why does relational supervision help? As detailed in the submitted method (Sec. 3 and Appendix B), Direct-future TTT independently matches predicted future images and proprioception to their realized targets. SG-TTT instead uses camera-conditioned joint DA3 features to represent both views and EEF pose in a common factual scene anchor: calibrated intrinsics/extrinsics relate the views, while EEF pose determines wrist-camera extrinsics. Each routed loss inserts one predicted future component into an otherwise factual scene and measures the resulting inconsistency with the other realized signals. For an image branch, the other factual view and EEF geometry constrain the predicted scene; for the pose branch, the predicted EEF pose changes wrist-camera geometry while both images remain factual. These component-specific gradients update shared WAM LoRA parameters, which also participate in action generation. SG-TTT thus extracts cross-view and EEF-geometric constraints from the same deployment evidence, providing spatial relational supervision beyond independent future-target matching.
9Abe 3/9 · W1 (A1)
Table R-S2. Spatial characteristics of LIBERO-Plus shifts and SG-TTT gains over Direct-future TTT (seeds 195–197; gains as absolute success-rate differences (%)). The full per-method mean ± SD (%) appears in Table R-S1.
| Perturbation | Spatial relevance | SG − Direct (%) |
|---|---|---|
| Layout | Object placement | +2.35 |
| Camera | View projection | +1.75 |
| Robot Init | EEF–scene configuration and wrist-camera viewpoint | +2.37 |
| Background | Appearance shift; geometry retained | +3.00 |
| Light | Illumination shift | 0.00 |
| Noise | Degraded visual evidence | +0.82 |
| Language | Instruction semantics | +0.87 |
| Avg. (7) | — | +1.59 |
The shift-specific results are consistent with this mechanism. Layout, Robot Init, and Camera change object placement, the EEF–scene configuration (including the wrist-camera viewpoint), or the projection of the scene into images; SG-TTT gains +2.35%/+2.37%/+1.75% over Direct-future, consistent with supervision against the realized spatial configuration. Background alters appearance while retaining useful scene geometry, enabling the multi-view/EEF factual reference to constrain predictions even with texture variation: SG-TTT achieves +3.00% over Direct-future and +4.50% over Visual-only. In contrast, Light already has 97.08% Direct-future SR and the two methods tie; Noise reduces the reliability of visual evidence (+0.82% gain), while Language primarily changes instruction semantics rather than scene geometry (+0.87%). The transferable insight is that relational supervision adds value when cross-view and EEF geometry supply reliable constraints beyond separate future targets.
What contributes to the gain?
Building on the submitted 65.6% → 66.4% → 67.3% staged progression from independent-component supervision to a shared factual reference and then component-wise routing, we added full-SG-TTT-centered design-element ablations on seeds 195–197 (Table R-M1) to examine the contributions of scene representation and component supervision. Across Camera, Robot Init, and Background, full SG-TTT reaches 67.98 ± 0.61% (SD). Removing cross-view joint encoding, camera conditioning, or the EEF-pose loss branch yields 66.17 ± 0.63%, 65.20 ± 0.20%, and 66.49 ± 0.69%, respectively (−1.81%/−2.78%/−1.49%); removing the agent-view or wrist-view branch yields 66.70 ± 0.31% and 66.78 ± 0.65%.
The representation controls clarify the complementary roles. − Cross-view joint encoding keeps both images, their camera parameters, and EEF-conditioned wrist geometry but encodes the two views independently through the same frozen DA3, disabling cross-view token exchange. − Camera conditioning keeps joint two-view encoding and attention, using DA3's image-only pathway without explicit camera inputs. Because predicted EEF pose enters scene features through wrist-camera extrinsics, this also disables the EEF-pose loss branch, while retaining both visual loss branches and the policy's proprioceptive input. In contrast, removing only the EEF-pose loss retains EEF-conditioned wrist geometry. These measured ablations show the contributions of cross-view interaction, the coupled camera/EEF geometry pathway, and EEF-pose supervision within the evaluated architecture.
9Abe 1/9 · W1 (A1)
Units: SR/Score in %; gains and Δ are absolute percentage-point differences (pp), not relative; gradient-norm changes are relative %.
W1: Added value of spatial relational supervision
Reviewer’s original comment
A1 (to W1)
Thank you for highlighting the importance of practical gains, matched TTT baselines on physical robots, and uncertainty and cost measurements. Our new matched hardware results show additional benefits from spatial relational supervision beyond Direct-future TTT, while the added geometry computation remains confined to offline adaptation rather than online action inference. We support this conclusion with four complementary additions: (i) Direct-future TTT comparisons on all three physical tasks; (ii) three-seed LIBERO-Plus results with SD and SE; (iii) full-method-centered representation ablations, fixed-weight routing controls, and branch-level gradient checks; and (iv) measured adaptation costs and inference latency.
Added value over Direct-future TTT on hardware
We added Direct-future TTT on all three physical tasks—Cube → Bowl, Pen → Box, and Cube → Top Drawer—under Nominal, Robot Init, and Camera. Within each task–condition pair, both TTT methods reuse the same cached 16 unlabeled samples, task-specific Cosmos Policy checkpoint, LoRA/optimizer configuration, and 16 updates; formal evaluation episodes are disjoint from adaptation data. This matched protocol directly compares the two supervision approaches under the same adaptation budget. Independent TTT in Table R-H1 denotes Direct-future TTT. We report both task Score and binary success rate (SR).
Table R-H1. Physical-robot Score/SR across three tasks and scenarios (%). Each task–condition–method cell has 10 evaluation rollouts; task SR is followed by approximate binomial SE in brackets (%); Avg. is the descriptive three-task mean. Independent TTT = Direct-future TTT.