acceptodds
Under review as a conference paper at ICLR 2027

SG-TTT: Scene-Grounded Supervision for Test-Time Training of World Action Models

Abstract

World action models (WAMs), which couple action generation with predictive world modeling, can degrade substantially under distribution shifts between training and deployment. Test-time training (TTT) offers a way to adapt these models using observations collected during deployment. For WAM adaptation under deployment shifts, future-prediction TTT typically matches each predicted future signal with its realized counterpart, without explicitly exploiting relationships among signals that describe the same realized future state, potentially limiting the informativeness of supervision. We formulate supervision for future-prediction TTT as a two-step process: organizing observed signals and their relationships into a structured supervisory reference, then translating this reference into adaptation objectives. We propose Scene-Grounded Test-Time Training (SG-TTT), a TTT method for WAMs with supervision grounded in scene geometry. SG-TTT first constructs a factual scene anchor from multi-view visual observations and the corresponding end-effector pose obtained after action execution, together with their spatial relationships. It then applies component-wise scene consistency to measure the scene-level spatial inconsistency introduced by each predicted future-state component and derives component-specific adaptation signals from the shared anchor. Extensive experiments on long-horizon LIBERO-10 suite under both standard and LIBERO-Plus distribution-shift settings, and real-world manipulation tasks show that SG-TTT achieves stronger overall performance than the frozen base WAM and representative future-prediction TTT baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Test-time Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9AbeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Scene-Grounded Test-Time Training (SG-TTT), a method for adapting world-action models to deployment-time distribution shifts using unlabeled robot interactions. The authors argue that existing future-prediction-based test-time training methods typically supervise predicted visual observations and proprioceptive states independently, without explicitly exploiting their spatial relationships. To address this limitation, SG-TTT introduces two components. First, a factual scene anchor is constructed using realized multi-view observations and end-effector poses, together with calibrated camera geometry. Second, component-wise scene consistency evaluates the spatial inconsistency introduced by each predicted future-state component while keeping the remaining components fixed to their observed values. The resulting scene-consistency objectives are1. A well-motivated approach to exploiting geometric relationships for test-time supervision. A major strength of this work is its formulation of test-time supervision through relationships among multiple observed signals rather than independent prediction targets. The use of multi-view images and end-effector poses to construct a shared factual scene anchor is intuitive and well aligned with the physical structure of robotic manipulation. The proposed component-wise routing mechanism is also interesting, as it allows individual future-state predictions to be evaluated in the context of other realized scene components. This provides a potentially useful framework for leveraging heterogeneous sensor observations during deployment without requiring additional task demonstrations or reward labels. 2. Meaningful validation under distribution shifts, including real-world experiments. The paper evaluates SG-TTT across seven distribution shifts in LIBERO-Plus, providing a relatively broad assessment of robustness to visual and robot-state variations. The experiments also include comparisons against different future-prediction TTT objectives and ablations of the shared scene reference and component-wise routing mechanisms. Importantly, the authors evaluate the method on a physical robot across three manipulation tasks under nominal conditions and two types of deployment shifts. The real-world results show meaningful improvements over the frozen Cosmos Policy, particularly under camera and robot-initialization variations. These experiments strengthen the practical relevance of the proposed approach and demonstrate its potential usefulness beyond simulation. used to adapt a pretrained Cosmos Policy through LoRA updates. The method is evaluated on LIBERO-10, LIBERO-Plus under seven distribution shifts, and three real-world manipulation tasks. SG-TTT improves the average LIBERO-Plus success rate from 80.7% to 82.5% compared with the frozen policy and achieves improvements in real-world manipulation under camera and robot-initialization shifts.

Critical strengths
  1. A well-motivated approach to exploiting geometric relationships for test-time supervision. A major strength of this work is its formulation of test-time supervision through relationships among multiple observed signals rather than independent prediction targets. The use of multi-view images and end-effector poses to construct a shared factual scene anchor is intuitive and well aligned with the physical structure of robotic manipulation.
  2. Meaningful validation under distribution shifts, including real-world experiments. The paper evaluates SG-TTT across seven distribution shifts in LIBERO-Plus, providing a relatively broad assessment of robustness to visual and robot-state variations. The experiments also include comparisons against different future-prediction TTT objectives and ablations of the shared scene reference and component-wise routing mechanisms. Importantly, the authors evaluate the method on a physical robot across three manipulation tasks under nominal conditions and two types of deployment shifts. The real-world results show meaningful improvements over the frozen Cosmos Policy, particularly under camera and robot-initialization variations.
Critical weaknesses
  1. Limited improvements over existing TTT approaches and insufficient real-world comparisons. Although SG-TTT consistently improves average performance over the compared methods, the improvements over existing test-time adaptation approaches are relatively modest in simulation. On LIBERO-Plus, SG-TTT achieves an average success rate of 82.5%, compared with 81.1% for Direct-future TTT and 81.4% for the strongest single-signal baseline. Similarly, the ablation study shows that introducing a shared factual scene reference improves average success from 65.6% to 66.4% on three selected shifts, while component-wise routing further increases it to 67.3%. These improvements provide evidence supporting the proposed design, but their practical significance remains unclear given the additional requirements for a pretrained geometry encoder, camera calibration, and multi-view geometric consistency computation. Furthermore, although the real-world results demonstrate meaningful improvements over the frozen Cosmos Policy, the paper does not compare SG-TTT against the existing future-prediction TTT baselines on physical robots. Consequently, it is difficult to determine whether the real-world improvements specifically arise from the proposed relational supervision or from test-time adaptation more generally. Additional real-world comparisons against matched TTT baselines, together with uncertainty estimates and computational-cost measurements, would substantially strengthen the evidence for the proposed method. Also, reporting standard error for each experiment would help identify the significance of the proposed method.
  2. Potential inconsistency between collected future observations and the updated policy during test-time adaptation. A methodological concern arises from the use of a fixed adaptation dataset collected using the frozen initial policy. During data collection, the initial policy generates an action sequence, which is executed to obtain realized future observations and end-effector states. These observations subsequently serve as supervision targets throughout adaptation. However, during optimization, the WAM regenerates its future predictions using updated LoRA parameters. Since the WAM jointly predicts actions and future states, adapting its parameters may also change the predicted action sequences. This creates a potential inconsistency: the realized future observations correspond to actions executed by the original policy, whereas the updated model may predict different actions whose actual future outcomes could differ. The paper does not explicitly analyze this issue or quantify how action predictions change during test-time adaptation. Although the approach empirically improves task performance, it is unclear whether this supervision remains consistent as the policy deviates from the behavior that generated the adaptation data.
Questions for authors

What is the computational cost of test-time adaptation? SG-TTT introduces a pretrained geometry encoder and computes multiple scene-consistency objectives during adaptation. Could the authors report the wall-clock time and GPU memory required for collecting the adaptation data and performing the 64 optimization updates? How do these costs compare with the direct future-prediction TTT baselines?

9 comments

9Abe 1/9 · W1 (A1)

Units: SR/Score in %; gains and Δ are absolute percentage-point differences (pp), not relative; gradient-norm changes are relative %.

W1: Added value of spatial relational supervision

Reviewer’s original comment

  1. Limited improvements over existing TTT approaches and insufficient real-world comparisons. Although SG-TTT consistently improves average performance over the compared methods, the improvements over existing test-time adaptation approaches are relatively modest in simulation. On LIBERO-Plus, SG-TTT achieves an average success rate of 82.5%, compared with 81.1% for Direct-future TTT and 81.4% for the strongest single-signal baseline. Similarly, the ablation study shows that introducing a shared factual scene reference improves average success from 65.6% to 66.4% on three selected shifts, while component-wise routing further increases it to 67.3%. These improvements provide evidence supporting the proposed design, but their practical significance remains unclear given the additional requirements for a pretrained geometry encoder, camera calibration, and multi-view geometric consistency computation. Furthermore, although the real-world results demonstrate meaningful improvements over the frozen Cosmos Policy, the paper does not compare SG-TTT against the existing future-prediction TTT baselines on physical robots. Consequently, it is difficult to determine whether the real-world improvements specifically arise from the proposed relational supervision or from test-time adaptation more generally. Additional real-world comparisons against matched TTT baselines, together with uncertainty estimates and computational-cost measurements, would substantially strengthen the evidence for the proposed method. Also, reporting standard error for each experiment would help identify the significance of the proposed method.

A1 (to W1)

Thank you for highlighting the importance of practical gains, matched TTT baselines on physical robots, and uncertainty and cost measurements. Our new matched hardware results show additional benefits from spatial relational supervision beyond Direct-future TTT, while the added geometry computation remains confined to offline adaptation rather than online action inference. We support this conclusion with four complementary additions: (i) Direct-future TTT comparisons on all three physical tasks; (ii) three-seed LIBERO-Plus results with SD and SE; (iii) full-method-centered representation ablations, fixed-weight routing controls, and branch-level gradient checks; and (iv) measured adaptation costs and inference latency.

Added value over Direct-future TTT on hardware

We added Direct-future TTT on all three physical tasks—Cube → Bowl, Pen → Box, and Cube → Top Drawer—under Nominal, Robot Init, and Camera. Within each task–condition pair, both TTT methods reuse the same cached 16 unlabeled samples, task-specific Cosmos Policy checkpoint, LoRA/optimizer configuration, and 16 updates; formal evaluation episodes are disjoint from adaptation data. This matched protocol directly compares the two supervision approaches under the same adaptation budget. Independent TTT in Table R-H1 denotes Direct-future TTT. We report both task Score and binary success rate (SR).

Table R-H1. Physical-robot Score/SR across three tasks and scenarios (%). Each task–condition–method cell has 10 evaluation rollouts; task SR is followed by approximate binomial SE in brackets (%); Avg. is the descriptive three-task mean. Independent TTT = Direct-future TTT.

MethodScenarioCube → Bowl ScoreCube → Bowl SR [SE]Pen → Box ScorePen → Box SR [SE]Cube → Top Drawer ScoreCube → Top Drawer SR [SE]Avg. ScoreAvg. SR
Cosmos Policy BaseNominal60.0%60.0 [15.5]50.0%50.0 [15.8]85.0%80.0 [12.6]65.0%63.3%
Cosmos Policy BaseRobot Init45.0%40.0 [15.5]35.0%30.0 [14.5]60.0%50.0 [15.8]46.7%40.0%
Cosmos Policy BaseCamera35.0%30.0 [14.5]30.0%20.0 [12.6]47.5%40.0 [15.5]37.5%30.0%
Cosmos Policy + Independent TTTNominal65.0%60.0 [15.5]50.0%50.0 [15.8]82.5%80.0 [12.6]65.8%63.3%
Cosmos Policy + Independent TTTRobot Init55.0%50.0 [15.8]40.0%40.0 [15.5]62.5%60.0 [15.5]52.5%50.0%
Cosmos Policy + Independent TTTCamera40.0%40.0 [15.5]40.0%40.0 [15.5]55.0%50.0 [15.8]45.0%43.3%
Cosmos + SG-TTTNominal70.0%70.0 [14.5]65.0%60.0 [15.5]90.0%90.0 [9.5]75.0%73.3%
Cosmos + SG-TTTRobot Init70.0%70.0 [14.5]45.0%40.0 [15.5]67.5%60.0 [15.5]60.8%56.7%
Cosmos + SG-TTTCamera55.0%50.0 [15.8]45.0%40.0 [15.5]57.5%50.0 [15.8]52.5%46.7%

9Abe 2/9 · W1 (A1)

Across the nine matched task–condition pairs, SG-TTT improves task Score in 9/9 and SR in 5/9, with SR tied in the other four. Score measures completion of ordered task stages, capturing execution progress even when binary success is unchanged. The three-task mean SR follows Frozen → Direct-future → SG-TTT: 40.0% → 50.0% → 56.7% under Robot Init and 30.0% → 43.3% → 46.7% under Camera, corresponding to +6.7% and +3.3% beyond Direct-future (differences calculated from task-level results before rounding). Under Nominal, SG-TTT reaches 73.3% vs. 63.3% for Direct-future. The mean task-score gains are +9.2%/+8.3%/+7.5% under Nominal/Robot Init/Camera. In particular, Cube → Bowl yields +10.0%/+20.0%/+10.0% SR across these conditions, while Pen → Box and Cube → Top Drawer gain in task Score under both shifts even when SR ties. These matched hardware comparisons provide direct evidence of gains beyond Direct-future TTT: higher task-stage Scores in all nine settings and higher full-task SR in five.

Uncertainty estimates. Table R-H1 reports each task's SR with its approximate binomial standard error (SE) from 10 evaluation rollouts per task–condition–method cell, computed as SE(%)=100p(1−p)/10SE(\%)=100\sqrt{p(1-p)/10} for observed success fraction pp. Three-task averages are reported separately as descriptive cross-task summaries. These task-level estimates complement the matched hardware comparisons by making the uncertainty of the observed success rates explicit.

Three-seed robustness and the spatial-relational insight

The submitted LIBERO-Plus protocol fixes benchmark evaluation seed 195 and reports three independent adaptation repetitions. We additionally evaluated benchmark seeds 196 and 197. Table R-S1 summarizes results across evaluation seeds 195–197 as mean ± SD [SE]; here the SD and SE reflect variation across the three seed-level results, distinct from the submitted within-seed adaptation repetitions. SE = SD/√3, calculated from the displayed SD. For Avg. (7), we first compute the equal-weight seven-shift average within each evaluation seed, then summarize those three averages. SG-TTT achieves 82.79 ± 0.35% SD [0.20 SE], versus 81.20 ± 0.07% for Direct-future, 81.49 ± 0.28% for Visual-only, and 80.84 ± 0.29% for Frozen Cosmos: +1.59%/+1.30%/+1.95%, respectively.

Table R-S1. Seven-shift LIBERO-Plus comparison across seeds 195–197 (%, mean ± SD [SE]). SE = SD/√3; Avg. (7) is formed within each seed before summarizing across seeds.

MethodLayoutCameraRobot InitLightBackgroundNoiseLanguageAvg. (7)
Cosmos Policy92.74 ± 0.74 [0.43]59.51 ± 0.77 [0.44]62.68 ± 0.64 [0.37]95.99 ± 0.36 [0.21]72.66 ± 1.83 [1.06]85.97 ± 0.67 [0.39]96.34 ± 0.94 [0.54]80.84 ± 0.29 [0.17]
Visual-only TTT93.70 ± 0.74 [0.43]61.73 ± 0.90 [0.52]64.72 ± 0.39 [0.23]97.45 ± 0.36 [0.21]70.47 ± 1.40 [0.81]87.16 ± 0.68 [0.39]95.21 ± 0.66 [0.38]81.49 ± 0.28 [0.16]
Proprio-only TTT93.38 ± 1.03 [0.59]61.81 ± 0.41 [0.24]64.55 ± 0.39 [0.23]93.07 ± 0.36 [0.21]72.43 ± 0.87 [0.50]85.82 ± 0.64 [0.37]96.17 ± 0.54 [0.31]81.03 ± 0.26 [0.15]
Direct-future TTT91.99 ± 0.85 [0.49]60.46 ± 0.60 [0.35]64.38 ± 0.25 [0.14]97.08 ± 0.00 [0.00]71.97 ± 1.25 [0.72]86.93 ± 0.26 [0.15]95.56 ± 0.52 [0.30]81.20 ± 0.07 [0.04]
SG-TTT94.34 ± 0.81 [0.47]62.21 ± 0.90 [0.52]66.75 ± 0.39 [0.23]97.08 ± 0.36 [0.21]74.97 ± 1.22 [0.70]87.75 ± 0.45 [0.26]96.43 ± 0.40 [0.23]82.79 ± 0.35 [0.20]

Why does relational supervision help? As detailed in the submitted method (Sec. 3 and Appendix B), Direct-future TTT independently matches predicted future images and proprioception to their realized targets. SG-TTT instead uses camera-conditioned joint DA3 features to represent both views and EEF pose in a common factual scene anchor: calibrated intrinsics/extrinsics relate the views, while EEF pose determines wrist-camera extrinsics. Each routed loss inserts one predicted future component into an otherwise factual scene and measures the resulting inconsistency with the other realized signals. For an image branch, the other factual view and EEF geometry constrain the predicted scene; for the pose branch, the predicted EEF pose changes wrist-camera geometry while both images remain factual. These component-specific gradients update shared WAM LoRA parameters, which also participate in action generation. SG-TTT thus extracts cross-view and EEF-geometric constraints from the same deployment evidence, providing spatial relational supervision beyond independent future-target matching.

9Abe 3/9 · W1 (A1)

Table R-S2. Spatial characteristics of LIBERO-Plus shifts and SG-TTT gains over Direct-future TTT (seeds 195–197; gains as absolute success-rate differences (%)). The full per-method mean ± SD (%) appears in Table R-S1.

PerturbationSpatial relevanceSG − Direct (%)
LayoutObject placement+2.35
CameraView projection+1.75
Robot InitEEF–scene configuration and wrist-camera viewpoint+2.37
BackgroundAppearance shift; geometry retained+3.00
LightIllumination shift0.00
NoiseDegraded visual evidence+0.82
LanguageInstruction semantics+0.87
Avg. (7)—+1.59

The shift-specific results are consistent with this mechanism. Layout, Robot Init, and Camera change object placement, the EEF–scene configuration (including the wrist-camera viewpoint), or the projection of the scene into images; SG-TTT gains +2.35%/+2.37%/+1.75% over Direct-future, consistent with supervision against the realized spatial configuration. Background alters appearance while retaining useful scene geometry, enabling the multi-view/EEF factual reference to constrain predictions even with texture variation: SG-TTT achieves +3.00% over Direct-future and +4.50% over Visual-only. In contrast, Light already has 97.08% Direct-future SR and the two methods tie; Noise reduces the reliability of visual evidence (+0.82% gain), while Language primarily changes instruction semantics rather than scene geometry (+0.87%). The transferable insight is that relational supervision adds value when cross-view and EEF geometry supply reliable constraints beyond separate future targets.

What contributes to the gain?

Building on the submitted 65.6% → 66.4% → 67.3% staged progression from independent-component supervision to a shared factual reference and then component-wise routing, we added full-SG-TTT-centered design-element ablations on seeds 195–197 (Table R-M1) to examine the contributions of scene representation and component supervision. Across Camera, Robot Init, and Background, full SG-TTT reaches 67.98 ± 0.61% (SD). Removing cross-view joint encoding, camera conditioning, or the EEF-pose loss branch yields 66.17 ± 0.63%, 65.20 ± 0.20%, and 66.49 ± 0.69%, respectively (−1.81%/−2.78%/−1.49%); removing the agent-view or wrist-view branch yields 66.70 ± 0.31% and 66.78 ± 0.65%.

The representation controls clarify the complementary roles. − Cross-view joint encoding keeps both images, their camera parameters, and EEF-conditioned wrist geometry but encodes the two views independently through the same frozen DA3, disabling cross-view token exchange. − Camera conditioning keeps joint two-view encoding and attention, using DA3's image-only pathway without explicit camera inputs. Because predicted EEF pose enters scene features through wrist-camera extrinsics, this also disables the EEF-pose loss branch, while retaining both visual loss branches and the policy's proprioceptive input. In contrast, removing only the EEF-pose loss retains EEF-conditioned wrist geometry. These measured ablations show the contributions of cross-view interaction, the coupled camera/EEF geometry pathway, and EEF-pose supervision within the evaluated architecture.

Reviewer_e2pjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of adapting world action models (WAMs) to deployment-time distribution shifts via test-time training (TTT). The authors argue that existing future-prediction TTT methods perform direct per-signal matching without exploiting relational structure among co-occurring deployment signals. To this end, they formulate future-prediction TTT as a two-step process: first constructing a structured supervisory reference from observed signals and their relationships, then translating this reference into component-specific adaptation objectives.

Critical strengths
  • Distribution shift during deployment is a critical bottleneck for deploying WAM-based robot policies. The proposed TTT approach requires neither task labels nor reward signals, making it suitable for real-world unlabeled deployment data.
  • The motivation of this paper is clear and the authors chose a straightforward but effective method to achieve the goal.
Critical weaknesses
  • SG-TTT relies on pre-calibrated camera intrinsics and extrinsics to build the geometry-aware scene anchor. In real deployment, however, camera calibration often drifts over time, or accurate calibration parameters may not be available. The paper does not evaluate or discuss how calibration errors would affect adaptation performance.
  • Test-time adaptation operates during deployment, so runtime and computational overhead are key practical metrics. The component-wise routing requires multiple forward passes through the geometry encoder per sample, which likely increases adaptation latency compared to single-branch direct-matching baselines. The paper provides no runtime or compute budget analysis, making it hard to assess the deployability of SG-TTT in latency-sensitive robot systems.
  • The paper states that scene consistency gradients propagate through future-state predictions to the adaptable parameters and thus improve action generation. However, the main text provides little intuition or analysis of why reducing scene-level spatial inconsistency in predicted futures leads to better action execution.
  • The real-world experiments include only 3 manipulation tasks on a single robot platform. While the results are directionally consistent with simulation, the small task set and limited hardware diversity provide weak evidence for real-world generalizability. A broader set of tasks or cross-platform validation would better support the claimed practical applicability.
  • SG-TTT shows almost no gain under language variation, which the authors acknowledge but do not elaborate on. Given that language distribution shift is a common challenge in open-world language-conditioned robot learning, a deeper discussion — for example, how scene-grounded supervision could be combined with language-level adaptation, or preliminary exploratory results — would improve the paper’s completeness.
Questions for authors
  • The geometry encoder is kept frozen during TTT adaptation. What is the reasoning behind this design choice? Would fine-tuning the geometry encoder alongside the WAM improve adaptation, or would it cause overfitting given the small TTT dataset?
  • The current method requires both agent-view and wrist-view images. How would SG-TTT perform in single-camera setups? Can the framework be adapted to single-view settings, and if so, how much performance degradation would be expected?
  • All experiments use Cosmos Policy as the base WAM. How portable is SG-TTT to other WAM architectures that may have different prediction interfaces (e.g., latent-space prediction instead of explicit multi-view image prediction)? Would the component-wise scene consistency still be applicable?
12 comments

e2pj 1/12 · W1

Units: SR/Score in %; gains in pp. Cosmos LIBERO-Plus evaluation seeds: 195–197.

W1: Camera calibration robustness

Reviewer’s original comment

  • SG-TTT relies on pre-calibrated camera intrinsics and extrinsics to build the geometry-aware scene anchor. In real deployment, however, camera calibration often drifts over time, or accurate calibration parameters may not be available. The paper does not evaluate or discuss how calibration errors would affect adaptation performance.

A1 (to W1)

Thank you for emphasizing calibration reliability in real-world deployment. Our new controlled evaluation demonstrates that SG-TTT retains high task success under the tested inaccuracies in its initial camera extrinsics. On LIBERO-10 “Put both moka pots on the stove”, accurate calibration yields 92.67 ± 1.15% success (Table R-Cal1).

With a 1°–5° fixed rotational error in either camera alone, the largest reduction in mean success rate is only 2.67 percentage points (pp).

Even when both cameras have 5° errors, SG-TTT achieves 88.00 ± 6.00% success, 4.67 pp below accurate calibration. These results directly quantify the method's tolerance to the tested initial calibration errors, while showing the additional impact of a larger simultaneous two-view bias.

Why the initial calibration matters—and why it is needed only once per rollout. SG-TTT does not require camera extrinsics to be re-estimated for every predicted future frame.

Initial calibration supplies intrinsics and reference extrinsics for the factual scene anchor and all component-wise routed scenes (Sec. 3; Appendix B).

In our physical-robot setup, intrinsics are calibrated offline, the agent-camera extrinsics are estimated once at the beginning of each rollout and then kept fixed, and the wrist camera maintains a rigid transform to the EEF.

Specifically, the initial wrist calibration CwC_w and calibration-time EEF pose TcalET_{\mathrm{cal}}^E determine TECw=(TcalE)−1Cw−1T_{EC_w}=(T_{\mathrm{cal}}^E)^{-1}C_w^{-1}.

At any realized or predicted EEF pose TET^E, the wrist-camera geometry is then obtained from TCwW(TE)=(TETECw)−1T_{C_wW}(T^E)=(T^E T_{EC_w})^{-1}. Thus, subsequent wrist-view geometry is updated from EEF motion, not from repeated camera recalibration. The accuracy of the initial reference geometry is therefore the relevant calibration dependency we evaluate.

New experiment: isolating first-frame extrinsic inaccuracies. We focus on rotation biases in the initial extrinsics, which directly enter the camera-conditioned scene representation and can arise from imperfect camera mounting or hand–eye calibration.

Unlike the existing physical-robot Camera shift, which physically rotates the agent camera by 5∘5^\circ and updates its calibration, this experiment keeps actual camera poses, rendered images, intrinsics, and measured EEF poses unchanged, and perturbs only the first-frame extrinsic metadata.

We apply 1∘/3∘/5∘1^\circ/3^\circ/5^\circ rotations about each affected camera's local Y axis to CaC_a (agent view), CwC_w (wrist view), or both, plus an accurate 0∘0^\circ reference.

Each perturbed calibration remains fixed and is used consistently for both the factual anchor and every routed branch. For each of evaluation seeds 195–197, each configuration undergoes a separate adaptation from the same base checkpoint, with 64 fixed unlabeled samples, batch size 1, and 64 optimization updates, followed by 50 disjoint evaluation episodes with matched initial states. This gives 30 independent adaptations and 1,500 evaluation episodes; Table R-Cal1 reports mean ± sample SD across the three seeds.

Why relational supervision can remain useful with a small fixed bias. The factual anchor and all routed scenes are encoded by the same frozen DA3 under the same initial calibration parameters.

Consequently, if a substituted prediction matches its corresponding realized component, the routed scene and factual anchor have identical inputs and therefore identical scene features even when the shared calibration metadata contains a fixed bias.

In other words, a common initial calibration error does not, by itself, create a spurious discrepancy between two otherwise identical scenes.

The error can still change the geometry-conditioned features and the gradients for imperfect predictions; it is not generally canceled. This distinction explains why SG-TTT's relational comparisons can remain informative under modest initial inaccuracies while more substantial errors across both views can reduce their reliability.

Measured pattern and deployment implication. Across the tested single-camera 1∘1^\circ–5∘5^\circ settings, mean SR remains within 2.67 pp below the accurately calibrated reference, and some estimates are slightly above it; the response is not monotonically related to rotation magnitude.

e2pj 2/12 · W1/W2

When both cameras are perturbed, success is 90.00 ± 2.00% at 1∘1^\circ, 92.00 ± 5.29% at 3∘3^\circ, and 88.00 ± 6.00% at 5∘5^\circ.

The main finding is that moderate fixed errors in the initial calibration do not prevent high post-adaptation task success in this tested setting, while simultaneous larger errors have a more visible effect on mean performance and variability.

As the real-robot protocol estimates the main-camera extrinsics at rollout start and subsequently relies on fixed calibration plus EEF-based wrist geometry, these results directly inform the precision required at that initialization step.

They evaluate fixed initial rotational biases; changes in physical calibration during a rollout, translation errors, and intrinsic errors remain distinct conditions.

Table R-Cal1. First-frame extrinsic-rotation calibration sensitivity of SG-TTT on LIBERO-10 “Put both moka pots on the stove” (seeds 195–197; mean ± sample SD of success rate, %). Each configuration uses 64 unlabeled samples, BS=1, 64 updates, and 50 matched evaluation episodes per seed.

No.MethodPerturbationAgent-view errorWrist-view errorSR, mean ± SD (%)
1SG-TTTBaseline0°0°92.67 ± 1.15
2SG-TTTAgent-only1°0°90.00 ± 2.00
3SG-TTTAgent-only3°0°93.33 ± 3.06
4SG-TTTAgent-only5°0°92.67 ± 1.15
5SG-TTTWrist-only0°1°91.33 ± 3.06
6SG-TTTWrist-only0°3°91.33 ± 1.15
7SG-TTTWrist-only0°5°92.00 ± 2.00
8SG-TTTBoth1°1°90.00 ± 2.00
9SG-TTTBoth3°3°92.00 ± 5.29
10SG-TTTBoth5°5°88.00 ± 6.00

Takeaway

SG-TTT relies on an initial geometric reference, not on repeated frame-by-frame camera recalibration. The same reference is used for the factual anchor and every routed comparison, while wrist-camera geometry follows measured or predicted EEF motion. Our controlled results show that 1°–5° single-camera initial rotation errors reduce mean success by at most 2.67 pp, and SG-TTT still attains 88.00% mean success with simultaneous 5° biases on the tested task. The practical implication is that useful scene-grounded supervision can persist with modest inaccuracies in the initial calibration, while maintaining reliable geometry across both views remains valuable—particularly for larger errors. This finding concerns fixed initial bias, rather than time-varying calibration drift.

W2: Runtime and computational overhead

Reviewer’s original comment

  • Test-time adaptation operates during deployment, so runtime and computational overhead are key practical metrics. The component-wise routing requires multiple forward passes through the geometry encoder per sample, which likely increases adaptation latency compared to single-branch direct-matching baselines. The paper provides no runtime or compute budget analysis, making it hard to assess the deployability of SG-TTT in latency-sensitive robot systems.

A2 (to W2)

Thank you for requesting a measured analysis of the computational overhead introduced by component-wise routing and its implications for latency-sensitive robot deployment.

Our new profiling separates the one-time cost of adapting the policy from the latency of generating actions after adaptation. On the physical-robot Cube → Bowl / Camera task, matched 16-window/16-update training takes 34.93 ± 0.06 s for SG-TTT versus 12.16 ± 0.18 s for Direct-future TTT; in the profiled LIBERO-Plus 64-sample/64-update setting, total adaptation takes 394.19 s versus 265.78 s (Tables R-C1 and R-C3).

This additional offline computation produces a more strongly adapted policy while leaving the original WAM-only online inference path unchanged, as confirmed by measured action-query latencies below.

Where the additional computation occurs. SG-TTT uses the frozen DA3 geometry encoder as an adaptation-time supervisor. For each collected transition, the factual multi-view/EEF scene anchor is encoded once and cached.

Each optimizer update then evaluates three prediction-dependent routed scenes: one predicted agent image, wrist image, or EEF pose is inserted into the otherwise factual scene, requiring three DA3 forward evaluations through the corresponding routes.

DA3's weights remain frozen, but the prediction-dependent path remains differentiable so that the relational losses update WAM LoRA, not DA3.

This explains the higher optimization time and training memory relative to independently matching future targets. The camera geometry used by these scenes comes from the initial calibration reference described in W1; SG-TTT does not require repeated camera calibration or DA3 evaluation at every online action query.

e2pj 3/12 · W2

Measured physical-robot cost and execution benefit. On Cube → Bowl under Camera, SG-TTT and Direct-future TTT share the same cached unlabeled deployment windows, task-specific checkpoint, LoRA/optimizer setup, and update count.

At the default 16 windows / 16 updates (one epoch, batch size 1), collection takes 23.47 ± 0.96 s for both methods. Training takes 34.93 ± 0.06 s for SG-TTT and 12.16 ± 0.18 s for Direct-future: 22.77 s additional one-time training.

At this matched budget, Score/SR rises from 40.0%/40.0% to 55.0%/50.0% (+15 pp Score, +10 pp SR; Table R-C1). Thus, the extra routing cost has a directly measured task-execution benefit in the same physical setting.

Performance across measured adaptation budgets. We also report the complete 8/16/32/64-window, one-epoch, batch-size-1 sweep, with the corresponding 8/16/32/64 optimizer updates and separately measured collection/training times (Table R-C1).

Across these settings, SG-TTT attains 40%/50%/60%/60% SR, while Direct-future attains 40%/40%/40%/50%. For a practical cross-budget reference, SG-TTT at 32 windows/32 updates reaches 60% SR with 119.91 s of collection plus training, whereas Direct-future at 64 windows/64 updates reaches 50% SR with 148.93 s.

These are sums of the reported stage means at different sample/update budgets, not an equal-wall-clock control. They show that, at these measured operating points, more informative offline supervision can compensate for its higher per-update cost in the observed performance–time trade-off.

Simulation and GPU-memory profile. For the profiled LIBERO-Plus stove-and-moka-pot task with 64 samples and 64 updates, Direct-future versus memory-optimized SG-TTT requires 210.19 vs. 223.45 s for collection and 55.32 vs. 169.94 s for training; independently measured total adaptation times are 265.78 vs. 394.19 s (Table R-C3).

Peak training memory is 7.65 vs. 22.63 GiB. On the physical Cube → Top Drawer task, peak training memory is 8.48 GiB for Direct-future and 22.53 GiB for memory-optimized SG-TTT, which runs on the 24 GiB RTX 3090 used for adaptation (Table R-C2).

Separately, across LIBERO-Plus evaluation seeds 195–197 and seven shifts, SG-TTT achieves 82.79 ± 0.35% mean SR versus 81.20 ± 0.07% for Direct-future (Table R-W4b).

These measurements disclose the additional training resources and the independently observed adaptation gains; the seven-shift performance summary is distinct from the single-task cost profile.

Why this remains suitable for the evaluated latency-sensitive control setting. After offline adaptation, the learned LoRA update is merged into the WAM, W′=W+(α/r)BAW'=W+(\alpha/r)BA.

The robot executes only the adapted WAM: neither DA3 nor the factual anchor and routed losses are evaluated inside the action-generation loop.

Measured Base / Direct-future / SG-TTT latency is 434.15 ± 0.91 / 434.05 ± 0.89 / 434.09 ± 0.93 ms per 16-step action-chunk call on the physical robot (Table R-C1), and 635.01 ± 3.22 / 634.72 ± 0.91 / 634.47 ± 1.26 ms/query on LIBERO-Plus (Table R-C3).

Neither measurement shows additional per-query latency from SG-TTT. The adaptation window has a measurable upfront cost, but the richer geometric supervision is not a recurring computation in the online control loop; the reported stage timings do not separately include calibration preparation.

Table R-C1. Cube → Bowl physical-robot cost and Camera-condition Score/SR across one-epoch adaptation budgets (8/16/32/64 windows correspond to 8/16/32/64 optimizer updates; batch size 1; mean ± SD for runtime; % for Score/SR). Eight windows per trajectory by default. Independent TTT denotes Direct-future TTT.

ConfigurationWindowsCollection time (s)Training time (s)Inference latency (ms/call)Score/SR: Camera (%)
Base---434.15 ± 0.9135.0 / 30.0
Independent TTT812.92 ± 1.426.13 ± 0.18434.05 ± 0.8945.0 / 40.0
Independent TTT1623.47 ± 0.9612.16 ± 0.18434.05 ± 0.8940.0 / 40.0
Independent TTT3250.13 ± 1.5324.31 ± 0.05434.05 ± 0.8940.0 / 40.0
Independent TTT6499.86 ± 3.0549.07 ± 0.17434.05 ± 0.8950.0 / 50.0
SG-TTT812.92 ± 1.4217.43 ± 0.19434.09 ± 0.9345.0 / 40.0
SG-TTT1623.47 ± 0.9634.93 ± 0.06434.09 ± 0.9355.0 / 50.0
SG-TTT3250.13 ± 1.5369.78 ± 0.20434.09 ± 0.9360.0 / 60.0
SG-TTT6499.86 ± 3.05139.60 ± 0.28434.09 ± 0.9360.0 / 60.0
Reviewer_Nit7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SG-TTT for deployment-time adaptation of world action models (WAMs). It uses realized multi-view observations and end-effector (EEF) poses to construct a common factual scene reference. A frozen DA3 geometry encoder produces supervision by replacing one predicted future component at a time while keeping the others factual. The resulting scene-consistency objective updates LoRA parameters of Cosmos Policy while freezing the base WAM.

Simulation uses 64 unlabeled trajectory samples and 64 optimizer steps per adaptation instance; the real-robot protocol uses 16 samples and 16 updates per task-condition pair. After batch adaptation, the model is frozen and evaluated on disjoint episodes. Experiments cover nominal LIBERO-10, seven LIBERO-Plus perturbations, and three B601-DM manipulation tasks (main paper §§3–4; Appendix B–F).

Critical strengths
  1. The supervision objective is concrete and well motivated. SG-TTT organizes realized views, EEF pose, and calibrated camera geometry into a common reference, and uses one-component-at-a-time substitution to constrain each future prediction. Appendices B and F clearly specify the feature discrepancy, gradient paths, and independent/joint supervision controls, making this a technically understandable and reusable design. Adaptation requires neither expert actions nor rewards.

  2. The evaluation covers relevant settings with reasonably controlled protocols. Simulation includes nominal LIBERO-10 and seven LIBERO-Plus shifts; compared adapters share the checkpoint, adaptation data, and update budgets, with disjoint evaluation episodes. SG-TTT achieves the highest LIBERO-Plus average success rate (82.5%). On three real-robot tasks, average success under both camera and robot-initialization shifts improves by 16.7 percentage points over the frozen policy.

Critical weaknesses
  1. Mechanism attribution and general insight remain limited. Appendix F.1 clearly defines the staged controls, but the authors correctly acknowledge that the Independent-components-to-shared-reference transition changes several factors together: single- versus multi-view encoding, camera-geometry conditioning, and the pose discrepancy. It is not a single-factor ablation. Thus, the evidence supports the overall supervision design without fully separating its constituent contributions. Moreover, the routing comparison contrasts a single joint feature loss with three weighted branch losses; the roles of supervision structure versus loss weighting or gradient scale could be better characterized. The work offers a reasonable concrete recipe, but yields relatively limited transferable insight into when and why it should outperform simpler objectives.

  2. The added practical value over simple TTT remains insufficiently established on hardware. On LIBERO-Plus, SG-TTT scores 82.5%, versus 81.4% for the strongest single-objective baseline, Visual-only TTT—a 1.1-point improvement. Although the real-robot gains are larger, the matched comparison is only against the frozen task-specific Cosmos Policy, not a Visual-only or Direct-future TTT baseline using the same deployment data and budget. Consequently, the extra benefit attributable to scene-grounded supervision, rather than deployment-time adaptation generally, remains uncertain. The appendix now specifies sample counts, LoRA rank, and recalibration, but not measured adaptation time or additional compute cost. The demonstrated scope is mainly adaptation of task-trained policies to controlled deployment shifts.

Questions for authors

1.Appendix F.1 clarifies that the independent-to-shared-reference transition is a staged, multi-factor comparison. Could the authors further separate the roles of joint multi-view encoding, camera geometry/EEF inputs, and supervision organization? For the routing control, how do relative objective weights and gradient scales affect the comparison, and can a more closely matched control be provided?

  1. Could the authors add a matched Visual-only or Direct-future TTT baseline on one existing representative physical task and shifted condition, using the same 16 unlabeled samples, LoRA configuration, and optimization budget? Could the authors also substantiate sample and computational efficiency by reporting success rates across adaptation-data sizes and update counts, varying these factors separately, together with measured data-collection and adaptation times? Comparison with simple TTT under matched wall-clock budgets would clarify whether the gains justify the additional cost.

  2. What target-task training data were used for the simulated Cosmos Policy LIBERO checkpoint? Appendix E makes clear that the physical checkpoints were fine-tuned separately for their corresponding tasks. If a robot-capable checkpoint without supervision on the evaluated target tasks is available, a matched frozen/simple-TTT/SG-TTT comparison on those held-out tasks would clarify dependence on prior task training. This is an extension to probe scope, not a prerequisite for the current deployment-shift claim.

10 comments

Nit7 1/10 · W1

Units: SR/Score in %; gains and Δ in absolute percentage points (pp); gradient-norm changes in relative %. Evaluation seeds: 195–197 for new LIBERO-Plus summaries (distinct from three adaptation repeats at fixed evaluation seed 195).

W1: Mechanism attribution and general insight

Reviewer’s original comment

  1. Mechanism attribution and general insight remain limited. Appendix F.1 clearly defines the staged controls, but the authors correctly acknowledge that the Independent-components-to-shared-reference transition changes several factors together: single- versus multi-view encoding, camera-geometry conditioning, and the pose discrepancy. It is not a single-factor ablation. Thus, the evidence supports the overall supervision design without fully separating its constituent contributions. Moreover, the routing comparison contrasts a single joint feature loss with three weighted branch losses; the roles of supervision structure versus loss weighting or gradient scale could be better characterized. The work offers a reasonable concrete recipe, but yields relatively limited transferable insight into when and why it should outperform simpler objectives.

A1 (to W1)

Thank you for the constructive suggestions on mechanism attribution and transferable insight. The submitted progression, Independent components → Shared factual reference → Component-wise routing (65.6% → 66.4% → 67.3%), established the value of the two-step supervision design. We now complement it with full-method-centered representation/branch ablations, matched routing × weighting controls, and gradient diagnostics to distinguish the contributors to the gain. We then explain the general TTT principle on which SG-TTT builds and use the seven-shift results as empirical evidence for why and when relational supervision is helpful.

New evaluation protocol. We extend the submitted LIBERO-Plus benchmark evaluation seed 195 with seeds 196 and 197. These are evaluation seeds, distinct from the submitted three adaptation repetitions at fixed evaluation seed 195. The new simulation comparisons use 64 unlabeled adaptation samples and 64 optimizer updates; within each controlled comparison, the methods share the base WAM checkpoint, adaptation records, and optimization settings apart from the specified intervention. We report success rate (SR) as mean ± SD over evaluation seeds 195–197. Avg. (3) and Avg. (7) are first computed by equally averaging shifts within each seed, then summarized across seeds. Gradient norms are reported means.

I. Mechanism attribution: what contributes to the improvement?

1. Joint representation, camera geometry, and supervision branches (new Table R-M1). We use full SG-TTT as a common reference across Camera, Robot Init, and Background (67.98 ± 0.61% Avg. (3)). When both calibrated views remain available but are encoded separately with the same frozen DA3, removing cross-view token interaction reduces Avg. (3) to 66.17 ± 0.63% (−1.81 pp); the differences are especially visible on Camera (−2.07 pp) and Background (−2.77 pp). Thus, the joint representation contributes information from the interaction between views, beyond having access to both images individually.

The camera-conditioning and EEF-pose paths can be examined through two complementary contrasts. Removing camera conditioning, while keeping joint image encoding, gives 65.20 ± 0.20% (−2.78 pp relative to full). In our implementation, predicted EEF pose enters the scene representation via the EEF-dependent wrist-camera extrinsics, so this intervention also makes the pose-loss branch inactive. We therefore additionally remove only the EEF-pose loss, retaining camera-conditioned visual geometry: SR becomes 66.49 ± 0.69%, a 1.49 pp difference from full. Importantly, the latter two configurations both have only the two visual loss branches. Comparing them gives a more specific estimate of the camera/EEF-conditioned geometry contribution within that two-branch setting: 66.49% vs. 65.20% (+1.29 pp Avg. (3)), with +3.26 pp on Camera, +1.61 pp on Background, and −1.02 pp on Robot Init. The differences reveal a useful interaction: on Robot Init, camera-conditioned geometry without the pose loss is 1.02 pp lower, but restoring the pose loss raises SR from 65.14% to 66.75% (+1.61 pp). Thus, geometry contributes both as visual context and as the pathway through which EEF-pose supervision becomes effective; the contributions vary with the shift. This is a conditional two-visual-loss contrast, rather than an additive decomposition of the full method. Removing only the agent-view or wrist-view loss also reduces Avg. (3), to 66.70 ± 0.31% (−1.28 pp) or 66.78 ± 0.65% (−1.20 pp); their factual images remain in the other branches' scene context.

Nit7 2/10 · W1

The added controls therefore distinguish cross-view interaction, camera-conditioned scene geometry, and the contribution of each predicted component within the evaluated WAM.

2. Factual routing, separated from branch weighting (new Tables R-M1 and R-R1). The submitted no-routing comparison uses one joint scene-feature loss involving all predicted components, whereas full SG-TTT uses three routed losses. To examine this distinction more closely, we add a joint-context, uniform-weight control (J-U) with three component losses: for each active prediction, the other prediction slots are detached, rather than filled with factual observations. J-U reaches 65.57 ± 0.27% Avg. (3), compared with 67.98 ± 0.61% for full SG-TTT. We further perform a Background-only 2 × 2 controlled study with the number and formulation of component losses fixed, crossing non-active context (detached predictions, J; factual observations, R) with branch weights (uniform U: (1,1,1); weighted W: (1,1,10)).

At fixed (1,1,1) weights, replacing predicted context with factual context improves Background SR from 71.16 ± 1.06% (J-U) to 73.93 ± 0.87% (R-U) (+2.77 pp). At fixed (1,1,10), SR increases from 70.59 ± 1.04% (J-W) to 74.97 ± 1.22% (R-W) (+4.38 pp). Changing the weights alone yields −0.57 pp under joint context and +1.04 pp under factual context. The decisive contrast is therefore between predicted and factual non-active scene context, at the same loss count and weights. It demonstrates that the factual relationships supplied to each branch contribute to the Background gains.

3. What the routing mechanism changes, and what the gradients show (new Tables R-R1 and R-G1). The factual anchor and component-wise routing have complementary roles. The anchor defines one geometrically coherent realized future; routing evaluates one predicted component in the context of the other realized components. For example, the agent-view branch compares a predicted agent image with the factual wrist image and EEF geometry. If the agent prediction equals its realized target, this branch reproduces the factual anchor even if the WAM's other predictions are inaccurate. A loss evaluated on an all-predicted scene can still reflect errors from those other components. Routing therefore gives each branch a more component-specific scene-consistency test, while preserving the physical relationships that independent target matching does not explicitly represent.

The gradient measurements reinforce this attribution. In both fixed-weight contrasts, factual routing gives higher SR with smaller mean gradient norms: 0.06014 → 0.05019 at uniform weights and 0.06086 → 0.05123 at weighted settings. In a separate Background branch-level diagnostic, dropping the EEF-pose loss increases the mean norm from 0.05123 to 0.05247 (+2.42%) while SR decreases from 74.97% to 72.43% (−2.54 pp); removing either visual branch also lowers SR. Together, these experiments show why supervision content and its assignment to a predicted component matter alongside objective weights and optimization scale. The fixed-weight routing attribution is measured on Background; the representation and branch controls cover the three specified shifts.

Mechanism conclusion. These new comparisons move beyond showing that the full recipe works: they identify two functional sources of useful supervision—a joint, geometrically conditioned factual reference and component-specific consistency checks against that reference. The gains depend on what spatial information the loss can test and which prediction is responsible for the tested discrepancy, rather than simply increasing the number, weight, or mean magnitude of gradients.

II. General insight: what does SG-TTT teach us about future-prediction TTT?

Why a small set of executed transitions can support TTT. An already task-trained WAM collects a small set of unlabeled context–future transitions using its initial policy (64 samples/64 updates in simulation; 16 samples/16 updates in the default physical setting). Every realized future is tied to the action actually executed, but the observations may also reflect partial characteristics shared across states in the shifted deployment observation distribution. Camera projection, background appearance, and the EEF/wrist-camera observations induced by new initial configurations can recur across different actions and trajectories. Future-prediction TTT uses these observed transitions to adjust the pretrained WAM's context-to-future mapping, and the updated parameters are then applied to new contexts. The collected futures serve as factual outcomes of executed actions, not action-imitation labels or counterfactual targets for future policy actions.

Nit7 3/10 · W1

Their potential to support transfer is evaluated on disjoint rollouts. Thus, the purpose of a small TTT set is to expose useful, recurring aspects of the new deployment conditions—not to enumerate all states or relearn the task from scratch.

The additional principle introduced by SG-TTT: from separate targets to conditional relational supervision. Access to deployment evidence is only the first step; the structure of supervision determines which relationships are explicitly tested and which adaptation gradients are obtained from the same evidence. Direct-future TTT matches each predicted image and proprioceptive target independently. But the two realized camera views and measured EEF pose are different observations of one physically coupled future. Their joint relationship is itself evidence: a future image may be a plausible individual prediction yet disagree with the other realized view or with the wrist-camera geometry implied by EEF pose. SG-TTT first uses the camera-conditioned joint DA3 factual anchor to represent this shared spatial structure. It then uses one-prediction-at-a-time routing so that each branch tests compatibility with the other factual signals. The anchor provides a factual relational context; routing makes the resulting supervision specific to the prediction being updated. These spatial gradients update the WAM's shared LoRA parameters, which also affect action generation (Appendix A). The methodological advance is therefore greater constraint specificity from the same observed transition: factual signals become both targets and context for checking other predictions.

Seven-shift evidence for this interpretation (expanded Tables R-S1 and R-M2). Across evaluation seeds 195–197, Avg. (7) SR is 80.84 ± 0.29% for Frozen Cosmos, 81.20 ± 0.07% for Direct-future, 81.49 ± 0.28% for Visual-only, and 82.79 ± 0.35% for SG-TTT (+1.59 pp over Direct-future; +1.30 pp over Visual-only). Relative to Direct-future, SG-TTT gains +2.35 pp on Layout, +2.37 pp on Robot Init, and +1.75 pp on Camera—shifts that change object placement, EEF–scene/wrist-camera configuration, or view projection, respectively. Background provides a particularly informative test: appearance changes while usable cross-view/EEF relations remain; here Visual-only and Direct-future fall 2.19 pp and 0.69 pp below Frozen (72.66%), whereas SG-TTT reaches 74.97% (+2.31 pp above Frozen, +3.00 pp above Direct-future, +4.50 pp above Visual-only). This pattern shows that using deployment transitions to adapt is not sufficient by itself; the way their evidence is organized can change the outcome of adaptation. It also aligns with the Background factual-context controls in Part I.

The remaining shifts help delineate the conditions. Noise (+0.82 pp over Direct-future) can weaken the visual evidence on which spatial constraints rely; Language (+0.87 pp) mainly changes instruction wording, for which the objective provides no direct semantic-alignment target. Under Light, Direct-future already reaches 97.08%, and SG-TTT ties it, leaving little headroom for an extra gain. Thus, the cross-shift performance pattern is consistent with the mechanism established by the controlled comparisons: relational supervision is especially useful when actual cross-view/EEF relations remain informative for the deployed task and independent future-target losses leave room for improvement. The shift comparison supports this interpretation as performance evidence, while the controlled ablations more directly identify the contributing design choices.

Transferable insight. For future-predictive models with multiple outputs describing the same underlying physical event, the informative unit of TTT supervision can be the structured factual event rather than a collection of unrelated targets. With scarce deployment samples, two decisions matter: which stable, action-relevant relationships the observations jointly reveal, and how those relationships are translated into prediction-specific consistency checks. SG-TTT implements these decisions by fixing the other components to the realized scene when checking one prediction. This can make each recorded transition more informative without adding expert labels or relying on a larger adaptation set. The broader supervision-design principle is to preserve reliable cross-signal structure in the reference, and use that factual structure to attribute discrepancies to individual predictions. Its value is greatest when the deployment observations still support those relationships; for shifts dominated by instruction semantics, degraded visual evidence, or near-saturated performance, the additional benefit is naturally more limited.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.