- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
A legged robot's response to an action depends on physical properties it cannot measure directly, such as terrain slope, friction and ground softness, so it must infer them from its recent interaction history. Prior history-based adaptation methods typically train the history encoder to predict the robot's next state. The authors argue that good next-step accuracy does not tell us whether the encoder has actually captured these properties. They propose an alternative training signal, Equivariant Counterfactual Residual (ECR) supervision. In simulation, they replay the same state and action while changing one physical property. They subtract the average response across those variations, leaving targets that contain only the differences caused by the property. A compact code computed from a history taken from a different trajectory in the same environment is then trained to predict these difference targets. The history is first summarized by a fixed random recurrent network, followed by a small trained network.
The experiments use a simulated quadruped across a grid of 60 slope/friction/softness combinations. ECR is compared against next-step prediction and against adapted versions of several prior objectives, including one trained directly on the true property labels. ECR codes make slope far easier to read out than next-step codes do, and they group histories from property values held out of training more faithfully. A matched ablation suggests the subtraction step contributes to this. However, when the frozen codes are used to train new predictors, the in-distribution gain is small. Next-step prediction wins on every held-out prediction task, and at unseen slopes ECR is worse than using no history code at all. The authors conclude that the supervision makes physical factors easier to read out, but that whether this helps prediction depends on the downstream task.
Critical strengths
- Slope is consistently easier to read out of ECR codes than out of next-step-prediction codes.
- The advantage over next-step prediction (NSP) and CaDM-style training appears in three different measurements: mutual information, linear probes and nearest-neighbour retrieval (Fig. 2).
- It holds when only one time step per trajectory is used, so repeated near-duplicate samples do not inflate it (Table 10: normalized mutual information 0.900 vs 0.148).
- It holds in all ten tests with a property value held out of training (Tables 11-13).
- Swapping in a history from a different slope raises ECR's slope-prediction error 1.60x but leaves NSP's unchanged (1.00x), so the ECR predictor does use the slope information (Table 8).
- The paper reports its negative results plainly. It states that NSP has lower error on all four held-out prediction tasks (Table 3). The discussion (Sec. 6) admits that the ablation does not isolate the effect of subtracting the mean. This makes the paper a useful counterexample: a code from which physical factors are easy to read out need not help a predictor.
Critical weaknesses
- Easier read-out of the factors does not improve prediction, and at unseen slopes the ECR code makes prediction worse than having no code.
- At the held-out 5° and 10° slopes, a predictor given the ECR code has 15% and 18% higher error than one given only the current state (0.1601 vs 0.1394 and 2.4578 vs 2.0897). NSP codes beat both (Table 3, Tables 14 and 17).
- In distribution, ECR's next-step error is only 1.8% lower than NSP's (App. H.3).
- The paper therefore shows no prediction task where easier read-out pays off, and it does not test control. The two baselines that also use environment information are LDG-style (groups histories from the same environment) and RMA-style (trained on the true labels). Both match or beat ECR on slope read-out (linear-probe R² 0.940 and 0.977 vs 0.907; Sec. 5.1).
- The evidence does not isolate what is new about ECR. The ablation meant to isolate mean-subtraction is confounded, NSP receives far less supervision than ECR, and the step beyond the friction-only predecessor is never tested.
- The control trains on the same targets with the mean left in. In some coordinates these targets are up to about 340 times larger, so the gradient is clipped on every update, with average pre-clipping norms above 3,100 (Table 25, App. H.3).
- The control ends up worse on slope than ordinary NSP (linear-probe R² 0.380 vs 0.555; Table 1). The paired gain may therefore come from a badly scaled baseline rather than from removing the shared component. The authors acknowledge this (Sec. 5.2.1), yet the abstract says the ablation "corroborates" the effect.
- ECR is trained to predict how the response changes when exactly the three factors (slope, friction, contact type) are varied. That uses simulator replays NSP never sees, and the probes and retrieval tests then try to recover those same factors. A code trained this way is expected to expose the factors more readably than a next-step code. The gap over NSP therefore shows that the extra supervision is used, not that the residual construction is a better learning principle.
- The paper presents itself as extending an earlier friction-only version of the method to slope and multiple factors (Sec. 2). The new ingredient, a replay that keeps the robot's position on the slope comparable across conditions, is never compared with a simpler replay, so the contribution beyond that predecessor is not shown.
- The results are hard to follow because the main text keeps recording where each number came from. Sentences and captions interrupt the findings with run lineage, reused encoders and sample counts (e.g. "Fresh 5000-update heads on the existing NSP and CaDM encoders provide context; their encoder training is not paired with the new ECR run", Sec. 5). Several terms are used without definition, such as "donor history", "receiving query", "anchor" and "accessibility". The title also promises an "equivariant" method, although the network is not equivariant (Sec. 2); equivariance is only a property checked in the simulator (Sec. 3.1).
Questions for authors
- (Weakness 1: no prediction benefit) Is there any downstream use where the ECR code beats both next-step-prediction codes and a predictor given no code, for example closed-loop locomotion or adaptation at the held-out slopes? At those slopes the ECR code currently makes prediction worse than no code (Tables 14 and 17). If the code helped a controller there, the main weakness would be largely resolved. If it hurts control as well, the claim that the code captures useful physical context does not hold.
- (Weakness 2: confounded ablation) Does subtracting the mean still help once the control is properly scaled? The control trains on the same replayed responses without the subtraction. It could, for example, standardize its targets with their own statistics, or have its clipping and learning rate tuned so it is not clipped on every update. If a control fixed this way matches ECR's slope read-out, the gain comes from the replay data rather than from the residual construction. The contribution would then reduce to "train on counterfactual replays".
- (Weakness 2: extra supervision) Why are counterfactual replays needed at all? The LDG-style baseline only needs to know which histories come from the same environment, and the RMA-style baseline uses the true labels. Both match or beat ECR on slope read-out, and RMA-style also beats it on retrieving the exact environment. Showing ECR ahead on some measure, at equal environment information and simulation cost, would justify re-simulating every factor setting. Without that, ECR is not the best choice among methods with similar access to the simulator.
- (Weakness 2: what is new beyond the friction-only predecessor) What does this paper establish that the earlier friction-only version of the method, which it says it extends, did not? If the new part is mainly the extension to slope and multiple factors, the paper should show that its matched replay for slopes matters, for example against a simpler replay that does not keep the robot's position on the terrain comparable. Without that, the contribution over the predecessor is small.
3 comments
Thank you for the new experiments. The transfer results are interesting. However, the gain in next-step prediction is very small, standard next-step prediction catches up once slope direction is available, and the comparison with the friction-only predecessor is still missing. My assessment therefore remains the same.
Thank you for the follow-up.
We do not claim that ECR beats NSP on next-step prediction. Our question is whether a history representation can retain the information needed to predict the robot’s next motion while allowing the same steepness read-out to work across different surfaces. Since NSP trains directly on the next physical response, it is the natural baseline for testing whether ECR preserves that predictive ability. Requiring ECR to outperform NSP on one-step prediction would test a different claim.
Toso, LeCun, Anderson, and Bounou (2026, Lemma 2) illustrate why both evaluations matter. In their linear JEPA example, an encoder can discard every unstable mode while predicting its next latent state with zero error. In our experiment, the same transformer architecture gave similar next-step errors on new rollouts from familiar conditions (0.0305 for ECR vs. 0.0315 for NSP). Yet when a steepness read-out fitted in one condition was applied in another, mean Spearman correlation across four friction and contact transfers was 0.878 for ECR vs. 0.099 for NSP at λ = 100.
With the original replays, inputs, and training budgets we tested, NSP cannot catch up to ECR on steepness read-out. Neither the encoder nor the training decoder explicitly received the direction of the replayed slope. After 20,000 transformer updates, steepness R² was 0.897 for ECR vs. 0.279 for NSP.
Our analysis in §3.4 explains why we tested direction. Uphill and downhill replays can give the decoder nearly identical terrain-relative queries while gravity produces different physical responses. We therefore gave only the training decoder a privileged two-dimensional vector indicating downhill direction relative to the robot’s heading. A probe given the query and this vector distinguished 5° from 10° at chance (50%). With the added direction input, reservoir NSP caught up (R² of 0.970 for NSP vs. 0.968 for ECR). For the transformer, ECR remained ahead at 6,000 updates (0.975 vs. 0.841). The transformer tie at 20,000 updates arose in a separate experiment that changed the replays to keep slope direction consistent (0.988 for NSP vs. 0.987 for ECR). Here, the geometry let us identify the missing variable and the simulator let us supply it. In another partially observed system, either step may be difficult.
Beyond the friction-only predecessor, we introduce matched slope replays and test whether steepness can be read across changes in friction and contact. Varying friction leaves the supporting terrain in place. For slope, we rotate the robot and terrain together so the feet begin in the same configuration relative to the ground while gravity remains fixed. The direct comparisons with friction-only training and terrain-only tilt remain open. The result we establish is that, under the matched replays, a steepness read-out fitted on one surface continues to rank slopes on another.
Reference: Toso, LeCun, Anderson, and Bounou (2026). Preserving Unstable Modes Through Inverse Dynamics in JEPA World Models, Lemma 2.
Response to Reviewer mHqY
We thank the reviewer. Numbered section and appendix references refer to the original manuscript.
W1, Q1. Does the representation improve a downstream task?
Yes, for next-step prediction in training conditions. To answer the reviewer's Q1, we trained new predictors from frozen representations. We added a trainable transformer to test dependence on the fixed reservoir, with the same encoder architecture across objectives.
With 6,000 encoder updates and 3,000 predictor updates, mean normalized prediction MSE was:
Representation results average three encoder seeds; the last row averages two predictor seeds.
We also asked whether a steepness predictor learned in one condition works in another. We fitted regularized linear predictors on source-condition training rollouts and evaluated target-condition test rollouts: friction 1.2↔0.2 and Rigid↔Soft+Dissipative contact profiles. Steepness is absolute slope. All conditions were included in encoder training and training-data preprocessing.
We added NSP trained only on recorded transitions, without counterfactual replays (factual-only NSP). Ordinary read-out fits a predictor across conditions; transfer fits in one and tests in another. Reservoir results average three encoder seeds; transfer averages four tests at regularization strength λ=100 (Spearman correlation).
Similar ordinary R² masks a substantial transfer difference. Across six regularization strengths, ECR's correlation spans 0.74–0.92 versus factual-only NSP's 0.11–0.19. ECR's mean transferred numerical R² is 0.71 at λ=100 and negative at every tested λ≤10.
The transfer test has no Query-only reference. Factual-only NSP slightly overtakes ECR in prediction after 20,000 encoder updates. NSP beats ECR on all four held-out prediction tasks; ECR loses to Query-only at both slopes. The stop-line control test did not establish a benefit: ECR minus no representation is +0.05 reward, with a paired 95% t interval [−2.58, 2.67] over five policy seeds using three representation seeds.
W2, Q2. Does the gain survive proper target normalization?
Yes. In the original paired comparison, ECR-Paired learns response differences and NSP-Matched learns full responses with the same training decoder architecture. We tested the reviewer's scaling and clipping repairs, keeping the remaining setup fixed.
These are three-run means. The standardized control clipped fewer than 0.6% of updates. Standard NSP is separate and already standardizes its own targets (App. B.2).
The original decoder is not explicitly given slope direction. ECR's advantage holds in this tested setting with partially observed inputs. Supplying direction to the reservoir decoder let NSP catch up: 0.970 versus ECR's 0.968. NSP also caught up after 20,000 transformer updates when replays kept slope direction consistent. ECR requires neither change.
W2, Q3. What do replays add beyond NSP, LDG-style and RMA-style?
NSP already received the same counterfactual data as ECR (§3.5; Apps. B.2–B.3). The history encoder architecture is shared; objectives retain their own output networks and normalization.
In transfer, ECR exceeds LDG-style's environment-grouping supervision and approaches RMA-style's direct physical-label supervision. ECR uses response differences, intervention metadata and histories paired by environment; slope replay also uses geometry. Computing its representation targets does not require numerical factor labels.
W2, Q4. What is new beyond the friction-only method?
The predecessor used residual supervision with matched friction interventions and histories from different rollouts. We extend this to slope and contact properties. For slope, we transform robot and terrain together to preserve their relative configuration while gravity remains fixed (§3.1). We have not completed the reviewer's proposed comparison with replay that tilts the terrain while leaving the robot fixed.
W3. How will we clarify presentation and equivariance?
We will add these results, revise the abstract's ablation claim, define the four flagged terms and move run details to the appendix. We will explain that the equivariance check rotates the robot, terrain and gravity together; the encoder is not an equivariant architecture.