- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies an additional training loss that supervises how the effect of switching actions actions changes across starting states. The authors analyse which prediction errors this loss penalizes by decomposing it into common and differential errors, and focus on differential errors that capture how action effect differs between states. They proceed to compare different ways of pairing states during training, including state shuffled pairs from the same actuator mode. They find that matching starting positions and velocities is unnecessary for prediction gains, while the benefit of preserving actuator modes depends on how those modes affect motion, in a synthetic dynamical system. They perform additional experiments on Catpole, Reacher, and Meta-World, and report that action-effect predictions can accompany worse absolute predictions and acknowledge that their results do not establish a general control benefit.
Critical strengths
None
Critical weaknesses
-
Insufficient motivation: it is not clear from the manuscript why cross-state action-effect supervision is an important object to analyse. I am not convinced as to whether there is any practically important failure that is effectively addressed by this supervision. The reported held-out action-effect improvements demonstrate benefits under the specific training budget used, but their implications for generalization, training stability or convergence speed, or downstream model use remain unclear. The authors acknowledge in Appendix A13 that zero-loss solutions do not change with this supervision, which makes it harder for me to understand why this additional supervision loss can be relevant for some prominent outstanding problems in robot learning such as data scarcity, memorization, or lack of generalization.
-
Limited novelty: the major contribution of the paper seems to be the controlled empirical analysis of the cross-state action-effect supervision. However, the tested settings are very limited, so the broader scientific contribution remains insufficiently compelling to me. Specifically, as written in Section 4.1, for the main correspondence experiment, true action effects are independent of the starting position and velocity coordinates being re-paired. Thus, their finding that exact alignment is unnecessary is limited only within this invariant setting, which the authors acknowledge in Section 6. Further, the physical intervention experiments establish that pairing benefits depend on the dynamics, but does not provide insights into how the dynamics informs the potential pairing benefits. It is not clear how the results can be used as guidance for choosing useful state pairings.
Questions for authors
-
What important problem or unresolved question in world-model training does this analysis help address? In particular, what should researchers understand or do differently based on these findings? If the main practical benefit is improved prediction under limited training budget, please clarify whether the authors interpret this as an optimisation benefit, or an improvement that persists with further training. If the former, I would be better convinced with either a theoretical analysis on how the supervision affects convergence speed or training stability, or a comprehensive empirical evidence over varying training settings.
-
How do the conclusions drawn from the experiments extend to settings not included in the manuscript? Is exact alignment not as necessary in settings where action effects are no longer independent of the position and velocity as in the tested settings? How should a user understand the importance of matched state pairs for supervision, given the dynamics of interest? Also, how can the physical intervention experiment results be used or extended to draw a general explanation or guidance for choosing good state pairs for supervision?