- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work proposes an algorithms for zero-shot reinforcement learning. Starting from TD-JEPA, the authors propose two additional losses for the encoder. These loss are computed by finding transitions whose initial state is approximately the same, and whose actions lead to different goals. The first loss regresses the difference in goal-conditioned reward between the two actions, while the second predicts which action is the best at goal-reaching. The motivation is that of encouraging state representation to capture information that is discriminative of the outcome of different actions, and can thus be useful for ranking them.
The method, called TwinJEPA, is then validated on a few environments spanning OGBench, EXORL, and D4RL. The experimental section (i) compares the method to TD-JEPA, (ii) breaks down the previous analysis by tasks and (iii) ablates the necessity of each of the two components.
Critical strengths
- The idea that state representation should be useful for ranking actions is interesting, and, to the best of my knowledge, this property is not encouraged explicitly in most zero-shot RL algorithms.
- The paper is overall clear, except for some imprecisions listed in the next section.
Critical weaknesses
- The method uses privileged information. The criterion in equation 3 requires access to the ground-truth state, which is not possible when learning from pixels; moreover, it assumes that the content of each feature is known. This gives the method a big advantage with respect to the baselines in Table 2. As a suggestion, the value function learned by TD-JEPA may provide a proxy for this criterion, as it can be used for evaluating goal-conditioned rewards. Modifying this criterion to be self-supervised may lift this weakness.
- The effectiveness of the additional objective does not appear to be sound. As reported, the reward gap in equation 5 is predicted directly from the encoded state, without access to the action pair. How can a regressor fit the target without this information? Similarly, the BCE loss is predicted without conditioning on the goal selected for the paired dataset. How can the model predict which actions has a higher reward if it does not see the goal? These issues seems to be confirmed on line 1737 and 1741.
- The experimental protocol is questionable, for two main reason. First, line 1618 states that individual seeds are excluded from the evaluation upon seeing their results. I cannot see an explanation of why this would be a defensible experimental protocol. Second, according to reported p-values, the difference with respect to TD-JEPA is hardly ever significant. The majority of the experimental results do not seem to provide sufficient statistical support for TwinJEPA, except for the quadruped environment. Similarly, section 4.4 does not seem to conclusively support the method, suggesting that often removing one of the two losses is actually beneficial. As a whole, the experimental evaluation appears rather inconclusive, despite being presented as supportive.
Minor:
- While the paper reads well, it often displays the symptoms of being generated by an LLM, especially in the Appendix. For instance, it often displays structures like ‘rather than X, we say Y’.
- Sections 2.1 and 2.2 are imprecise. Line 138 introduces goal-conditioned control, but the setting appears to be zero-shot RL. Equation 2 suggests that predictions are for the next step (t+1), but this is not the case; furthermore the targets are not introduced. The orthonormalization loss on line 168 should have a coefficient (and be introduced mathematically). Explaining TD-JEPA or successor features in a more precise and self-contained way could be helpful - there is no information about the reward inference procedure at the moment, for instance.
- A precise description of how goals are sampled and rewards are generated (line 212) would be helpful. This is not entirely clear.
- The presentation of results in section 4 could improve. Figure 2, 3 and Table 1 overlap significantly, and present the same outcomes several times.
- On line 207, and are introduced, but they seem to overlap completely with and .
- On line 219, is defined and only used once.
- The number of seeds is inconsistent, spanning from 2 to 10.
- Citations could use \citep instead of \citet
Questions for authors
- How can the additional heads help with shaping representations, when they do not receive sufficient information to minimize their losses?
- What is the logic of putting so much emphasis on per-seed results and seed-matched evaluations? Similarly, why does it make sense to report the numerical values of the seeds (e.g. line 1610)?
3 comments
(continued)
Question 2 Matched seeds and seed-level reporting
We appreciate the reviewer’s question about matched seeds and seed-level reporting. Matching the random seeds reduces comparison noise because TD-JEPA and TwinJEPA are evaluated under the same seed. The primary statistic is the within-seed method difference. Seed-direction counts show whether the aggregate effect is shared across runs. Numerical seed IDs are only needed for auditability in cases such as the Walker replacement check. The main text will emphasize paired effects, intervals, and direction counts.
Minor and presentation comments
We thank the reviewer for the detailed presentation comments. The revision will make the following changes.
- Clarify offline goal relabeling during training and zero-shot task inference at evaluation.
- Correct the time indices in Eq. 2 and define the target-encoder outputs.
- Include the orthonormality coefficient in the objective.
- Add a self-contained description of TD-JEPA task and reward inference.
- Specify goal sampling and reward recomputation in pair construction.
- Add concise pair-mining pseudocode and benchmark-specific matching details.
- Reduce repeated presentation across Figures 2 and 3 and Table 1.
- Report seed counts and evaluation budgets consistently.
- Clean up overlapping notation and standardize citation style.
- Rewrite repetitive Appendix phrasing into shorter technical prose.
We thank the reviewer again for the detailed suggestions and hope the added experiments and revisions make the method and evaluation clearer.
Thank you for the response. Unfortunately, Weakness 1 and 3 was only slightly alleviated, and the other two largely remain. I also fundamentally disagree on the utility of matched seeds: I do not think they reduce comparison noise, they simply condition on that noise. I'm not sure if there should be such thing as a "replacement check". I would suggest redesigning the experimental protocol to be sound and rigorous, and I do not recommend acceptance in the current state.
Official Comment by Authors to Reviewer nBdG
We thank the reviewer for the detailed feedback and for recognizing the interest of explicitly encouraging action-ranking information in zero-shot control representations. We clarify the auxiliary-head inputs, RGB training setup, evaluation protocol, and statistical evidence below.
Weakness 1 Simulator information in the RGB experiments
We thank the reviewer for identifying the simulator-state dependency in the RGB pipeline. We audited the implementation and separated network observations from training-time pair construction.
The network receives RGB observations, while simulator physics is used only in training-time matching and target generation. The pair buffer and auxiliary heads are not used at inference. We therefore describe the setting as RGB-input learning with simulator-assisted pair construction and target generation. This wording keeps the inference-time claim unchanged and makes the training-time information source explicit.
Following the reviewer’s suggestion, we also completed a representation-based matching diagnostic with a frozen official TD-JEPA RGB encoder on Walker frame stacks. It retains 19,999 of 20,000 pairs at the selected threshold. This shows that visual representations can support pair retrieval without the physics-state matching key. Reward labels in this diagnostic remain simulator-derived, so the result addresses the matching component and is reported separately from a complete pixel-only pipeline.
Weakness 2 Information available to the auxiliary heads
We thank the reviewer for pointing out the ambiguity in Eq. 5. The preference head receives (Z,aA,aB), while the gap head predicts E[Δr∣Z] on the mined-pair distribution. The Cheetah input ablations directly test whether adding the omitted variables improves the auxiliary pathway. The submitted configuration has the highest mean among the tested variants. Maze2D provides a complementary component test in a different reward structure, where ∣Δr∣ is constant and preference-only reaches the highest mean.
Weakness 3 Evaluation protocol and statistical support
We appreciate the reviewer’s concern about run selection and statistical support. We restored the original Walker runs and recomputed the affected results. We also audited the other main benchmarks and found no analogous substitutions. The additional Cheetah extra-seed runs remain separate from the primary matched set and are not mixed into the main estimate.
For statistical reporting, we use matched mean differences, 20,000-resample bootstrap intervals, and seed-direction counts. This is particularly useful for five-seed settings, where the exact two-sided sign-flip test has a minimum value of 0.0625. The Quadruped and Maze2D preference-only results are therefore presented with their effect sizes, intervals, and paired directions without using a binary significance claim.
Question 1 How the auxiliary heads shape the shared representation
We thank the reviewer for asking how the auxiliary losses affect the shared representation. Both auxiliary losses backpropagate through the shared encoder. Preference classification requires the shared representation together with the two candidate actions to preserve information that predicts the retained ordering. Reward-gap regression requires Z to preserve information associated with local contrast magnitude on the mined-pair distribution. The Quadruped shuffle intervention provides direct downstream evidence for this supervision. When the pair-target association is destroyed and the rest of the pipeline is unchanged, the +40.4 gain disappears and the mean returns near TD-JEPA.