acceptodds
Under review as a conference paper at ICLR 2027

TwinJEPA: Action-Preferred Predictive Representations for Goal-Conditioned Control

Abstract

Joint-Embedding Predictive Architectures (JEPAs) have recently emerged as a promising paradigm for representation learning by predicting future latent states without reconstructing observations. Recent work has adapted JEPA-style latent prediction to offline zero-shot control through action-conditioned temporal prediction, enabling representations to capture long-horizon dynamics from fixed behavioral data. However, transition-level predictive objectives largely supervise actions in isolation and provide limited information about which actions are preferable when similar states admit different goal-conditioned outcomes. We introduce TwinJEPA, a framework for learning action-preferred predictive representations by augmenting JEPA-based control with offline-mined action-preference supervision. TwinJEPA identifies approximately matched states across offline trajectories and constructs preference pairs via goal-conditioned reward relabeling. It then learns two complementary objectives: reward-gap regression, which preserves the magnitude of outcome differences, and preference classification, which captures the relative ordering of alternative actions. Both objectives are used only during training and incur no additional inference-time cost. We evaluate TwinJEPA across long-horizon navigation and continuous-control benchmarks, with both state-based and pixel-based observations. TwinJEPA yields positive benchmark-level mean differences across all matched state-based evaluations, while analyses across domains and observation modalities indicate that larger gains tend to arise when local action alternatives provide more informative outcome contrasts. The results suggest that local action-comparison supervision can complement temporal prediction by encouraging JEPA-based representations to retain both long-horizon temporal structure and fine-grained distinctions among locally observed actions for offline zero-shot control.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Joint-Embedding Predictive Architectures, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nBdGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes an algorithms for zero-shot reinforcement learning. Starting from TD-JEPA, the authors propose two additional losses for the encoder. These loss are computed by finding transitions whose initial state is approximately the same, and whose actions lead to different goals. The first loss regresses the difference in goal-conditioned reward between the two actions, while the second predicts which action is the best at goal-reaching. The motivation is that of encouraging state representation to capture information that is discriminative of the outcome of different actions, and can thus be useful for ranking them.

The method, called TwinJEPA, is then validated on a few environments spanning OGBench, EXORL, and D4RL. The experimental section (i) compares the method to TD-JEPA, (ii) breaks down the previous analysis by tasks and (iii) ablates the necessity of each of the two components.

Critical strengths
  • The idea that state representation should be useful for ranking actions is interesting, and, to the best of my knowledge, this property is not encouraged explicitly in most zero-shot RL algorithms.
  • The paper is overall clear, except for some imprecisions listed in the next section.
Critical weaknesses
  1. The method uses privileged information. The criterion in equation 3 requires access to the ground-truth state, which is not possible when learning from pixels; moreover, it assumes that the content of each feature is known. This gives the method a big advantage with respect to the baselines in Table 2. As a suggestion, the value function learned by TD-JEPA may provide a proxy for this criterion, as it can be used for evaluating goal-conditioned rewards. Modifying this criterion to be self-supervised may lift this weakness.
  2. The effectiveness of the additional objective does not appear to be sound. As reported, the reward gap in equation 5 is predicted directly from the encoded state, without access to the action pair. How can a regressor fit the target without this information? Similarly, the BCE loss is predicted without conditioning on the goal selected for the paired dataset. How can the model predict which actions has a higher reward if it does not see the goal? These issues seems to be confirmed on line 1737 and 1741.
  3. The experimental protocol is questionable, for two main reason. First, line 1618 states that individual seeds are excluded from the evaluation upon seeing their results. I cannot see an explanation of why this would be a defensible experimental protocol. Second, according to reported p-values, the difference with respect to TD-JEPA is hardly ever significant. The majority of the experimental results do not seem to provide sufficient statistical support for TwinJEPA, except for the quadruped environment. Similarly, section 4.4 does not seem to conclusively support the method, suggesting that often removing one of the two losses is actually beneficial. As a whole, the experimental evaluation appears rather inconclusive, despite being presented as supportive.

Minor:

  • While the paper reads well, it often displays the symptoms of being generated by an LLM, especially in the Appendix. For instance, it often displays structures like ‘rather than X, we say Y’.
  • Sections 2.1 and 2.2 are imprecise. Line 138 introduces goal-conditioned control, but the setting appears to be zero-shot RL. Equation 2 suggests that predictions are for the next step (t+1), but this is not the case; furthermore the targets yty_t are not introduced. The orthonormalization loss on line 168 should have a coefficient (and be introduced mathematically). Explaining TD-JEPA or successor features in a more precise and self-contained way could be helpful - there is no information about the reward inference procedure at the moment, for instance.
  • A precise description of how goals are sampled and rewards are generated (line 212) would be helpful. This is not entirely clear.
  • The presentation of results in section 4 could improve. Figure 2, 3 and Table 1 overlap significantly, and present the same outcomes several times.
  • On line 207, aAa_A and aBa_B are introduced, but they seem to overlap completely with aia_i and aja_j.
  • On line 219, δr\delta r is defined and only used once.
  • The number of seeds is inconsistent, spanning from 2 to 10.
  • Citations could use \citep instead of \citet
Questions for authors
  • How can the additional heads help with shaping representations, when they do not receive sufficient information to minimize their losses?
  • What is the logic of putting so much emphasis on per-seed results and seed-matched evaluations? Similarly, why does it make sense to report the numerical values of the seeds (e.g. line 1610)?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors to Reviewer nBdG

We thank the reviewer for the detailed feedback and for recognizing the interest of explicitly encouraging action-ranking information in zero-shot control representations. We clarify the auxiliary-head inputs, RGB training setup, evaluation protocol, and statistical evidence below.

Weakness 1 Simulator information in the RGB experiments

We thank the reviewer for identifying the simulator-state dependency in the RGB pipeline. We audited the implementation and separated network observations from training-time pair construction.

RGB pipelineImplementation
Network input64 x 64 RGB 3-frame stack
Pair matchingSimulator physics state
Reward targetsSimulator-derived task reward
InferenceOriginal policy path no auxiliary heads

The network receives RGB observations, while simulator physics is used only in training-time matching and target generation. The pair buffer and auxiliary heads are not used at inference. We therefore describe the setting as RGB-input learning with simulator-assisted pair construction and target generation. This wording keeps the inference-time claim unchanged and makes the training-time information source explicit.

Following the reviewer’s suggestion, we also completed a representation-based matching diagnostic with a frozen official TD-JEPA RGB encoder on Walker frame stacks. It retains 19,999 of 20,000 pairs at the selected threshold. This shows that visual representations can support pair retrieval without the physics-state matching key. Reward labels in this diagnostic remain simulator-derived, so the result addresses the matching component and is reported separately from a complete pixel-only pipeline.

Weakness 2 Information available to the auxiliary heads

We thank the reviewer for pointing out the ambiguity in Eq. 5. The preference head receives (Z,aA,aB)(Z,a_A,a_B), while the gap head predicts E[Δr∣Z]E[\Delta r \mid Z] on the mined-pair distribution. The Cheetah input ablations directly test whether adding the omitted variables improves the auxiliary pathway. The submitted configuration has the highest mean among the tested variants. Maze2D provides a complementary component test in a different reward structure, where ∣Δr∣|\Delta r| is constant and preference-only reaches the highest mean.

Weakness 3 Evaluation protocol and statistical support

We appreciate the reviewer’s concern about run selection and statistical support. We restored the original Walker runs and recomputed the affected results. We also audited the other main benchmarks and found no analogous substitutions. The additional Cheetah extra-seed runs remain separate from the primary matched set and are not mixed into the main estimate.

For statistical reporting, we use matched mean differences, 20,000-resample bootstrap intervals, and seed-direction counts. This is particularly useful for five-seed settings, where the exact two-sided sign-flip test has a minimum value of 0.0625. The Quadruped and Maze2D preference-only results are therefore presented with their effect sizes, intervals, and paired directions without using a binary significance claim.

Question 1 How the auxiliary heads shape the shared representation

We thank the reviewer for asking how the auxiliary losses affect the shared representation. Both auxiliary losses backpropagate through the shared encoder. Preference classification requires the shared representation together with the two candidate actions to preserve information that predicts the retained ordering. Reward-gap regression requires ZZ to preserve information associated with local contrast magnitude on the mined-pair distribution. The Quadruped shuffle intervention provides direct downstream evidence for this supervision. When the pair-target association is destroyed and the rest of the pipeline is unchanged, the +40.4 gain disappears and the mean returns near TD-JEPA.

AuthorsOpenReview ↗

(continued)

Question 2 Matched seeds and seed-level reporting

We appreciate the reviewer’s question about matched seeds and seed-level reporting. Matching the random seeds reduces comparison noise because TD-JEPA and TwinJEPA are evaluated under the same seed. The primary statistic is the within-seed method difference. Seed-direction counts show whether the aggregate effect is shared across runs. Numerical seed IDs are only needed for auditability in cases such as the Walker replacement check. The main text will emphasize paired effects, intervals, and direction counts.

Minor and presentation comments

We thank the reviewer for the detailed presentation comments. The revision will make the following changes.

  • Clarify offline goal relabeling during training and zero-shot task inference at evaluation.
  • Correct the time indices in Eq. 2 and define the target-encoder outputs.
  • Include the orthonormality coefficient in the objective.
  • Add a self-contained description of TD-JEPA task and reward inference.
  • Specify goal sampling and reward recomputation in pair construction.
  • Add concise pair-mining pseudocode and benchmark-specific matching details.
  • Reduce repeated presentation across Figures 2 and 3 and Table 1.
  • Report seed counts and evaluation budgets consistently.
  • Clean up overlapping notation and standardize citation style.
  • Rewrite repetitive Appendix phrasing into shorter technical prose.

We thank the reviewer again for the detailed suggestions and hope the added experiments and revisions make the method and evaluation clearer.

Thank you for the response. Unfortunately, Weakness 1 and 3 was only slightly alleviated, and the other two largely remain. I also fundamentally disagree on the utility of matched seeds: I do not think they reduce comparison noise, they simply condition on that noise. I'm not sure if there should be such thing as a "replacement check". I would suggest redesigning the experimental protocol to be sound and rigorous, and I do not recommend acceptance in the current state.

Reviewer_TDKEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TwinJEPA, which adds action-comparison supervision to TD-JEPA. It mines transitions with different actions from similar states, then uses goal relabeling to construct reward-gap and preference targets. Two auxiliary heads train the shared representation, while the policy and inference procedure remain unchanged. Experiments cover navigation and continuous control with state and pixel observations. The results show positive average changes across the state-based benchmarks, although their size and consistency vary substantially. Component ablations also show that combining both objectives is not always better than using either alone.

Critical strengths
  1. The method is a simple addition to an existing control framework. It derives supervision from offline data without human preference labels or additional inference cost, making the approach practically appealing (Section 3).
  2. The matched-seed comparisons and task-level breakdowns are useful. The paper reports uncertainty and negative results openly, and the Quadruped results provide the clearest evidence that the additional supervision can help across multiple tasks (Table 1; Appendix F).
Critical weaknesses
  1. The quality of the mined action preferences is not sufficiently established. In AntMaze, states are matched using position alone, so outcome differences may also reflect differences in posture or velocity rather than the compared actions (Section 3.1). The theory analyzes prediction of the mined labels but does not validate this matching assumption (Appendix I).
  2. The empirical benefit is limited and inconsistent. The AntMaze and Walker comparisons have substantial uncertainty, individual objectives outperform the combined loss on some navigation settings, and pixel-based Cheetah favors the baseline (Tables 1, 3, and 17). These results leave me unconvinced that the proposed supervision provides a sufficiently reliable improvement.
Questions for authors
  1. How are relabeled rewards and pixel-based matches constructed, and how is the reliability of the resulting preferences checked?
  2. Since preferences depend on the selected goal, what motivates excluding that goal from the preference head?
  3. What caused the unstable Walker runs to be replaced, and how would retaining the original runs affect the comparison (Appendix D.1)?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors to Reviewer TDKE

We thank the reviewer for the constructive feedback and for highlighting the practical appeal of TwinJEPA and the value of the matched-seed evaluation. We provide additional analyses of pair reliability, goal conditioning, empirical consistency, and the Walker evaluation protocol below.

Weakness 1 Reliability of mined action preferences

We thank the reviewer for raising the reliability of XY matching. We compared the original AntMaze XY rule with stricter and pair-budget-matched full-state alternatives.

AntMaze matchingPairsMedian full-state dist.
XY matching5,5017.88 on remaining dims
Strict full-state τ=1.0\tau=1.01680.47
Pair-budget full-state τ=4.0\tau=4.05,5014.52
Pair-budget kNN5,5014.14

The strict full-state rule retains only 168 pairs. Matching the original 5,501-pair budget requires a looser full-state neighborhood. This quantifies the trade-off between local-state fidelity and the amount of comparative supervision available for training.

AntMaze pair sourceMean successPaired result
XY matching71.6%Reference
Pair-budget full-state64.4%-7.2 pp CI [-27.9, 13.5]

At a matched pair budget, XY matching reaches 71.6% and full-state matching reaches 64.4%. The five-seed interval includes zero. We therefore treat this experiment as an audit of the matching assumption and pair-coverage trade-off, while the original XY rule remains competitive under the matched-budget evaluation.

We also measured label predictability and replay-based outcome structure.

AntMaze diagnosticObserved value
Held-out label AUROC XY0.976 to 0.985
Held-out label AUROC full-state0.958 to 0.969
Held-out label AUROC kNN0.937 to 0.952
Replay tie rate H=197.9%
Replay tie rate H=5,10,2595.4% / 93.6% / 96.4%
Untied longer-horizon agreement0.45 to 0.48

The held-out AUROC values measure whether the stored labels are predictable under each mined distribution. The replay diagnostics address a different question. Longer-horizon comparisons are tied in 93.6% to 97.9% of cases, and agreement on the untied subset is 0.45 to 0.48. This helps explain why AntMaze yields weaker gains than Maze2D and supports the paper’s conclusion that the utility of pairwise supervision depends on the informativeness of the mined comparisons.

Question 1 Relabeled rewards, pixel matching, and preference checks

We appreciate the reviewer’s request for a precise description of pair construction and reward relabeling. Each pair starts from an observed anchor transition. A second observed transition is selected from the local matching neighborhood and must pass the action-diversity filter. Candidate goals are sampled from the fixed offline training data. For each candidate goal, we recompute the goal-conditioned rewards of the two observed next states. We select the goal with the largest absolute contrast and store the corresponding gap and preference targets. No held-out evaluation goal is used.

The two candidate actions come from locally matched observed transitions. The AntMaze full-state audit checks sensitivity to the matching representation, and the held-out AUROC measures consistency of the constructed labels. In the RGB experiments, pair matching uses simulator physics state and target generation uses simulator-derived reward, which we state explicitly in the response to Reviewer nBdG.

Question 2 Excluding the selected goal from the preference head

We thank the reviewer for asking about the omission of the selected goal. The goal defines the offline comparison target, while the preference head receives the anchor representation and both candidate actions. This design keeps the auxiliary comparison objective separate from the original TD-JEPA task-conditioning pathway. The Cheetah goal-augmented run provides a direct input ablation and reaches 649.8 compared with 668.5 for the submitted heads, so explicit goal input does not improve this evaluated configuration.

AuthorsOpenReview ↗

(continued)

Weakness 2 Consistency of empirical improvements

We thank the reviewer for emphasizing consistency across domains. We use the Quadruped shuffle control and Maze2D preference-only result as the primary added evidence and report the other matched results as directional context with their uncertainty.

Additional matched resultTwin - TD95% CISeed direction
PointMass retuned τ\tau+42.4[-24.3, 110.2]4/5 +
Cheetah state 10 seeds+33.7[-1.0, 68.8]7/10 +
AntMaze large-stitch+9.0 pp[-0.4, 18.4]7/10 +

PointMass, Cheetah state, and AntMaze large-stitch all have positive matched means, with 4/5, 7/10, and 7/10 positive seed directions. Their intervals include zero, so they are not used as headline evidence. Their role is to show that the positive direction is not confined to a single benchmark, while Quadruped and Maze2D provide the clearest added support.

Question 3 Walker run replacement

We thank the reviewer for asking us to examine the Walker run replacements. We audited the three affected seed-configuration pairs. The reruns used the same nominal seeds and hyperparameter configurations as the corresponding original runs, and the matched seed set itself was unchanged. The available records contain no pre-specified exclusion criterion or documented numerical failure, corrupted checkpoint, crash, or incomplete training that would justify excluding the original runs. We therefore compared the aggregate results under both run sets to assess sensitivity to these individual training realizations.

Walker 3M aggregateTwinJEPATD-JEPAPaired Difference
30-pair with replacements771.3765.5+5.7
30-pair originals restored764.7765.5-0.8

The comparison shows that the 30-pair aggregate is sensitive to the three affected training realizations. We therefore interpret the Walker grid as a sensitivity analysis and do not rely on its positive aggregate from the rerun set as evidence of a consistent performance improvement. The paper-aligned Fixed-HP configuration is reported separately from the six-configuration aggregate, using its corresponding five-seed paired statistics. We also audited the other main domains and found no analogous run substitutions in Cheetah, Quadruped, AntMaze, Maze2D, or PointMass.

We appreciate the reviewer’s constructive comments and hope the new evidence and protocol clarifications resolve the remaining questions.

Reviewer_ffSfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to improve the representation learning for JEPA-based methods used in goal-conditioned control. The authors argue that by using pairwise preferences between actions and learning to predict the maximum reward gap between actions taken in approximately the same state, one can make the representations action-discriminative, which helps at inference time. The authors propose TwinJEPA, which incorporates these auxiliary losses into training, while making no modifications to the inference-time algorithm to eliminate any additional cost from the changes. The authors evaluate the proposed method on continuous control benchmarks (navigation and locomotion) with both state-based and pixel-based observations. In the experiments, the proposed method obtains better results on average than those of a reproduced TD-JEPA baseline.

Critical strengths
  • The proposed idea is simple and aimed at improving the representation learning, without adding any additional cost at inference time.
  • The high-level idea of utilizing pairwise action preferences for improving the training JEPA-based representations is intuitive and makes sense.
Critical weaknesses
  • I find the proposed auxiliary losses to not be well motivated. The proposed reward-gap regression is learned purely from the state. Goals do not impact it. One could interpret it as a signal of "how much do the actions matter here.", but it is not clear how that should improve the learned representation. L222 states that pairs with minimal differences are filtered, so if the rewards are binary (as often in goal-conditioned tasks, see e.g. OGBench), the reward contrast equals 0 for most pairs, and for the remaining ones, learning to predict 1 is the optimal solution. For the preference classification, we aim to predict which of the two actions is better. However, doing this without access to the goal is surprising. How can one learn to say, which action is better without information about the goal? In the conclusion, the authors state that the proposed objectives encourage the learned representation to preserve action-discriminative information. I failed to find much support for this claim in the paper.
  • The results are inconclusive and mostly within the variance. First off, many of the reported TD-JEPA results are inferior to those in the original paper (e.g. Quadruped 559.4 vs 691.4). Even then, most confidence intervals (7 of 9) in Table 1 include 0, and the smallest reported p-value is 0.0625. Unfortunately, this could be consistent with the proposed auxiliary losses not working as desired (see weakness #1 above). The ablations lack clear signal, e.g. the value of \lambda behaves in a surprisingly non-monotone fashion in Table 8. Each component alone harms the performance in quadruped (Table 16), but the combination outperforms TD-JEPA.
Questions for authors
  1. Did you try including the action and/or goal as input in the reward-regression task and if yes, what was the outcome?
  2. Did you try including the goal in the preference classification task and if yes, what was the outcome?
  3. There is an official implementation of TD-JEPA (https://github.com/facebookresearch/td_jepa), and you state that you used it. Nevertheless, the results you report are well below the published results. Do you have any intuition, what explains the gap, and in particular, what kind of hyperparameter tuning did you do?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors to Reviewer ffSf

We thank the reviewer for the thoughtful feedback and for recognizing the simplicity and intuitive motivation of TwinJEPA. We clarify the auxiliary objectives, additional input ablations, baseline reproduction, and new empirical evidence below.

Weakness 1 Auxiliary objectives and their inputs

We thank the reviewer for raising this point. The key distinction is the information available to each head and the quantity each head is trained to predict. The implemented objectives are summarized below.

HeadInputsTargetSupervision
Reward-gapZZΔr\Delta rLocal contrast magnitude
PreferenceZ,aA,aBZ,a_A,a_Byprefy_{pref}Ordering of retained pair

For the reward-gap head, squared-loss regression targets E[Δr∣Z]E[\Delta r \mid Z] on the mined-pair distribution. This encourages ZZ to retain information that predicts the magnitude of local outcome contrast around the anchor representation. The preference head receives both candidate actions and predicts the stored ordering of the retained pair. The selected relabeled goal is used when constructing the target, while the classifier input is (Z,aA,aB)(Z,a_A,a_B). We will make these input-target relationships explicit in the revision and use the same scope in the theoretical discussion.

We thank the reviewer for pointing out the binary-reward case. Maze2D provides a direct diagnostic since all 7,946 retained pairs have ∣Δr∣=1|\Delta r|=1 while the preference labels remain close to balanced at 0.510 and 0.490.

Maze2D Phase AMeanPaired comparison
TD-JEPA9.2%Reference
Full TwinJEPA8.4%-0.8 pp vs TD
Preference-only13.6%+5.2 pp vs full CI [3.2, 7.2] 5/5 +
Preference-only13.6%+4.4 pp vs TD CI [0.8, 7.6] 4/5 +
Shuffled labels10.8%Pref - shuffle +2.8 pp CI [0.0, 5.6]

Preference-only reaches 13.6%, which is +5.2 pp over full TwinJEPA with CI [3.2, 7.2] and 5/5 positive seeds, and +4.4 pp over TD-JEPA with CI [0.8, 7.6] and 4/5 positive seeds. Since the gap magnitude is constant in this environment, the result isolates the contribution of preference ordering when magnitude supervision carries no variation. The preference-only versus shuffled-label comparison is +2.8 pp with CI [0.0, 5.6], so we use the shuffled run as a control and base the component conclusion on the matched comparisons with full TwinJEPA and TD-JEPA.

To address the representation-level concern with a training-time intervention, we shuffled the stored Quadruped targets while keeping the pair buffer, architecture, optimizer, and seeds unchanged. The shuffled model reaches 546.8, which is 53.0 points below real-target TwinJEPA with CI [-72.8, -28.7], and its difference from TD-JEPA is -12.6 with CI [-37.9, 12.8]. This isolates target alignment and shows that the observed Quadruped gain depends on the correspondence between each mined pair and its comparative target.

Question 1 Action and goal inputs for reward-gap regression

We appreciate the reviewer’s suggestion to test additional inputs directly. We evaluated two auxiliary-input variants on Cheetah using five matched seeds at 1M steps.

Cheetah auxiliary inputMean return
TD-JEPA660.7
Submitted TwinJEPA heads668.5
Actions added to gap head658.7
Goal added to both heads649.8

The submitted head design has the highest mean among the tested variants. Adding the candidate actions to the gap head reaches 658.7, and adding the goal to the auxiliary pathway reaches 649.8, compared with 668.5 for the submitted heads. These results indicate that the simpler input structure is sufficient in this evaluated setting and that the additional variables do not improve downstream return.

Question 2 Goal input for preference classification

We thank the reviewer for highlighting the role of goal conditioning. In the submitted method, the selected goal is used to construct the offline pair label, and the preference head predicts that label from ZZ and the two candidate actions. The goal-augmented auxiliary configuration reaches 649.8 compared with 668.5 for the submitted heads. The implemented goal-augmented run changes the auxiliary pathway consistently across both heads, so we report it as a joint input ablation. Within this test, explicit goal conditioning does not improve the submitted configuration.

AuthorsOpenReview ↗

(continued)

Weakness 2 Statistical evidence and component sensitivity

We thank the reviewer for emphasizing statistical strength and component sensitivity. We added a target-shuffling control and report the original-tolerance comparison alongside the retuned result.

Quadruped 1MReturnPaired comparison
TD-JEPA559.4Reference
TwinJEPA real targets599.8+40.4 CI [20.6, 60.3] 5/5 +
TwinJEPA shuffled targets546.8-53.0 vs real CI [-72.8, -28.7]
Shuffle vs TD-JEPA-12.6CI [-37.9, 12.8] 3/2/0
TwinJEPA original tolerance543.9-15.5 vs TD CI includes 0

On retuned Quadruped, real-target TwinJEPA improves over TD-JEPA by +40.4 with CI [20.6, 60.3], and all five paired differences are positive. Shuffling the targets lowers return by 53.0 points relative to real-target TwinJEPA with CI [-72.8, -28.7]. The shuffled model is close to TD-JEPA, with a difference of -12.6 and CI [-37.9, 12.8]. Together, these controls separate the effect of target alignment from the extra auxiliary training path and show that the positive result is specific to the retuned matching condition.

For five matched seeds, the smallest attainable two-sided exact sign-flip value is p=0.0625p=0.0625. We therefore report effect size, bootstrap interval, and seed direction together. The λ\lambda sweep and component experiments are sensitivity analyses. Their ordering varies across environments, while the full objective is strongest on Quadruped.

Question 3 TD-JEPA reproduction and hyperparameter tuning

We thank the reviewer for calling attention to the gap between our reproduced baseline and the published TD-JEPA number. We completed a 3M-step baseline audit.

TD-JEPA Quadruped checkMean return
Published reference691.4
3M paper-matched ψ=0.1\psi=0.1548.8
3M completed setting ψ=0.01\psi=0.01622.0
Completed grid cells10/30

With the paper-matched setting ψ=0.1\psi=0.1, TD-JEPA reaches 548.8 at 3M steps, which is close to the 1M matched baseline of 559.4. A completed alternative setting with ψ=0.01\psi=0.01 reaches 622.0, showing that the reproduced implementation can obtain substantially higher return under a different setting. The grid contains 10 completed cells out of 30, so we report 622.0 as a completed alternative setting and do not treat it as an exhaustive optimum. This audit places the 1M baseline in context and supports using the matched TD-JEPA/TwinJEPA comparison for the method effect.

The primary TwinJEPA experiments use a shared Fixed-HP protocol with λ=0.1\lambda=0.1. The reported λ\lambda sweep is a sensitivity analysis. The retuned Quadruped matching tolerance was selected using an offline matching and coverage probe, not evaluation return. This protocol distinction will be stated clearly in the revision.

We thank the reviewer again for the careful assessment and hope the additional analyses and clarifications address the main concerns.

I thank the authors for their responses. Unfortunately I remain mostly unconvinced by the soundness of the proposed method and the strength of the experimental results, and the concerns about experimental validity raised by the other reviewers also seem valid. I believe the paper is not yet ready for publication.

AuthorsOpenReview ↗

Additional Maze2D Evidence on Auxiliary-Objective Contributions and Evaluation Protocol

We would like to provide further details on the Maze2D component analysis reported above, particularly regarding the distinct roles of reward-gap regression and preference classification under binary reward contrasts.

1. Component behavior under constant reward-gap magnitude.

In the additional Maze2D experiment (denoted as Phase A in our response), all 7,946 retained comparison pairs have ∣Δr∣=1|\Delta r|=1, while the preference labels remain approximately balanced (0.510/0.490). Consequently, reward-gap regression receives a constant magnitude target across the retained pairs, whereas preference classification retains a nonconstant ordering target.

Within this experiment, preference-only TwinJEPA achieves 13.6% success, compared with 8.4% for the corresponding full TwinJEPA and 9.2% for TD-JEPA. The preference-only variant differs from the Phase A full configuration only by setting the reward-gap coefficient to zero. Its improvement over the full model is +5.2 percentage points, with positive differences across all five matched seed identifiers (95% paired-bootstrap CI: [3.2, 7.2]). Relative to the corresponding TD-JEPA baseline, the difference is +4.4 percentage points (95% CI: [0.8, 7.6], 4/5 positive seeds).

These results provide additional evidence that the two auxiliary objectives contribute differently depending on the reward structure. In this particular setting, preference ordering remains useful even when the reward-gap magnitude contains no variation. This supports interpreting the two losses as complementary forms of supervision whose relative utility is domain-dependent, rather than assuming that their combination must always perform best.

2. Experimental independence and evaluation protocol.

For completeness, we also clarify that Phase A was conducted independently from the Maze2D experiments reported in the submitted paper. The submitted paper reports 14.4% success for TwinJEPA and 9.2% for TD-JEPA, and these results remain unchanged.

Both experiments used the same five nominal seed identifiers, 1M training steps, and Fixed-HP hyperparameters, but they involved separate training runs and different execution and evaluation configurations. Specifically, the submitted experiments used compile=True and shift_reward=0, whereas Phase A used compile=False and shift_reward=1.

Importantly, shift_reward is applied before reward inference and can affect the actions selected during evaluation, rather than merely changing the reported scores. The two TD-JEPA baselines also have different per-seed outcomes despite sharing the same 9.2% mean. Therefore, the two experiments should be interpreted separately, and their full-model results should not be directly compared or pooled.

3. Interpretation and scope.

The Phase A results are intended as an additional within-protocol component diagnostic, not as a replacement for the submitted Maze2D evaluation. They help distinguish the contribution of preference ordering from that of reward-gap magnitude under the specific constant-gap condition. They do not establish that preference-only supervision universally outperforms the combined objective, nor do they alter the paper's original 14.4% TwinJEPA result.

We will explicitly distinguish the two experimental configurations in the revised manuscript and report Phase A separately as supplementary component evidence.

We sincerely thank the reviewer again for the thoughtful and constructive feedback, which has helped us better characterize the contributions of the auxiliary objectives and improve the clarity and transparency of our experimental reporting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.