- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a conceptual study of whether the JEPA framework works in language reasoning. They formulate reasoning over language as a partially observed decision process. They construct synthetic decision problems: the model receives a catalogue of intent phrases that serve as the admissible actions. The model's action appends the corresponding phrase into the sentence. There are many valid phrases that are equivalent in terms of semantics. So the reasoning trajectories form a quotient set. It is natural to apply the JEPA framework to learn this equivalence relation. The authors then go on to implement LR-JEPA that slightly deviates from the original JEPA idea. Learning an energy function by contrastive learning over negative pairs. On FSA deduction, LM overshoot reaches 1.69 to 1.91 while LR-JEPA gives 1.09.
Critical strengths
- The paper presents a controlled environment so the authors can cleanly compare different approaches in it.
- The paper raises an interesting and important research question on whether reasoning in language requires the functional separation of state representation, prediction of consequences, and decision making.
Critical weaknesses
- LR-JEPA actually deviates from the general JEPA framework. I am very concerned about bringing this method to the JEPA family. It seems to me the authors borrow the JEPA name without actually following its recipe. The original JEPA is essentially training an energy-based model with compatible pairs/relations, where the energy is defined by the prediction error or similarity metrics (e.g. LeJEPA) in a latent space. Following this direction, the typical JEPAs in robotic control implements an encoder and a predictor that predicts the future latent state conditioned on actions. While, in this paper, the authors define an energy function as the predictive distance to the terminal state in a task, and learns the energy function with a contrastive learning approach (a positive sample plus several negative pairs). This is actually "counter" JEPA or non-JEPA in the sense that the general JEPA framework does not introduce a contrastive approach (JEPA learns with compatible pairs not negative pairs). I will recommend acceptance if and only if the authors can confront and answer this weakness point.
- LR-JEPA only works in synthetic and condition-controlled experiments instead of real world language reasoning tasks. It is difficult to directly extend this paper's implementation to typical language reasoning benchmarks.
- For the tasks that are closer to real world reasoning, the paper fails to prove that reasoning in language requires the functional separation of state representation, prediction of consequences, and decision making. Applying the same ranking head to a frozen LM works better than LR-JEPA. Single step search works better than increasing depths. In iGSM, LM achieves 1.05 while LR-JEPA achieves 1.45. The performance gets worse and closer to random policy under increasing search depths.
- On FSA deduction, LM overshoot reaches 1.69 to 1.91 while LR-JEPA gives 1.09. But the gap comes from counterfactual data and ranking, not from the JEPA components.
Questions for authors
- It would be interesting to discuss a bit about the relation between state-prediction separation and JEPA in language modeling. Specifically, one can refer to the paper "The State-Prediction Separation Hypothesis" by Monea et al. Their paper also studies a similar research question to your paper: separating different modeling objectives. They separate state and prediction in the token-level while your paper separates thinking and acting.
4 comments
Addendum on LR-JEPA as a standalone reasoner
You pointed out that LR-JEPA has so far only been shown in controlled settings, so we ran one more experiment in which LR-JEPA solves FSA problems without any access to the environment [new].
In the paper, the planner receives the list of applicable rules from the environment at each step. In the new experiment we instead let the model work out this list from the text it has seen so far. We take every rule stated in the problem whose premises have already been written down, either in the problem statement itself or in one of the conclusions the model has produced in earlier steps, in a similar way to H-JEPA (Zhang et al., 2026), which also does not take its candidate actions from the environment but derives them from the actions seen in training. LR-JEPA then chooses among these rules by goal distance, as in the paper, writes down the chosen rule together with its conclusion, and repeats this until it has written the target. We only use the environment once the model has finished, in order to check that every step of the proof is valid and that the proof reaches the target.
With this procedure all proofs are valid and reach the target, both on problems of the training length (ID) and on longer problems (OOD). The overshoot, which is the number of steps divided by the length of the shortest solution, is also the same as with the environment.
| FSA, overshoot | ID | OOD |
|---|---|---|
| LR-JEPA with the environment | 1.84 | 1.83 |
| LR-JEPA without the environment [new] | 1.84 | 1.83 |
| Random choice among the same candidates [new] | 2.03 | 2.02 |
This shows that LR-JEPA can be used on its own in the same way as a language model, reading a problem and writing out the full solution.
In addition, we trained LR-JEPA with the same objective on plain text (FineWeb-Edu, Appendix D). There it costs 2.3 points of perplexity, and when we use its world model to pick among possible continuations, the continuations it picks come closer to a given goal than those picked by any of the baselines (unigram F1 0.212, compared with 0.152 for a random pick). Taken together, the two experiments show that the world model can be learned from text and that planning with it does not require an environment. We will add the new experiment to the appendix.
If any concern remains, please let us know, and we are happy to run further experiments during the discussion period. If our replies have addressed your concerns, we would appreciate it if you would consider updating your score and confidence accordingly.
2nd Addendum on the impact of search and length generalization
You noted that single-step search works as well as deeper search. On the original test bands this was the case, because the one-step choice was already close to the shortest solution, which left a deeper search little room to improve. We therefore evaluated the already trained models, without retraining, on harder iGSM problems with 17 to 24 operations and wider dependency graphs, which lie beyond the training lengths (steps per necessary step, 1.0 is the shortest solution, lower is better):
| Planner | depth 1 | depth 2 | depth 4 | depth 8 |
|---|---|---|---|---|
| Frozen LM + value head | 2.07 +- 0.04 | 2.08 +- 0.04 | 2.09 +- 0.05 | 2.09 |
| LR-JEPA, oracle goal | 2.12 +- 0.01 | 1.96 +- 0.11 | 1.88 +- 0.14 | 1.96 +- 0.07 |
| LR-JEPA, TD head | 2.01 +- 0.03 | 1.85 +- 0.07 | 1.81 +- 0.05 | 1.78 +- 0.10 |
Increasing the search depth from 1 to 8 removes about a fifth of the steps beyond the shortest solution. Since a value head on a frozen language model does not benefit from the same search, the improvement can be attributed to the LR-JEPA latent dynamics rather than to the search procedure itself.
On FSA, planning towards the goal state benefits from depth up to 32 on problems beyond the training lengths of 6 to 10 rules:
| FSA, oracle goal | depth 1 | depth 2 | depth 4 | depth 8 | depth 16 | depth 32 |
|---|---|---|---|---|---|---|
| 11 to 16 rules | 1.91 | 1.77 | 1.72 | 1.69 | 1.57 | 1.39 |
| 17 to 24 rules | 1.81 | 1.87 | 1.81 | 1.76 | 1.71 | 1.59 |
A language model has no state that it can roll forward in this way, and this is where a latent world model has an advantage over a one-step policy, even a strong one. We will add these results to the appendix.
We hope this addresses your concern. If it does, we would be grateful if you would consider raising your score.
As the discussion period ends soon, we would be grateful to hear whether our responses and the revised PDF (changes marked in blue) address the reviewers' concerns, and which points remain open, so that we can still respond to them.
Thank you for the detailed review and for stating clearly what would change your recommendation. Your main point made us rethink how the paper presents its contribution, and the general response summarizes the resulting changes.
We agree that the energy head of the submission is trained contrastively. We added it because we believed a first study had to present a complete reasoning system, but the question is posed more cleanly in the line of LeJEPA and LeWM, where a JEPA is trained only to predict the next latent state and plans toward the encoding of a goal. We therefore removed all energy heads and retrained the model with smooth-l1 prediction and SIGReg as the only losses, with no contrastive term and no EMA teacher, the target being the encoding of the observed consequence by the same encoder and the goal being given as an observation, as LeWM receives a goal image [new]. Planning by distance to this goal without any learned cost, LR-JEPA reaches 1.83 OOD on FSA against 2.01 for a random policy and 1.96 for the same network at initialization, and 1.62 on iGSM against 1.74 and 1.80, while removing either loss brings it back to the untrained level on both domains (general response, Table A). Its imagined states stay closest to the true successor among sibling branches in 76 to 96 percent of cases up to eight steps on FSA, against 31 to 46 percent without the prediction loss, and without SIGReg the encoder collapses to an effective rank of 1.01 (Section 6, Appendix C).
The energy heads are now optional cost modules in the sense of LeCun (2022), trained on top of this backbone. A head trained by temporal-difference learning reaches 1.00 ID and 1.04 to 1.05 OOD at every depth from 1 to 8 (Table B) [new]. The method section and Figure 1 now present LR-JEPA as a JEPA world model for reasoning in language, with the heads as optional modules that read its latent states.
We tested both [new]. A language model trained on the serialized counterfactual stream gets 1.00 ID but 2.04 OOD (Table 6), and the temporal-difference head trained on the frozen JEPA backbone, which never saw a ranking label, reaches 1.15 OOD against 1.48 on a frozen random-init backbone, so the gap lies in the state the policy reads. The hidden state of the language models predicts the number of remaining steps with R^2 0.99 ID but only 0.39 OOD, and although a linear probe still finds a correct step OOD at 0.976 top-1, their likelihood readout imitates gold solutions of the trained lengths and plans at about 1.9. LR-JEPA is trained to predict the consequence of every step, so its latent keeps tracking the problem state beyond the trained lengths, and the planner chooses steps by these imagined consequences (Appendix C).
The frozen-LM ranking head is a strong one-step policy on FSA (1.01 OOD), and we report it as such. The claim of the paper concerns lookahead, where a language model has no state to roll forward. A state-value head on frozen-LM states goes from 1.57 at depth 1 to 2.03 at depth 8 and a semantic-world-model LM stays at 1.78 to 1.85, while LR-JEPA with the TD head keeps 1.04 to 1.05 (Table B, manuscript Table 3) [new].
Synthetic domains give exact shortest solutions and test problems longer than any training problem, a choice V-JEPA 2-AC and LeWM also make. As a first test on real text, a 124M LR-JEPA trained on FineWeb-Edu chooses among 16 sampled continuations the one closest to the goal in imagined end state with a unigram F1 of 0.212 against the true text, where a random pick gets 0.152 and the most likely sample 0.123 (Appendix D) [new].
Thank you for this pointer. We ask the same question at the level of reasoning steps, and our probes are in line with it, since the separated JEPA state keeps tracking the problem where the language model state does not, and it can be rolled forward without generating text. We added this to the related work (Section 2).
Further changes, marked in blue: the world-model probes (Appendix C, Table 7), the TD sweep and the component ablations (Appendix C), and the natural-text experiment (Appendix D).
If there is any further evidence we could provide to address your concerns, please let us know.