acceptodds
Under review as a conference paper at ICLR 2027

Language-Reasoning JEPA: Searching Imagined Futures over Language

Abstract

A language model reasons by generating text about the future with the same likelihood readout that chooses its actions, and it has no separate mechanism for imagining what an action would do. Action-conditioned joint-embedding predictive architectures (JEPAs) supply such a mechanism in robotic control, and we study whether the same construction supports reasoning over language. We read a sentence as a partial observation of a controlled process and a derivation step as an intervention, whose observed consequence is the supervision a JEPA is trained to predict. Transition prediction determines the dynamics of the latent space but leaves its metric free, so we add a goal-conditioned energy head that we train only by ranking outcomes the environment has executed, and we plan by model-predictive control over imagined latent trajectories. We call the model Language-Reasoning JEPA (LR-JEPA) and evaluate it in two synthetic environments rendered entirely in language, where every policy selects its next step from the same menu of admissible actions and we measure overshoot, the number of executed actions relative to the shortest solution. On chain-structured deduction, language models trained on the same trajectories are optimal inside the trained lengths and reach overshoot to beyond them, where a random policy reaches . The same ranking head on the frozen language model reaches at search depth one and falls to the random level by depth four, because a language model cannot roll its state forward and appending the true consequences of imagined steps to its context does not restore search reliably, while LR-JEPA improves from to over those depths. On dependency graphs the search degrades with depth, which we trace to the drift of imagined trajectories.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World Models, Language Models, JEPA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_axwsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a conceptual study of whether the JEPA framework works in language reasoning. They formulate reasoning over language as a partially observed decision process. They construct synthetic decision problems: the model receives a catalogue of intent phrases that serve as the admissible actions. The model's action appends the corresponding phrase into the sentence. There are many valid phrases that are equivalent in terms of semantics. So the reasoning trajectories form a quotient set. It is natural to apply the JEPA framework to learn this equivalence relation. The authors then go on to implement LR-JEPA that slightly deviates from the original JEPA idea. Learning an energy function by contrastive learning over negative pairs. On FSA deduction, LM overshoot reaches 1.69 to 1.91 while LR-JEPA gives 1.09.

Critical strengths
  1. The paper presents a controlled environment so the authors can cleanly compare different approaches in it.
  2. The paper raises an interesting and important research question on whether reasoning in language requires the functional separation of state representation, prediction of consequences, and decision making.
Critical weaknesses
  1. LR-JEPA actually deviates from the general JEPA framework. I am very concerned about bringing this method to the JEPA family. It seems to me the authors borrow the JEPA name without actually following its recipe. The original JEPA is essentially training an energy-based model with compatible pairs/relations, where the energy is defined by the prediction error or similarity metrics (e.g. LeJEPA) in a latent space. Following this direction, the typical JEPAs in robotic control implements an encoder and a predictor that predicts the future latent state conditioned on actions. While, in this paper, the authors define an energy function as the predictive distance to the terminal state in a task, and learns the energy function with a contrastive learning approach (a positive sample plus several negative pairs). This is actually "counter" JEPA or non-JEPA in the sense that the general JEPA framework does not introduce a contrastive approach (JEPA learns with compatible pairs not negative pairs). I will recommend acceptance if and only if the authors can confront and answer this weakness point.
  2. LR-JEPA only works in synthetic and condition-controlled experiments instead of real world language reasoning tasks. It is difficult to directly extend this paper's implementation to typical language reasoning benchmarks.
  3. For the tasks that are closer to real world reasoning, the paper fails to prove that reasoning in language requires the functional separation of state representation, prediction of consequences, and decision making. Applying the same ranking head to a frozen LM works better than LR-JEPA. Single step search works better than increasing depths. In iGSM, LM achieves 1.05 while LR-JEPA achieves 1.45. The performance gets worse and closer to random policy under increasing search depths.
  4. On FSA deduction, LM overshoot reaches 1.69 to 1.91 while LR-JEPA gives 1.09. But the gap comes from counterfactual data and ranking, not from the JEPA components.
Questions for authors
  1. It would be interesting to discuss a bit about the relation between state-prediction separation and JEPA in language modeling. Specifically, one can refer to the paper "The State-Prediction Separation Hypothesis" by Monea et al. Their paper also studies a similar research question to your paper: separating different modeling objectives. They separate state and prediction in the token-level while your paper separates thinking and acting.
4 comments

Thank you for the detailed review and for stating clearly what would change your recommendation. Your main point made us rethink how the paper presents its contribution, and the general response summarizes the resulting changes.

LR-JEPA actually deviates from the general JEPA framework. ... This is actually "counter" JEPA or non-JEPA ... I will recommend acceptance if and only if the authors can confront and answer this weakness point.

We agree that the energy head of the submission is trained contrastively. We added it because we believed a first study had to present a complete reasoning system, but the question is posed more cleanly in the line of LeJEPA and LeWM, where a JEPA is trained only to predict the next latent state and plans toward the encoding of a goal. We therefore removed all energy heads and retrained the model with smooth-l1 prediction and SIGReg as the only losses, with no contrastive term and no EMA teacher, the target being the encoding of the observed consequence by the same encoder and the goal being given as an observation, as LeWM receives a goal image [new]. Planning by distance to this goal without any learned cost, LR-JEPA reaches 1.83 OOD on FSA against 2.01 for a random policy and 1.96 for the same network at initialization, and 1.62 on iGSM against 1.74 and 1.80, while removing either loss brings it back to the untrained level on both domains (general response, Table A). Its imagined states stay closest to the true successor among sibling branches in 76 to 96 percent of cases up to eight steps on FSA, against 31 to 46 percent without the prediction loss, and without SIGReg the encoder collapses to an effective rank of 1.01 (Section 6, Appendix C).

The energy heads are now optional cost modules in the sense of LeCun (2022), trained on top of this backbone. A head trained by temporal-difference learning reaches 1.00 ID and 1.04 to 1.05 OOD at every depth from 1 to 8 (Table B) [new]. The method section and Figure 1 now present LR-JEPA as a JEPA world model for reasoning in language, with the heads as optional modules that read its latent states.

On FSA deduction ... the gap comes from counterfactual data and ranking, not from the JEPA components.

We tested both [new]. A language model trained on the serialized counterfactual stream gets 1.00 ID but 2.04 OOD (Table 6), and the temporal-difference head trained on the frozen JEPA backbone, which never saw a ranking label, reaches 1.15 OOD against 1.48 on a frozen random-init backbone, so the gap lies in the state the policy reads. The hidden state of the language models predicts the number of remaining steps with R^2 0.99 ID but only 0.39 OOD, and although a linear probe still finds a correct step OOD at 0.976 top-1, their likelihood readout imitates gold solutions of the trained lengths and plans at about 1.9. LR-JEPA is trained to predict the consequence of every step, so its latent keeps tracking the problem state beyond the trained lengths, and the planner chooses steps by these imagined consequences (Appendix C).

... Applying the same ranking head to a frozen LM works better than LR-JEPA. Single step search works better than increasing depths. ...

The frozen-LM ranking head is a strong one-step policy on FSA (1.01 OOD), and we report it as such. The claim of the paper concerns lookahead, where a language model has no state to roll forward. A state-value head on frozen-LM states goes from 1.57 at depth 1 to 2.03 at depth 8 and a semantic-world-model LM stays at 1.78 to 1.85, while LR-JEPA with the TD head keeps 1.04 to 1.05 (Table B, manuscript Table 3) [new].

LR-JEPA only works in synthetic and condition-controlled experiments instead of real world language reasoning tasks.

Synthetic domains give exact shortest solutions and test problems longer than any training problem, a choice V-JEPA 2-AC and LeWM also make. As a first test on real text, a 124M LR-JEPA trained on FineWeb-Edu chooses among 16 sampled continuations the one closest to the goal in imagined end state with a unigram F1 of 0.212 against the true text, where a random pick gets 0.152 and the most likely sample 0.123 (Appendix D) [new].

... "The State-Prediction Separation Hypothesis" by Monea et al.

Thank you for this pointer. We ask the same question at the level of reasoning steps, and our probes are in line with it, since the separated JEPA state keeps tracking the problem where the language model state does not, and it can be rolled forward without generating text. We added this to the related work (Section 2).

Further changes, marked in blue: the world-model probes (Appendix C, Table 7), the TD sweep and the component ablations (Appendix C), and the natural-text experiment (Appendix D).

If there is any further evidence we could provide to address your concerns, please let us know.

AuthorsOpenReview ↗

Addendum on LR-JEPA as a standalone reasoner

You pointed out that LR-JEPA has so far only been shown in controlled settings, so we ran one more experiment in which LR-JEPA solves FSA problems without any access to the environment [new].

In the paper, the planner receives the list of applicable rules from the environment at each step. In the new experiment we instead let the model work out this list from the text it has seen so far. We take every rule stated in the problem whose premises have already been written down, either in the problem statement itself or in one of the conclusions the model has produced in earlier steps, in a similar way to H-JEPA (Zhang et al., 2026), which also does not take its candidate actions from the environment but derives them from the actions seen in training. LR-JEPA then chooses among these rules by goal distance, as in the paper, writes down the chosen rule together with its conclusion, and repeats this until it has written the target. We only use the environment once the model has finished, in order to check that every step of the proof is valid and that the proof reaches the target.

With this procedure all proofs are valid and reach the target, both on problems of the training length (ID) and on longer problems (OOD). The overshoot, which is the number of steps divided by the length of the shortest solution, is also the same as with the environment.

FSA, overshootIDOOD
LR-JEPA with the environment1.841.83
LR-JEPA without the environment [new]1.841.83
Random choice among the same candidates [new]2.032.02

This shows that LR-JEPA can be used on its own in the same way as a language model, reading a problem and writing out the full solution.

In addition, we trained LR-JEPA with the same objective on plain text (FineWeb-Edu, Appendix D). There it costs 2.3 points of perplexity, and when we use its world model to pick among possible continuations, the continuations it picks come closer to a given goal than those picked by any of the baselines (unigram F1 0.212, compared with 0.152 for a random pick). Taken together, the two experiments show that the world model can be learned from text and that planning with it does not require an environment. We will add the new experiment to the appendix.

If any concern remains, please let us know, and we are happy to run further experiments during the discussion period. If our replies have addressed your concerns, we would appreciate it if you would consider updating your score and confidence accordingly.

AuthorsOpenReview ↗

2nd Addendum on the impact of search and length generalization

You noted that single-step search works as well as deeper search. On the original test bands this was the case, because the one-step choice was already close to the shortest solution, which left a deeper search little room to improve. We therefore evaluated the already trained models, without retraining, on harder iGSM problems with 17 to 24 operations and wider dependency graphs, which lie beyond the training lengths (steps per necessary step, 1.0 is the shortest solution, lower is better):

Plannerdepth 1depth 2depth 4depth 8
Frozen LM + value head2.07 +- 0.042.08 +- 0.042.09 +- 0.052.09
LR-JEPA, oracle goal2.12 +- 0.011.96 +- 0.111.88 +- 0.141.96 +- 0.07
LR-JEPA, TD head2.01 +- 0.031.85 +- 0.071.81 +- 0.051.78 +- 0.10

Increasing the search depth from 1 to 8 removes about a fifth of the steps beyond the shortest solution. Since a value head on a frozen language model does not benefit from the same search, the improvement can be attributed to the LR-JEPA latent dynamics rather than to the search procedure itself.

On FSA, planning towards the goal state benefits from depth up to 32 on problems beyond the training lengths of 6 to 10 rules:

FSA, oracle goaldepth 1depth 2depth 4depth 8depth 16depth 32
11 to 16 rules1.911.771.721.691.571.39
17 to 24 rules1.811.871.811.761.711.59

A language model has no state that it can roll forward in this way, and this is where a latent world model has an advantage over a one-step policy, even a strong one. We will add these results to the appendix.

We hope this addresses your concern. If it does, we would be grateful if you would consider raising your score.

As the discussion period ends soon, we would be grateful to hear whether our responses and the revised PDF (changes marked in blue) address the reviewers' concerns, and which points remain open, so that we can still respond to them.

Reviewer_GyjzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the action-conditioned JEPA recipe used for robot planning can drive step-by-step reasoning in text. A sentence is treated as a partial observation and a derivation step, drawn from a fixed catalogue of intent phrases, as an action. LR-JEPA trains an encoder and a predictor to predict the encoded consequence of executed and counterfactual actions, plus a small energy head trained only to rank executed outcomes. At test time a receding-horizon beam search scores imagined latent rollouts with this head and executes the first action. Two propositions argue that transition prediction alone does not order states by distance to the goal, and that a correct local order is enough for greedy planning. Experiments use two synthetic, templated environments, FSA chain deduction and iGSM-style dependency graphs. Every policy picks from the same menu of feasible actions and is scored by overshoot, executed steps relative to the shortest solution, on problems longer than those seen in training. On FSA, LR-JEPA beats language models that pick actions by likelihood and improves with search depth, while the same head on a frozen LM is best at depth 1 and gets worse with depth. On iGSM, deeper search makes LR-JEPA worse.

Critical strengths
  1. The evaluation is careful and the controls are good. All policies choose from the same shuffled menu of feasible actions, so the comparison measures action selection rather than generation errors, and random and first-feasible levels anchor it. The paper runs most of the controls I would have asked for: the same head on a frozen LM, TD heads, latent-distance readouts, looped and sentence-level LMs, larger LMs, and a FLOP axis. It also reports results that cut against its own story, such as the iGSM depth inversion. So I trust the numbers, including the negative ones.

  2. As a first attempt at an action-conditioned JEPA over text, it gives one clear contrast on FSA. With the same ranking-trained head and the same beam, LR-JEPA's out-of-distribution overshoot falls with depth, while the head on frozen LM features climbs to the random level. The probes add a second point: the LM's failure beyond the trained lengths sits in the likelihood readout, not the features, since a probe or a ranking head on the same features reaches 1.01 to 1.03. Both results matter to anyone working on latent world models or on how LMs choose actions.

Critical weaknesses
  1. The main causal claim is not isolated, and one of the paper's own controls points against the stated reason. The abstract says the frozen-LM head fails with depth "since a language model cannot roll its state forward". But Table 4 also rolls the LM forward with the true consequence after every imagined step, and the head still climbs to the random level with depth. The paper reports this control, yet the abstract keeps the original reason. The two systems also differ in more than the predictive state. The LM backbone saw only gold shortest paths and is frozen, while LR-JEPA's backbone is trained jointly with the ranking losses on executed off-path branches. Off-path histories are out of distribution for the LM, which could explain its useless deeper states. No ablation removes the prediction loss. LR-JEPA with search also does not beat the simplest policy with the same kind of supervision: the frozen-LM head at depth 1 gets 1.03 out of distribution against 1.09 for the 10M LR-JEPA, with fewer FLOPs.

  2. The evidence that latent search helps is narrow and fragile, which limits the impact. Apart from one depth-8 point on near-chain graphs, the depth gain shows up only on FSA, where each rule states its own conclusion, so one-step imagination is easy by design. On iGSM, where the consequence has to be computed, deeper search makes LR-JEPA worse, nearly to the random level, and LR-JEPA trails both the plain LM and the looped LM. Even on FSA, the depth gain hangs on one training recipe: the wide-batch default and the 90M model at the default learning rate get worse with depth. The depth-1 reference has a large standard deviation, and the depth-8 cell uses n=10. The setting also strips out much of what makes real language reasoning hard: a fixed action menu, an oracle executor for counterfactuals, templated text, and small models. I see this as good early work, but from these results it is hard to tell how much it will matter for reasoning in real language.

Questions for authors
  1. (Clarification) Algorithm 1 and the setup section say that deeper imagined steps are drawn from the root menu of feasible actions. On FSA the next gold rule is usually not feasible at the root, since its antecedent comes from the current gold rule, so no depth-d beam can follow the gold chain past step 1. What is the typical menu size, what happens when d exceeds it, and if depth doesn't look ahead along the solution, what does it add?
  2. (Suggestion) To back the claim that the predictive state is what turns test-time compute into better plans, please add: LR-JEPA with the prediction loss removed; a JEPA backbone trained without the ranking losses, then frozen, with the same head trained afterwards, which mirrors the frozen-LM baseline; and an LM trained on the serialized counterfactual stream, or fine-tuned with the same ranking and horizon losses, then searched with oracle consequences under the same beam.
  3. (Clarification) What do the ± values measure, and are they training seeds for every row, including the n=10 and n=20 cells? Please give confidence intervals for the two comparisons the paper rests on: LR-JEPA at depth 1 against depths 4 and 8, and LR-JEPA against the frozen-LM head at depth 1.
  4. (Clarification) Overshoot is computed on solved episodes only, and the text says that every menu policy, including random, succeeds within the budget. Please back this up with solve rates for every cell, especially iGSM, the TD rows, the quasimetric head and the distance readouts.
  5. (Suggestion) The drift explanation for iGSM could be tested directly: report the predictor's latent error against the EMA encoding of the true successor as a function of depth and graph width, or run the iGSM beam on oracle latents. If oracle latents remove the inversion, drift is the cause; if not, the head or the beam is the more likely culprit. A related question: on iGSM, do equally good topological orders get negative ranking labels when they differ from the demonstrated order?
  6. (Clarification) The appendix calls the main depth grid the "learned-embedding rows", while elsewhere it says both model families use rotary embeddings, and a rotary run gives slightly different numbers. Which run is the main-text model, and how did you choose it?
  7. (Suggestion) The appendix reports gold-action identification at AUC 1.0 even from a randomly initialized encoder. Does the ranking head on a random-init encoder also reach overshoot near 1.0 out of distribution at depth 1? If so, FSA step selection may be solvable from local cues once ranking labels are given, and the LM vs JEPA gap would mostly come down to the training signal.
  8. (Suggestion) Semantic World Models (arXiv 2510.19818) should be discussed and ideally compared. It fine-tunes a vision-language model to answer questions about the state reached after an action sequence and plans by maximizing the likelihood of the desired answers, so it is an action-conditioned world model built on a vision-language model and speaks directly to the claim that a language model cannot roll its state forward. An SWM-style baseline would fit your setup, and peer-reviewed work that uses LMs as world models for planning, such as SWAP (ACL 2025) and Web Agents with World Models (ICLR 2025), should also be discussed.
5 comments

Part 1 of 2

Thank you for a careful and constructive review. Your central question was whether the gains of LR-JEPA come from the world model or from what surrounds it, namely the ranking losses, the off-path data and the joint training of backbone and head. The original design made this hard to answer, so we changed the design. The revised LR-JEPA is trained with prediction and SIGReg as its only losses, as in LeJEPA and LeWM, and energy heads are optional cost modules trained afterwards on the frozen backbone. This allowed us to run the experiments you asked for directly, and they changed how we explain the results. All changes in the manuscript are marked in blue, and [new] marks results obtained since the reviews.

... No ablation removes the prediction loss. ... please add ... a JEPA backbone trained without the ranking losses, then frozen ...; and an LM trained on the serialized counterfactual stream ...

We ran all three experiments with the new backbone, which sees no ranking label during training. Planning by goal distance with no head at all, it reaches 1.83 OOD on FSA [new]. Removing the prediction loss gives 1.95, removing SIGReg gives 1.91, and the same network at initialization gives 1.96, so the planner needs both losses to move away from the untrained level.

We then froze this backbone and trained only a TD head on top, the same protocol as for the frozen LM. It reaches 1.15 OOD at depth 1, against 1.48 for the same head on a frozen random-init backbone [new]. Since the backbone never saw a ranking label, most of the gain comes from the representation that prediction and SIGReg learn.

An LM trained on the serialized counterfactual stream, which contains the same off-path data, reaches 1.00 ID and 2.04 OOD [new]. Off-path data alone therefore does not produce the generalization.

Linear probes on the two models locate the difference in the state. The LM state predicts the number of remaining steps with R^2 0.99 ID and 0.39 OOD, so within the training lengths the LM tracks how far it is from the solution and beyond them it largely loses track. The imagined JEPA state, rolled forward without executing any step, identifies the true successor among sibling branches in 76 to 96 percent of cases (Appendix C). We use this difference to explain the remaining results, since a planner that compares imagined futures needs states that keep encoding progress beyond the training lengths, which the JEPA states do and the LM states largely do not.

The abstract says the frozen-LM head fails with depth "since a language model cannot roll its state forward" ...

You are right that the setting of Table 4, where the true consequences are appended to the LM context, contradicts that sentence. We tested the alternative explanation with a state-value head on frozen-LM states [new]. It also moves to the random level with depth (OOD 1.57 at depth 1, 2.03 at depth 8), and appending the true consequences does not prevent this. Since the true future states are supplied in this setting, the failure lies in how the LM represents them, which agrees with the probe above. The abstract now states the finding in these terms.

... the frozen-LM head at depth 1 gets 1.03 out of distribution against 1.09 for the 10M LR-JEPA ...

On 100 test problems the frozen-LM ranking head gets 1.01 OOD at depth 1, and the revision reports it as a strong one-step baseline. This result is consistent with the probes, because scoring the next step only needs the LM state at the current position, while lookahead needs a judgement of states several steps ahead, and there heads on LM states break down. The state-value head on a frozen LM goes to 2.03 at depth 8, and an SWM-style LM stays at 1.78 to 1.85 at every depth [new]. LR-JEPA with the TD head keeps 1.04 to 1.05 OOD at every depth (Table B). The claim of the paper concerns this lookahead regime, and the revision states it that way.

Part 2 of 2

... the depth gain shows up only on FSA ... Even on FSA, the depth gain hangs on one training recipe ... the depth-8 cell uses n=10.

We agree with the first point, and the revision states the depth result for FSA only. On FSA all cells now use 100 test problems, and the TD head reaches 1.00 ID at every depth, so the result no longer depends on the one ranking recipe you pointed to.

On iGSM the world model still plans better than the baselines without step supervision, reaching 1.62 by goal distance against 1.74 to 1.81 for the random policy and the untrained network, and it again needs both losses. The language models reach 1.16 on iGSM, but they are trained on gold solutions with supervision on every step, which LR-JEPA never receives. What fails on iGSM is the gain from depth with the ranking head, and the test you proposed below traces it to the ranking labels.

The drift explanation for iGSM could be tested directly ... do equally good topological orders get negative ranking labels?

We ran the drift test by rolling the predictor forward along OOD trajectories without executing any step [new]. On iGSM the rollout error is 0.11 to 0.39, against 0.74 to 1.17 without the prediction loss, and on FSA the rollout error stays below the distance between sibling branches up to eight steps. Drift is therefore too small to explain the iGSM depth profile, and we removed that explanation. The data agree better with your second suggestion, as hindsight ranking treats every other feasible step as a negative, so equally good orders do receive negative labels. On identical checkpoints the effect of depth follows the width of the dependency graph, with depth eight below depth one on near-chains (1.39) and the harm growing as graphs widen. Wider graphs admit more equally good orders and hence more wrong negatives, and a deeper search accumulates their effect. Section 7 now attributes the iGSM depth profile to these labels, which the goal-distance planner does not use.

The setting also strips out much of what makes real language reasoning hard ...

We agree that the setting is simplified, and we chose it on purpose for a first study, as V-JEPA 2-AC and LeWM did. Synthetic domains give exact shortest solutions, so overshoot is measured exactly, and they allow test problems longer than any training problem, which is where LM and JEPA states differ. The fixed menu keeps the quality of a step proposer out of the comparison. To check that the objective is not tied to templated text, we trained a 124M LR-JEPA on FineWeb-Edu with the same objective [new]. Given a held-out next sentence as the goal, it chooses among 16 continuations sampled from a language model the one with the closest imagined end state. The chosen continuation reaches a unigram F1 of 0.212 with the true text, against 0.152 for a random pick and 0.123 for the most likely sample (Appendix D).

What is the typical menu size, what happens when d exceeds it ...?

The menu holds 10.8 applicable rules on average ID and 16.2 OOD. In the revised planner later steps are drawn from those admissible after the preceding ones, so the remaining derivation bounds depth and the menu size never does. Depth lets the planner check whether a step taken now keeps useful steps available later, which a one-step planner cannot see.

What do the +/- values measure ...? ... solve rates for every cell ...

They are standard deviations over training runs, for every row. For example, goal distance gives 1.83 +- 0.02 on FSA OOD against 1.91 +- 0.01 without SIGReg. Every cell with the feasible-menu interface solves all test problems, so overshoot is never averaged over a subset.

Which run is the main-text model, and how did you choose it?

The main model is the 10M backbone trained with prediction and SIGReg, for 3000 steps on FSA and 12.5k steps on iGSM. Head hyperparameters were chosen on 10 ID problems, and all reported numbers come from fresh runs after this selection. The sentence about rotary embeddings was wrong and is corrected.

Does the ranking head on a random-init encoder also reach overshoot near 1.0 ...?

With predictor and head trained, it reaches 1.35 OOD at depth 1 [new], so a trainable predictor and head recover part of the structure from random features. With the backbone frozen, which isolates the representation, the TD head reaches 1.15 on the LR-JEPA backbone against 1.48 on the random one.

Semantic World Models ... SWAP ... Web Agents with World Models ...

We thank you for these references, which are now discussed in Section 2. All three predict future states in text, while LR-JEPA predicts them in latent space, which keeps lookahead cheap and lets the planner compare futures by distance. As a direct comparison we added an SWM-style LM baseline under the same search, which reaches 1.78 to 1.85 OOD on FSA (Table 3) [new].

If there is any further evidence we could provide to address your concerns, please let us know.

AuthorsOpenReview ↗

Addendum on planning without the action menu

You pointed out that our setting relies on a fixed action menu, so we ran one more experiment in which LR-JEPA solves FSA problems without any access to the environment [new].

In the paper, the planner receives the list of applicable rules from the environment at each step. In the new experiment we instead let the model work out this list from the text it has seen so far. We take every rule stated in the problem whose premises have already been written down, either in the problem statement itself or in one of the conclusions the model has produced in earlier steps, in a similar way to H-JEPA (Zhang et al., 2026), which also does not take its candidate actions from the environment but derives them from the actions seen in training. LR-JEPA then chooses among these rules by goal distance, as in the paper, writes down the chosen rule together with its conclusion, and repeats this until it has written the target. We only use the environment once the model has finished, in order to check that every step of the proof is valid and that the proof reaches the target.

With this procedure all proofs are valid and reach the target, both on problems of the training length (ID) and on longer problems (OOD). The overshoot, which is the number of steps divided by the length of the shortest solution, is also the same as with the environment.

FSA, overshootIDOOD
LR-JEPA with the environment1.841.83
LR-JEPA without the environment [new]1.841.83
Random choice among the same candidates [new]2.032.02

This shows that the planning results do not depend on the action menu that the environment provides.

In addition, we trained LR-JEPA with the same objective on plain text (FineWeb-Edu, Appendix D). There it costs 2.3 points of perplexity, and when we use its world model to pick among possible continuations, the continuations it picks come closer to a given goal than those picked by any of the baselines (unigram F1 0.212, compared with 0.152 for a random pick). Taken together, the two experiments show that the world model can be learned from text and that planning with it does not require an environment. We will add the new experiment to the appendix.

If any concern remains, please let us know, and we are happy to run further experiments during the discussion period. If our replies have addressed your concerns, we would appreciate it if you would consider updating your score and confidence accordingly.

Reviewer_GNi8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LR-JEPA, which borrows the JEPA idea to the language domain. The proposed model combines consequence prediction with a goal-conditioned ranking head to guide receding-horizon search over reasoning steps. Experiments on two synthetic environments investigate when latent search improves length generalization and when it degrades on more complex dependency structures.

Critical strengths
  • A novel and interesting framework that separates predicting action consequences from evaluating their usefulness.
  • Extensive ablations and diagnostic comparisons help isolate the roles of ranking supervision, predictive representations, and search depth.
Critical weaknesses
  • The evaluation is limited to two synthetic, deterministic environments with predefined action menus. Although these settings enable controlled analysis, they omit important challenges in real-world language reasoning, such as proposing actions, handling ambiguous observations, and evaluating uncertain consequences. Without experiments on realistic tasks or applications, the method’s practical effectiveness and generalizability remain unclear.
  • If I understand correctly, in applications the method assumes access to an externally supplied, finite set of feasible candidate actions at each decision step.
Questions for authors

If I understand correctly, in applications the method assumes access to an externally supplied, finite set of feasible candidate actions at each decision step. I would like to know the authors' insights on how this work can be extended to wide language reasoning applications.

4 comments

Thank you for the review and for the question about how the method extends to open-ended language reasoning. The general response summarizes the changes, which are marked in blue in the manuscript.

The evaluation is limited to two synthetic, deterministic environments with predefined action menus. ... Without experiments on realistic tasks or applications, the method's practical effectiveness and generalizability remain unclear.

We chose synthetic domains because an oracle gives the shortest solution and test problems can be longer than any training problem, so we can measure exactly whether a latent world model over sentences supports planning, a choice V-JEPA 2-AC and LeWM also make for video and control. Within them we test generalization on problems longer than any seen in training, and at fixed step budgets of 1.25 to 1.5 times the shortest solution on such problems LR-JEPA solves at least 90 percent of FSA problems, where a 50M language model trained on the same text solves 5 to 16 percent (Section 6).

For the revision we also trained a new 124M LR-JEPA on natural text, FineWeb-Edu, with the same objective, as a first test outside synthetic domains [new]. Through a language-model head its perplexity is 40.6, against 38.3 for a language model of the same size, so the predictive objective costs little language modeling. We then gave it the encoding of a held-out next sentence as the goal and let it choose among 16 continuations sampled from a fixed language model, none of them the true one, by the distance of the imagined end state to the goal. The chosen continuation lies at 0.16 of the mean candidate distance from the goal and reaches a unigram F1 of 0.212 with the true text, against 0.152 for a random pick. The same selection with a random-init model (0.136) or a model trained without SIGReg (0.148) stays near the random level, and the sample the language model itself finds most likely falls below it (0.123), so the gain comes from the learned world model, which tells candidates apart by where they lead (Appendix D).

If I understand correctly, in applications the method assumes access to an externally supplied, finite set of feasible candidate actions at each decision step. I would like to know the authors' insights on how this work can be extended to wide language reasoning applications.

Your understanding is correct, and the consequences of each candidate, including those of later steps in a candidate sequence, are imagined by the predictor without an executor. Since the planner only requires a set of candidates, the menu can be replaced by a proposal mechanism, as in the FineWeb-Edu experiment above, where a language model proposes and LR-JEPA evaluates. Proposing steps with a language model, as in Tree of Thoughts and SWAP, or with a proposal head trained jointly on the latent state, is well studied and straightforward to add, and so is handling ambiguous observations and uncertain consequences with a predictor that outputs several latent successors, as in stochastic world models. Many applications also come with finite action spaces already, such as tool and API calls in agents or tactics in theorem provers. We left these additions out of the study deliberately, since each would introduce a further component whose quality affects the results, and the research question here is whether a latent world model over sentences supports planning. The limitations section (Section 7) now states this scope and the extensions.

Further changes, marked in blue in the manuscript:

  • Method and Figure 1 present LR-JEPA as a JEPA world model with optional cost heads (Section 4).
  • World-model probes of the imagined states (Appendix C, Table 7) and the natural-text experiment (Appendix D).
  • Related work on textual world models (Section 2) and the revised limitations (Section 7).

If there is any further evidence we could provide to address your concerns, please let us know.

AuthorsOpenReview ↗

Addendum on planning without an external set of candidates

You asked whether the method needs an externally supplied set of candidate actions at each step, and to answer this we ran one more experiment in which LR-JEPA solves FSA problems without any access to the environment [new].

In the paper, the planner receives the list of applicable rules from the environment at each step. In the new experiment we instead let the model work out this list from the text it has seen so far. We take every rule stated in the problem whose premises have already been written down, either in the problem statement itself or in one of the conclusions the model has produced in earlier steps, in a similar way to H-JEPA (Zhang et al., 2026), which also does not take its candidate actions from the environment but derives them from the actions seen in training. LR-JEPA then chooses among these rules by goal distance, as in the paper, writes down the chosen rule together with its conclusion, and repeats this until it has written the target. We only use the environment once the model has finished, in order to check that every step of the proof is valid and that the proof reaches the target.

With this procedure all proofs are valid and reach the target, both on problems of the training length (ID) and on longer problems (OOD). The overshoot, which is the number of steps divided by the length of the shortest solution, is also the same as with the environment.

FSA, overshootIDOOD
LR-JEPA with the environment1.841.83
LR-JEPA without the environment [new]1.841.83
Random choice among the same candidates [new]2.032.02

This shows that we do not need to supply the candidate actions from outside, because the model can read them from the problem and from its own output, and it is the world model that decides which of them to take.

In addition, we trained LR-JEPA with the same objective on plain text (FineWeb-Edu, Appendix D). There it costs 2.3 points of perplexity, and when we use its world model to pick among possible continuations, the continuations it picks come closer to a given goal than those picked by any of the baselines (unigram F1 0.212, compared with 0.152 for a random pick). Taken together, the two experiments show that the world model can be learned from text and that planning with it does not require an environment. We will add the new experiment to the appendix.

If any concern remains, please let us know, and we are happy to run further experiments during the discussion period. If our replies have addressed your concerns, we would appreciate it if you would consider updating your score and confidence accordingly.

AuthorsOpenReview ↗

2nd Addendum on the impact of search and length generalization

As a further test of how the method generalizes, we evaluated the already trained models, without retraining, on harder iGSM problems with 17 to 24 operations and wider dependency graphs, which lie beyond the training lengths (steps per necessary step, 1.0 is the shortest solution, lower is better):

Plannerdepth 1depth 2depth 4depth 8
Frozen LM + value head2.07 +- 0.042.08 +- 0.042.09 +- 0.052.09
LR-JEPA, TD head2.01 +- 0.031.85 +- 0.071.81 +- 0.051.78 +- 0.10

On these problems, searching deeper in the latent space removes about a fifth of the steps beyond the shortest solution, while a value head on a frozen language model does not benefit from the same search. The imagined consequences of later steps come from the predictor alone, without an executor, so the gain reflects what the latent world model has learned about the task.

On FSA, planning towards the goal state benefits from depth up to 32 on problems beyond the training lengths of 6 to 10 rules:

FSA, oracle goaldepth 1depth 2depth 4depth 8depth 16depth 32
11 to 16 rules1.911.771.721.691.571.39
17 to 24 rules1.811.871.811.761.711.59

We will add these results to the manuscript.

Thanks for your response and the additional experiments. I maintain my positive rating, as I believe this is a nice investigation of a new paradigm.

The reason I am not giving the highest score is that I would like to see stronger evidence of the approach’s benefits beyond the current settings. For example, this could take the form of advantages across a range of open-ended language tasks, such as long dialogue or narrative understanding and general instruction following. Alternatively, even without such broad evaluation, I would consider a score of 4 warranted if the approach demonstrated a clear advantage over standard techniques on a compelling application.

If other reviewers have concerns about the paper, I would be happy to discuss them and share the reasons for my positive assessment.

Comments on the paper

2 comments

We thank all reviewers for their careful reviews. The main concerns shared across reviews were whether LR-JEPA is a JEPA world model or a contrastive critic, whether its advantage comes from the world model or from the ranking supervision and counterfactual data, and whether it extends beyond synthetic domains. To answer the first two directly, we removed all energy heads, retrained the model with prediction and SIGReg as the only losses (no contrastive term, no EMA teacher, no stop-gradient), and tested it by planning toward a goal observation, the solved problem state, as in the LeJEPA and LeWM line of work. All changes in the manuscript are marked in blue. (Just for context, we report overshoot as executed steps divided by the shortest solution, so 1.00 is optimal, ID and OOD refer to test problems within and beyond the training lengths, and [new] marks results obtained since the reviews.)

Table A. Planning by goal distance with one imagined step and no energy head (manuscript Table 1) [new].

PolicyFSA IDFSA OODiGSM IDiGSM OOD
Random policy2.022.011.811.74
Same network at initialization1.981.961.791.80
LR-JEPA, goal distance1.841.831.621.62
without SIGReg1.961.911.801.78
without prediction loss1.961.951.791.78

On both domains the world model alone, with no learned cost and no supervision on correct steps, plans clearly better than a random policy and than the same network at initialization, and removing either loss returns it to the untrained level, which the imagined states explain. Rolled forward along held-out OOD trajectories without executing any step, the imagined state after up to eight steps lies closer to the true successor than to three sibling branches in 76 to 96 percent of cases on FSA, against 31 to 46 percent without the prediction loss, and on iGSM the rollout error is three to seven times lower than without it (Appendix C, Table 7) [new].

Table B. FSA OOD overshoot by the number of imagined steps (manuscript Tables 1, 3 and 4) [new].

Imagined steps1248
Language models (20M to 200M), no lookahead1.69 to 1.91
LM likelihood lookahead (RAP, ToT)1.831.81
State-value head on a frozen LM1.571.862.022.03
Semantic-world-model LM1.851.781.811.83
LR-JEPA with TD head1.051.041.041.05

Energy heads are now optional cost modules on this backbone. A head trained by temporal-difference learning (expectile regression as in IQL) reaches 1.00 ID at every depth and keeps 1.04 to 1.05 OOD, while every cost read from a language model moves toward the random level (2.01) with depth. Trained afterwards on the frozen backbone, which never saw a ranking label, the same head reaches 1.15 OOD, against 1.48 on a frozen random-init backbone [new], and a language model trained on the serialized counterfactual stream reaches 2.04 OOD (Table 6) [new]. The language models do encode the correct step beyond the trained lengths (a linear probe finds it at 0.976 top-1 accuracy), yet plan at about 1.9 through their likelihood readout, which imitates gold solutions of the trained lengths. LR-JEPA instead chooses steps by evaluating their imagined consequences, and the results above show that this rests on the representation learned by the JEPA objective.

Natural text [new]. A 124M LR-JEPA trained on FineWeb-Edu with the same objective costs 2.3 points of perplexity (40.6 against 38.3). Given the encoding of a held-out next sentence as the goal, it picks among 16 continuations sampled from a language model the one whose imagined end state lies closest. The pick lies at 0.16 of the mean candidate distance from the goal and reaches a unigram F1 of 0.212 with the true text, against 0.152 for a random pick, 0.136 for a random-init model, 0.148 without SIGReg and 0.123 for the most likely sample (Appendix D).

Further changes (marked in blue in the manuscript revision):

  • Method and Figure 1 present LR-JEPA as a JEPA world model with optional heads (Section 4).
  • Related work on textual world models (SWM, SWAP, WMA) and the state-prediction separation hypothesis (Section 2), and a revised limitations section (Section 7).
  • Complete results, the TD sweep and the component ablations (Appendix C).

Dear Reviewers,

The authors have now submitted their rebuttal and responded to the concerns raised in the reviews. We would greatly appreciate it if you could take some time to review their responses and participate in the discussion, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.