acceptodds
Under review as a conference paper at ICLR 2027

Enterprise-JEPA: Learning Latent Tool-Use Dynamics for Efficient Agent Planning

Abstract

Enterprise workflows require tool-using agents to coordinate actions across applications, maintain state over long horizons, and avoid costly execution errors. World models can improve planning by predicting the consequences of candidate tool actions, but existing LLM-based approaches rely on autoregressive generation of imagined observations, making repeated planning computationally expensive. We introduce Enterprise-JEPA, the first action-conditioned latent world model for enterprise tool-using environments. Given the task context, interaction history, and a candidate action, Enterprise-JEPA predicts the latent representation of the resulting tool observation and decodes seven interpretable operational signals: action type, execution status, error, progress, side effects, information sufficiency, and task completion. A two-stage training procedure first learns latent dynamics from unlabeled trajectories and then fits operational-state heads using annotated transitions. Scaling unlabeled pretraining improves structured next-state prediction and surpasses a similarly sized LLM-based world model. Enterprise-JEPA also delivers strong downstream performance. With beam search, it consistently outperforms a no-world-model baseline across all five benchmarks, while its latent rollout mechanism is substantially faster than an LLM-based world model. These results demonstrate that latent world modeling enables more effective and efficient planning for enterprise tool-using agents, improving decision quality while reducing inference latency. Code is available at https://anonymous.4open.science/r/enterprise_jepa-8421.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, joint-embedding predictive architectures, LLM agents, model-based planning, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_du5bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Enterprise-JEPA is an action-conditioned latent world model for text-based tool-using agents in enterprise environments. Where existing LLM-based world models autorgressively generate the imageined next tool observation, Enterprise-JEPA predicts that observation's representation in a single forward pass, and decodes the predictor's final hidden state into seven interpretatble categorical fileds describing the action's operational consequence: action type, execution staus, error signature, progress signal, side-effect type, inforation sufficiency, and task completion. The model have a Qwen3-embedding-0.6B encoder feeds a 1024 linear projector with LayerNorm then a Transformer predictor over a nine-token sequence: eight recent tool-output representations plus one context token. Actions are not separate tokens. Stage 1 learns latent dynamics self-supervised, aligning the predicted zt; stage 2 freezes encoder, projector and predictor and fits the seven heads with calss balanced cross entropy.

Critical strengths
  1. it is the first action-conditioned JEPA model for text-based llm tool agents in enterprise settings, predicting tool0induced transitinos in latent space rather than reconstructing future observations.
  2. The stage 1 ablation is internal consistent and makes it convincing, The default skeptical hypotehesis for any two stage paper is that the labelled stage 2 does not work and the representation learing is decoration. They have tested that the collapse to 0.404 falls below both the small corpus model and the LLM -WM baseline. More telling than the magnitude is the pattern, it degrades hardest on execution status and error signature.
  3. The paper also ran planning control that most planning paper skip, they have shown that the gains are from the world model rather than from candidate generation plus reflection.
  4. The annoation schema was pruned by measured reliability rather than intuition. and Appendix C is better than most human annotation sections, let alone LLM - annotation ones:two passes, three strong raters plus a separte tie break rater.
Critical weaknesses
  1. The headline claim is broader than Table 3 supports, and the one regression is never metioned. The abstract's beam-seach sentence is narrowly accurate, but the paper proposes three harnesses, and Enterprise-JEPA with ITP-I on automationbench scores 0.136 against 0.174, there is a drop.

  2. The hand engineered planner may be doingmuch of the work. Appendix D deines the ranking function with roughly fifteen free constants: five field weights, six category utilities, seven lambdas, read pressire K-2 , veto threshold at 0.6 and 0.5 and an acceptance margin. No tuning protocol is stated, no help out split for selectionand no senstivitiy analysis beyond the candiate count and horizon.

  3. there is no decisive baseline; the no stage 1 leaves the projector and predictor at random initilisation, which makes it a strawman for question the paper's framing actually raises. The needed control is the same seven heads, same architecture, with the action representatino available, trained on the same labels but without the stage 1 alignment objective.

Questions for authors
  1. May need to explain the table 3 0.136 vs 0.174 better, the 3.8pp drop and the largest regressino in the table.

  2. Table 4, both controls fall below the no WM baseline(0.342 and 0.338 and 0.371). which implies the beam-search scaffold is net harmful without informative predictions. Could you state the decomposition explicitly: how much of the 0.404 is recovering harness overhead, and how much is net new value?

  3. how were the roughly fifteen constants in appendix D is selected, and on what data, the five field weights, the category utilities, and etc. are they tuned on enterprise ops gym? how much of the 0.404 holds on a benchmark they were not tuned on?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer du5b (Part 1)

We thank Reviewer du5b for the encouraging assessment of our work as the first action-conditioned JEPA model for text-based tool-use agents. We also appreciate their insightful comments on our ablation choices (including Stage 1 and the planning controls), as well as the thorough description of the annotation protocol. We address your concerns and questions below.

Item 1: W1/Q1, Scope of the main claim; ITP-I on AutomationBench

Thank you for pointing out this omission. We agree that our headline claim should distinguish performance across planning harnesses. On AutomationBench, Enterprise-JEPA with ITP-I scores 0.136, below the no-WM baseline of 0.174, a drop of 3.8 percentage points. We will explicitly report this regression and revise the claim to state that Enterprise-JEPA improves over the baseline on all five benchmarks under beam search, rather than across all harnesses.

Our analysis of the AutomationBench logs suggests that the outcome depends on the interaction between the world model, harness, and benchmark:

  • Performance varies across harnesses. The same Enterprise-JEPA checkpoint scores 0.180 with Revision and 0.192 with beam search, compared with the 0.174 no-WM baseline. Of the 26 tasks lost under ITP-I, 19 are solved by Enterprise-JEPA under Revision.

  • ITP-I is challenging for all three evaluated world models on AutomationBench. It is the lowest-scoring harness for Enterprise-JEPA, the state-output LLM-WM (0.167), and Qwen-AgentWorld (0.151). This pattern suggests a harness-related effect, although it does not rule out a contribution from the world model.

  • The logs reveal problematic rollout and execution behavior. ITP-I asks the policy to generate a four-step hypothetical rollout and returns it as free-text foresight. On AutomationBench, 11% of these rollouts end early with an unparseable action, and 18% repeat the same action at every imagined step. During execution, repeated tool calls account for 14% of calls versus 3.3% for the baseline, and 21.7% of tasks reach the 50-turn limit versus 8.7% for the baseline. Some lost tasks also complete with partial scores because of incorrect arguments or omitted writes.

These observations are consistent with ITP-I being less effective on this benchmark, but they do not establish a single cause for the regression. Beam search uses predictions differently, to rank and veto candidate plans, and achieves 0.192 with Enterprise-JEPA on the same benchmark. We will add the regression, the log analysis, and a per-task breakdown to the revision, and qualify the paper's claims accordingly.

Item 2: W2/Q3, Selection of the planner constants and gains on untuned benchmarks

We thank the reviewer for highlighting the need for further clarification.

  • Scoring constants: Field weights, category utilities, penalty coefficients λ\lambda, the read-pressure threshold K=2K=2, veto thresholds, and the acceptance margin were fixed a priori using the design principles below, independently of evaluation results.
  • Planning budget: We used open-loop planning and selected the candidate budget B=8B=8 and horizon H=3H=3 using the ablations in Table 10 on EnterpriseOps-Gym and AutomationBench.
  • Transfer across benchmarks: CRMArena-Pro, WorkBench, and Terminal-Bench were not used to select the scoring constants or planning budget. With the same configuration, beam search improves success over the no-WM baseline by 6.1, 2.4, and 6.4 percentage points, respectively. Mean success across these three benchmarks is 0.498 versus 0.449. These improvements exceed those on the two benchmarks used for budget selection (3.3 and 1.8 percentage points), supporting transfer of the selected configuration across benchmarks.

Design principles. Execution status (1.0) and progress signal (1.2) receive the largest weights because they directly indicate whether an action succeeds and advances the task. Complementary fields receive smaller weights: side-effect type (0.6), information sufficiency (0.5), and error signature (0.4). Category utilities use a common scale from −1 to +1, representing unfavorable and favorable outcomes, respectively. Neutral progress receives a utility of −0.3 to favor actions that advance the task, while intended state changes receive neutral utility. Read-pressure and required-operation coverage terms balance information gathering with the state-changing operations needed to complete the task. We will add these design principles and the rationale for the numerical values to Appendix D.

AuthorsOpenReview ↗

Author Response to Reviewer du5b (Part 2)

Item 3: Q2, Decomposition of the 0.404 success rate

Thank you for raising this critical point. Relative to the no-feedback control (0.342), informative predictions add 6.2 points: 2.9 recover the baseline level (0.342 → 0.371) and 3.3 are a net gain over it (0.371 → 0.404); relative to shuffled predictions, the split is 3.3 + 3.3. To decompose the contribution of the predictions further, we added a ranking-only control.

We added two controls on EnterpriseOps-Gym (80 tasks, B=8B=8, H=3H=3, three runs per condition). The actions-only control retains JEPA-based candidate ranking and gating but shows the policy only the selected actions, withholding the predicted states. The unranked actions-only control assigns equal scores to all candidates and selects the first generated candidate; the policy again receives only the selected actions when the plan passes the confidence gate. Both controls retain the full planner's candidate generation, scoring rules, vetoes, confidence-gate rules, and policy-call budget. These comparisons distinguish the contribution of prediction-guided selection and gating from that of explicitly displaying predicted states. Table R combines these controls with the published results.

ConditionCandidate rankingShown to the policySuccess (%)
No world model——37.1 ± 0.6
State-output LLM-WMLLM-WM predictionsactions + predicted states37.1 ± 1.9
JEPA, unranked actions onlynoneselected actions only36.2 ± 2.2
JEPA, actions onlyJEPA predictionsselected actions only39.6 ± 0.7
Enterprise-JEPAJEPA predictionsactions + predicted states40.4 ± 0.7

Table R. Candidate selection versus explicit predicted-state advice. Success is reported as mean ± standard deviation across three runs.

The actions-only control retains most of the full planner's performance: 39.6% versus 40.4% success rate. It exceeds the unranked actions-only control by 3.4%, suggesting that JEPA-guided plan selection and gating contribute to the improvement beyond supplying candidate actions and additional deliberation alone. Adding the predicted states and advice provides a further mean improvement of 0.8%. These results support the value of using learned world-model predictions for plan selection. We include this analysis in Appendix F.9 and the discussion of Table 4.

Item 4: W3, Architecture-matched control without the alignment objective

Thank you for proposing this valuable experiment.; we ran exactly this control. We added this baseline (Architecture-matched baseline). It starts from the same weights as the "no Stage 1" row, keeps the architecture, AdaLN action conditioning, and heads, and trains the same parameters as Stage 1 (top four encoder layers, projector, predictor) together with the heads on the 17,066 labeled transitions, with the Stage-2 data, loss, and schedule but without the latent-prediction objective. We also fine-tuned the Stage-1 checkpoint end to end on the labels (JEPA init + sup. FT).

We report the mean ± standard deviation across 5 seeds below and will include the results in the revised version of the paper along with Table 2.

Model (5 seeds)Avg. macro-F1Avg. macro-recallExec. status F1Seeds exec. ≥ 0.6
LLM-WM, expanded0.557 ± 0.0030.556 ± 0.0030.387 ± 0.0030/5
JEPA, no Stage 10.403 ± 0.0020.417 ± 0.0020.264 ± 0.0050/5
JEPA, small corpus0.578 ± 0.0060.606 ± 0.0010.583 ± 0.0464/5
Architecture-matched baseline (new)0.587 ± 0.0130.608 ± 0.0120.458 ± 0.0931/5
JEPA init + sup. FT (new)0.567 ± 0.0530.585 ± 0.0630.475 ± 0.1262/5
JEPA, expanded0.602 ± 0.0030.623 ± 0.0020.611 ± 0.0055/5

Table S. Next-state prediction, mean ± std over five seeds.

  • Enterprise-JEPA with the expanded corpus is the best and the most stable model. It achieves the highest average macro-F1 (0.602 ± 0.003) and average macro-recall (0.623 ± 0.002).
  • Stage 1 makes execution-status prediction reliable: Enterprise-JEPA achieves a mean F1 of 0.611 and exceeds 0.60 in all five seeds, whereas the architecture-matched baseline reaches this threshold in only one of five seeds (mean 0.458 ± 0.093), with a nearly 20× larger standard deviation. Predicting whether an action succeeds is exactly what next-observation prediction learns.

These results show that the latent-prediction objective, combined with unlabeled data, improves both execution-status prediction and seed stability.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the time and effort you have devoted to reviewing our submission. We have responded to your comments and revised the manuscript to address the concerns you raised. Our responses to the other reviewers may also provide useful clarification on related points.

As the discussion period draws to a close, we would welcome any further thoughts or remaining concerns and would be happy to address them.

Thank you again for your thoughtful consideration.

Reviewer_EAjfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Enterprise-JEPA, an action-conditioned latent world model for enterprise tool-using agents. It learns to predict embeddings of future tool observations from unlabeled trajectories, then trains seven classification heads that expose operational signals for planning. Evaluation covers next-state prediction and downstream execution across five benchmarks using revision, Imagine-Then-Plan, and beam-search harnesses. The paper reports average next-state macro-F1 of 0.603, compared with 0.556 for a similarly sized autoregressive world model; beam-search task success increases from 0.378 to 0.418 over the no-world-model baseline. Optimized world-model rollouts achieve up to 44.8× lower latency

Critical strengths
  1. It provides a practically useful adaptation of latent world modeling to enterprise tools. Prior work such as LeWorldModel establishes action-conditioned latent prediction for visual control, while WebDreamer establishes world-model-assisted planning for web agents. Enterprise-JEPA connects these directions through text-based tool dynamics and a compact, reusable operational-state interface. This is a meaningful application contribution with a clear computational motivation. arxiv.org

  2. The paper conducts broad downstream evaluation with informative prediction controls. Table 3 reports higher beam-search mean success rates than the no-WM baseline on all five benchmarks, with three-run reporting. Table 4 additionally shows that correctly associated predictions outperform removed or shuffled feedback, supporting the usefulness of prediction content within the planner. The optimized and shared-backend timing comparisons strengthen the evidence for a substantial world-model rollout efficiency benefit.

Critical weaknesses
  1. The controls insufficiently isolate the contribution of latent-dynamics learning. The “no Stage 1” ablation freezes the randomly initialized projector and predictor and trains only the classification heads (4.4). This establishes the importance of training those components, but leaves open whether a directly trained discriminative state predictor would provide similar benefits. Furthermore, beam search consumes JEPA’s soft probabilities but converts the LLM-WM’s generated labels into one-hot distributions (5.3). Because these distributions determine both expected-utility rankings and failure/deletion vetoes, the comparison changes the planning interface alongside the model. These confounds weaken attribution of the reported advantage to the proposed learning approach.

2 The multi-step inference protocol is insufficiently specified and internally inconsistent. Equation 1 defines one-step prediction, while Appendix D provides no explicit recurrence explaining what enters the predictor at later imagined steps or how result-dependent symbolic arguments are handled. Separately, 5.2 states that all ITP variants alternate policy actions and world-model predictions, whereas Appendix G.4 describes a single generative-WM call producing the entire trajectory and separate policy calls for JEPA. If the latter procedure applies to the state-output baseline in Table 3, both the action generator and policy-computation budget differ. This matters because ITP produces several of the strongest downstream gains.

Questions for authors
  1. Which ITP procedure was used for each world model in Tables 3 and 5? Please reconcile 5.2 with Appendix G.4 and specify the corresponding policy-call and token budgets.

  2. What is the exact multi-step latent recurrence, and how are symbolic dependencies such as $step1.field represented during imagined rollouts? For the default planning configuration, what is the full inference-time breakdown, including encoding, candidate generation, world-model evaluation, and reflection?

  3. Do the advantages persist against a fully trained discriminative state predictor and when both world models supply equivalent planner inputs—for example, hard-label JEPA versus hard-label LLM-WM? Please distinguish the effects of additional unlabeled data, the pretraining objective, and the probability interface.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer EAjf (Part 1)

We thank Reviewer EAjf for their thoughtful feedback and careful observations. We appreciate your recognition of Enterprise-JEPA's contribution in linking latent world modeling to text-based agents. Your comments regarding the timing comparisons, which support our claims about the benefits of rollout efficiency, are encouraging. We have carefully considered your points and addressed them below.

Item 1: W2/Q1, ITP-I procedure; policy-call and token budgets

Thank you for asking for this important clarification. All three world models in Tables 3 and 5 used the identical alternating procedure of §5.2; we revise App. G.4 to present exactly this procedure. At each imagined step k=1,…,Kk=1,\dots,K (K∈{1,…,4}K\in\lbrace 1,\dots,4\rbrace chosen by the policy):

  1. the policy proposes one hypothetical action, conditioned on the real history and the imagined prefix including predictions so far;
  2. the world model makes one prediction for that action (operational state for Enterprise-JEPA and the LLM-WM, tool output for Qwen-AgentWorld).

The imagined trajectory is then returned to the policy with the same reflection prompt. The action generator, prompts, and call counts are therefore identical (per event: 4 proposal calls, 1 reflect-and-act call, 4 world-model calls). Only the predictions and the cost of a world-model call differ (a latent forward pass vs. autoregressive generation).

Item 2: Q2, Multi-step recurrence, symbolic dependencies, inference time

Thank you for requesting clarification on these important points.

Recurrence. Given context latent zcz_c, real observation latents zt−N,…,zt−1z_{t-N},\dots,z_{t-1}, candidate action latents u(1),…,u(H)u^{(1)},\dots,u^{(H)}, and z^(0)=zt−1\hat z^{(0)}=z_{t-1}, for k=1,…,Hk=1,\dots,H, we predict the next observation latent at each step using Eq. (1):

(z^(k),s(k))=Tθ(zc,{(xj,uj+1)}),pf(k)=softmax(Cf(s(k))),(\hat z^{(k)}, s^{(k)}) = T_\theta\big(z_c,\\ \lbrace (x_j, u_{j+1}) \rbrace\big),\qquad p^{(k)}_f=\mathrm{softmax}(C_f(s^{(k)})),

where xx is the real history followed by z^(1),…,z^(k−1)\hat z^{(1)},\dots,\hat z^{(k-1)} (the N=8N=8 most recent entries), and z^(k−1)\hat z^{(k-1)} is conditioned on u(k)u^{(k)} through AdaLN as in Eq. (1). Only latents are fed back: no text is decoded and the context is fixed. All BB candidates are rolled out in one batch per step.

Symbolic dependencies. A reference such as $step1.field means "use the value of field returned by step 1." During imagination, we keep this placeholder in the encoded action text. It is replaced with the actual value only after step 1 has been executed. Before scoring a plan, we check that each reference has valid syntax, points to an earlier step, and provides an output type compatible with the receiving tool's argument. Invalid references are rejected or penalized (App. D).

Inference-time breakdown.

PhaseTimeShare of the step
Candidate generation (policy model, n = 8)3.08 s99.5%
JEPA world-model evaluation, total16.1 ms0.5%
↳ history encoding8.5 ms
↳ action encoding (cached)0.2 ms
↳ latent prediction + heads6.4 ms
↳ parsing / scoring / ranking~1 ms

For the default beam-search configuration with eight candidates, candidate generation takes 3.08 s, while JEPA evaluation and candidate scoring take approximately 16.1 ms per planning event. Thus, JEPA evaluation and scoring account for only 0.5% of the measured planning time, demonstrating that our world model adds minimal computational overhead relative to candidate generation. We will update this inference-time analysis in the paper.

AuthorsOpenReview ↗

Author Response to Reviewer EAjf (Part 2)

Item 3: W1/Q3, Discriminative predictor; equivalent planner inputs; data vs. objective vs. interface

We appreciate your thoughtful suggestion. We ran an architecture-matched supervised baseline and evaluated JEPA with hard-label planner inputs. The results show that JEPA's next-state prediction advantage persists against the supervised predictor.

Fully trained discriminative predictor. We added this baseline (Architecture-matched baseline). It starts from the same weights as the "no Stage 1" row, keeps the architecture, AdaLN action conditioning, and heads, and trains the same parameters as Stage 1 (top four encoder layers, projector, predictor) together with the heads on the 17,066 labeled transitions, with the Stage-2 data, loss, and schedule but without the latent-prediction objective. We also fine-tuned the Stage-1 checkpoint end to end on the labels (JEPA init + sup. FT).

We report the mean ± standard deviation across 5 seeds below and will include the results in the revised version of the paper along with Table 2.

Model (5 seeds)Avg. macro-F1Avg. macro-recallExec. status F1Seeds exec. ≥ 0.6
LLM-WM, expanded0.557 ± 0.0030.556 ± 0.0030.387 ± 0.0030/5
JEPA, no Stage 10.403 ± 0.0020.417 ± 0.0020.264 ± 0.0050/5
JEPA, small corpus0.578 ± 0.0060.606 ± 0.0010.583 ± 0.0464/5
Architecture-matched baseline (new)0.587 ± 0.0130.608 ± 0.0120.458 ± 0.0931/5
JEPA init + sup. FT (new)0.567 ± 0.0530.585 ± 0.0630.475 ± 0.1262/5
JEPA, expanded0.602 ± 0.0030.623 ± 0.0020.611 ± 0.0055/5

Table S. Next-state prediction, mean ± std over five seeds.

  • Enterprise-JEPA with the expanded corpus is the best and the most stable model. It achieves the highest average macro-F1 (0.602 ± 0.003) and average macro-recall (0.623 ± 0.002).
  • Stage 1 makes execution-status prediction reliable: Enterprise-JEPA achieves a mean F1 of 0.611 and exceeds 0.60 in all five seeds, whereas the architecture-matched baseline reaches this threshold in only one of five seeds (mean 0.458 ± 0.093), with a nearly 20× larger standard deviation. Predicting whether an action succeeds is exactly what next-observation prediction learns.

These results show that the latent-prediction objective, combined with unlabeled data, improves both execution-status prediction and seed stability.

Equivalent hard-label planner inputs. On EnterpriseOps-Gym (80 tasks, eight candidates, horizon three; average over three runs), we convert JEPA's predicted field distributions to one-hot labels before scoring, vetoes, and policy advice.

ConditionSuccess (%)
No world model37.1
LLM-WM hard37.1
JEPA hard35.8
JEPA soft40.4

Table P. Published baselines and the new hard-label JEPA control. Values are mean task-success rates.

Hardening the labels in JEPA transforms the probability distribution to one-hot vectors, resulting in loss of fine-grained discriminative information necessary for distinguishing between candidate plans. Thus, the success rate drops from 40.4% to 35.8% for the hard-label JEPA baseline.

Disentangling the factors.

  • Objective (Table S): at equal labels and architecture, Stage 1 gives reliable execution-status prediction and a more stable model.
  • Unlabeled data (Table S): with JEPA pretraining, expanding the corpus increases average macro-F1 from 0.578 to 0.602 (+2.4% improvement).
  • Interface (Table P): using JEPA's probability distributions instead of one-hot labels improves task success by 4.6% (40.4% vs. 35.8%), with performance higher or equal in every run.
AuthorsOpenReview ↗

Official Comment by Authors

We truly appreciate the time and effort you’ve dedicated to reviewing our submission. We have replied to your comments and made revisions to address the concerns raised. Additionally, we have responded to other reviewers’ feedback, which might also help clarify any additional questions.

Since the discussion phase ends soon, we wanted to follow up and would greatly value any further thoughts or concerns you might have so we can address them appropriately.

Thank you again for your time and commitment to the review process.

Reviewer_3qiEOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Enterprise-JEPA, an action-conditioned latent world model designed to improve planning efficiency for LLM agents operating in enterprise tool-use environments. Rather than autoregressively generating future tool observations, the proposed architecture predicts their latent representations and uses seven classification heads to estimate operational outcomes, including execution status, errors, progress, side effects, information sufficiency, and task completion.

The model is trained in two stages, self-supervised latent prediction on unlabeled action–observation trajectories, followed by supervised learning of operational-state classifiers using annotated transitions. The authors evaluate next-state prediction against an autoregressive LLM-based world model and downstream agent performance using three inference mechanisms, revision, Imagine-Then-Plan, and beam-search planning. The results show that expanded latent pretraining improves average next-state macro-F,, outperforming the LLM-based baseline

Critical strengths

The paper addresses an important limitation of autoregressive world models for agent planning, the computational cost of repeatedly simulating future observations. Predicting compact operational states rather than generating full tool outputs is a useful design choice, particularly for applications requiring frequent evaluation of candidate actions. The reported latency improvements seems to be substantial, and the inclusion of a shared-backend comparison strengthens the efficiency evidence.

On the other han, the evaluation covers five benchmarks, including unseen enterprise environments and an out-of-domain terminal benchmark, and examines three inference harnesses. Enterprise-JEPA with beam search achieves higher mean success rates than the no-world-model baseline on all five benchmarks. Importantly, the prediction-removal and prediction-shuffling controls provide evidence that informative world-model feedback contributes to downstream improvements, rather than the gains arising entirely from the existence of an additional planning procedure.

Critical weaknesses

I think the central methodological claim is that JEPA-style next-observation prediction learns useful action-conditioned dynamics for planning. However, the seven operational-state heads operate on the Transformer's contextual hidden state rather than directly on the predicted next-observation embedding. Consequently, the reported improvements in operational-state prediction do not establish that the model learns accurate or compositional latent transitions, as opposed to effective action-conditioned representations for classification.

The main Stage-1 ablation does not resolve this ambiguity. Removing Stage 1 leaves the projector and predictor randomly initialized and frozen, while training only the classification heads. This demonstrates the importance of learning the intermediate representations, but does not isolate the benefit of the latent prediction objective. A more informative comparison would use the same architecture trained directly to predict operational states, or an alternative representation-learning objective under comparable training conditions. Am I missing something?

Furthermore, the paper evaluates next-state prediction primarily on recorded transitions, but uses the model for multi-step imagination during planning. It provides no direct evaluation of free-running latent rollout accuracy, accumulated prediction error, or the relationship between predicted plan scores and realized outcomes. This omission is particularly concerning because increasing rollout horizon or candidate count reduces downstream performance in Appendix F.5. These findings may indicate model error accumulation or exploitation of inaccurate predictions, but the paper does not investigate these possibilities directly. I would like this to be more expanded.

The planning procedure combines learned operational predictions with several manually designed utility weights, penalties, veto thresholds, candidate-generation strategies, and additional policy deliberation. Although the prediction-removal and shuffling controls are valuable, they do not separately quantify the contributions of candidate ranking, heuristic scoring, and predicted-state feedback to the acting agent. In particular, the no-feedback control selects the first generated candidate, while shuffling simultaneously changes candidate ranking and the feedback presented to the policy.

The qualitative analysis also illustrates this ambiguity. In the WorkBench email example in Appendix F.3, the world model predicts that an action with an incorrect recipient would succeed, while the eventual improvement comes from the agent performing an additional lookup after reflection. This suggests that some gains may result from additional deliberation rather than accurate prediction of tool-action consequences.

The evaluation protocol raises further concerns. Certain supervised labels, including progress and information sufficiency, are constructed using future trajectory information and final outcomes, potentially making them dependent on the recorded continuation policy. The paper also does not clearly establish the validation procedure used to select planning hyperparameters independently of final evaluation tasks. Finally, downstream performance is reported over only three runs, with several relatively small improvements lacking paired confidence intervals or task-level significance analysis.

These issues do not invalidate the reported results, but they substantially limit the strength of the causal and generalization claims. Also, I would strongly encourage the authors to select the main experimental comparisons and repeat them across more independent random seeds, reporting mean performance, standard deviations, and preferably confidence intervals for the improvement over the baseline. Paired task-level comparisons would be particularly valuable, since they could reveal whether the average gains arise from consistent improvements across tasks or from a relatively small number of successful trajectories. I consider this important for evaluating the reliability of the conclusions, rather than merely an additional desirable experiment.

Questions for authors
  1. Can the authors compare Enterprise-JEPA against an architecture-matched baseline that uses the same pretrained encoder, projector, Transformer predictor, and classification heads, but trains directly on the seven operational-state labels without the Stage-1 next-observation prediction objective? This would help establish whether the latent-dynamics objective is essential to the contribution.

  2. How accurate are the model's free-running multi-step latent rollouts? The next-state results evaluate recorded transitions, while planning requires predicting consequences of hypothetical action sequences. Can the authors clarify exactly how predicted latent embeddings are propagated across imagined steps and report prediction accuracy or calibration as a function of rollout horizon?

  3. How much of the downstream improvement is attributable to learned predictions rather than additional deliberation or planner heuristics? Can the authors distinguish the contribution of model-based candidate ranking from that of predicted-state advice to the acting policy?

  4. How were evaluation tasks and planner hyperparameters separated? Can the authors clarify how the 80 EnterpriseOps-Gym tasks were selected, which validation data were used to choose the beam-search configuration and utility weights, and whether any reported evaluation tasks influenced those choices?

  5. How reliable are the hindsight-derived operational labels and downstream performance differences? For labels such as progress and information sufficiency, which use future trajectory information, how sensitive are the targets to alternative continuation policies?

Your sentence is grammatically correct, but this sounds more natural and professional:

Please also refer to the Critical Weaknesses section.

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3qiE (Part 1)

We sincerely thank Reviewer 3qiE for their comprehensive and insightful review, as well as for raising critical questions that greatly enhance the quality of our research. We appreciate their recognition of our efficiency comparisons and control measures that highlight the meaningful feedback from the world model contributing to our downstream results. Below, we have outlined our responses to your concerns.

Item 1: W2/Q1, Architecture-matched baseline without the Stage-1 objective

Thank you for suggesting an important baseline. We added this baseline (Architecture-matched baseline). It starts from the same weights as the "no Stage 1" row, keeps the architecture, AdaLN action conditioning, and heads, and trains the same parameters as Stage 1 (top four encoder layers, projector, predictor) together with the heads on the 17,066 labeled transitions, with the Stage-2 data, loss, and schedule but without the latent-prediction objective. We also fine-tuned the Stage-1 checkpoint end to end on the labels (JEPA init + sup. FT).

We report the mean ± standard deviation across 5 seeds below and will include the results in the revised version of the paper along with Table 2.

Model (5 seeds)Avg. macro-F1Avg. macro-recallExec. status F1Seeds exec. ≥ 0.6
LLM-WM, expanded0.557 ± 0.0030.556 ± 0.0030.387 ± 0.0030/5
JEPA, no Stage 10.403 ± 0.0020.417 ± 0.0020.264 ± 0.0050/5
JEPA, small corpus0.578 ± 0.0060.606 ± 0.0010.583 ± 0.0464/5
Architecture-matched baseline (new)0.587 ± 0.0130.608 ± 0.0120.458 ± 0.0931/5
JEPA init + sup. FT (new)0.567 ± 0.0530.585 ± 0.0630.475 ± 0.1262/5
JEPA, expanded0.602 ± 0.0030.623 ± 0.0020.611 ± 0.0055/5

Table S. Next-state prediction, mean ± std over five seeds.

  • Enterprise-JEPA with the expanded corpus is the best and the most stable model. It achieves the highest average macro-F1 (0.602 ± 0.003) and average macro-recall (0.623 ± 0.002).
  • Stage 1 makes execution-status prediction reliable: Enterprise-JEPA achieves a mean F1 of 0.611 and exceeds 0.60 in all five seeds, whereas the architecture-matched baseline reaches this threshold in only one of five seeds (mean 0.458 ± 0.093), with a nearly 20× larger standard deviation. Predicting whether an action succeeds is exactly what next-observation prediction learns.

These results show that the latent-prediction objective, combined with unlabeled data, improves both execution-status prediction and seed stability.

AuthorsOpenReview ↗

Author Response to Reviewer 3qiE (Part 2)

Item 2: W1, Do the heads reflect learned latent transitions?

Table 2 alone cannot establish whether Stage 1 learns to predict the next observation: its heads read the hidden state sts_t, and the architecture-matched baseline achieves similar classification performance. We therefore directly test whether the predicted latent z^t\hat z_t aligns with the latent of the actual next observation.

With each representation model frozen, we conduct two complementary probes. First, we rank all 5,854 test-observation latents by similarity to z^t\hat z_t and report the true next observation's median rank (1 is best). We also restrict candidates to observations from the same task and report top-1 accuracy. Second, we train probe heads on real-observation latents and apply them unchanged to predicted latents, reporting the percentage of their real-observation macro-F1 retained on z^t\hat z_t.

ModelClassification macro-F1 (heads on sts_t)Observation matching: median rank ↓ (of 5,854)Observation matching: within-task top-1 ↑Probe transfer: macro-F1 retained ↑
JEPA, expanded corpus0.585669%76%
JEPA, small corpus0.562675%78%
Architecture-matched baseline0.5672,95922%36%
JEPA, no Stage 10.3532,82822%29%

Table L. Alignment between predicted latents and real next observations. Macro-F1 is averaged over six categorical fields. Task completion is excluded because its head receives identical inputs across probe conditions. Results are averaged over four seeds.

The Stage-1 models strongly match the actual next observation: its median rank is 6 among 5,854 candidates, and within-task top-1 accuracy is 69–75%, compared with 18% by chance. They also outperform copying the previous observation latent (median rank approximately 400). Observation-trained heads retain 76–78% of their macro-F1 on JEPA's predicted latents, further supporting their alignment with real observations.

In contrast, the architecture-matched baseline achieves similar classification macro-F1, yet its matching performance is near chance (18%) and its probe retention is only 36% (p=3×10−9p=3\times10^{-9}). Thus, classification performance alone does not establish next-observation prediction. These probes support the role of Stage 1 in learning predicted latents aligned with actual next observations, which the planner then reuses during multi-step imagination.

AuthorsOpenReview ↗

Author Response to Reviewer 3qiE (Part 3)

Item 3: W3/Q2, Free-running multi-step accuracy; how are latents propagated?

Thank you for this question. We clarify the latent rollout procedure and add a free-running evaluation across horizons 1–4. On the same 2,737 test transitions, macro-F1 decreases from 0.572 at one step to 0.552 at two steps and remains at 0.552 through four steps.

How latent rollouts work. Given context latent zcz_c, real observation latents zt−N,…,zt−1z_{t-N},\dots,z_{t-1}, candidate action latents u(1),…,u(H)u^{(1)},\dots,u^{(H)}, and z^(0)=zt−1\hat z^{(0)}=z_{t-1}, for k=1,…,Hk=1,\dots,H, we predict the next observation latent at each step using Eq. (1):

(z^(k),s(k))=Tθ(zc,{(xj,uj+1)}),pf(k)=softmax(Cf(s(k))),(\hat z^{(k)}, s^{(k)}) = T_\theta\big(z_c,\\ \lbrace (x_j, u_{j+1}) \rbrace\big),\qquad p^{(k)}_f=\mathrm{softmax}(C_f(s^{(k)})),

where xx is the real history followed by z^(1),…,z^(k−1)\hat z^{(1)},\dots,\hat z^{(k-1)} (the N=8N=8 most recent entries), and z^(k−1)\hat z^{(k-1)} is conditioned on u(k)u^{(k)} through AdaLN as in Eq. (1). Only latents are fed back: no text is decoded and the context is fixed. All BB candidates are rolled out in one batch per step.

Horizon-dependent evaluation. We evaluate all horizons on the same set of target test transitions, restricting the analysis to 2,737 transitions with complete four-step rollout windows so that all horizons use identical targets. For each target transition jj and horizon kk, we initialize the model with the real interaction history preceding action aj−k+1a_{j-k+1}. We then supply the logged actions aj−k+1,…,aja_{j-k+1},\ldots,a_j, updating the imagined history with predicted observation latents rather than real observations. We evaluate the final prediction against the operational labels of transition jj. The teacher-forced reference predicts the same transition using its real preceding history. At k=1k=1, both modes use identical inputs and the same prediction procedure, so their results coincide.

Horizon kk1 (teacher forcing)234
Macro-F10.5720.5520.5520.552
Macro-recall0.5910.5870.5870.586
Task completion macro-F10.6610.5180.5190.522

Table M. Free-running rollout accuracy, EnterpriseOps-Gym and CRMArena-Pro test transitions.

Macro-recall remains within 0.5 percentage points of the one-step reference. The largest degradation is in task completion. Each of the other six fields remains within 2 percentage points of its teacher-forced score. Task completion provides advisory guidance to the agent and is excluded from the trajectory utility score.

These results show limited additional degradation between horizons 2 and 4 on the test set, following an initial drop at horizon 2. We include this analysis in the revised §5.6 and Appendix F.2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Discussion

We thank all three reviewers for their careful and constructive reviews. In response, we conducted the following additional experiments and analyses (details in each reviewer's thread):

  1. Architecture-matched supervised baseline (Link: 3qiE, EAjf, du5b). Holding the architecture, labeled data, and seven output heads fixed, Enterprise-JEPA achieves the highest and most stable next-state accuracy (macro-F1 0.602 ± 0.003 vs. 0.587 ± 0.013 over five seeds) and reliable execution-status prediction (F1 ≥ 0.6 in 5/5 vs. 1/5 seeds).
  2. Predicted latents vs. real next observations (Link: 3qiE). JEPA's predicted latent ranks the true next observation at median 6 of 5,854 candidates (2,959 for the architecture-matched baseline), and observation-trained probe heads retain 76% of their macro-F1 on it (36%). Similar classification accuracy therefore does not imply learned latent dynamics; Stage 1 provides them.
  3. Free-running multi-step accuracy (Link: 3qiE). Macro-F1 drops from 0.572 to 0.552 at horizon 2 and then stays at 0.552 through horizon 4; macro-recall stays within 0.5 points.
  4. Ranking vs. predicted-state advice (Link: 3qiE, du5b). With predicted states withheld from the policy, JEPA-ranked plans still reach 39.6% success on EnterpriseOps-Gym, 3.4 points above unranked candidates (36.2%); showing the predicted states adds a further 0.8 points (40.4%). Most of the gain comes from prediction-guided plan selection.
  5. Soft vs. hard planner interface (Link: EAjf). Converting JEPA's predicted distributions to one-hot labels lowers success from 40.4% to 35.8%; class probabilities give higher or equal success in every run.
  6. Hindsight labels (Link: 3qiE). Without future context, about 91% of progress labels and 79–84% of information-sufficiency labels are unchanged (excess disagreement 3–9 points). Removing both hindsight-derived fields from planning retains about half of the gain over the no-WM baseline (38.8% vs. 37.1% and 40.4%).
  7. Inference-time breakdown (Link: EAjf). In beam search, JEPA evaluation and scoring take 16.1 ms per planning event, 0.5% of planning time; candidate generation by the policy takes 3.08 s.

Clarifications and revisions to the paper.

  • Scope of the main claim (Link: du5b). We will report the AutomationBench ITP-I regression (0.136 vs. 0.174) explicitly, with a log analysis, and state that Enterprise-JEPA improves over the no-WM baseline on all five benchmarks under beam search, rather than under every harness. On the same benchmark, the same checkpoint scores 0.180 with revision and 0.192 with beam search.
  • Planner constants (Link: 3qiE, du5b). Scoring weights, utilities, and thresholds were fixed a priori from design principles that we will state in App. D; only the budget (B=8B=8, H=3H=3) was chosen, on EnterpriseOps-Gym and AutomationBench. On the three benchmarks not used for any choice, beam search improves success by 6.1, 2.4, and 6.4 points, although the size of the gain varies across benchmarks.
  • ITP-I procedure (Link: EAjf). All three world models used the same alternating procedure with identical prompts and policy-call counts; we revise App. G.4 accordingly.
  • Latent recurrence and symbolic dependencies (Link: 3qiE, EAjf). We specify how predicted latents are fed back during rollouts and how step references are validated.

We address each reviewer's remaining points individually below. We also thank the AC for their time and careful consideration of this discussion.

AuthorsOpenReview ↗

General Response to Reviewers

Dear Reviewers and Area Chair,

We have uploaded a revised version of the manuscript. Revised text is colored according to the reviewer whose comments it addresses:

  • Reviewer 3qiE: 🟥 red
  • Reviewer EAjf: 🟩 green
  • Reviewer du5b: 🟪 purple
  • General changes, or changes that address comments from more than one reviewer: 🟦 blue

The main changes are:

  1. Architecture-matched supervised baseline and five-seed results (§4.4, Table 2, App. F.1). Table 2 now reports mean ± standard deviation over five seeds. It also includes a baseline with the same architecture and labeled data, trained without the Stage-1 latent-prediction objective.
  2. Predicted-latent analysis and multi-step rollout accuracy (§4.4, App. F.2, F.3). We compare predicted latents with the observations that actually followed, and we report free-running accuracy at horizons 1–4.
  3. Planner details (§5.2, App. D). We describe how predicted latents are fed back during rollouts and how symbolic step references are checked. We also state how the scoring constants were chosen.
  4. Hindsight-derived labels (§4.1, App. C, F.8). We measure how much the progress and information-sufficiency labels depend on future context, and we evaluate planning with these fields removed.
  5. Controls on how predictions are used (§5.5, App. F.7). We add a ranking-only control that separates candidate selection from showing predicted states to the policy, and a comparison of probability inputs with one-hot inputs.
  6. Analysis and clarification of ITP-I (§5.5, App. F.9, G.4). We analyze the AutomationBench ITP-I result, and App. G.4 now gives the exact ITP-I procedure used for all three world models.
  7. Inference-time breakdown (§5.6, App. F.4). We break down the time in one planning step. We also clarify that the speedups refer to world-model latency.

We thank the reviewers again for their constructive feedback, which has substantially improved the paper. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.