acceptodds
Under review as a conference paper at ICLR 2027

Granularity Is a Decision: Joint Horizon–Description Selection in Latent World Models

Abstract

Embodied agents increasingly plan with world models, rolling candidate actions forward and executing the one whose predicted outcome best reaches the goal. Such models are typically trained as Markov decision process (MDP) transitions , binding every prediction to one environment step and one state representation. However, many physical interactions are action-sparse with persistent effects: a single throw sends an object through flight, collision, and settling, a cascade that unfolds without further action yet decides success. Step-wise prediction of such cascades compounds error, coarse horizons skip decisive contacts, and full detail wastes capacity. All three costs stem from one fixed choice of granularity, made once at design time. Thus, we propose making granularity a variable that the world model chooses per prediction. We formalize the idea as an MDP while every model query carries a prediction request that advances the imagined rollout steps at description level . Across three physics benchmarks, granularity proves consequential: predicting in long jumps cuts compounding rollout error by orders of magnitude, yet no fixed request suits every test set: the one that best separates successful from failed launches on two sets ranks worst on a third. That variance is exploitable: choosing the request per prediction beats a fixed request tuned on other test sets, and choosing both axes matches a horizon-only adaptive baseline at a third of its compute. These results establish granularity as a decision variable rather than a design constant: consequential, unsolvable by any fixed setting, and cheap to adapt.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
world models, physical reasoning, adaptive temporal abstraction, latent prediction, model-based planning, open-world generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vSA9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose a "shared latent predictor" that takes as input a request—defined by a specific time span and a descriptive mode (such as continuous, microscopic, or macroscopic). Symbolic heads feed predicted predicates back into this predictor. The system employs dynamic programming to perform labeling and extract a policy, which is optimized based on "duration-weighted prediction error" and the number of linear MAC (multiply-accumulate) operations. Our experimental evaluation is multifaceted: it includes performance comparisons over fixed time spans across three physics-based benchmarks; an assessment of candidate ranking for nine distinct types of requests using the NovPhy dataset; and a comparison between our adaptive joint model and a system trained independently, which considers only the prediction time horizon.

Critical strengths
  1. By employing a shared carrier, it becomes possible to switch between temporal and symbolic feedback within a single rollout, without requiring transformations between independently learned latent spaces.

  2. This paper address the matching of training budgets, the use of three random seeds, explicit computational accounting, and the standardization of evaluation inventories.

Critical weaknesses
  1. The data processed by the NovPhy predictor includes carrier frames that span both a single engine step and 50 engine steps, with request horizons also defined in terms of these same carrier frames. This method of temporal-rate encoding is poorly characterized, making it difficult to evaluate its transferability.
  2. The comparison of computational costs for the "adjusting horizons only" scenario actually involves simultaneous changes to the predictor width, the tokenization training method, the utilization of representations, and the training dataset—all within a "development state" context. Consequently, this does not substantiate the claim that "jointly selecting the representation for each prediction" yields actual savings in computational cost. Furthermore, the dynamic programming approach optimizes an "additive error proxy" based on a fixed-context carrier, rather than the actual, state-dependent recursive rollout cost.
Questions for authors
  1. Do the training windows and evaluation contexts mentioned on page 15 originate from the same CLEVRER scenes? Please provide the scene IDs used for fitting (training) and evaluation, ensuring there is absolutely no overlap between the two sets, and recalculate Table 1.
  2. Regarding the predictor input, what specific metric (e.g., rate or elapsed time variable) is used to capture the distinction between a "one-engine-step" and a "50-engine-step" execution?
  3. Using the same checkpoints and an equivalent decision budget, how does the joint policy perform compared to the following approaches: fixed macro-step size with adaptive horizons, fixed horizon with adaptive descriptions, and a time-only request scheduling strategy?
  4. For the NovPhy task comparisons, please report the results pairing uncertainty with task success rate (or Top-1 decision quality). Provide a complete list of results—covering all successful and failed cases—rather than just the data summarized by the AUC metric. Does the previously observed marginal gain of 0.004 from the offset-set still hold?
  5. What is the specific objective function that the dynamic programming (DP) approach is guaranteed to optimize? Furthermore, for requests that were not visited during the rollouts in "round 1," how are their successor contexts constructed for "round 2"?
Reviewer_MSwmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an idea for letting a world model decide how far ahead to jump and how much detail to predict at each prediction step. Simply to achieve this goal, each query carries a request composed of two terms: one showing jump size in frames (1, 5, 15) and the other for description level. Since the agent acts only once, a long jump skips no action, and the setting stays an MDP. Under the hood, the authors used 1) one predictor on a frozen object-slot encoder for all nine requests, and 2) a small policy, trained using dynamic programming labels to achieve the trade-off between error and compute to finally pick the request at each step. In Table 1, the authors showed lower rollout error for 15-frame jumps compared to frame by frame steps on 3 benchmarks.

Critical strengths
  1. The idea of letting each query decide its own horizon and level of detail is simple and well-motivated when no actions happen between two decisions and a long jump can still be a valid transition. It could be easily reused with a single request-conditioned predictor for all nine requests and a small policy trained on dynamic-programming labels, with no RL needed.

  2. Experiments are reported honestly, and details about them are shared clearly; this helps us to have a fair judgment and evaluation.

  3. The shared diagnostics are insightful; for example, in the case of a 15-frame request with the lowest rollout error in T1, while it's not the best in Fig. 4, suggesting that the right granularity depends on how the prediction is used.

Critical weaknesses
  1. As shown in Fig. 4, all ordering AUCs are between 0.40 and 0.63, where 0.5 is chance, and the policy's gains over r†r^\dagger (+0.004 to +0.070) are measured against a reference that is itself near or below chance. Given the fact that each set has only 7 to 14 scored levels and only one of the four intervals in Fig. 1(b2) clears zero, this gain is weak.

  2. If we look at Chatterjee and Khardon (NeurIPS 2025), we can see that the idea of choosing the horizon per step is introduced by them. Also, Neitz et al. (NeurIPS 2018) talked about passive dynamics; however, none of them is cited. Given them, we see what is mostly new is about the description axis and the joint choice; however, still no ablation tests a cont-only policy on the same hybrid checkpoint, and part of the compute saving is just the narrower trunk (1.27 times cheaper calls). Therefore, the scope of novelty is narrower.

  3. According to Sec. 4, 2400 of the 2409 training lineages count Δ\Delta in engine steps, while the NovPhy test seems to use 50-step frames; therefore, test jumps may be about 50 times longer than most training jumps. Also, the CLEVRER test contexts in Table 1 come from the 12 training scenes, while with longer rollouts, and there are no details about the Physion-Dominoes data or split. So two things are required: one, the time base should be clarified; two, Table 1 should use documented, held out scenes.

Some minor points I have:

  1. Time-Aware World Model for Adaptive Prediction and Control (TAWM) is published in the ICML 2025 main track, not a workshop.
  2. Please define terms like "lineage, base corpus, N1 captures, development states", I didn't find definitions for these terms.
Questions for authors

Q1. Considering weakness 3, what does Δ\Delta count in training and the NovPhy test? If training counts engine steps (Sec. 4) and the test counts 50-step N1 frames, does the model rely on the elapsed-time entry of the carrier (Eq. 9)?

Q2. Can you please report the number of scored levels and successful launches for each launch set? Also, what are the numeric intervals for all Fig. 4 cells and the four Fig. 1(b2) differences?

Q3: Can Table 1 be recomputed on held-out CLEVRER scenes and documented held-out Physion-Dominoes scenes, with a persistence or constant-velocity baseline and the error of the policy's own schedules?

Q4: Does the compute saving come from the description level, the horizon, or the narrower trunk?

Reviewer_MMedOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an adaptive-granularity world model that jointly selects prediction horizons and description levels. A shared latent predictor handles different prediction requests, while a learned policy chooses the granularity at each step. Experiments on three physics benchmarks investigate rollout accuracy, action ranking, and computational efficiency. They also compare the shared predictor with specialized models and evaluate adaptive selection against fixed-granularity baselines. However, I do not trust the results of this paper, see reason below.

Critical strengths

The use of a shared predictor for different prediction requests is interesting.

Critical weaknesses
  1. The experimental evaluation is not convincing. The CLEVRER experiments appear to use overlapping training and evaluation scenes, while the NovPhy results rely on a very limited number of evaluation levels. These issues seriously weaken the generalization claims.

  2. Several action-ranking results are close to random performance, and the additional description levels do not consistently improve prediction accuracy. The figures, particularly Figures 1 and 3, are overcrowded and poorly presented. The main text also feels underdeveloped despite not using the full available space.

Questions for authors
  1. Why are CLEVRER training and evaluation samples drawn from the same scenes? How is generalization assessed?

  2. What benefit do the symbolic description levels offer over continuous prediction alone?

  3. Do the relatively weak action-ranking results translate into meaningful improvements in task success?

  4. Can the authors evaluate the proposed approach in more complex or closed-loop environments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.