acceptodds
Under review as a conference paper at ICLR 2027

Structured World Models for Sample-Efficient Planning in Partially Known Dynamical Systems

Abstract

Model-based decision making often involves partially known dynamical systems, where structural knowledge (e.g., stock-flow relations, conservation laws, delays, and mechanistic dependencies) is available while some dynamics remain unknown. Yet general-purpose world models often learn system dynamics largely from data without explicitly encoding this known structure. We propose a structured world-modeling framework that separates supplied structural knowledge from the dynamics that remain to be learned, exposes agent-controlled variables as actions, retains the state and memory needed for recursive multi-step prediction, and learns unknown parameters and, when needed, residual dynamics. The resulting transition model is not tied to a particular planner; in our discrete-action experiments, we primarily pair it with Monte Carlo tree search. In a realizable setting, we provide a finite-sample analysis showing how low-dimensional structural parameterization reduces system-identification complexity and how the resulting transition error propagates through multi-step rollouts to planning-value error and, under a sufficient action gap, preservation of the selected action. Experiments on system dynamics and control benchmarks, including larger-scale supply-chain and epidemic-intervention tasks, show strong planning performance under limited data and competitive performance as interaction budgets increase. These results suggest that explicitly preserving known system structure can provide a practical route to more sample-efficient model-based decision making.

Primary area:
reinforcement learning
Keywords:
structured world models, model-based decision making, sample-efficient planning, structural priors, structured dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KTPXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the decision problems when the dynamics structure is known. By learning a small amount of parameters within the true model structure, the proposed method can achieve better performance with less data compared to those baselines with no structural prior knowledge.

Critical strengths
  • The paper is a relatively complete study, where the problem definition, method, experiment, theory, and the additional informations in the appendix are well-presented. These give me a whole picture of the method, and how it works in practice.
Critical weaknesses
  • My main concern is the novelty. This approach (use a known dynamics structure, and only learn the corresponding parameters within) is called system/model identification in some literatures. I think this is kind of well-understood by the academia: With structural bias, the model learning will surely be more efficient, but when the assumed structure is not accurate, it will result in large model error. So for me this work delivers limited new insight.
  • The applicability is another problem. This kind of approach is very useful when there is a simple and clear dynamics structure. However, when the system is complex, like multi-agent system, where we need to predict the behavior of other agents, there is hardly any useful structure to impose, and the bottleneck of learning will still be the non-structural part.
Questions for authors
  • What new methodological or conceptual contribution does the proposed method offer, compared with standard system identification works?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KTPX

We thank the reviewer for recognizing the completeness of this study. The setups and full results of the supplementary experiments R1 and R2 referred to below are given in the General Response.

W1. Relative to system identification, what new insight does this paper provide?

We agree that "structural priors improve sample efficiency" is already known; once the structural model and the state–input interface are given, the parameter learning in our paper is also structured system identification.

A standard SD model usually embeds decision rules and keeps intertemporal effects in delay and pipeline variables. Our paper focuses on constructing an existing SD model into a world model for planning: Section 4.1 and Appendix A.2 give a six-step procedure for action externalization and dynamic-memory retention, together with a state-transition completeness criterion. Section 4.2 and Appendix B.6 discuss residual correction and error propagation under structural mismatch. The supplementary experiments test these construction requirements and the effect of residual correction:

  • R1: Under the same three-parameter identification and MCTS setup, not fully externalizing the action and omitting the shipping, order, and source-node pipeline memory reduce the mean return from −3,315.4 to −5,208.1 and −18,426.3, respectively; the equivalent state-space implementation gives the same result as the full construction.
  • R2: After introducing a 25% transit loss outside the original parameter family, the return stays at about −9,458 when only the original parameters are identified; with the residual added, 49.1% / 57.3% / 88.1% of the gap to the structure-matched reference is closed at 1k / 3k / 10k. At 1k and 3k, both misspecified-structure models obtain higher returns than the black-box model (−23,500.0 and −17,582.0).

Therefore, under this controlled mismatch, the structural prior still retains its planning benefit with limited data, and residual learning further reduces the loss that parameter identification alone does not remove.

W2. What if a complex system lacks a clear structure?

The applicability of our method depends on whether reliable structure covers the key mechanisms that affect planning, and on how hard the remaining unknown part is to learn. For systems that still have reliable local mechanisms, the current framework allows these mechanisms to be kept and systematic deviations not represented by the structural model to be corrected through residual learning. R2 provides evidence for this under a controlled mismatch.

We agree that, if the main prediction difficulty comes from the behavior of other agents for which no reliable model exists, the learning bottleneck may remain in the unstructured part, and the sample-efficiency advantage we observe cannot be directly generalized to such cases. Our paper targets partially known dynamical systems with mechanism structure that can be exploited; R2 is not sufficient to show effectiveness for general multi-agent systems. How to start from agent-level mechanisms and interactions to build hierarchical structured world models is a direction we hope to explore in future work. Section 7 of the revised paper adds this discussion of the scope of applicability.

Q1. Relative to standard system identification, what is the methodological and conceptual contribution?

Conceptually and methodologically, our paper explicitly separates the organization of existing structural knowledge, the identification of unknown dynamics, and planning, and connects them through a complete one-step transition interface. The construction procedure (Section 4.1, Appendix A.2) enables the model to accept actions specified by the planner and to support recursive prediction; after construction, the structural parameters can still be learned with standard system-identification methods, and the resulting model can be called by different planners (Sections 4.2–4.3).

Theoretically, our paper links the identification accuracy of the structural parameters to the planning-value error and the stability of action selection. Section 5 (Eqs. 6–8, Appendix B), using standard identification and Lipschitz error-propagation tools within a realizable structural model family and under the corresponding assumptions, gives a finite-sample identification error bound that depends on the effective parameter dimension. For the same candidate action sequences, it propagates the error through one-step transitions and multi-step prediction to the planning value, while accounting for the terminal-value approximation error.

If, at the current state, the true HH-step planning value QHπQ_H^\pi has a unique maximizing action over the finite candidate action set, and the value-estimation error on all candidate actions is smaller than half of the action-value gap, then exact maximization of the estimated values still selects the same action (Eq. 9).

Reviewer_KTPXOpenReview ↗

Response

Thank you for the response. I understand that the paper’s core contribution is a procedure for converting existing simulation models into world models suitable for planning. However, I still find the ideas and techniques straightforward and largely consistent with standard engineering practice. The procedure is described at a high level, while its implementation appears to rely heavily on manual, task-specific modeling choices. The rebuttal has therefore not resolved my concerns about the methodological contribution, and my score remains unchanged.

AuthorsOpenReview ↗

Follow-up Response to Reviewer KTPX

We thank Reviewer KTPX for taking the time to read our response and clarify the remaining concern.

Regarding manual, task-specific choices and applicability to multi-agent systems (W2, where we noted hierarchical structured world models as an ongoing direction), we share a brief preliminary follow-up check. Because multi-agent hierarchical coordination goes beyond the single-level scope of the current manuscript, we do not add new claims to the revised paper, but report the comparison here in case it is informative for the discussion.

Using the same construction principle from Section 4.1—preserving conservation/kinematic structure and delay/contact memory in the state interface while restricting learning to unresolved bottleneck closures—we instantiated a two-level (macro-micro) SD world model (SD-WM) across two distinct multi-agent testbeds:

  1. Discrete-event fleet coordination (10–50 warehouse robots with queueing and path-blocking, 6 regimes × 5 blind seeds);
  2. Continuous multi-joint contact control (MAMuJoCo Hopper 3x1, 3 regimes × 5 blind seeds).

To eliminate task-specific planner confounding, all methods below share the exact same rule-free candidate grid, decision cadence, and model-free Greedy fallback (Candidate 0), and are trained on the exact same multi-regime offline datasets (Warehouse: 60 trajectories covering all 6 regimes—30 base and 6 per non-base regime including 50-AGV, sampled uniformly across regimes; Hopper: 48 trajectories covering all 3 regimes—24 base, 12 1.15g, and 12 goal_only). To test whether the learning bottleneck in multi-agent systems lies mainly in the unstructured part (W2), we compared SD-WM against an unstructured neural transition model (Pure-BB-WM), the implicit latent-value model TD-MPC2 [1], and a state-space adaptation of the recent end-to-end hierarchical latent world model H-JEPA [2] (retaining its two-level temporal predictors, SIGReg anti-collapse regularization, inverse dynamics, and top-down subgoal projection), all trained for 3,000 updates to convergence (3.75×–15× our update budget):

Method (shared planner & Greedy Cand. 0)WH Params / StepsWH 10-AGV (base)WH 50-AGV (high_load)WH 6-Case MeanHopper Params / StepsHopper 1.15gHopper 3-Case Mean
Fixed + Greedy (no world model)0 / 0229.7-69.0189.60 / 0630.8668.1
Pure-BB-WM (converged)23.7k / 3000228.2-250.2162.224.7k / 3000562.1630.0
TD-MPC2 [1] (converged)50.4k / 3000173.3229.5241.951.1k / 3000583.6644.0
H-JEPA [2] (converged)33.3k / 3000209.7-394.236.728.1k / 3000572.1583.9
SD-WM (Ours)13.1k / 800227.1452.7324.717.5k / 200751.4751.0

Two observations directly speak to W2 and the role of the structured interface:

  • In low-coupling nominal regimes (e.g., 10-robot base), converged black-box models match SD-WM (228.2 vs. 227.1), confirming that the unstructured baselines fit passive in-distribution transitions well.
  • Under emergent multi-agent bottlenecks (50-robot congestion or 1.15g contact overload), even with target-regime trajectories in the training set, converged unstructured and latent models misrank counterfactual macro actions not covered by healthy offline rollouts (triggering backlog penalties or fleet gridlock in 50-AGV, often falling below Fixed + Greedy), whereas retaining stock/physical conservation and explicit delay/contact closure (SD-WM) yields consistent gains (+82.8 to +135.1 over the best baseline/heuristic across both domains) at a fraction of the update budget.

We appreciate the reviewer's candid feedback, which has helped sharpen both the boundary of the current paper and this multi-agent extension.

References

[1] Hansen, Su, and Wang. TD-MPC2: Scalable, Robust World Models for Continuous Control. ICLR, 2024.

[2] Zhang, Terver, Rabbat, LeCun, and Balestriero. H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning. arXiv:2610.06805, Oct. 2026.

Thank you for providing these additional experiments. I agree that partial structural knowledge can be beneficial even in complex settings compared with having no structural prior.

To clarify my earlier comment, my motivation for mentioning multi-agent systems was to highlight settings where reliable structural knowledge is limited. For example, other agents may act independently, without a reliable model of their behavior being available. Learning their behavior from data may then remain the dominant bottleneck in the unstructured component. From the description provided, it remains unclear whether the additional tasks capture this situation, so I do not view the results as directly addressing this particular concern.

Overall, the additional experiments strengthen the discussion of applicability and help me better understand the paper. However, my main concern about the methodological novelty remains unresolved, and my rating remains unchanged.

Reviewer_zk6pOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the world modeling for partially known dynamics as a decision-making problem, which learns the structural parameters and residual dynamics. It connects the parameters to finite-sample identification and multi-step prediction and planning accuracy. It also does the evaluation through various data settings.

Critical strengths
  1. The motivations and the presentations are clear. The problem is well-defined, as well as the proposed approach. The proposed MDP model for the partially known dynamics setting is simple but effective. I believe it captures the difficulty of this world-modeling setting.
  2. The experiment settings are clear, and the result supports the claim. It also studies core questions Q1-Q3 proposed in Section 6. The result of the experiment shows the effectiveness of the proposed approach.
Critical weaknesses

My major concerns come from theoretical side. The major theoretical contribution is in the appendix. I suggest to summarize the theoretical analysis into section 5. Also, I believe the theoretical analysis needs to address some questions as proposed in the next section.

Questions for authors
  1. How should one model ξt\xi_t for a given problem setting?
  2. In section B.1, Assumptions T1-T6 seems important for theoretical analysis. Could the authors explain why these assumptions are reasonable using some real-world application examples?
  3. Can the analysis of Δ∗=0\Delta^* = 0 extend to nonzero Δ∗\Delta^*?
  4. In real-world application, how do we determine σ\sigma, the sub-Gaussian scale of the transition noise in the finite-sample analysis?
  5. In the model, if the part Δ∗\Delta^* is nonzero, one uses RϕR_\phi to approximate it. Essentially, this is also parameterization. What is the difference between the proposed approach and one directly uses a parametric model?
2 comments
AuthorsOpenReview ↗

Response to Reviewer zk6p (Part 1/2)

We thank the reviewer for the positive assessment of the problem formulation, the method, and the experiments. We respond point by point below, with references to the relevant sections and supplementary results.

W1. Summarize the main theoretical results in Section 5

Agreed. Section 5 of the revised paper presents together the main conclusions of Propositions 1–3 and Corollary 1, how they relate, and the assumptions they require.

Q1. How is ξt\xi_t modeled?

ξt\xi_t is the zero-mean stochastic fluctuation given the state, the action, and the available exogenous input; predictable systematic deviation is assigned to Δ⋆\Delta^\star. Let m⋆(s,a,e)=E[st+1∣st=s,at=a,et=e]m^\star(s,a,e)=\mathbb E[s_{t+1}\mid s_t=s,a_t=a,e_t=e]; then

\xi_t=s_{t+1}-m^\star(s_t,a_t,e_t),\qquad \mathbb E[\xi_t\mid s_t,a_t,e_t]=0.$$ For example, if each susceptible individual is infected conditionally independently with the same probability $p_t$, then $X_t\mid s_t,a_t\sim\mathrm{Binomial}(S_t,p_t)$, and the centered flow $X_t-S_tp_t$ is one source of state disturbance. The formal epidemic experiments in our paper use deterministic transitions, so $\xi_t=0$. The revised paper adds this criterion in Section 3. ## Q2. Why are Assumptions T1–T6 reasonable? T1–T3 are used for finite-sample parameter identification, T4–T5 for propagating model error to the planning value, and T6 gives a sufficient condition for action selection to remain unchanged. In both settings they correspond to checkable quantities: - **T1 (independent observations, conditionally sub-Gaussian noise):** a generative simulator resettable to a specified state supports independent transition sampling; the centered binomial infection flow in a finite population is an example of conditionally zero-mean, sub-Gaussian noise. - **T2 (local identifiability):** different intervention strengths, infection states, or demand and inventory conditions can provide information to distinguish nearby parameters; the condition needs to be checked via local sensitivity or $\lambda_{\min}(I(\theta^\star))\ge\mu>0$. - **T3 (parameter regularity):** the structural transition needs sufficient local smoothness and the regularity required for consistent estimation near the true parameters. In SEIHR, infection probabilities and compartment flows vary smoothly with these parameters; capacity truncation requires checking the piecewise behavior within the parameter neighborhood considered. - **T4 (Lipschitz transition):** within the bounded operating region involved in planning, the effects of state and parameter perturbations on the next state need to be controlled. Linear pipeline shifts and the addition, subtraction, and min/max operations in inventory balances and capacity truncation provide a basis for checking this condition. - **T5 (reward/value regularity):** state errors need to have bounded effects on rewards and subsequent value; inventory holding costs and backlog penalties are examples. The main experiments extend the horizon to $H'$ with a fixed, model-independent rollout policy and use a zero terminal value, corresponding to Proposition 3 at horizon $H'$ with $\epsilon_V=0$. - **T6 (positive action-value gap):** at the current state, the true planning value needs a unique maximizing action and a positive gap. Discrete intervention or ordering actions give settings in which this condition can be checked. The revised paper adds these explanations in Appendix B.1.
AuthorsOpenReview ↗

Response to Reviewer zk6p (Part 2/2)

Q3. Can the analysis be extended to Δ⋆≠0\Delta^\star\neq0?

The error-propagation part can be extended; the identification bound of Proposition 1 still applies only to realizable low-dimensional structural models. Here PP denotes the structural transition FGF_G, with the known or predicted exogenous input sequence omitted, as in Appendix B.6.

For the hybrid model T^hyb=P(⋅;θ^)+Rϕ\hat T_{\mathrm{hyb}}=P(\cdot;\hat\theta)+R_\phi, let ϵR=sup⁡s,a∥Rϕ−Δ⋆∥\epsilon_R=\sup_{s,a}\lVert R_\phi-\Delta^\star\rVert, and let ηNhyb\eta_N^{\mathrm{hyb}} denote its supremum one-step systematic prediction error relative to the true combined transition P(⋅;θ⋆)+Δ⋆P(\cdot;\theta^\star)+\Delta^\star. Under the parameter Lipschitz condition of Assumption T4,

ηNhyb≤Lθ∥θ^−θ⋆∥2+ϵR.\eta_N^{\mathrm{hyb}}\le L_\theta\lVert\hat\theta-\theta^\star\rVert_2+\epsilon_R.

If the true combined transition still satisfies the state-regularity condition of Proposition 2, and the reward/value regularity of Assumption T5 holds, this error can be propagated further to multi-step prediction and the planning value (Appendix B.6 of the original submission). Finite-sample convergence of jointly learning θ\theta and RϕR_\phi further depends on the residual function class, the data distribution, and the optimization process.

Empirically, in R2 of the General Response, after a 25% transit loss is introduced, the return is about −9,458 when only the original parameters are identified, and with the residual added it rises to −8,300.7 at 10k, closing 88.1% of the gap to the structure-matched reference. The revised paper adds the nonzero-mismatch error-propagation result to Section 5.

Q4. How is σ\sigma determined?

σ\sigma is the scale of the conditionally sub-Gaussian noise in Assumption T1, i.e., E[exp⁡(λξt)∣st,at,et]≤exp⁡(λ2σ2/2)\mathbb E[\exp(\lambda\xi_t)\mid s_t,a_t,e_t]\le\exp(\lambda^2\sigma^2/2) for all λ\lambda (in the vector case, uniformly for the projection onto any unit direction). It can be determined as follows:

  • Known stochastic mechanism: a conservative upper bound follows from its concentration properties, e.g., for the centered binomial infection flow with St≤Smax⁡S_t\le S_{\max} one can take Smax⁡/2\sqrt{S_{\max}}/2.
  • Known bounded support: for a conditionally zero-mean scalar disturbance, or a vector disturbance projected onto any unit direction, with a conditional interval of length at most BB, Hoeffding's lemma gives σ=B/2\sigma=B/2.
  • Only observed data: residuals can be used for empirical calibration, but a rigorous upper bound needs additional tail or support assumptions.

Model training does not need σ\sigma. Turning Proposition 1 into a numerical finite-sample error bound also requires justified constants such as μ\mu and Assumptions T1–T3. The revised paper adds this explanation in Appendix B.1.

Q5. RϕR_\phi is also a parameterization; how does it differ from directly using a parametric model?

The difference is that the SD construction first uses the stock–flow, feedback, and delay/pipeline structure to determine the dynamics skeleton, the state memory, and the action boundary, and then leaves the unknown part to parameter identification or to RϕR_\phi. RϕR_\phi and a directly used parametric model can have the same functional form. R1 in the General Response shows that, with an incomplete interface, the return falls from −3,315.4 to −5,208.1 or −18,426.3.

In R2, the residual branch and the black-box model use the same neural architecture (170→256→256→80170\to256\to256\to80, 130,128 neural parameters); the grey-box model additionally retains FGF_G, jointly learns 3 structural parameters, and uses residual regularization and feasibility handling. At 1k, the mechanism + regularized residual model obtains a return of −8,816.4, versus −23,500.0 for the black-box model.

Reviewer_vT6sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce a procedure for learning a state transition function when some structural knowledge about the environment is known. This structural knowledge is modeled using an adapted framing of System Dynamics. The primary benefit expressed is sample efficiency; by exploiting and refining underlying structure in the problem, downstream planning and learning approaches can converge to better results quicker and with fewer data samples. Empirically, the authors convincingly demonstrate the sample-efficiency of their framework across three task families with partially known "grey-box" settings with finite-data.

Critical strengths
  1. Representation. The idea is intuitive enough and well executed. System Dynamics provides a strong foundation on which to model partially understood dynamics, due to its ability to represent accumulation, transfer, feedback, and delays. Also, the manuscript appears to be well written, the first figure is illustrative, and the appendix is thorough.
  2. Experiments. The empirical results are convincing, through the authors' comprehensive evaluation across finite-data regimes, graph structures, and integration interfaces. It is evident that this framing is effective in improving sample-efficiency, although there are baseline omissions, per the below.
Critical weaknesses
  1. Novelty. While gains over black-box learning are evident, it's not clear to me that the approach presented is meaningfully novel. Modeling partially known structure into learned transition functions is well-trodden ground. The primary contribution of the paper appears to be the reliance on the System Dynamics representation adapted to decision making. Still, those same dynamics would seem to feasibly be implemented in a state-space model or standard transition function. It would be helpful for the authors to clarify exactly why the SD representation provides over existing structured modeling or identification approaches.
  2. Baselines. The authors compare against data-pure learning to demonstrate sample efficiency. But there is no comparison against other approaches for modeling partially known dynamics. This connects with the concern in 1, where there is desire to clarify exactly why the SD framing is especially beneficial.
Questions for authors
  1. Adaptability and Generalization. A natural question for this type of work would be the extent to which "getting it wrong" on the underlying partially known structure can be remedied with sufficient data. In other words, how much misspecification can be corrected while maintaining a sample-efficiency advantage.
2 comments
AuthorsOpenReview ↗

Response to Reviewer vT6s (Part 1/2)

We thank the reviewer for the positive assessment. To address the three concerns on novelty, baselines for partially known dynamics, and structural mismatch, we added three experiments: R1 (decision-interface ablation) and R2 (structural mismatch and residual correction) address several reviewers at once, with full results in the General Response; R3 (external grey-box baseline) specifically addresses W2 and is reported below. R3 uses a four-step MPC and an extended training protocol with validation-based model selection, which differ from the MCTS setup of R1 and R2, so returns are not compared across experiments.

W1. Compared with existing structured modeling and system identification, what does the SD representation additionally provide?

Compared with a modeling starting point in which the state–action transition function is given directly, SD provides explicit mechanism semantics for constructing that function: which relations describe the environment's response, which rules generate the agents' decisions, and which delays and pipelines carry intertemporal effects. For example, in the Beer Game, the ordering rule and the shipping pipeline respectively state "who decides the order" and "how an order affects future arrivals".

Using this mechanism information, Section 4.1 and Appendix A.2 give a six-step procedure and a state-transition completeness criterion for constructing a planning interface from an existing SD model. This role of the SD representation lies in the interface-construction stage. We agree that, once the action boundary and the recursive state are fully determined, the same dynamics can be implemented equivalently in an augmented state-space form, and the parameters can be learned with standard system-identification methods.

R1 tests the practical effect of action externalization and dynamic-memory retention: the full construction and the equivalent state-space implementation give the same result (−3,315.4 ± 148.0), while not fully externalizing the action and omitting the shipping, order, and source-node pipeline memory reduce the return to −5,208.1 and −18,426.3, respectively.

AuthorsOpenReview ↗

Response to Reviewer vT6s (Part 2/2)

W2. Comparison with other methods for partially known dynamics

We added a "nominal dynamics + Gaussian-process (GP) residual" baseline, implemented following Hewing, Kabzan, and Zeilinger, Cautious Model Predictive Control Using Gaussian Process Regression (IEEE TCST, 2020; the revised paper adds this citation). Its nominal logistics transition is written independently and does not call our SD module; it uses the same topology, shipping delays, allocation rule, and initial demand prior. Both methods use the same MPC with four steps and 1,250 candidate trajectories, share the training data and a validation set split by whole episodes, and select models by 8-step validation error.

When the structure is correct (the only unknown part is the exogenous demand), both methods obtain an MPC return of −2,986.4 ± 139.3 at 1k / 3k / 10k, and the returns agree on all 150 paired evaluation episodes.

Under structural mismatch (the same 25% transit loss as in R2; neither method knows the loss rate or where the loss occurs), GP uses the full state and order quantities to correct the non-clock states with an independent-output sparse GP with 64 inducing points. Our method uses joint parameter learning, a regularized neural residual, and non-negativity handling. The table reports the mean ± sample standard deviation over five seeds, with ten common evaluation episodes per seed.

Interaction budgetSD one-step validation RMSE ↓GP one-step validation RMSE ↓SD MPC return ↑GP MPC return ↑
1k0.507 ± 0.0410.458 ± 0.020−10,037.7 ± 270.7−9,424.9 ± 284.1
3k0.356 ± 0.0190.441 ± 0.026−10,103.1 ± 385.4−9,418.1 ± 200.2
10k0.207 ± 0.0170.429 ± 0.035−9,520.8 ± 193.7−9,431.8 ± 179.4
30k0.088 ± 0.0040.406 ± 0.014−8,974.6 ± 185.0−9,307.4 ± 202.1
100k0.046 ± 0.0020.369 ± 0.015−8,985.3 ± 141.4−9,263.7 ± 139.5

GP is better at low budgets, and the two are similar at 10k; at 30k and 100k, SD outperforms GP on all five paired seeds, leading by 278.4 on average at 100k (paired 95% interval [107.2, 449.6]). SD at 30k and 100k approaches the reference model with the true loss coefficient known and the demand process accurately recovered (−8,981.8 ± 177.1).

At every budget, all five GP seeds reach the preset stable plateau: over the last five checkpoints, the relative range of both the training objective and the 8-step validation error is at most 1%, and the best validation error has not improved materially for at least eight checks. SD reaches the update limit in all cases; all models are selected by validation metrics only (Appendix H).

For the models selected at 30k, batched prediction for 1,250 states takes about 6.4 ms for SD and 127.5 ms for GP, measured on the same machine with a single CPU thread, double precision, and no query cache. A single-state query takes about 0.48 ms for both; the computational advantage lies mainly in batched planning queries.

Q1. How much structural mismatch can be corrected while keeping the sample efficiency?

Under the 25% transit loss in R2, the residual model closes 49.1% / 57.3% / 88.1% of the mean-return gap to the structure-matched reference at 1k / 3k / 10k, and its return at 1k (−8,816.4) is higher than that of the black-box model at 10k (−10,861.1). When only the original parameters are identified, the return stays at about −9,458 from 1k to 10k, showing that, within the current three-parameter model family, more data did not repair the omitted transit-loss mechanism.

Whether correction is possible depends on whether the omitted mechanism can be represented by the current state–action information and whether the data cover its effects; if what is omitted is necessary dynamic memory or unobserved factors, the state or observations need to be extended.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank the three reviewers for their reviews. The shared comments center on three points: novelty relative to system identification (KTPX W1/Q1, vT6s W1, zk6p Q5); whether structural mismatch can be corrected (vT6s Q1, KTPX W1, zk6p Q3); and a consolidated presentation of the theoretical results (zk6p). Below we state the positioning of our contribution and report R1 and R2, which address several reviewers at once; point-by-point responses are in the individual replies. The revised paper is submitted together with this response.

Positioning of the contribution. We agree that "structural priors improve sample efficiency" is already known; once the structural model and a complete state–action interface are given, the parameter learning in our paper is structured (grey-box) system identification.

A standard SD model usually writes decision rules inside the model and keeps intertemporal effects in delay and pipeline variables. Our focus is how to construct such an existing SD model into a world model for planning. Section 4.1 and Appendix A.2 give a six-step procedure and a state-transition completeness criterion: externalize the focal decision as a planning action, keep the dynamic memory needed for prediction in the recursive state, and check whether the one-step transition is complete.

Section 4.2 and Appendix B.6 discuss residual correction and error propagation when the structural parameters cannot describe the true dynamics. Section 5 links parameter-identification accuracy to multi-step prediction error, planning-value error, and the condition under which action selection is preserved.

Common setup. Both are run on the T4 topology (a ten-node supply network; the topology labels are unrelated to the theoretical assumptions T1–T6), using fixed offline data collected with 50% random policy and 50% base-stock policy; within each experiment, all methods use the same data from the same seeds. Results are the mean ± sample standard deviation over five random seeds, with ten common evaluation episodes per seed. R1 and R2 use different dynamics, so their returns are not compared across experiments. The main experiments in the paper collect data online, so the return of the full construction in R1 differs slightly from the −3,404.5 ± 183.1 in Table 15; the difference is within one sample standard deviation.

R1: Decision-interface ablation (nominal dynamics, 1k interactions, three-parameter identification, MCTS)

Model construction8-step prediction RMSE ↓MCTS mean return ↑
Full construction0.588−3,315.4 ± 148.0
Equivalent augmented state-space implementation of the same interface (positive control)0.588−3,315.4 ± 148.0
Action not fully externalized; part of the built-in ordering rule kept14.678−5,208.1 ± 185.0
Shipping, order, and source-node pipeline memory omitted17.963−18,426.3 ± 205.3

Under the same parameter-identification setup, action externalization and dynamic-memory retention both have a practical effect; the same complete interface can be implemented in different representational forms.

AuthorsOpenReview ↗

General Response (continued): R2 and Paper Revisions

R2: Structural mismatch and residual correction (fixed 25% transit loss: true arrival rate 0.75, nominal model 1.0; MCTS mean return ↑)

Model1k3k10k
Misspecified structure, original parameters only−9,459.7 ± 205.6−9,456.1 ± 174.5−9,457.9 ± 177.3
Same misspecified structure + regularized residual with non-negativity constraint (ours)−8,816.4 ± 176.9−8,690.5 ± 167.2−8,300.7 ± 175.4
Purely black-box neural network−23,500.0 ± 2,194.7−17,582.0 ± 960.7−10,861.1 ± 2,068.7
Structure-matched reference with the true loss coefficient known; other parameters still learned−8,149.1 ± 221.8−8,120.0 ± 76.7−8,145.0 ± 156.7

The transit loss lies outside the original three-demand-parameter model family. When only the original parameters are identified, the return stays at about −9,458 from 1k to 10k. With the residual added, 49.1% / 57.3% / 88.1% of the mean-return gap to the reference is closed at 1k / 3k / 10k. At 10k, the paired mean gain over the purely parametric model is +1,157.3 (95% bootstrap confidence interval [1,043.8, 1,270.7]); the residual model's return at 1k is also higher than that of the black-box model at 10k.

Correction depends on whether the missing mechanism can be represented by the current state–action information and whether the data cover its effects. If what is omitted is necessary dynamic memory or unobserved factors, the state or observations need to be extended accordingly.

Main changes in the revised paper (using the extra page of main text allowed in the discussion phase; original conclusions unchanged): Sections 1 and 4.1 add the positioning of the contribution; Section 2 adds related work on grey-box system identification and residual modeling; Section 3 adds the conditional zero-mean criterion for the stochastic disturbance; Section 5 presents together the main conclusions of Propositions 1–3 and Corollary 1, the assumptions they require, and the error propagation under nonzero structural mismatch.

A new Section 6.5 summarizes R1–R3; Section 7 adds a discussion of the scope of applicability; Appendices A.2, B.1, and B.6 add the corresponding explanations, and a new Appendix H gives the full setups and results of R1–R3. The external grey-box baseline comparison R3 is reported in the reply to vT6s W2.

AuthorsOpenReview ↗

Thank You to the Area Chair and Reviewers

As the author–reviewer discussion period comes to a close, we would like to sincerely thank all reviewers and the Area Chair for the time, effort, and thoughtful feedback you have dedicated to our submission. Your constructive comments have helped us significantly strengthen the positioning, theoretical clarity, and empirical evaluation of our work. We have revised the manuscript accordingly and provided additional clarifications and experimental evidence during the discussion. We truly appreciate your patience and consideration throughout this process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.