acceptodds
Under review as a conference paper at ICLR 2027

Delayed Heuristic-Behavior-Target Coupling for Off-Policy Reinforcement Learning in Inventory Control

Abstract

Many reinforcement learning methods tie data collection to the task policy, while coordinating independently optimized behavior and target policies remains challenging. Policies learned by these methods can perform unreliably in long-horizon tasks under uncertainty, such as inventory control, where failures can incur extreme costs. We introduce Heuristic Behavior Policy Optimization (HBPO), an off-policy framework that routes heuristic guidance through persistent behavior learning, coupled with target-policy, updating separately with residual-based advantage estimation and emphatic weighting. Under matched environment interaction budgets on DSIS and its risk-augmented variant DSIS-RB, HBPO recovers many high-cost initial policies and improves most non-collapse comparison groups. Cold starts achieve competitive costs without final cost collapse across all tested seeds. We identify a transmission chain in the delayed heuristic–behavior–target coupling as a key driver of recovery and improvement. Generation-level diagnostics trace heuristic uptake and subsequent selective target responses, while component ablations support the contribution of this transmission mechanism to policy recovery and improvement.

Primary area:
reinforcement learning
Keywords:
off-policy reinforcement learning, behavior policy optimization, policies transmissions, inventory control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_y1AjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript introduces Heuristic Behavior Policy Optimization (HBPO), an off-policy RL framework that keeps separately learned behavior and target policies. The behavior policy is shaped by heuristic guidance (I think this is section 4.1, but I am not really sure) and this informs the target through a delayed, advantaged-filtered transfer term. On two inventory benchmarks (DSIS and DSIS-RB), HBPO lowers mean cost in 17 of 18 warm-start settings and recovers 35 of 36 collapsed baseline runs. Ablations support the design decisions, though validation is limited to a single domain.

Critical strengths

None.

Critical weaknesses
  1. The writing is lacking precision, filled with jargon, and is incredibly difficult to follow intuitively. Overall, there are many variables to keep track of (5 pages, 14-19!) which adds to the difficulty of parsing this work, and their definition lacks any motivation.

Here are a few examples from the initial pages:

  • Lines 053-056 aim to describe HBPO at a high level but use terms like "delayed heuristic–behavior–target transmission", "persistent behavior policy" (what does persistent mean in this context?), "residual-based advantage estimation supporting both policies and emphatic weighting carrying heuristic confidence forward". The abstract is similarly worded, and it is not clear at all what any of these terms mean in the paper's context, making these descriptions non-useful to the reader.
  • Lines 070: so the "data source cannot outgrow the target?" Once again, what does this mean? The target is a policy, the data source is not so it's not clear what growing a data source means and how this relates to the target policy. If this refers to exploring states beyond those visited by the target policy, this should be mentioned more explicitly.
  • L037: "can heuristic guidance applied to it and behavior policy persistently influence the target policy?" What is "it" in this context
  • Lines 075: "Correction schemes persistently track the behavior...reconcile it with target updates without shaping it, ... so correction grows with the gap" Which gap, reconcile how, shape what? In what way?
  • Line 093: introduces symbols without definition
  • Line 116: mentions the action space, but not what these indices represent - what is a regular/expedited order in this context.
  • Lines 110-112: acronyms (DSIS, IID, DSIS-RB) are mentioned without being introduced.
  • Lines 149-150: subscript g is mentioned, and only defined later as a generation.
  • Lines 152-153: long-horizon comparisons of alternative first actions followed by πg\pi_g construct a temporary heuristic policy qH,gq_{H, g}. How, and what is HH? Ah, defined later as heuristic guidance, but it is still not clear how it factors into qq.
  • Line 168: Why define EHE_H for expectation under another distribution?
  • Line 158-160: What are Vg,QgV_g, Q_g?
  • Lines 154: What are the mathematical definitions of A,J,PA, J, P?
  • Lines 160-161, why are masses being introduced 3 sections ahead? What is the reader meant to take from this information, how does they related to emphatic weights and how are emphatic weights used?

There are more concerns, for the remaining pages but with the background material here being hard to follow - everything that follows is immediately hard to understand as well.

There are >60 pages of appendices, with several expanded definitions and theorems. These are referred to several times in the paper, but if they are actually important, these results (namely theorems) should be present in the paper itself and motivated. Definitions are provided repeatedly as well, but appear heuristic rather than motivated.

Overall, the work is filled with arbitrary design decisions without clear explanation. It is not clear to me why certain things work and others do not, which makes me concerned about HBPO's generalization and how it may be applied to other methods in the future.

Questions for authors

I found the paper to be extremely dense, containing many variables/terms without precise definition, making the work incredibly hard to follow. There are too many questions/clarifications to ask for from this standpoint, and 1 or 2 answers would not change my assessment. I do not think this work is ready for publication in the current form.

4 comments

Thank you for your careful reading and detailed comments. We appreciate the specific examples identifying where the terminology, notation, and organization make the paper difficult to follow. We will revise the presentation throughout so that the method’s intuition, definitions, and design rationale are clear from the main text. This includes simplifying the abstract and introduction, defining notation before use, explaining the motivation behind each component, and presenting the central theoretical statements and assumptions alongside their practical implications. We will also reorganize the appendix, consolidate repeated definitions, and carefully proofread the entire manuscript for clarity, grammar, and consistency.

HBPO is motivated by the question of how improving the policy that collects experience can produce sustained improvement in the policy ultimately evaluated. In long-horizon problems, the collector determines both the actions sampled and the downstream states encountered. HBPO therefore learns a behavior policy separately from the target. Model-based lookahead provides guidance about long-term action consequences, which the behavior policy integrates with target-advantage feedback across training rounds. The target learns from the collected experience and an explicit transfer objective: the magnitude of behavior–target disagreement determines the strength of the transfer signal, while estimated target advantages determine its direction. This gives behavior learning a direct role in target optimization. Shared value estimates provide a consistent basis for judging actions, and state weights determine where the target is improved and transfer receives emphasis. These components address distinct questions: how to improve collection, which behavioral changes should influence the target, and where that influence should be strongest.

We provide the answers for the specific questions below:

  • Lines 053–056: High-level terminology. “Persistent behavior policy” means a separately parameterized policy whose learned parameters are retained and updated across training rounds. The heuristic controller is temporary and supplies fitting targets for this policy.

    “Delayed transmission” describes when newly introduced heuristic guidance reaches the target through behavior learning. In generation gg, μg\mu_g collects Dg\mathcal{D}_g, and the target update producing πg+1\pi_{g+1} uses these data together with the frozen parent policies and estimates. Meanwhile, πg\pi_g defines a temporary heuristic controller qH,gq_{H,g}, whose guidance combines with target-advantage feedback to produce μg+1\mu_{g+1}. An immediate coupling would use this newly updated behavior policy in the same generation’s target update. HBPO instead retains μg+1\mu_{g+1} as an independently learned policy: in generation g+1g+1, it collects Dg+1\mathcal{D}_{g+1} and enters the explicit advantage-filtered transfer term used to produce πg+2\pi_{g+2}. The chain πg→qH,g→μg+1→πg+2\pi_g \to q_{H,g} \to \mu_{g+1} \to \pi_{g+2} therefore specifies how heuristic guidance is carried across generations through both subsequent data collection and direct policy transfer. The delay concerns the use of newly learned behavior; the current batch Dg\mathcal{D}_g already contributes to the current target update. This timing is central to the interaction among the heuristic controller, persistent behavior policy, and target policy in HBPO.

    “Residual-based advantage estimation” uses accumulated model-based Bellman residuals to correct critic-based action-value estimates used by both policies.

    “Emphatic weighting carrying heuristic confidence forward” means that evidence of heuristic usefulness enters learned state weights, propagates through behavior transitions, and affects later target updates.

  • Line 070: “The data source cannot outgrow the target.” “Data source” refers to the behavior policy, the data-collecting policy. The intended meaning is that, when collection uses the current task/target policy, the collector cannot be independently optimized to favor different actions and thereby change the distribution of visited states. “Outgrow” is an imprecise description of this restriction, but I just want to offer an intuitive picture here.

  • Line 037: The referent of “it.” “It” refers to the behavior policy. The question is whether heuristic guidance applied to a separately learned behavior policy can continue to influence target-policy improvement across successive updates.

  • Line 075: “Gap,” “reconcile,” and “shape.” The “gap” refers to differences between behavior and target action probabilities. “Reconcile” refers to accounting for those differences when using behavior-generated data to estimate quantities for target learning. “Shaping behavior” means directly changing the collector’s action distribution through a learning objective.

  • Line 093: Symbols in the transmission chain. In πg→qH,g→μg+1→πg+2\pi_g \to q_{H,g} \to \mu_{g+1} \to \pi_{g+2}, gg indexes training rounds, πg\pi_g is the current target policy, qH,gq_{H,g} is the temporary heuristic controller constructed using that target, and μg+1\mu_{g+1} is the updated behavior policy. Each round computes its updates from frozen parent policies. Consequently, μg+1\mu_{g+1} first enters the following round’s target update, producing πg+2\pi_{g+2}.

  • Line 116: Action components and ordering channels. In at=(qtr,qte)a_t=(q_t^r,q_t^e), tt indexes the decision period, and the superscripts rr and ee identify regular and expedited orders. Each component is an order quantity between 0 and 8 units. Regular orders arrive four periods later. Expedited orders arrive immediately at higher unit cost.

  • Lines 110-112: Acronyms. DSIS denotes the Dual-Sourcing Inventory System; IID means independent and identically distributed, describing demand in DSIS; and DSIS-RB denotes DSIS with Risky Behavior, the extension incorporating persistent demand and supply disruptions.

  • Lines 149-150: The generation index. A “generation” is one complete training round: the current behavior policy collects data, update signals are computed using the current frozen policies and supporting estimates, and the resulting successors become the inputs to the next round. The subscript gg identifies these successive versions.

  • Lines 152-153: Constructing the temporary heuristic policy. At a queried state, each candidate first action is evaluated through 1,000-step model simulations, with all subsequent actions following the frozen target policy. The estimated returns are centered relative to the target’s action distribution to obtain action utilities. The controller qH,gq_{H,g} maximizes expected utility subject to a regularization penalty for departing from the target distribution. Thus, it assigns action probabilities using their estimated long-term consequences. The subscript HH identifies heuristic guidance. The corresponding behavior-learning component fits the behavior policy toward this controller.

  • Line 168: The expectation notation. EH\mathbb{E}_H is shorthand for expectation in the heuristic evaluation. Its randomness comes from the simulation model, subsequent actions sampled from the target policy, and the synthetic-risk draw in DSIS-RB. The subscript labels the purpose of the expectation; the underlying trajectory distribution is specified by these sampling rules.

  • Lines 158-160: Critics and action values. Vg,j(s)V_{g,j}(s) is the state-value estimate from critic jj, with j=1,…,5j=1,\ldots,5. Their average, vg(s)=15∑jVg,j(s)v_g(s)=\frac{1}{5}\sum_j V_{g,j}(s), is held fixed during the current round and supports heuristic evaluation and action-value estimation. Qgπg(s,a)Q_g^{\pi_g}(s,a) denotes the expected discounted return under the simulation model after taking action aa at state ss and following πg\pi_g thereafter. The residual correction accounts for continuation-value error in the critic-based estimate within that model.

  • Line 154: Mathematical definitions of A,J,PA,J,P. Let μ,π\mu,\pi be the frozen behavior and target policies, and m,pm,p their candidate updates. Write A^(s,a)\widehat{A}(s,a) for the estimated target advantage, centered under π\pi, and ED\mathbb{E}_D for averaging over collected states. The objectives are

    A(m) &=\frac{1}{\tau}\mathbb{E}_D\\!\left[ \sum_a(m(a\mid s)-\mu(a\mid s))\widehat{A}(s,a)\right],\\\\ J(p) &=\mathbb{E}_D\\!\left[ w^J(s)\sum_a(p(a\mid s)-\pi(a\mid s))\widehat{A}(s,a)\right],\\\\ P(p) &=\frac{1}{\tau}\mathbb{E}_D\\!\left[ w^P(s)\sum_a|\mu(a\mid s)-\pi(a\mid s)| \left(\frac{p(a\mid s)}{\pi(a\mid s)}-1\right)\widehat{A}(s,a)\right]. \end{aligned}$$ Here, $\tau > 0$ is the heuristic regularization coefficient, also used to scale the objectives, and $w^J,w^P$ are the state weights. $A$ improves behavior using target advantages, $J$ directly improves the target, and $P$ supplies the additional disagreement-weighted transfer signal.
  • Lines 160–161: Masses and emphatic weights. The sentence previews the state weighting used by the learning objectives. The masses MJ,MPM^J,M^P are positive learned state-importance functions. Normalization and shrinkage convert them into mean-one weights wJ,wPw^J,w^P, which determine each state’s contribution to JJ and PP, respectively. MJM^J propagates importance through target-policy continuation; MPM^P incorporates heuristic-improvement confidence and propagates importance through behavior transitions. Their role is therefore to determine where the two target-learning signals receive emphasis.

I thank the authors for their detailed rebuttal to clarify the concerns I raised regarding the writing of the manuscript. While I think the idea of optimizing the behavioural policy to ultimately improve the target policy is an interesting and important direction, I still cannot recommend this paper for acceptance in its current form due to concerns with clarity. These concerns are shared by the other reviewers and are too substantial, in my opinion, to be considered within the scope of rebuttal. Not to mention, the authors have not provided a revised manuscript towards addressing these concerns.

I encourage the authors to revise the work to improve the communication of their ideas and incorporate the clarifications they have provided above, which I believe improve the work and will facilitate reproduction and future developments.

Reviewer_8w8SOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the long-horizon inventory control problem by proposing the HBPO framework, which decouples behavior-policy training from target-policy training. HBPO employs model-based long-horizon action evaluation to construct a temporary heuristic policy. It combines this heuristic guidance with target-policy advantage estimates to update the behavior policy. The target policy is updated using its own advantage estimates, together with an advantage-weighted transfer mechanism that accounts for discrepancies between the behavior and target policies across successive generations. The framework also incorporates residual-based advantage estimation and emphatic state weighting. Experiments in two simulated dual-sourcing inventory environments, DSIS and DSIS-RB, comparing against nine baselines under the same real-environment interaction budget. The results show that HBPO improves final policy performance across various experimental settings and facilitates recovery from high-cost policies. Furthermore, in cold-start scenarios, the method avoids final performance collapse across all tested random seeds.

Critical strengths

Strong empirical performance in recovering unstable policies.HBPO shows promise in recovering poorly performing policies and mitigating catastrophic failures in long-horizon inventory control tasks. Under matched real-environment interaction budgets, it improves 35 of 36 seed-matched cases where the original baselines exhibit high final costs, with 29 recovering to the non-collapse regime. All evaluated cold-start runs also avoid final cost collapse in both DSIS and DSIS-RB.

Critical weaknesses

Why is a separately trainable behavior policy necessary? The behavior and target actors use different update terms, but this alone does not show that heuristic information must pass through a persistent behavior actor. A more direct alternative is to apply the same qHq_H, model-based action-value estimates, and heuristic-update budget directly to the target policy. The current ablations do not test this alternative. A direct heuristic-to-target baseline is necessary to determine whether the intermediate behavior actor is responsible for the improvement.

The learned behavior actor is still not separated from stronger heuristic-guided exploration. No-P shows that explicit behavior-to-target transfer can help beyond behavior-generated data alone. Still, it does not show whether the separately learned behavior policy does more than produce better trajectories. A cleaner comparison would use the same heuristic controller to guide data collection while removing the persistent behavior actor and the explicit behavior-target transfer. Without such an exploration-only baseline, it is difficult to tell whether HBPO introduces a distinct learning mechanism or mainly improves the state-action distribution seen during training.

Questions for authors

Why is a separately trainable behavior policy necessary? The behavior and target actors use different update terms, but this alone does not show that heuristic information must pass through a persistent behavior actor. A more direct alternative is to apply the same qHq_H, model-based action-value estimates, and heuristic-update budget directly to the target policy. The current ablations do not test this alternative. A direct heuristic-to-target baseline is necessary to determine whether the intermediate behavior actor is responsible for the improvement.

The comparison does not separate the effect of HBPO’s coupling mechanism from the large amount of model-based lookahead available to it. At each queried state, HBPO evaluates the full 81-action space using multiple long-horizon model rollouts with a 1000-step horizon. This provides much richer counterfactual action information than most of the native RL baselines receive. Matching the number of real environment interactions therefore does not make the comparison computation- or information-matched.

A directly relevant model-based inventory-control baseline is missing. DSIS adapts the dual-sourcing inventory problem of Böttcher et al. (2023), who optimize inventory policies by embedding system dynamics in a neural controller. Because HBPO also relies heavily on access to, or estimation of, the transition model, comparing it only with standard RL methods and GPS is insufficient to establish its advantage over existing model-based inventory optimization.

The learned behavior actor is still not separated from stronger heuristic-guided exploration. No-P shows that explicit behavior-to-target transfer can help beyond behavior-generated data alone. Still, it does not show whether the separately learned behavior policy does more than produce better trajectories. A cleaner comparison would use the same heuristic controller to guide data collection while removing the persistent behavior actor and the explicit behavior-target transfer. Without such an exploration-only baseline, it is difficult to tell whether HBPO introduces a distinct learning mechanism or mainly improves the state-action distribution seen during training.

PPO methods that incorporate trajectory modeling—such as through the use of RNNs—are employed to address so-called long-horizon tasks.

3 comments

We highly appreciate your constructive comments. We clarify below the role of behavior learning, the computational setting, and the scope of our experiments.

1. Why retain a separately trainable behavior policy?

The behavior policy integrates two complementary signals: long-horizon heuristic guidance HH and target-advantage feedback AA. The target then combines its own learning objective JJ with selective transfer PP. This design addresses two distinctions: actor mismatch, where behavior and target action distributions differ, and state-emphasis mismatch, where collection frequencies differ from the importance assigned to states for target learning. Coupled behavior and target policy is designed to solve the actor mismatch and emphatic weighting is designed to solve the state-emphasis mismatch.

The temporary qHq_H already incorporates target-policy information through its continuation policy and regularization prior. Using it directly as behavior would remove the persistent integration of AA and HH. Our existing No-A ablation supports this integration: Full achieves lower costs in 35/40 paired runs. This result interprets that single heuristic guidance may not perform well and the necessity of a persistent behavior to exist while heuristic guidance provides simulation information for it.

2. Does behavior learning contribute beyond better trajectories?

No-P retains behavior-generated data while removing the explicit transfer objective. In Cold experiments, Full achieves lower costs than No-P on every seed in both environments; mean costs increase from 45.18 to 77.47 on DSIS and from 70.67 to 82.19 on DSIS-RB when P is removed. Full also outperforms KL-P and Near-on-policy on every Cold seed. These results support explicit, selective transfer within HBPO beyond the behavior policy’s data-collection role.

3. Computation and information budgets

Our experiments match real-interaction budgets, while HBPO uses additional simulation. This computation supports reliable heuristic guidance and value estimates for the transmission πg→qH,g→μg+1→πg+2\pi_g \to q_{H,g} \to \mu_{g+1} \to \pi_{g+2}. HBPO and GPS share structural priors in DSIS-RB. The baselines involve different learning architectures and computational workloads, making aligning the computation costs difficult.

To test sensitivity to simulation intensity, we evaluated HBPO-Lite on both environments with all five PPO warm-start seeds. Lite reduces simulation repetitions for heuristic utility, residual continuation, and confidence estimation from 32 to 8, retaining the initialization, planning horizon, and coupling mechanism. All three methods use the same 2M real-interaction budget and evaluation protocol.

Final costs across five paired training seeds (lower is better):

DSIS

MethodSeed 0Seed 1Seed 2Seed 3Seed 4
HBPO-Full20.7321.2651.41158.8820.76
HBPO-Lite20.7221.6451.6383.5320.77
PPO21.0121.1756.8913817.3021.33

DSIS-RB

MethodSeed 0Seed 1Seed 2Seed 3Seed 4
HBPO-Full52.6881.935893.0155.30112.01
HBPO-Lite51.7884.135865.0560.44107.10
PPO70.0675.4666970.7575.1374.88

Lite preserves the large mean-cost improvements over PPO continuation, with median costs close to Full, despite a fourfold reduction in these per-estimate sampling quotas.

4. Inventory-control baselines and benchmark scope

Böttcher et al.’s dynamics-embedded neural controller is a relevant model-based comparator. Our focus is the RL learning mechanism: how heuristic guidance shapes behavior learning and subsequent target improvement. DSIS provides transparent inventory dynamics with delayed consequences, while DSIS-RB introduces hidden persistent risk, allowing this interaction to be examined under uncertainty.

The nine baselines assess improvement across different RL policy sources. We will clarify that our empirical contribution concerns an effective RL framework in these inventory environments. Here is the baseline scope and why we compare them with HBPO.

  • GPS: model-based guided policy learning, adapted to the categorical action space.
  • TPO: a recent approach that fits policies to improved target distributions.
  • PPO, A2C, and TRPO: standard policy-optimization references.
  • PPO-81: controls for the 81-action categorical parameterization.
  • Discrete SAC: off-policy actor-critic learning with replay and entropy regularization.
  • ACE: emphatic weighting in off-policy actor-critic learning.
  • GRPO: uses the updated policy directly for subsequent data collection.

5. Recurrent PPO and presentation

We highly appreciate your kind advice to mention PPO through the use of RNN Recurrent PPO , which adds memory to policy representation. HBPO addresses long-term action consequences through model-based evaluation and coordinated policy updates. These are complementary design dimensions; memory may be particularly useful for the hidden risk in DSIS-RB. For example, risk remembered and re-used in next simulations, combined with the same structural prior, will definitely cause a simulated trajectory to be recognized as a re-appear of the past trajectories. This is avoided in all baseline and HBPO algorithms. And this will bring more information that can mis-leading to the interpretation of better performance.

I thank the authors for their response. However, the authors did not directly address my questions 1, 3, and 4.

I could not understand the argument in point 5. Recurrent PPO does not use model simulations, so I do not see where the statement "risk remembered and re-used in next simulations" comes from.

Regarding DSIS-RB, the cost reported in the response is 70.67, whereas Table 1 and the appendix report 72.1 and 72.115, respectively. There is a discrepancy.

The statement in the authors' rebuttal, "Coupled behavior and target policy is designed to solve the actor mismatch and emphatic weighting is designed to solve the state-emphasis mismatch.", contradicts the statement in the paper, "The ablations in Appendix D.3 show mixed endpoint effects, so a consistent final-cost benefit is not established in the evaluated settings. We adopt this pathway considering its theoretical advantages, which not obviously show in the experimental endpoint."

In addition, the authors did not upload a revised manuscript.

Reviewer_nYjLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Heuristic Behavior Policy Optimization (HBPO), which maintains separate behavior and target policies and uses a temporary model-based controller to guide behavior learning. Target updates combine advantage optimization with selective transfer based on behavior-target disagreement. Frozen-parent updates create a delayed heuristic-behavior-target pathway. The framework also incorporates residual-completed value estimates, learned emphatic weights, and fused natural-gradient updates. Experiments on two inventory environments show substantial improvements over several unstable RL baselines, especially when switching from pretrained policies.

Critical strengths
  1. The motivation of this paper is clear and convincing: improper coordination in training the behavior policy and the target policy can cause unreliable performance, which is practically important in inventory control.
  2. The selective-transfer mechanism is appealing: HBPO addresses this problem through a coupling mechanism that allows planning guidance to shape data collection while giving the target policy a selective way to use the resulting behavioural deviations.
  3. The warm start experiments show promising improvements in recovering poor policies.
Critical weaknesses
  1. The manuscript requires thorough proofreading. Two specific examples are highlighted below, and the entire text should be carefully reviewed and revised: a. Many symbols and abbreviations are used before explained: the symbols in line 093 are introduced two sections later, DSIS and DSIS-RB are not explained what they are short for. Although readers can eventually find these in later sections or cited literature, defining them earlier would significantly improve readability for an interdisciplinary audience. b. Some sentences are incomplete or contain grammar mistakes. For example, at line 37-38, “which is the policy finally want to optimize” should be “which is the policy we/users finally want to optimize. Such issues frequently occur in key sections, making the paper’s methodology and contributions unnecessarily difficult to follow.

  2. HBPO uses counterfactual action evaluation, long simulated continuations, synthetic critic training, and substantial initialization. Even with equal real-interaction budgets, HBPO still receives more information and computation than its baselines. Ensuring a fairer comparison would make the paper's conclusions much more convincing.

  3. There are no comparisons with closely related methods. The paper compares with 9 RL baselines. However, most of them are generic, naïve policy gradient/actor–critic methods. There is no comparison to recently proposed algorithms that directly address model based policy improvement and guided policy learning.

  4. It is not clear what contributions the authors want to claim. If it is a new off policy RL framework that perform well in long horizon tasks under uncertainty, then the authors should evaluate their method on well adopted RL benchmarks such as MuJoCo, to evaluate whether their algorithm outperform others. If, however, the contribution is a better RL method for inventory control problems, then the authors need to do comparisons against established inventory control heuristics, which would help show the competitiveness of the proposed HBPO on solving this particular problem.

  5. The DSIS and DSIS-RB are similar inventory control environments. Such inventory control problems are not well known to general AI researchers. This is itself not a problem. However, since inventory control is not usually used as benchmark problems in RL, the authors need to help readers understand how difficult these problems are. Can you get optimal policies using dynamic programming?if not, what are state of the art heuristics for this problem, and how your method compares?

Questions for authors

See weakness.

2 comments

We highly appreciate your constructive comments. Our central contribution is explicit, delayed heuristic-behavior-target coupling for policy recovery and stable improvement in inventory control. We respond point by point below.

1. Presentation and definitions.

We will thoroughly proofread the manuscript, correct grammatical errors, and define symbols and abbreviations at first use. DSIS stands for “Dual-Sourcing Inventory System”; DSIS-RB stands for “DSIS with Risky Behavior,” extending DSIS with persistent demand and supply disruptions. We will explain these environments when first introduced.

2. Interaction budgets and computation.

Our experiments match real-interaction budgets, while HBPO uses additional simulation. This computation supports reliable heuristic guidance and value estimates for the transmission πg→qH,g→μg+1→πg+2\pi_g \to q_{H,g} \to \mu_{g+1} \to \pi_{g+2}. HBPO and GPS share structural priors in DSIS-RB. The baselines involve different learning architectures and computational workloads, making aligning the computation costs difficult.

To test sensitivity to simulation intensity, we evaluated HBPO-Lite on both environments with all five PPO warm-start seeds. Lite reduces simulation repetitions for heuristic utility, residual continuation, and confidence estimation from 32 to 8, retaining the initialization, planning horizon, and coupling mechanism. All three methods use the same 2M real-interaction budget and evaluation protocol.

Final costs across five paired training seeds (lower is better):

DSIS

MethodSeed 0Seed 1Seed 2Seed 3Seed 4
HBPO-Full20.7321.2651.41158.8820.76
HBPO-Lite20.7221.6451.6383.5320.77
PPO21.0121.1756.8913817.3021.33

DSIS-RB

MethodSeed 0Seed 1Seed 2Seed 3Seed 4
HBPO-Full52.6881.935893.0155.30112.01
HBPO-Lite51.7884.135865.0560.44107.10
PPO70.0675.4666970.7575.1374.88

Lite preserves the large mean-cost improvements over PPO continuation, with median costs close to Full, despite a fourfold reduction in these per-estimate sampling quotas.

3. Relevance of the baselines.

Thank you for highlighting the relevance of the comparisons. HBPO combines off-policy learning, model-based guidance, emphatic weighting, and explicit policy coupling. We selected complementary baselines to contextualize these design choices across method families, including GPS for guided policy learning and the recent TPO (2026) for target-distribution fitting. Each baseline therefore serves a specific comparison role:

  • GPS: model-based guided policy learning, adapted to the categorical action space.
  • TPO: a recent approach that fits policies to improved target distributions.
  • PPO, A2C, and TRPO: standard policy-optimization references.
  • PPO-81: controls for the 81-action categorical parameterization.
  • Discrete SAC: off-policy actor-critic learning with replay and entropy regularization.
  • ACE: emphatic weighting in off-policy actor-critic learning.
  • GRPO: uses the updated policy directly for subsequent data collection.

4–5. Benchmark choice and task difficulty.

DSIS and DSIS-RB offer transparent dynamics that let us trace how action quality and heuristic guidance affect behavior and target learning. Four-period delivery delays, accumulating backlogs, severe shortage penalties, and persistent disruptions in DSIS-RB make policy recovery and stability meaningful challenges. Evaluation on MuJoCo would strengthen evidence of broader applicability and is an important extension. We highly appreciate your mention of MuJoCo. But our present empirical contribution concerns policy improvement and its coupling mechanism in inventory control.

For DSIS, finite-state approximations permit dynamic programming, as in prior dual-sourcing work; established heuristics include dual-index and capped dual-index policies. Böttcher et al.. DSIS-RB additionally requires accounting for hidden risk, which cannot be solved with pure dynamic programming. HBPO’s good performance in both benchmarks shows that HBPO is a relatively good RL algorithm in inventory control, with clear mechanisms analyzed, compared with other RL algorithms.

I appreciate the effort the authors put in rebuttal. However, I still have concerns about the benchmarked methods and contribution/position of this work. The authors mention that the contribution is "delayed heuristic-behavior-target coupling for policy recovery and stable improvement in inventory control". The current experiments don't provide sufficient evidence on the proposed method improves on inventory control methods.

In addition, if the main contribution is to demonstrate the effectiveness of an RL method for inventory control, i believe a venue focused on inventory management/operation management may be a better fit. For a broad AI/ML venue such as ICLR, the paper would benefit from demonstrating its benefits extend beyond the specific inventory control settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.