- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This manuscript introduces Heuristic Behavior Policy Optimization (HBPO), an off-policy RL framework that keeps separately learned behavior and target policies. The behavior policy is shaped by heuristic guidance (I think this is section 4.1, but I am not really sure) and this informs the target through a delayed, advantaged-filtered transfer term. On two inventory benchmarks (DSIS and DSIS-RB), HBPO lowers mean cost in 17 of 18 warm-start settings and recovers 35 of 36 collapsed baseline runs. Ablations support the design decisions, though validation is limited to a single domain.
Critical strengths
None.
Critical weaknesses
- The writing is lacking precision, filled with jargon, and is incredibly difficult to follow intuitively. Overall, there are many variables to keep track of (5 pages, 14-19!) which adds to the difficulty of parsing this work, and their definition lacks any motivation.
Here are a few examples from the initial pages:
- Lines 053-056 aim to describe HBPO at a high level but use terms like "delayed heuristic–behavior–target transmission", "persistent behavior policy" (what does persistent mean in this context?), "residual-based advantage estimation supporting both policies and emphatic weighting carrying heuristic confidence forward". The abstract is similarly worded, and it is not clear at all what any of these terms mean in the paper's context, making these descriptions non-useful to the reader.
- Lines 070: so the "data source cannot outgrow the target?" Once again, what does this mean? The target is a policy, the data source is not so it's not clear what growing a data source means and how this relates to the target policy. If this refers to exploring states beyond those visited by the target policy, this should be mentioned more explicitly.
- L037: "can heuristic guidance applied to it and behavior policy persistently influence the target policy?" What is "it" in this context
- Lines 075: "Correction schemes persistently track the behavior...reconcile it with target updates without shaping it, ... so correction grows with the gap" Which gap, reconcile how, shape what? In what way?
- Line 093: introduces symbols without definition
- Line 116: mentions the action space, but not what these indices represent - what is a regular/expedited order in this context.
- Lines 110-112: acronyms (DSIS, IID, DSIS-RB) are mentioned without being introduced.
- Lines 149-150: subscript g is mentioned, and only defined later as a generation.
- Lines 152-153: long-horizon comparisons of alternative first actions followed by construct a temporary heuristic policy . How, and what is ? Ah, defined later as heuristic guidance, but it is still not clear how it factors into .
- Line 168: Why define for expectation under another distribution?
- Line 158-160: What are ?
- Lines 154: What are the mathematical definitions of ?
- Lines 160-161, why are masses being introduced 3 sections ahead? What is the reader meant to take from this information, how does they related to emphatic weights and how are emphatic weights used?
There are more concerns, for the remaining pages but with the background material here being hard to follow - everything that follows is immediately hard to understand as well.
There are >60 pages of appendices, with several expanded definitions and theorems. These are referred to several times in the paper, but if they are actually important, these results (namely theorems) should be present in the paper itself and motivated. Definitions are provided repeatedly as well, but appear heuristic rather than motivated.
Overall, the work is filled with arbitrary design decisions without clear explanation. It is not clear to me why certain things work and others do not, which makes me concerned about HBPO's generalization and how it may be applied to other methods in the future.
Questions for authors
I found the paper to be extremely dense, containing many variables/terms without precise definition, making the work incredibly hard to follow. There are too many questions/clarifications to ask for from this standpoint, and 1 or 2 answers would not change my assessment. I do not think this work is ready for publication in the current form.
4 comments
We provide the answers for the specific questions below:
-
Lines 053–056: High-level terminology. “Persistent behavior policy” means a separately parameterized policy whose learned parameters are retained and updated across training rounds. The heuristic controller is temporary and supplies fitting targets for this policy.
“Delayed transmission” describes when newly introduced heuristic guidance reaches the target through behavior learning. In generation , collects , and the target update producing uses these data together with the frozen parent policies and estimates. Meanwhile, defines a temporary heuristic controller , whose guidance combines with target-advantage feedback to produce . An immediate coupling would use this newly updated behavior policy in the same generation’s target update. HBPO instead retains as an independently learned policy: in generation , it collects and enters the explicit advantage-filtered transfer term used to produce . The chain therefore specifies how heuristic guidance is carried across generations through both subsequent data collection and direct policy transfer. The delay concerns the use of newly learned behavior; the current batch already contributes to the current target update. This timing is central to the interaction among the heuristic controller, persistent behavior policy, and target policy in HBPO.
“Residual-based advantage estimation” uses accumulated model-based Bellman residuals to correct critic-based action-value estimates used by both policies.
“Emphatic weighting carrying heuristic confidence forward” means that evidence of heuristic usefulness enters learned state weights, propagates through behavior transitions, and affects later target updates.
-
Line 070: “The data source cannot outgrow the target.” “Data source” refers to the behavior policy, the data-collecting policy. The intended meaning is that, when collection uses the current task/target policy, the collector cannot be independently optimized to favor different actions and thereby change the distribution of visited states. “Outgrow” is an imprecise description of this restriction, but I just want to offer an intuitive picture here.
-
Line 037: The referent of “it.” “It” refers to the behavior policy. The question is whether heuristic guidance applied to a separately learned behavior policy can continue to influence target-policy improvement across successive updates.
-
Line 075: “Gap,” “reconcile,” and “shape.” The “gap” refers to differences between behavior and target action probabilities. “Reconcile” refers to accounting for those differences when using behavior-generated data to estimate quantities for target learning. “Shaping behavior” means directly changing the collector’s action distribution through a learning objective.
-
Line 093: Symbols in the transmission chain. In , indexes training rounds, is the current target policy, is the temporary heuristic controller constructed using that target, and is the updated behavior policy. Each round computes its updates from frozen parent policies. Consequently, first enters the following round’s target update, producing .
-
Line 116: Action components and ordering channels. In , indexes the decision period, and the superscripts and identify regular and expedited orders. Each component is an order quantity between 0 and 8 units. Regular orders arrive four periods later. Expedited orders arrive immediately at higher unit cost.
-
Lines 110-112: Acronyms. DSIS denotes the Dual-Sourcing Inventory System; IID means independent and identically distributed, describing demand in DSIS; and DSIS-RB denotes DSIS with Risky Behavior, the extension incorporating persistent demand and supply disruptions.
-
Lines 149-150: The generation index. A “generation” is one complete training round: the current behavior policy collects data, update signals are computed using the current frozen policies and supporting estimates, and the resulting successors become the inputs to the next round. The subscript identifies these successive versions.
-
Lines 152-153: Constructing the temporary heuristic policy. At a queried state, each candidate first action is evaluated through 1,000-step model simulations, with all subsequent actions following the frozen target policy. The estimated returns are centered relative to the target’s action distribution to obtain action utilities. The controller maximizes expected utility subject to a regularization penalty for departing from the target distribution. Thus, it assigns action probabilities using their estimated long-term consequences. The subscript identifies heuristic guidance. The corresponding behavior-learning component fits the behavior policy toward this controller.
-
Line 168: The expectation notation. is shorthand for expectation in the heuristic evaluation. Its randomness comes from the simulation model, subsequent actions sampled from the target policy, and the synthetic-risk draw in DSIS-RB. The subscript labels the purpose of the expectation; the underlying trajectory distribution is specified by these sampling rules.
-
Lines 158-160: Critics and action values. is the state-value estimate from critic , with . Their average, , is held fixed during the current round and supports heuristic evaluation and action-value estimation. denotes the expected discounted return under the simulation model after taking action at state and following thereafter. The residual correction accounts for continuation-value error in the critic-based estimate within that model.
-
Line 154: Mathematical definitions of . Let be the frozen behavior and target policies, and their candidate updates. Write for the estimated target advantage, centered under , and for averaging over collected states. The objectives are
A(m) &=\frac{1}{\tau}\mathbb{E}_D\\!\left[ \sum_a(m(a\mid s)-\mu(a\mid s))\widehat{A}(s,a)\right],\\\\ J(p) &=\mathbb{E}_D\\!\left[ w^J(s)\sum_a(p(a\mid s)-\pi(a\mid s))\widehat{A}(s,a)\right],\\\\ P(p) &=\frac{1}{\tau}\mathbb{E}_D\\!\left[ w^P(s)\sum_a|\mu(a\mid s)-\pi(a\mid s)| \left(\frac{p(a\mid s)}{\pi(a\mid s)}-1\right)\widehat{A}(s,a)\right]. \end{aligned}$$ Here, $\tau > 0$ is the heuristic regularization coefficient, also used to scale the objectives, and $w^J,w^P$ are the state weights. $A$ improves behavior using target advantages, $J$ directly improves the target, and $P$ supplies the additional disagreement-weighted transfer signal. -
Lines 160–161: Masses and emphatic weights. The sentence previews the state weighting used by the learning objectives. The masses are positive learned state-importance functions. Normalization and shrinkage convert them into mean-one weights , which determine each state’s contribution to and , respectively. propagates importance through target-policy continuation; incorporates heuristic-improvement confidence and propagates importance through behavior transitions. Their role is therefore to determine where the two target-learning signals receive emphasis.
I thank the authors for their detailed rebuttal to clarify the concerns I raised regarding the writing of the manuscript. While I think the idea of optimizing the behavioural policy to ultimately improve the target policy is an interesting and important direction, I still cannot recommend this paper for acceptance in its current form due to concerns with clarity. These concerns are shared by the other reviewers and are too substantial, in my opinion, to be considered within the scope of rebuttal. Not to mention, the authors have not provided a revised manuscript towards addressing these concerns.
I encourage the authors to revise the work to improve the communication of their ideas and incorporate the clarifications they have provided above, which I believe improve the work and will facilitate reproduction and future developments.
Thank you for your careful reading and detailed comments. We appreciate the specific examples identifying where the terminology, notation, and organization make the paper difficult to follow. We will revise the presentation throughout so that the method’s intuition, definitions, and design rationale are clear from the main text. This includes simplifying the abstract and introduction, defining notation before use, explaining the motivation behind each component, and presenting the central theoretical statements and assumptions alongside their practical implications. We will also reorganize the appendix, consolidate repeated definitions, and carefully proofread the entire manuscript for clarity, grammar, and consistency.
HBPO is motivated by the question of how improving the policy that collects experience can produce sustained improvement in the policy ultimately evaluated. In long-horizon problems, the collector determines both the actions sampled and the downstream states encountered. HBPO therefore learns a behavior policy separately from the target. Model-based lookahead provides guidance about long-term action consequences, which the behavior policy integrates with target-advantage feedback across training rounds. The target learns from the collected experience and an explicit transfer objective: the magnitude of behavior–target disagreement determines the strength of the transfer signal, while estimated target advantages determine its direction. This gives behavior learning a direct role in target optimization. Shared value estimates provide a consistent basis for judging actions, and state weights determine where the target is improved and transfer receives emphasis. These components address distinct questions: how to improve collection, which behavioral changes should influence the target, and where that influence should be strongest.