- Rating
- 1(−1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper proposes a novel task-triggering framework for dynamic task-resource assignment, reformulating repeated reinforcement learning triggering decisions into estimating the potential benefit of waiting.
- The framework combines a supervised delay-benefit estimator trained using rollout-generated labels with an RL policy that selects task-specific triggering thresholds. This decomposition aims to alleviate sparse rewards and ambiguous credit assignment while preserving global resource coordination.
- Experiments on DJSP, WTA, and Warehouse Replenishment demonstrate consistent improvements over RL-based and heuristic-based baselines. Ablation studies further examine the effectiveness of delay-benefit estimation and adaptive threshold selection.
Critical strengths
- The paper identifies a fundamental difficulty in repeated RL triggering decisions and introduces an intuitive decomposition between delay-benefit estimation and threshold selection. This provides a promising perspective on improving learning efficiency in dynamic task-resource assignment.
- The method consistently outperforms competing approaches across three distinct optimization problems and multiple difficulty settings. The ablation studies also support the complementary contributions of delay-benefit estimation and task-specific threshold selection.
Critical weaknesses
- The delay-benefit estimator relies on counterfactual future triggering opportunities, but the procedure for generating these returns is insufficiently specified, particularly regarding future task interactions and rollout policies. This raises concerns about the accuracy and potential bias of the supervision signals used to support the central methodological contribution.
- The proposed method requires additional rollout-based data generation and supervised pretraining before RL optimization. However, the paper does not provide comprehensive comparisons of total computational costs, environment interactions, or training efficiency. Consequently, it remains unclear whether the improvements stem from the proposed decision reformulation or additional computational supervision.
Questions for authors
- How are counterfactual returns for different future triggering times computed while accounting for interactions among competing tasks and resources? What policies govern the remaining decisions during these rollouts?
- Can the authors report the computational overhead of rollout generation and estimator pretraining, and compare all methods under matched total environment-interaction or computational budgets?
- Can the authors provide learning curves or sample-efficiency comparisons to directly substantiate the claim that the proposed decomposition alleviates sparse rewards and improves RL credit assignment?
3 comments
I appreciate the authors' detailed responses and additional experiments. While these revisions partially address my original concerns, several important issues remain.
- The authors clarify that counterfactual rollouts use an earliest-feasible (EF) policy for other tasks. However, the resulting delay-benefit estimates remain policy-dependent and may not accurately reflect the returns under the learned RL policy. The additional relabeling experiment is limited to one DJSP setting and does not establish that EF-based supervision is sufficiently reliable across all three benchmarks.
- The new experiments account for preprocessing costs and compare methods under matched wall-clock budgets, which is a useful improvement. However, comparisons under matched total environment-interaction budgets remain absent, and the new results are difficult to independently reproduce from the submitted code.
- The additional learning curves provide empirical evidence of improved learning efficiency. Nevertheless, they do not directly establish that the improvement results from alleviating sparse rewards or improving credit assignment, as claimed.
More importantly, my examination of the supplementary code revealed two additional concerns:
-
Inconsistent enforcement of safety boundaries. Section 4.2 defines as the latest safe triggering epoch. However, the implementations in all three environments do not enforce triggering at this boundary when the estimated delay benefit exceeds the selected threshold. The submitted unit tests explicitly confirm this behavior. This raises concerns about the consistency between the proposed formulation and its implementation.
-
Potentially biased DJSP evaluation. The implementation updates
total_tardinessonly when jobs are completed, while episodes terminate atmax_timewithout accounting for unfinished jobs in this metric. Consequently, overdue but unfinished jobs may contribute nothing to the reported objective, potentially favoring policies that delay task execution. The authors should report job completion rates and clarify whether this issue affects the experimental comparisons.
Although the rebuttal improves the empirical presentation, these implementation-level concerns raise more fundamental questions about the validity of the evaluation and the consistency of the proposed framework. The additional evidence does not sufficiently resolve these issues. I am therefore lowering my score and maintaining a negative recommendation.
Dear Reviewer,
1. Policy dependence of EF supervision.
Thank you for this concern. We have now extended the relabeling experiment to WTA and Warehouse Replenishment.
We used the same relabeling procedure described in our previous response. The settings are DJSP (100 jobs, IAT 5, DDT 2.0), WTA (150 targets, IAT 5, ATK Type 2), and Warehouse Replenishment (500 orders, high supplier heterogeneity, tight time windows).
| Benchmark | Ours | Relabeling | Label time × | Total steps × | Total time × |
|---|---|---|---|---|---|
| DJSP ↓ | 376.80±30.16 | 375.32±38.80 | 14.11 | 2.00 | 4.38 |
| WTA ↑ | 498.15±15.20 | 495.63±21.52 | 12.70 | 2.00 | 5.04 |
| Warehouse ↑ | 92.09±3.64 | 93.30±4.76 | 12.93 | 2.00 | 4.80 |
Relabeling achieves similar mean performance to Ours across all three benchmarks, while requiring twice the environment steps and 4.38–5.04 times the total training time.
These results show that EF rollouts already provide enough information for the estimator to support effective threshold selection in the tested settings across all three domains, offering a favorable balance between performance and training cost.
2. Interactions and computational cost.
Our comparison matches total computational cost, including preprocessing. On DJSP (100 jobs, IAT 5, DDT 2.0), labeling uses 23,760,500 simulator steps and takes 1.83 h. PPO training uses 5,400,000 environment steps and takes 7.41 h; labeling takes only 24.7% of PPO time. Rule-based rollouts require no PPO policy inference or gradient updates, so equal numbers of interactions do not represent equal computational costs.
Training all baselines to match a total budget of 29,160,500 environment interactions would require substantial additional training time. Moreover, the late-stage plateaus observed in the learning curves suggest limited further performance gains for most baselines. We therefore evaluated computational efficiency through matched total-time comparisons, including data collection, label generation, estimator training and PPO (Appendix A.9).
3. Advantage estimation and credit assignment.
Thank you for highlighting the need for evidence on credit assignment. Our additional diagnostics provide empirical support for improved credit assignment under the proposed formulation. Specifically, PPO's advantage differences more faithfully reflect the long-term return differences between candidate actions in the evaluated states.
For each method, we froze policies from five independent training seeds and evaluated 20 states per seed (100 total). At each state, we compared mean PPO advantage differences for a prespecified action pair with differences in Monte Carlo system returns. Each rollout forced the initial action, then followed the frozen policy and its normal execution mechanism. Within-state differences remove action-independent offsets. A pair was distinguishable if its return-difference confidence interval excluded zero. Counts and agreement rates are pooled across seeds:
| Metric | Repeated Delay-or-Trigger | Ours |
|---|---|---|
| Distinguishable action pairs | 71/100 | 93/100 |
| Preference agreement on distinguishable pairs | 74.6% | 91.4% |
| Normalized action-difference error ↓ | 0.31 | 0.15 |
These results provide empirical support for improved credit assignment through better action ranking and return-difference estimation.
4. Meaning and use of .
The description in Section 4.2 defines what denotes; it does not prescribe a release action at that epoch. Once defined, this value is used to delimit the meaningful candidate release times considered during rollout-based label generation, rather than to impose a mandatory-release boundary. The actual release condition is given in Eq. (9).
During training and deployment, release follows the selected triggering criterion and feasibility conditions. The implementation therefore does not automatically release a task at when its threshold criterion remains unmet.
If a task's predicted delay benefit never falls below its selected threshold, it remains unreleased. Any resulting loss of reward is reflected in the system return and informs policy updates. This helps RL learn to trade off the benefits of waiting against the cost of leaving tasks unserved, guiding the selection of appropriate task-specific thresholds.
5. DJSP evaluation and job completion.
The max_time parameter is configurable. For each tested problem size, we configure an appropriate value to ensure that all jobs complete. All reported DJSP evaluations achieved a job completion rate of 100%.
The purpose of this design is to control the simulation interval used for test-runtime measurement, avoiding shorter measurement intervals caused solely by some methods completing jobs earlier.
Because all jobs completed in these evaluations, the reported objectives account for every job, including those completed after their due dates.
Counterfactual Rollouts, Computational Costs, and Learning Efficiency
Dear Reviewer, thank you for highlighting the rollout-policy and training-cost questions.
1. Counterfactual returns and rollout policies (Q1).
Regarding which policy controls other tasks during rollouts, we use an earliest-feasible (EF) policy. Only the task being evaluated is held until its candidate release time. All other tasks, including new arrivals, enter the same downstream heuristic as soon as feasible. Each rollout simulates the full system, updating assignments and shared resource occupancy. We average system returns for each candidate release time before maximizing across candidates to construct the label in Eq. (6).
It seems that a better alternative is to let the learned RL policy govern all other tasks' releases, as in deployment. This can be implemented by freezing the initially trained RL policy for a second labeling round and then updating the estimator and RL policy. However, our experiments show that this alternative offers little improvement over EF while incurring substantially higher training costs. Specifically, we evaluated this alternative on DJSP (100 jobs, IAT 5, DDT 2.0), with the results shown below:
Compared with our method, relabeling with the learned RL policy reduces mean tardiness by only 0.39%, while requiring 4.38 times as much total training time. These results suggest that EF rollouts already provide enough information for the estimator to support effective threshold selection in this setting.
Our method remains effective with EF rollouts because RL and the estimator work together. RL uses information about all current tasks and resources to coordinate task-specific threshold choices, with training guided by the original system reward. It thus learns how to use these reference estimates under actual task competition, rather than directly executing potentially incompatible single-task optima.
2. Preprocessing costs and comparison at matched budgets (Q2).
We have added total-time comparisons in Section 5.5, accounting for data collection, label generation, estimator training and PPO. The total preprocessing time is 1.91 h for DJSP, 1.42 h for Warehouse Replenishment and 1.48 h for WTA.
We have added Table 16 in Appendix A.9 to compare Ours with RL, Global Delay, MoE Delay and Hyper-Heuristic under matched total time budgets using comparable hardware and parallelism. Budgets are 5/6/7/8 h for DJSP and 3/3.5/4 h for WTA and Warehouse Replenishment, including preprocessing. Ours achieves the best mean objective at every reported budget. This comparison matches total time, not total environment interactions.
3. Learning curves and evidence of learning efficiency (Q3).
We have added Figure 4 in Section 5.5 to compare Ours with Repeated Delay-or-Trigger (Repeated) and Repeated + Delay Benefit during PPO training. Both baselines retain repeated binary decisions; the latter also receives the same frozen estimator's prediction, testing whether this information alone explains the gains. The figure reports mean episodic returns and confidence intervals across independent training seeds. Across all three benchmarks, Ours reaches higher-return regimes earlier and maintains higher mean returns during intermediate/later training. The axis counts PPO environment steps only, excluding label-generation interactions.
Figure 5 retains total elapsed time, including preprocessing; only the initial preprocessing interval is cropped from view. Both estimator-based methods incur the costs listed above, while Repeated trains during this interval. Ours subsequently overtakes both baselines in all three benchmarks and remains ahead over the remaining displayed budgets.
These results support improved PPO-stage learning efficiency and end-to-end time performance after accounting for preprocessing.