acceptodds
Under review as a conference paper at ICLR 2027

Don't Learn When to Trigger, Learn When Waiting Is Not Worthwhile: A New Framework for Dynamic Task–Resource Assignment

Abstract

For large-scale task–resource assignment problems in dynamic environments with hard constraints, a common solution is a hierarchical RL–heuristic framework, in which an upper-layer reinforcement learning (RL) policy determines which tasks to trigger at each decision epoch, while a lower-layer heuristic assigns available resources to them. However, since most pending tasks remain untriggered at most decision epochs, RL receives extremely sparse reward signals, leading to ambiguous credit assignment and ineffective triggering policies. Our key insight is that, instead of asking RL to repeatedly decide whether to trigger at each decision epoch, we can reformulate the problem as determining how much can still be gained from waiting. A task is triggered when its delay benefit, defined as the expected-return difference between its best triggering opportunity in the remaining life cycle and triggering it now, falls below a learned threshold. Based on this insight, we propose a new triggering framework that decomposes the original triggering problem into two complementary learning problems. First, a delay-benefit estimator is trained with rollout-generated supervision to estimate how much can still be gained by waiting. Second, RL selects a task-specific threshold that determines when the estimated delay benefit is sufficiently small to trigger the task, while accounting for global competition among tasks and resources. This decomposition replaces repeated RL triggering decisions with supervised delay-benefit estimation and one-time RL threshold selection, providing dense supervision for delay-benefit estimation while allowing RL to focus on long-horizon coordination. Experiments on Dynamic Job-Shop Scheduling, Weapon-Target Assignment, and Warehouse Replenishment demonstrate consistent improvements over pure RL and hybrid RL–heuristic approaches.

Primary area:
optimization
Keywords:
Dynamic task–resource assignment, adaptive task triggering, delay-benefit estimation, supervised learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MCfdOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper proposes a novel task-triggering framework for dynamic task-resource assignment, reformulating repeated reinforcement learning triggering decisions into estimating the potential benefit of waiting.
  2. The framework combines a supervised delay-benefit estimator trained using rollout-generated labels with an RL policy that selects task-specific triggering thresholds. This decomposition aims to alleviate sparse rewards and ambiguous credit assignment while preserving global resource coordination.
  3. Experiments on DJSP, WTA, and Warehouse Replenishment demonstrate consistent improvements over RL-based and heuristic-based baselines. Ablation studies further examine the effectiveness of delay-benefit estimation and adaptive threshold selection.
Critical strengths
  1. The paper identifies a fundamental difficulty in repeated RL triggering decisions and introduces an intuitive decomposition between delay-benefit estimation and threshold selection. This provides a promising perspective on improving learning efficiency in dynamic task-resource assignment.
  2. The method consistently outperforms competing approaches across three distinct optimization problems and multiple difficulty settings. The ablation studies also support the complementary contributions of delay-benefit estimation and task-specific threshold selection.
Critical weaknesses
  1. The delay-benefit estimator relies on counterfactual future triggering opportunities, but the procedure for generating these returns is insufficiently specified, particularly regarding future task interactions and rollout policies. This raises concerns about the accuracy and potential bias of the supervision signals used to support the central methodological contribution.
  2. The proposed method requires additional rollout-based data generation and supervised pretraining before RL optimization. However, the paper does not provide comprehensive comparisons of total computational costs, environment interactions, or training efficiency. Consequently, it remains unclear whether the improvements stem from the proposed decision reformulation or additional computational supervision.
Questions for authors
  1. How are counterfactual returns for different future triggering times computed while accounting for interactions among competing tasks and resources? What policies govern the remaining decisions during these rollouts?
  2. Can the authors report the computational overhead of rollout generation and estimator pretraining, and compare all methods under matched total environment-interaction or computational budgets?
  3. Can the authors provide learning curves or sample-efficiency comparisons to directly substantiate the claim that the proposed decomposition alleviates sparse rewards and improves RL credit assignment?
3 comments
AuthorsOpenReview ↗

Counterfactual Rollouts, Computational Costs, and Learning Efficiency

Dear Reviewer, thank you for highlighting the rollout-policy and training-cost questions.

1. Counterfactual returns and rollout policies (Q1).

Regarding which policy controls other tasks during rollouts, we use an earliest-feasible (EF) policy. Only the task being evaluated is held until its candidate release time. All other tasks, including new arrivals, enter the same downstream heuristic as soon as feasible. Each rollout simulates the full system, updating assignments and shared resource occupancy. We average system returns for each candidate release time before maximizing across candidates to construct the label in Eq. (6).

It seems that a better alternative is to let the learned RL policy govern all other tasks' releases, as in deployment. This can be implemented by freezing the initially trained RL policy for a second labeling round and then updating the estimator and RL policy. However, our experiments show that this alternative offers little improvement over EF while incurring substantially higher training costs. Specifically, we evaluated this alternative on DJSP (100 jobs, IAT 5, DDT 2.0), with the results shown below:

MethodObjective ↓Label timeTotal stepsTotal time
Fixed-estimator training (Ours)376.80±30.161.001.001.00
Relabeling with the learned RL policy375.32±38.8014.112.004.38

Compared with our method, relabeling with the learned RL policy reduces mean tardiness by only 0.39%, while requiring 4.38 times as much total training time. These results suggest that EF rollouts already provide enough information for the estimator to support effective threshold selection in this setting.

Our method remains effective with EF rollouts because RL and the estimator work together. RL uses information about all current tasks and resources to coordinate task-specific threshold choices, with training guided by the original system reward. It thus learns how to use these reference estimates under actual task competition, rather than directly executing potentially incompatible single-task optima.

2. Preprocessing costs and comparison at matched budgets (Q2).

We have added total-time comparisons in Section 5.5, accounting for data collection, label generation, estimator training and PPO. The total preprocessing time is 1.91 h for DJSP, 1.42 h for Warehouse Replenishment and 1.48 h for WTA.

We have added Table 16 in Appendix A.9 to compare Ours with RL, Global Delay, MoE Delay and Hyper-Heuristic under matched total time budgets using comparable hardware and parallelism. Budgets are 5/6/7/8 h for DJSP and 3/3.5/4 h for WTA and Warehouse Replenishment, including preprocessing. Ours achieves the best mean objective at every reported budget. This comparison matches total time, not total environment interactions.

3. Learning curves and evidence of learning efficiency (Q3).

We have added Figure 4 in Section 5.5 to compare Ours with Repeated Delay-or-Trigger (Repeated) and Repeated + Delay Benefit during PPO training. Both baselines retain repeated binary decisions; the latter also receives the same frozen estimator's prediction, testing whether this information alone explains the gains. The figure reports mean episodic returns and confidence intervals across independent training seeds. Across all three benchmarks, Ours reaches higher-return regimes earlier and maintains higher mean returns during intermediate/later training. The axis counts PPO environment steps only, excluding label-generation interactions.

Figure 5 retains total elapsed time, including preprocessing; only the initial preprocessing interval is cropped from view. Both estimator-based methods incur the costs listed above, while Repeated trains during this interval. Ours subsequently overtakes both baselines in all three benchmarks and remains ahead over the remaining displayed budgets.

These results support improved PPO-stage learning efficiency and end-to-end time performance after accounting for preprocessing.

I appreciate the authors' detailed responses and additional experiments. While these revisions partially address my original concerns, several important issues remain.

  • The authors clarify that counterfactual rollouts use an earliest-feasible (EF) policy for other tasks. However, the resulting delay-benefit estimates remain policy-dependent and may not accurately reflect the returns under the learned RL policy. The additional relabeling experiment is limited to one DJSP setting and does not establish that EF-based supervision is sufficiently reliable across all three benchmarks.
  • The new experiments account for preprocessing costs and compare methods under matched wall-clock budgets, which is a useful improvement. However, comparisons under matched total environment-interaction budgets remain absent, and the new results are difficult to independently reproduce from the submitted code.
  • The additional learning curves provide empirical evidence of improved learning efficiency. Nevertheless, they do not directly establish that the improvement results from alleviating sparse rewards or improving credit assignment, as claimed.

More importantly, my examination of the supplementary code revealed two additional concerns:

  • Inconsistent enforcement of safety boundaries. Section 4.2 defines TisafeT_i^{\mathrm{safe}} as the latest safe triggering epoch. However, the implementations in all three environments do not enforce triggering at this boundary when the estimated delay benefit exceeds the selected threshold. The submitted unit tests explicitly confirm this behavior. This raises concerns about the consistency between the proposed formulation and its implementation.

  • Potentially biased DJSP evaluation. The implementation updates total_tardiness only when jobs are completed, while episodes terminate at max_time without accounting for unfinished jobs in this metric. Consequently, overdue but unfinished jobs may contribute nothing to the reported objective, potentially favoring policies that delay task execution. The authors should report job completion rates and clarify whether this issue affects the experimental comparisons.

Although the rebuttal improves the empirical presentation, these implementation-level concerns raise more fundamental questions about the validity of the evaluation and the consistency of the proposed framework. The additional evidence does not sufficiently resolve these issues. I am therefore lowering my score and maintaining a negative recommendation.

Dear Reviewer,

1. Policy dependence of EF supervision.

Thank you for this concern. We have now extended the relabeling experiment to WTA and Warehouse Replenishment.

We used the same relabeling procedure described in our previous response. The settings are DJSP (100 jobs, IAT 5, DDT 2.0), WTA (150 targets, IAT 5, ATK Type 2), and Warehouse Replenishment (500 orders, high supplier heterogeneity, tight time windows).

BenchmarkOursRelabelingLabel time ×Total steps ×Total time ×
DJSP ↓376.80±30.16375.32±38.8014.112.004.38
WTA ↑498.15±15.20495.63±21.5212.702.005.04
Warehouse ↑92.09±3.6493.30±4.7612.932.004.80

Relabeling achieves similar mean performance to Ours across all three benchmarks, while requiring twice the environment steps and 4.38–5.04 times the total training time.

These results show that EF rollouts already provide enough information for the estimator to support effective threshold selection in the tested settings across all three domains, offering a favorable balance between performance and training cost.

2. Interactions and computational cost.

Our comparison matches total computational cost, including preprocessing. On DJSP (100 jobs, IAT 5, DDT 2.0), labeling uses 23,760,500 simulator steps and takes 1.83 h. PPO training uses 5,400,000 environment steps and takes 7.41 h; labeling takes only 24.7% of PPO time. Rule-based rollouts require no PPO policy inference or gradient updates, so equal numbers of interactions do not represent equal computational costs.

Training all baselines to match a total budget of 29,160,500 environment interactions would require substantial additional training time. Moreover, the late-stage plateaus observed in the learning curves suggest limited further performance gains for most baselines. We therefore evaluated computational efficiency through matched total-time comparisons, including data collection, label generation, estimator training and PPO (Appendix A.9).

3. Advantage estimation and credit assignment.

Thank you for highlighting the need for evidence on credit assignment. Our additional diagnostics provide empirical support for improved credit assignment under the proposed formulation. Specifically, PPO's advantage differences more faithfully reflect the long-term return differences between candidate actions in the evaluated states.

For each method, we froze policies from five independent training seeds and evaluated 20 states per seed (100 total). At each state, we compared mean PPO advantage differences for a prespecified action pair with differences in Monte Carlo system returns. Each rollout forced the initial action, then followed the frozen policy and its normal execution mechanism. Within-state differences remove action-independent offsets. A pair was distinguishable if its return-difference confidence interval excluded zero. Counts and agreement rates are pooled across seeds:

MetricRepeated Delay-or-TriggerOurs
Distinguishable action pairs71/10093/100
Preference agreement on distinguishable pairs74.6%91.4%
Normalized action-difference error ↓0.310.15

These results provide empirical support for improved credit assignment through better action ranking and return-difference estimation.

4. Meaning and use of TisafeT_i^{\mathrm{safe}}.

The description in Section 4.2 defines what TisafeT_i^{\mathrm{safe}} denotes; it does not prescribe a release action at that epoch. Once defined, this value is used to delimit the meaningful candidate release times considered during rollout-based label generation, rather than to impose a mandatory-release boundary. The actual release condition is given in Eq. (9).

During training and deployment, release follows the selected triggering criterion and feasibility conditions. The implementation therefore does not automatically release a task at TisafeT_i^{\mathrm{safe}} when its threshold criterion remains unmet.

If a task's predicted delay benefit never falls below its selected threshold, it remains unreleased. Any resulting loss of reward is reflected in the system return and informs policy updates. This helps RL learn to trade off the benefits of waiting against the cost of leaving tasks unserved, guiding the selection of appropriate task-specific thresholds.

5. DJSP evaluation and job completion.

The max_time parameter is configurable. For each tested problem size, we configure an appropriate value to ensure that all jobs complete. All reported DJSP evaluations achieved a job completion rate of 100%.

The purpose of this design is to control the simulation interval used for test-runtime measurement, avoiding shorter measurement intervals caused solely by some methods completing jobs earlier.

Because all jobs completed in these evaluations, the reported objectives account for every job, including those completed after their due dates.

Reviewer_P41HOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper extends the common hierarchical RL–heuristic framework for dynamic task resource assignment by estimating the delay benefit of each pending task. The proposed framework trains a small MLP via supervised learning, using rollout-generated labels, to predict the delay benefit, i.e., the additional expected return that can still be gained by waiting for the best future triggering epoch instead of triggering the task now. A high-level RL agent then selects a task-specific threshold from a set of quantile-based candidates for each pending task. The threshold remains fixed until the task is triggered, which happens once its estimated delay benefit falls below the threshold.

Critical strengths

The decomposition into supervised delay-benefit estimation and one-shot RL threshold selection is intuitive. A delay-benefit estimator trained via supervised learning provides a dense, task-level signal for determining the triggering time.

Critical weaknesses
  1. The position of this paper is ambiguous. The authors do not discuss how their approach differs from closely related work on learning when to trigger tasks or matching operations (e.g., Zhong et al., 2024; Bao et al., 2026). The experimental comparison is also insufficient. Many classic DJSP methods are missing, such as dispatching rules.

  2. The key component, delay-benefit estimation, is not clearly described. The estimator relies heavily on the quality of labels, which are generated by rollouts, yet the authors do not explain how the rollouts are performed. The label D_i is implicitly defined with respect to some policy π_b. Once the estimator is deployed under the learned RL policy, the semantics of the labels may shift. Moreover, it is unclear whether stochastic events are resampled in each rollout. Labels would leak hindsight information. The rollout-based labeling requires a resettable simulator and substantial additional environment interaction, neither of which is available to the baselines or accounted for in the comparison. The experiments suggest that the estimator contributes little on its own. With a fixed global threshold, it improves only marginally over earliest-feasible triggering. It raises the question of whether the gains mainly come from RL-based threshold selection.

Questions for authors
  1. How does this paper differ from Zhong et al. (2024) and Bao et al. (2026), and what benefits does this difference bring?
  2. What is the complete training process of the proposed framework, e.g., how are the delay benefits estimated and how are the heuristics chosen?
  3. Are baselines fully trained as the proposed method?
3 comments
AuthorsOpenReview ↗

Method Distinctions, Estimator Contributions, and Baseline Comparisons

Thank you for your constructive comments.

1. Differences from prior triggering methods (Q1).

WATTER (Zhong et al.) learns order-specific extra-time thresholds using TD loss and distribution-derived targets. Bao et al. learn repeated platform-level match/skip decisions with reward shaping. We instead use supervised estimates of return gains from waiting and one persistent RL threshold per task under the original reward. This reduces repeated RL decisions. We added these comparisons to Section 2.2.

2. Training procedure and heuristic choices (Q2).

Training proceeds in two stages. First, we generate delay-benefit labels by averaging LL rollout returns for each candidate release time and subtracting the immediate-release mean from the largest candidate mean (Eq. 6). We standardize these labels and train an MLP using squared loss. Second, we freeze the estimator and train PPO to choose, once per task, either earliest-feasible release or one of K=10K=10 thresholds derived from quantiles of the estimator's predictions. Selected thresholds remain fixed while predicted benefits and feasibility are reevaluated at subsequent decision epochs. Each task is released when its selected triggering rule is satisfied.

The fixed lower-layer heuristics are minimum remaining slack for DJSP; highest-threat target then cheapest feasible weapon-ammunition option for WTA; urgency/value/shortage priority then cheapest feasible supplier for Warehouse Replenishment (Appendix A.8).

3. Policy controlling other tasks during rollouts.

Regarding which policy controls other tasks during rollouts, we use an earliest-feasible (EF) policy. Only the task being evaluated is held until its candidate release time. All other tasks, including new arrivals, enter the same downstream heuristic as soon as feasible. Each rollout simulates the full system, updating assignments and shared resource occupancy.

It seems that a better alternative is to let the learned RL policy govern all other tasks' releases, as in deployment. This can be implemented by freezing the initially trained RL policy for a second labeling round and then updating the estimator and RL policy. However, our experiments show that this alternative offers little improvement over EF while incurring substantially higher training costs. Specifically, we evaluated this alternative on DJSP (100 jobs, IAT 5, DDT 2.0), with the results shown below:

MethodObjective ↓Label timeTotal stepsTotal time
Fixed-estimator training (Ours)376.80±30.161.001.001.00
Relabeling with the learned RL policy375.32±38.8014.112.004.38

Compared with our method, relabeling with the learned RL policy reduces mean tardiness by only 0.39%, while requiring 4.38 times as much total training time. These results suggest that EF rollouts already provide enough information for the estimator to support effective threshold selection in this setting.

Our method remains effective with EF rollouts because RL and the estimator work together. RL uses information about all current tasks and resources to coordinate task-specific threshold choices, with training guided by the original system reward. It thus learns how to use these reference estimates under actual task competition, rather than directly executing potentially incompatible single-task optima.

4. Rollout sampling and future information.

For each snapshot, future events are resampled across LL repetitions, with the same scenarios used across candidate times. Estimator and PPO inputs contain no unrevealed events, and labeling and evaluation data are independent.

5. Estimator contribution.

A fixed global threshold tests the estimator without task-specific RL coordination; we also assess its contribution with learned thresholds. With RL threshold selection retained, replacing the hand-crafted Tuned Continuous Score with our estimator improves DJSP/WTA/Warehouse Replenishment by 5.0%/4.4%/5.6% (Table 4). We added Repeated + Delay Benefit in Section 5.5: a Repeated Delay-or-Trigger policy receives the same frozen estimator's prediction as a state feature but still trails Ours (Figures 4–5). These results support combining the estimator with persistent RL thresholds.

6. Missing classical DJSP baselines.

We added seven classical dispatching rules: FCFS, SPT, WSPT, EDD, CR, MS, and ATC. Means and SDs are provided in Appendix A.4, Tables 10–12. Ours achieves the lowest mean total weighted tardiness in all 18 configurations.

7. Are baselines sufficiently trained? (Q3)

We added training curves for the main learning baselines in Appendix A.9, Figure 6. They show late-stage plateaus or fluctuations, while Ours retains its advantage.

We also added comparisons under matched total time budgets in Appendix A.9, Table 16, accounting for data collection, labeling, estimator training and PPO. Ours has the best mean objective at every budget across all three benchmarks.

Thanks for the answers and additional results. Some of my concerns were addressed, however, the following remains:

  • the approach is much more complex than an end-to-end DRL approach like L2D (Zhang 2020). How is the total time of your approach in comparison to L2D?
  • the comparison to the state-of-the-art DRL approaches to DJSP is still missing. Many e2e DRL approaches for JSP/FJSP, which can also be used to solve the dynamic JSP, outperform L2D. Since the framework of e2d DRL approaches is much simpler than the one proposed in the paper, it is important to compare to the SOTA approaches, to justify the added complexity of the approach and additional computational burden of the labelling and rollouts.

In addition:

  • Although the related work by Zhong et al. (2024) and Bao et al. (2026) is now discussed, no empirical comparison is provided. This is particularly important because Zhong et al. (2024) also learns order-specific thresholds.
  • The core contribution, the delay-benefit estimator, remains questionable. The relabeling experiment shows that changing the policy under which the labels are defined has virtually no effect on performance. This suggests that the estimator acts as a learned task-ranking feature rather than a delay-benefit estimate, and the central claim of the paper is not well supported by the current evidence.

I will update my evaluation based on these points.

Reviewer_bEJNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies dynamic task–resource assignment with a two-layer architecture in which an upper-layer RL policy decides when a pending task is released to a lower-layer domain heuristic that builds the assignment. The authors argue that asking RL to make a delay-or-trigger decision for every task at every epoch leads to long decision sequences with sparse feedback and poor credit assignment. They instead define the delay benefit of a task as the expected return from the best remaining trigger epoch minus the return from triggering now, train an MLP to predict it from labels generated by environment rollouts, and let PPO pick once per task a persistent threshold from a set of empirical quantiles of the predicted values. A task is triggered when its predicted delay benefit falls below its threshold. On three synthetic benchmarks (54 configurations) the method obtains the best mean objective in every configuration against baselines. The authors also did ablations to separate persistent control, the triggering signal and task-specific thresholds, and the appendix adds an estimator reliability study and a sensitivity study over the number of levels K.

Critical strengths
  1. The idea is simple and intuitive. Replacing a stream of per-epoch trigger decisions by one persistent threshold per task on an estimated benefit of waiting is easy to implement on top of an existing RL–heuristic pipeline, and keeps the RL action space small.

  2. The evaluation is broad for a methods paper of this kind.

  3. The paper is clearly written and easy to follow. The figures explain the pipeline well.

Critical weaknesses
  1. The central claim is not tested. The motivation is that the reformulation makes the RL problem easier to optimize (sparse feedback, credit assignment, exploration), but the paper reports only final performance. There are no learning curves or sample-efficiency or wall-clock comparison, and final-performance tables cannot separate "easier to optimize" from "a different policy class".

  2. Statistical reporting is weak for an RL paper. The reported std is over 100 evaluation repetitions, i.e. environment stochasticity, not over training seeds, and it is unclear whether the tables report a single trained policy or an average over the ten training seeds listed in Table 12.

  3. The delay benefit and its labels are under-specified. The rollouts that define G_i(tau) must also decide all other tasks and everything after tau, but the paper never says which policy does this, and since the estimator is trained before PPO and then frozen, the labels are relative to some pre-RL behaviour that the final policy no longer follows.

  4. Some related literature is missing like Bao et al. (“Timing the Match”, arXiv 2503.13200).

Questions for authors
  1. Which policy controls the other tasks and the decisions after tau during label rollouts? How does the total simulation cost of label generation compare with the cost of PPO training?

  2. What happens to a task whose predicted delay benefit never falls below its threshold before T_safe, and how is T_safe defined in each benchmark, in particular for DJSP?

  3. How does Repeated Delay-or-Trigger perform when it is given the estimated delay benefit as a state feature, and can you show training curves or sample-efficiency comparisons between your method and the repeated-decision baselines?

1 comment
AuthorsOpenReview ↗

Rollout Policies, Triggering Boundaries, and Learning Efficiency

Dear Reviewer, thank you for your constructive feedback.

1. Policy controlling other tasks during rollouts (Q1-1).

Regarding which policy controls other tasks during rollouts, we use an earliest-feasible (EF) policy. Only the task being evaluated is held until its candidate release time. All other tasks, including new arrivals, enter the same downstream heuristic as soon as feasible.

It seems that a better alternative is to let the learned RL policy govern all other tasks' releases, as in deployment. This can be implemented by freezing the initially trained RL policy for a second labeling round and then updating the estimator and RL policy. However, our experiments show that this alternative offers little improvement over EF while incurring substantially higher training costs. Specifically, we evaluated this alternative on DJSP (100 jobs, IAT 5, DDT 2.0), with the results shown below:

MethodObjective ↓Label timeTotal stepsTotal time
Fixed-estimator training (Ours)376.80±30.161.001.001.00
Relabeling with the learned RL policy375.32±38.8014.112.004.38

Compared with our method, relabeling with the learned RL policy reduces mean tardiness by only 0.39%, while requiring 4.38 times as much total training time. These results suggest that EF rollouts already provide enough information for the estimator to support effective threshold selection in this setting.

Our method remains effective with EF rollouts because RL and the estimator work together. RL uses information about all current tasks and resources to coordinate task-specific threshold choices, with training guided by the original system reward. It thus learns how to use these reference estimates under actual task competition, rather than directly executing potentially incompatible single-task optima.

2. Label-generation cost versus PPO training (Q1-2).

We measured the costs of label generation and PPO training on DJSP (100 jobs, IAT 5, DDT 2.0). Indeed, label generation used 23,760,500 simulator steps, approximately 4.40 times the 5,400,000 environment steps used by PPO.

However, label generation took 1.83 h and estimator training took 0.08 h, totaling 1.91 h of preprocessing—approximately 0.26 times the 7.41 h required for PPO training. This difference reflects the lower per-step cost of heuristic-controlled rollouts, which require no PPO policy inference or gradient updates.

3. What if no trigger occurs by TisafeT_i^{\mathrm{safe}}? (Q2)

In WTA/Warehouse Replenishment, untriggered tasks expire without completion reward. These forgone rewards provide feedback to PPO, encouraging thresholds that avoid missing valuable completion opportunities. DJSP due dates are soft constraints: late completion incurs weighted tardiness, but operations do not expire.

WTA uses the latest admissible release time over compatible options; feasibility checks still apply. Warehouse Replenishment uses Tisafe=diT_i^{\mathrm{safe}}=d_i, the order expiration. Existing stock permits fulfillment, so supplier lead time is not subtracted.

For DJSP, TisafeT_i^{\mathrm{safe}} is fixed at control entry tt:

Tisafe=max⁡dj(i),t+bi+∑q∈Qipq.T_i^{\mathrm{safe}}=\max\\{d_{j(i)},t+b_i+\sum_{q\in\mathcal Q_i}p_q\\}.

Here dj(i)d_{j(i)} is the due date of the job containing operation ii, bib_i remaining machine processing (zero if idle), and Qi\mathcal Q_i its precedence-ready queue, including ii and excluding the executing operation; pqp_q is processing time. Beyond TisafeT_i^{\mathrm{safe}}, operations remain pending, and the estimator and triggering gate continue to be evaluated.

4. Repeated + Delay Benefit and learning curves (Q3).

We have added Section 5.5 to introduce Repeated + Delay Benefit, which adds the same frozen estimator's output as a state feature to Repeated Delay-or-Trigger (Repeated). As shown in Figure 4, the benefit of this feature varies across benchmarks and training stages, while Ours achieves higher mean returns during intermediate and later training.

To evaluate time costs, Figure 5 compares performance against total elapsed time. Both estimator-based methods incur the same preprocessing costs. The initial preprocessing interval is omitted from view but remains included in elapsed time; Repeated trains during this interval. Ours subsequently overtakes both baselines across all benchmarks.

Following your comment, we have revised Section 5.1 to clarify that the main-table means average evaluation results from 10 independently trained policies. Evaluation uses 10 seeds, with 100 repetitions per seed. Figure 4 reports means and confidence intervals across independent training runs.

5. Related work.

Thank you for pointing us to Bao et al. (2026). We have added this work to Section 2.2. Their method learns repeated platform-level match/skip decisions with potential-based reward shaping. We use supervised delay-benefit estimation and one persistent task-specific threshold under the original reward.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Summary of revisions

We have revised the manuscript in response to the reviews:

  • Section 2.2: added discussions of WATTER and Bao et al.
  • Section 5.1: clarified the evaluation protocol.
  • Section 5.5, Figures 4–5: added learning curves and total-time comparisons.
  • Appendix A.4, Tables 10–12: added comparisons with seven classical DJSP dispatching rules.
  • Appendix A.9, Figure 6 and Table 16: added baseline training curves and comparisons under matched total training-time budgets.

Changes in the revised manuscript are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.