acceptodds
Under review as a conference paper at ICLR 2027

Hybrid multi-objective reinforcement learning algorithm for hybrid flow shop with limited workers and time-of-use electricity tariffs

Abstract

With the increasing requirements of lower carbon emission, energy-related objectives have gained increasing attention in both industrial and management decision makers. This study proposed a Q-learning-driven hybrid reinforcement learning algorithm to solve the hybrid flow shop scheduling problem, wherein limited resources, worker energy consumption, and time-of-use (TOU) electricity tariffs are considered simultaneously. Three objectives are to be minimized, including makespan, total energy consumption, and TOU electricity pricing. First, a mathematical model based on the mixed integer linear programming (MILP) is formulated. A dynamic decoding approach is then developed to choose suitable machines and workers. Next, six variations of Nawaz–Enscore–Ham heuristics are proposed to enhance the initialization approach with diversity capabilities. Considering the multi-objective handling method, an adaptive-space-based crowding distance heuristic is embedded to select solutions resulted in higher global searching abilities. Additionally, two lemmas considering machine turn off and worker rest are designed to improve certain objectives. Furthermore, a Q-learning driven hyper-heuristic combining with VNS and IG is developed to balance the exploration and exploitation abilities. Finally, detailed experimental comparisons with state-of-the-art algorithms are tested to verify the efficiency and effectiveness of the proposed algorithm.

Primary area:
optimization
Keywords:
Q-learning-driven hyper-heuristic, hybrid flow shop, worker energy consumption, time-of-use electricity pricing, multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_trL8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a MOEA + DRL algorithm for a multi-objective hybrid flowshop scheduling problem. The DRL is to do operator selection, and based on Q-learning.

Critical strengths

None

Critical weaknesses

The problem is very similar to the standard benchmark scheduling problems and no obvious extra challenge is shown.

The algorithm has no novelty - it is very similar to existing methods for NSGA-II and operator selection.

The design is very ad hoc - there is no reason to split the population into sub-populations.

There are some description errors. In line 3 of algorithm 3, the population is divided into eight sub-populations, but in line 6 it is divided into four sub-populations.

Ther is no algorithm 3.

The experiments are too simple.

Questions for authors

Why is it good to divide the population?

What are the main advantages of your algorithm over existing algorithms such as NSGA-II?

What are the main differences and advantages of your algorithm over existing operator selection algorithms?

Are there any state-of-the-art problem specific methods rather than generic methods to compare with?

Reviewer_3vLkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a hybrid multi-objective optimization algorithm to address the Hybrid Flow Shop Scheduling Problem under constraints of limited workforce availability and time-of-use electricity pricing. To enhance population diversity and solution quality, the initialization phase incorporates a mutation-based variant of the NEH heuristic; furthermore, a Q-learning–driven adaptive mechanism is introduced to dynamically select the most effective neighborhood search operator during evolution. While the experimental evaluation demonstrates promising performance across multiple benchmark instances, several methodological concerns warrant clarification and revision.

Critical strengths

The paper addresses a practically relevant multi-objective scheduling problem involving limited workforce availability and time-of-use electricity pricing. The experimental evaluation also demonstrates promising performance across multiple benchmark instances.

Critical weaknesses
  1. The mathematical formulation of the multi-objective optimization problem remains incomplete: objective functions f_2 and f_3 are either undefined or lack explicit expressions in the main text.

  2. Figure 1 (Algorithm 1 flowchart) exhibits conceptual inconsistencies. First, the bottom Gantt chart represents the output of the scheduling process—i.e., the final feasible schedule to be deployed in practice—not an input derived from the scenario. Second, the upper-left component depicts a Q-table, yet Algorithm 1’s initialization step relies solely on the NEH-based heuristic without any Q-learning involvement at this stage. This misalignment between visual representation and procedural description undermines the clarity and fidelity of the proposed framework.

  3. The dual Q-table architecture lacks essential design specifications. Specifically: (i) no definition is provided for the state space—neither its cardinality nor the semantic meaning of individual states; (ii) the action set a_1 remains unspecified, precluding assessment of exploration breadth and policy interpretability; and (iii) with only one state per Q-table, convergence guarantees vanish, and learned policies risk degenerating into arbitrary or non-generalizable mappings. A principled state-action formulation grounded in scheduling semantics is indispensable.

  4. Critical implementation details are omitted: (i) the termination criteria for all algorithms (e.g., maximum iterations, computational time limit, or convergence threshold) are not stated; (ii) Table 2 contains two distinct entries labeled “n”, yet their respective meanings (e.g., number of jobs vs. number of iterations) are ambiguous and unexplained; (iii) the origin and characteristics of the test instances (e.g., generation procedure, scale, realism) are undocumented; and (iv) citations for all baseline algorithms used in comparative experiments are absent, impeding scholarly traceability and fair evaluation.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

Reviewer_2nKaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies hybrid flow shop scheduling with limited workers and time-of-use electricity prices, aiming to reduce makespan, energy use, and electricity cost. It combines Q-learning with evolutionary search, variable neighborhood search, and iterated greedy search.

Critical strengths
  1. The problem brings worker availability and changing electricity prices into the same scheduling task. These constraints create useful trade-offs between factors.
  2. The paper includes ablations for several components . The reported results favor the full method in many cases, providing some support for the combined approach, though the modeling and testing concerns below limit confidence.
Critical weaknesses
  1. The model does not fully enforce machine assignment or processing time: Eq. (20) allows no assignment, and Eqs. (41)–(43) leave the time constraint loose. Eq. (60) also omits machine power. These errors weaken the link between the reported results and the stated problem.
  2. The learning contribution remains unclear beyond earlier Q-learning scheduling methods.
Questions for authors

Were the model errors corrected in experiments, and did the shared time budget include training and all subpopulations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.