acceptodds
Withdrawn from ICLR 2027

One State Variable Was the Whole Effect: A Controlled Study of Offline-to-Online Reinforcement Learning for Cluster Scheduling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretraining a cluster scheduler on the production scheduler's placement log, fine-tuning online, and protecting the learned policy with a safety layer and an out-of-distribution fallback is a common recipe, yet its offline premise is rarely tested. We evaluate this pipeline on a Google production trace replayed in a capacity-constrained simulator. The offline premise fails: production placements show little predictable dependence on system load and cannot be reliably recovered from any shortlist we can construct. The deployed -slot shortlist recalls of them, and a -candidate shortlist containing every machine a job has ever used reaches only , making behaviour cloning the poorest-performing model we train. Two implementation defects initially obscured this failure: using only one gradient step per update and a policy head degenerate at initialisation. The dispersion of candidate logits, rather than policy loss, reveals both defects. After correcting the pipeline, the learned policy outperforms the production scheduler. However, its apparent advantage over heuristic baselines disappears when those baselines are given the co-runner count, a state variable omitted from the original feature set: adding this variable shifts the performance frontier by 5.61 percentage points and reduces the learned policy's apparent wins from of operating points to of , while denser sampling within the same heuristic family does not change the result. We therefore argue that baseline feature coverage should be established before attributing performance gains to learning. Replication on two additional traces, including one from a different operator, yields the same qualitative finding while showing that the magnitude of the gain depends strongly on simulator capacity: the gain is 0.29 percentage points at the trace's native cluster size but 43.3 percentage points in the capacity-shrunk setting. Thus, the observed improvement should not be interpreted as a general ranking of the production scheduler. Finally, a deployable takeover-and-fallback gate reduces catastrophic episodes from of to of . The source code and the result artifacts behind every number reported here are provided in the supplementary material.

Primary area:
reinforcement learning
Keywords:
offline-to-online reinforcement learning, cluster scheduling, imitation learning, datacenter workloads, safety-critical deployment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wRv8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a controlled study of the offline-to-online reinforcement learning (RL) pipeline for cluster scheduling. It examines the utility of production traces for imitation learning, training implementation challenges, the state-information coverage of baselines, and the role of safe fallback mechanisms. Under the specific candidate construction and evaluation objectives considered, direct imitation of production logs performs poorly; after two training flaws are corrected, however, the learned policy outperforms the initial heuristic-based baseline.

The key finding is that adding the number of concurrent tasks on candidate machines to the heuristic rules improves the rule-based frontier by 5.61 percentage points. Across the dual objectives (SLA and hotspots), the number of scenarios in which the learned policy beats the rules drops from 9/20 to 0/20, and hyperparameter tuning yields no further gains. The authors accordingly emphasize the importance of verifying the information coverage of baselines before attributing benefits to learning-based methods. The paper also analyzes how capacity settings affect outcomes and shows that safe gating mechanisms can mitigate catastrophic failures during online learning.

Critical strengths
  1. The paper addresses a meaningful performance-attribution problem: by separating feature coverage from hyperparameter search density, it provides a more informative experimental analysis than a straightforward comparison between learning-based and rule-based methods.
  2. The authors are transparent about identifying and fixing training implementation issues and offer a thoughtful analysis of the limitations of log replay and capacity configuration. These efforts strengthen the credibility of the experiments and provide valuable practical insights.
Critical weaknesses
  1. The representativeness of the core findings, and their contribution to existing knowledge, remain unclear. The number of concurrent tasks is intuitive and readily available state information. Although the paper offers this empirical attribution, it is not fully clear whether the finding reflects a representative evaluation challenge in the field or mainly corrects the authors' own baseline design. It is also unclear whether comparable existing methods already use this variable (or equivalent information).
  2. The analysis of the two training flaws rules out ineffective training as a confound, which strengthens the validity of the experiments. The paper does not yet articulate, however, what new and transferable insights these diagnostics offer as contributions. The respective contributions of the core attribution findings and the training diagnostics—and the incremental knowledge gained from their combination—therefore require further clarification.
  3. The impact of the environment simplifications on the core conclusions remains to be verified. The authors commendably discuss the simulator's simplifications and the scope of their experiments in the appendix, but disclosing these limitations delimits the applicability of the findings without demonstrating that the conclusions are robust to the modeled features. For instance, co-location costs affect only new tasks and do not slow down existing ones, which may alter the performance trade-offs between learning strategies and rules. The current analysis does not sufficiently address whether the primary conclusions would still hold if resource contention also affected task completion times and resource release.
Questions for authors
  1. Which representative existing methods fail to account for the number of concurrent tasks (or equivalent information)? Why was this factor omitted from the paper's original baseline? Is this omission representative enough to support conclusions beyond the specific experimental setup presented here?
  2. Do the learning-based strategies and the heuristic rules have access to the same information and the same set of candidate machines? Can the full comparison—with and without this variable—be replicated on a different time window or a separate trace?
  3. When a task is migrated to another machine (or placed on a machine already running other tasks), how does the simulator determine the task's execution duration and the timing of its resource release? Have these predictions been validated?
  4. Could the same set of tasks be run on a small-scale real cluster to compare the learning-based strategies against the heuristic rules, verifying whether the methods that win in simulation also win in practice? If real-world experiments are infeasible, could the simulation at least model the mutual slowdown caused by co-located tasks to check whether the primary conclusions remain valid?
Reviewer_CbBEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that learned scheduling policies outperform those rule-based scheduling policies commonly used in production systems, but the performance gains often are a result that the learned policies have access to information that traditional rule-based policies in production systems do not. The paper evaluates scheduling policies trained through offline learning and online reinforcement learning using production traces in a simulated environment. It compares the learned policies against a parameterised family of rule-based schedulers, progressively extending the rules with additional state variables and decision mechanisms to investigate the sources of the learned policies' apparent performance advantages.

Critical strengths

The paper compares learned scheduling policies and rule-based policies using a production trace. It identifies sources of the performance gains of the learned schedulers as being able to access certain information that rule-based schedulers do not use, like co-runner count and queue length.

Critical weaknesses

In this study, the authors construct the rule-based baselines and then improve them by incorporating additional information. The improvement is done manually, which perhaps can be seen as a manual distillation from RL scheduling results. It is not surprising that these customised heuristic scheduling policies can achieve good performance with such progressive feature addition and improved heuristic (previous work showed a distilled simple scheduling policy outperform RL-based scheduling method under certain circumstance: arxiv.org/abs/2405.19131).

From another angle, it is a bit difficult to understand why these features are not included at the first place in designing rule-based policies. The effect of these features can be evaluated through ablation studies of RL scheduling algorithms to measure the performance degradation after removing some features. This may be more convincing to show the feature coverage plays a role in the performance gap in scheduling policies, including between RL-based scheduling policies, and between heuristic rule-based scheduling policies and RL-based scheduling policies.

In general, the paper's novelty and contribution are limited mainly to reporting an observation.

Questions for authors

see weaknesses

Reviewer_Yc28OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests a pipeline for learned cluster scheduling: imitate the production scheduler’s log, fine-tune online with RL, and guard the policy with a safety layer and an out-of-distribution fallback. On the Google 2011 trace, production placements prove a poor imitation target, and two implementation bugs initially prevented learning. After the fixes, the learned policy beats the replayed production scheduler, by a large margin only when the simulated cluster is shrunk, but loses its advantage over heuristic rules once the rules can read the co-runner count. The rule-versus-production comparison replicates on two more traces, and a takeover-and-fallback gate limits catastrophic failures of an online RL policy.

Critical strengths
  1. A simple, reusable test of where a learned scheduler’s advantage comes from: give the baseline rules the learner’s state variables one at a time and check whether the advantage survives. Here it vanishes once the rules see the co-runner count, while a finer rule grid changes nothing.
  2. I consider the evaluation honest and transparent. A paired comparison separates real scheduling gains from the effect of shrinking the cluster, and the paper reports its own bugs and the conditions that would overturn its claims.
Critical weaknesses
  1. The paper builds its study around a “common recipe” (imitate the production scheduler’s log, fine-tune online, add a safety guard), but gives no evidence from the literature or from practice that this recipe is used. By the paper’s own account, prior imitation-based schedulers imitate heuristic or solver experts, not production logs. The assumption the paper refutes, that production placements are a useful imitation target, therefore does not seem to be one that existing work relies on. In addition, the learning pipeline is tested on a single 2011 trace, which is too narrow to support general conclusions about learned schedulers versus simple rules.

  2. The tested learned scheduler is weak. It is a small pipeline built by the authors (it chooses among 64 candidate machines and is trained for 15 PPO rounds), not an established learned scheduler. Given the same information as the simple rules, it still performs worse, which suggests that it has not learned to use that information. The negative result may therefore reflect an under-trained learner rather than a limitation of learned scheduling. A more convincing test would use established learned schedulers in this field.

Questions for authors
  1. The learning pipeline and the imitability measurements use only the 2011 Google trace, so the conclusion that production placements are a poor imitation target concerns Borg in 2011. The paper selects traces that record which machine each task was placed on (Sec. 3.1), and newer public traces also do: the 2019 trace used in Sec. 5.6, whose capacity is recoverable, and Alibaba’s PAI 2020 GPU trace, which also gives true machine capacities. Why is the 2011 trace the main testbed? What prevents running the pipeline on a newer trace, and do the Sec. 4 measurements hold on the 2019 trace, as Table 11 itself proposes? A short comparison of candidate traces (placement records, capacities, submission times, workload type, year) would justify the choice.

Minor: The introduction opens with the recipe without motivating it. While the paper claims this is a common pipeline, no direct evidence supports this statement. The opening paragraph does not explain why one would imitate a production scheduler, so the starting point is hard to follow, even for readers familiar with cluster scheduling. I strongly recommend that the authors rewrite the opening of the introduction to clarify the background and the motivation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.