acceptodds
Under review as a conference paper at ICLR 2027

Offline Learning of Flexible Autoscaling Policies from Controller Logs

Abstract

Autoscaling is the primary mechanism through which cloud platforms adapt compute capacity to changing demand. Yet many production systems continue to rely on hand-designed heuristics. For example, Kubernetes primarily scales resources based on CPU utilization. Reinforcement learning (RL) offers a way to learn autoscaling policies from logs, but online exploration is impractical in these systems. Moreover, deployment objectives can vary substantially across operators and workloads, particularly since different users have different latency-cost trade-offs. Training a separate policy for each objective is computationally expensive and scales poorly with the number of users. We therefore study whether logs generated by autoscaling controllers can enable offline policy training and support different downstream objectives without retraining. Using strong controller baselines and realistic workloads in a distributed-system simulator, we answer both questions affirmatively. Our experiments show that pooling logs from multiple controllers and applying bounded local perturbations to their actions enable offline learning that matches the performance of an objective-aligned expert. Building on this result, we adapt Forward–Backward representations to learn a reward-free model from the same controller logs. After training, when queried with rewards, it comes within of policies trained for each objective on latency and rejection and outperforms a preference-conditioned baseline on a new capacity objective ( versus ), while utilization remains harder ( versus ). To our knowledge, this is the first reward-free offline policy extraction for autoscaling trained entirely from offline controller logs.

Primary area:
reinforcement learning
Keywords:
Autoscaling, Reinforcement Learning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EogkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies offline autoscaling where objectives are specified after training. First, it shows that pooling the logs of four heuristic controllers and perturbing the actions they take improves an Implicit Q-Learning (IQL) agent focused on utilization. It then trains, on the same logs, a Forward-Backward (FB) model that is reward-free, using centered reward queries plus occupancy-target capping, as well as an optional search across policy-conditioning vectors. Tests run on a distributed search-engine simulator: 266 windows held out for evaluation, five seeds used in training, and about 402,000 training transitions. A single frozen FB model nearly matches reward-specific IQL on rejection and latency, while falling short on utilization. On a fresh capacity-threshold reward it also beats preference-conditioned IQL. Results are less favorable on three further objective changes.

Critical strengths
  1. Objectives that are supplied after training are supported by the model, with no parameter updates and no further environment interaction needed for policy selection. For example, Table 3 reports searched FB reaching 22.27±0.18 on the capacity-threshold reward with Q=20, against 21.36±0.19 for preference-IQL. That baseline is handed its best tested preference, picked using test returns. Evidence that FB copes with changes beyond a reweighted fixed reward basis comes from keeping the same checkpoints while varying capacity allowances.

  2. The comparison of data collection is useful. With pooled dataset sizes matched approximately, local exploration lifts the utilization return of IQL to 6.94 from 6.54, while behavior cloning's return falls to 4.01 from 5.86 (Table 1). So reward-aware learning benefits from the extra experience even if imitation worsens. Five seeds are used for the main experiments, with a shared evaluation suite, split off from training along source time series.

Critical weaknesses
  1. The gain is inconsistent on new objectives. Across the three primary objectives of Table 3, preference-IQL posts higher mean returns compared to either FB variant. Its reported means are also higher on the three further objective changes of Appendix A.6, even though its reward inputs cannot express those changes directly. Whether these differences reach statistical significance is unclear, yet they weaken support for whatever advantage a reward interface this flexible brings. A positive quota result stands, though no policy for quota-specific IQL is reported, so what avoiding retraining costs in performance remains unclear. The paper thus demonstrates useful adaptation within the quota family, but does not establish when that benefit reaches other objectives.

  2. Within the final setup, the FB changes that are proposed never get isolated. Ablations for capping and centering rely on a plain-MLP model from earlier, and a separate corpus for training (Appendix A.3). Residual networks appear in the final results. This matters most where the occupancy cap is concerned: a 24-step horizon is what motivates setting it at 21, although what training bootstraps is a continuing task whose discounted occupancy mass is 100. By the time training finishes, that cap is still rescaling 47% to 58% of target rows (Appendix D.7). To learn whether these changes lift stability and returns, and how much the results depend on the cap, ablation runs at the level of the final architecture are required. Absent that, what the proposed modifications contribute stays unclear.

Questions for authors
  1. How would a policy for quota-specific IQL fare on that same evaluation suite and corpus? Which preference vectors, in full, did the quota comparison test? Answers would clarify both what avoiding retraining costs in performance, and how strong the preference baseline is.

  2. What happens if capping and centering get removed or varied, each independently, on both the final residual model and the main training corpus? Please report downstream returns as well as training stability.

  3. How well does FB do, without exploration, on a pooled corpus of approximately equal size? Since exploration alters 51.7% of the actions a controller proposes (Appendix K), such a comparison would show how far the result rests on intentionally perturbed data instead of ordinary controller logs.

1 comment

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

Reviewer_4TQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether logs from existing autoscalers can support offline policy learning for objectives chosen after training. The setting is a discrete-event simulator of a distributed search engine, with BOOM-derived workloads and 252 distinct allocations. The main contributions of the paper are the data collection methodology and the reward-free model. All results are produced via simulation using five seeds.

Critical strengths
  1. The paper shows that combining logs from multiple controllers and adding action exploration improves the diversity of available actions. An approach based on implicit Q-learning (IQL) benefits from this exploration, while behaviour cloning performs worse. This suggests that reward-aware learning can make better use of diverse logged data than imitation learning.

  2. The authors report results across all seeds, investigate weak performance, and include unfavourable results. Reward centering also provides a simple way to improve the stability of the forward-backward (FB) model.

Critical weaknesses
  1. FB does not consistently match preference-conditioned IQL on the tested objectives and performs worse on all three in-family objectives. Its out-of-family advantage is limited to one of the four tested cases.

  2. Additionally, considering latency and rejection, the logging controller outperforms the learned methods. These results contradict the paper's claims of broad generalisation.

  3. The authors do not compare FB against simply selecting the best existing controller by evaluating its logged data under each new reward. This baseline requires no additional training and could outperform FB on several objectives. Without this comparison, it is unclear whether FB learns genuinely useful new policies or just reproduces behaviours already present in the data.

Questions for authors

1.⁠ ⁠Were the quota reward and the additional targets in Table 5 chosen before evaluation? Why is quota highlighted in the main text while the other targets are only reported in the supplementary material?

  1. Can you test the mass cap and reward centering on the residual model? Why use a discount factor of 0.99 and not the one matched to the 24-step evaluation horizon? How does the frequently active cap affect the accuracy of the learned successor measure?

  2. Can you evaluate IQL and FB on utilisation without CPU-ScaleX, using a comparable dataset size? Also include the missing ScaleX-only exploration result referenced in Table 1 and Table 17.

  3. Simulated latency is approximately ten times higher than real-system latency. How does this affect the conclusions? Can you also report results across different demand levels, given the differences between training and test workloads?

1 comment

The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.

AC

Reviewer_kFfiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this work the authors study whether autoscaling logs can support policies whose operating objectives are supplied after training. They first compare offline learning from a single controller, a pool of controllers, and the same pool with local action perturbations. They then adapt Forward-Backward (FB) representations to learn a reward-free policy family from the pooled, perturbed logs. At inference, a reward evaluated on logged states selects a policy without changing the model parameters. An optional search chooses among nearby policy indices using the model's predicted values. The experiments use a distributed search-engine simulator, workloads constructed from BOOM time-series patterns, and five training seeds. FB approaches reward-specific IQL on latency and rejection and improves over a preference-conditioned baseline on a new capacity-threshold reward. Utilization remains harder, and additional changes to service-level objectives give mixed results. The paper offers a useful empirical study of reusable offline control, but the breadth of reward flexibility and the contribution of the proposed adaptations need stronger support.

Critical strengths
  • The problem is important and the learning settings are clearly distinguished. Training a separate autoscaler every time an operator changes the objective is unattractive. The authors separate reward-specific learning, preference conditioning within a known reward family, and supplying a reward only after pretraining. This distinction gives the paper a concrete question rather than treating all forms of adaptation as equivalent.
  • The data-collection analysis is useful. Workload diversity and action diversity are not the same thing. The factored formulation makes this clear, and the experiments connect it to learning outcomes. In the approximately volume-matched comparison, adding local exploration raises IQL's utilization return from 6.54 to 6.94 while reducing behavior cloning from 5.86 to 4.01. This is informative evidence that experience useful for reward-based learning need not be useful for imitation.
  • The reward-free model produces meaningful results without downstream parameter updates. With the same checkpoint reused across objectives, FB search reaches 18.56 versus IQL's 18.80 on latency and 23.27 versus 23.04 on rejection. These results support the feasibility of separating representation learning from objective specification, even though they do not establish uniformly strong performance.
  • The capacity experiment goes beyond reweighting the original penalties. The thresholded capacity reward cannot generally be expressed through the preference baseline's fixed linear basis. At an allowance of 20 compute units, searched FB returns 22.27 versus 21.36 for the best tested preference. Re-querying the same checkpoints at different allowances is a useful test of the intended interface.
  • The evaluation protocol is comparatively careful. The authors split source series before generating workloads, use common evaluation windows and controller resets, report five-seed variation, and discuss paired and trace-clustered uncertainty. They reuse final checkpoints rather than selecting a different model for each reward, and retain the weak utilization seed in the reported averages.
  • The paper reports limitations and negative results openly. These include the utilization gap, the lack of consistent benefit from wider search, failures on additional target changes, and the heuristic nature of the occupancy cap. The distinction between a higher predicted score and a higher realized return is also stated clearly. Similarly, the authors do not present the capacity penalty as a hard constraint
Critical weaknesses
  • The evidence for broad reward flexibility is narrower than the main presentation suggests. The quota experiment is a positive result, but Appendix A.6 is important to the central claim and should not remain peripheral. The preference model has higher mean return on all three additional changes: tightened latency, a utilization setpoint of 85%, and an availability cliff. For example, searched FB obtains 4.79 versus 5.40 on the changed utilization target. Even on the original utilization objective, closed-form FB reaches 5.59 versus the specialist's 6.94, a roughly 19% relative gap. I do not expect a reward-free learner to dominate informed specialists, and the ability to accept a reward is valuable in itself. However, accepting a new reward and selecting a good policy for it are different claims. Please bring these additional target changes into the main evaluation and explain which kinds of changes the representation handles well. A quota-specific IQL policy trained on the same logs would also provide a missing reference: the current comparison shows an advantage over a restricted preference interface, but not how close FB comes to an objective-specific policy on that reward.

  • The related work should position the reuse and selection mechanisms against closer alternatives. Yu et al. (2020), MOPO: Model-based Offline Policy Optimization: discuss frozen-dynamics planning as an alternative for reward reuse, without treating MOPO itself as zero-shot policy extraction. Barreto et al. (2020), Fast reinforcement learning with generalized policy updates: distinguish fixed-policy search from statewise generalized policy improvement in the main discussion. Paine et al. (2020), Hyperparameter Selection for Offline Reinforcement Learning: connect candidate ranking to offline policy selection and fitted Q-evaluation.

  • The positioning should also consider existing risk- and uncertainty-aware resource optimization frameworks. Overprovisioning and oversubscription already have a substantial optimization literature spanning offline learning and online adaptation. COIN (2024): learns oversubscription policies from historical telemetry using chance-constrained imitation learning to account for uncertainty and congestion risk. ProtoRAIL (2025): combines risk-cognizant imitation learning with active knowledge feedback to balance resource efficiency and overload risk. These address related but different control problems, and neither should be assumed to provide zero-shot reward reuse. The authors should discuss what extensions would let such frameworks learn autoscaling policies from controller logs, whether adapting them to changed objectives would require retraining or online feedback, and what FB adds relative to those possibilities. A focused discussion of these extensions would better position the claims without requiring a full implementation of each alternative.

  • The useful logs are actively constructed, not simply inherited from ordinary controller operation. The offline learning protocol is valid: the learners do not collect data during training. However, the main FB model is trained on deliberately perturbed simulator trajectories. Appendix K reports that exploration changes the commanded action on 51.7% of decisions, with no safety guard beyond the allocation bounds. A two-position move on the capacity-tier axis can also be a substantial resource change. This matters because the motivation is that online exploration is impractical. Please distinguish what is demonstrated with existing unmodified logs from what requires an exploratory data-collection procedure. An FB comparison on the unperturbed pooled corpus would directly test that dependence. The pooling analysis also changes data volume from about 100k to 402k transitions and adds a controller already targeting 70% CPU. The paper acknowledges both points. Table 11 provides a useful three-controller comparison, but its corpus also differs from the primary explored corpus in volume and exploration. Matched-size subsets and removing the objective-aligned controller while holding the other collection choices fixed would better isolate these effects.

  • The proposed FB adaptations need ablations on the model used for the headline results. Occupancy-mass clipping and reward centering are sensible practical ideas, but Appendix A.3 evaluates their necessity on an earlier plain-MLP model and a different corpus. The main residual model inherits them without the same controlled comparisons. This leaves open whether the gains come from the adaptations, the backbone, or their interaction. The cap is also active on 47-58% of target rows, so it is not an occasional numerical safeguard. Training bootstraps a continuing process with discount 0.99, whose total discounted occupancy is 100, while the cap of 21 is motivated by the 24-step evaluation horizon. The authors correctly call this heuristic. I would like sensitivity to the cap and evaluation horizon, together with centering and backbone-matched ablations. The actor update also differs from the IQL references in advantage weighting and normalization. The paper should isolate its contribution or present the method more explicitly as an evaluated recipe rather than attributing the result to a small number of identified changes.

  • The simulation supports a useful feasibility study, but limited evidence of generality. Appendix F validates directional behavior against two short OpenSearch workloads, not quantitative policy rankings or the latency, rejection and scaling costs that determine the rewards. This is a reasonable starting point, but it does not establish that a policy that wins in the simulator would retain its advantage under modest changes to those dynamics. BOOM supplies real temporal patterns, while arrival rates, request costs and request types are synthesized; the generated workloads use GET requests with a fixed resource profile within each workload. Please test sensitivity to scaling overhead or service-cost parameters, and report results by the existing calm, borderline and overload strata. Longer rollouts would also help, since collection resets after 12 decisions and evaluation scores 24. I do not regard a production deployment as necessary for this paper, but some evidence that the conclusions survive changes within the simulator would make the empirical contribution more convincing.

  • The computational advantage is motivated but not measured. Equal update counts do not imply equal training costs: FB uses a larger batch and pairwise occupancy computations, and its searched inference evaluates 257 policy indices on 512 scoring states. The paper provides detailed hyperparameters but not the training time, query time, memory use or amortization point relative to training specialists. Since avoiding repeated training is a main motivation, please report these costs for direct FB, searched FB, preference-IQL and an IQL specialist. This would show when learning once is actually cheaper, and whether the modest search gains justify the additional query cost.

Questions for authors
  • Can exitsing cloud resource management / optimization policies (offline/online) be adapted or extended to leanring autoscaler policies?
  • Which adaptations remain necessary on the final residual backbone, and how sensitive are the results to the occupancy cap and evaluation horizon?
  • What are the actual training and reward-query costs, including search, relative to the specialist and preference-conditioned alternatives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.