- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies offline autoscaling where objectives are specified after training. First, it shows that pooling the logs of four heuristic controllers and perturbing the actions they take improves an Implicit Q-Learning (IQL) agent focused on utilization. It then trains, on the same logs, a Forward-Backward (FB) model that is reward-free, using centered reward queries plus occupancy-target capping, as well as an optional search across policy-conditioning vectors. Tests run on a distributed search-engine simulator: 266 windows held out for evaluation, five seeds used in training, and about 402,000 training transitions. A single frozen FB model nearly matches reward-specific IQL on rejection and latency, while falling short on utilization. On a fresh capacity-threshold reward it also beats preference-conditioned IQL. Results are less favorable on three further objective changes.
Critical strengths
-
Objectives that are supplied after training are supported by the model, with no parameter updates and no further environment interaction needed for policy selection. For example, Table 3 reports searched FB reaching 22.27±0.18 on the capacity-threshold reward with Q=20, against 21.36±0.19 for preference-IQL. That baseline is handed its best tested preference, picked using test returns. Evidence that FB copes with changes beyond a reweighted fixed reward basis comes from keeping the same checkpoints while varying capacity allowances.
-
The comparison of data collection is useful. With pooled dataset sizes matched approximately, local exploration lifts the utilization return of IQL to 6.94 from 6.54, while behavior cloning's return falls to 4.01 from 5.86 (Table 1). So reward-aware learning benefits from the extra experience even if imitation worsens. Five seeds are used for the main experiments, with a shared evaluation suite, split off from training along source time series.
Critical weaknesses
-
The gain is inconsistent on new objectives. Across the three primary objectives of Table 3, preference-IQL posts higher mean returns compared to either FB variant. Its reported means are also higher on the three further objective changes of Appendix A.6, even though its reward inputs cannot express those changes directly. Whether these differences reach statistical significance is unclear, yet they weaken support for whatever advantage a reward interface this flexible brings. A positive quota result stands, though no policy for quota-specific IQL is reported, so what avoiding retraining costs in performance remains unclear. The paper thus demonstrates useful adaptation within the quota family, but does not establish when that benefit reaches other objectives.
-
Within the final setup, the FB changes that are proposed never get isolated. Ablations for capping and centering rely on a plain-MLP model from earlier, and a separate corpus for training (Appendix A.3). Residual networks appear in the final results. This matters most where the occupancy cap is concerned: a 24-step horizon is what motivates setting it at 21, although what training bootstraps is a continuing task whose discounted occupancy mass is 100. By the time training finishes, that cap is still rescaling 47% to 58% of target rows (Appendix D.7). To learn whether these changes lift stability and returns, and how much the results depend on the cap, ablation runs at the level of the final architecture are required. Absent that, what the proposed modifications contribute stays unclear.
Questions for authors
-
How would a policy for quota-specific IQL fare on that same evaluation suite and corpus? Which preference vectors, in full, did the quota comparison test? Answers would clarify both what avoiding retraining costs in performance, and how strong the preference baseline is.
-
What happens if capping and centering get removed or varied, each independently, on both the final residual model and the main training corpus? Please report downstream returns as well as training stability.
-
How well does FB do, without exploration, on a pooled corpus of approximately equal size? Since exploration alters 51.7% of the actions a controller proposes (Appendix K), such a comparison would show how far the result rests on intentionally perturbed data instead of ordinary controller logs.
The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.
AC