- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes FSPO, an adaptive controller for LLM RL post-training that aims to improve training performance while maintaining stability and satisfying computational resource constraints. FSPO combines three components: policy-consistent risk estimation for predicting future training instability, decision-conditioned trajectory calibration (DCTC) for calibrating risk on selected actions, and a Pareto resource continuation certificate (PRCC) for ensuring future resource feasibility. The controller selects the highest-utility configuration satisfying the risk and resource constraints. The authors evaluate FSPO on a synthetic benchmark and GRPO post-training tasks, reporting improvements in held-out and OOD accuracy and reduced training-instability rates compared with several baselines. Ablations are provided to support their design choices.
Critical strengths
-
The paper addresses balancing big performance gains with training stability and resource allocation, which is an important problem especially in the context of LLM post training where training is costly.
-
The algorithm is mostly reasonable, has a coherent decomposition of the training issues and the existence of components are justified; future risk, calibration, multi-resource feasibility etc. all address specific, well-motivated problems. The paper provides several ablations and baseline comparisons to justify these design choices.
Critical weaknesses
-
Action space being limited to a small number of fixed, hand-crafted configurations is a significant drawback. For example, the current formulation can learn that an exploratory config is unsafe but since each config has multiple different hyperparameters it says nothing about which hyperparameter/s cause that unsafety (Table 39 does explore this approach, but the deployed controller cannot really exploit this information). Another consequence of this limitation is that if the filtering step filters out all 3 actions then FSPO can do nothing to improve the training. It would be a much more interesting formulation to be able to augment this action space either by expanding the set of configurations over the course of training or by tuning the existing configurations online.
-
The scalability and impact of the Pareto step are not sufficiently established. The number of non-dominated cumulative resource reservations can grow exponentially with the size of the action catalog and planning horizon, which limits extension to higher-dimensional action spaces (see Weakness 1). Although the paper evaluates catalogs with up to 18 actions, it is unclear how well this approach would scale to substantially larger catalogs. Furthermore, in the original synthetic benchmark, the conservative configuration has the lowest declared costs across all resource dimensions, making the Pareto check equivalent to a much simpler resource-reservation test. Thus it is unclear how much of the observed benefit requires the Pareto formulation rather than prospective budget checking.
Questions for authors
-
Regarding Weakness 1, why did you use this action space, and how would you propose to handle this drawback? An interesting extension of your work would be combining FSPO with, for example, Bayesian optimization to propose new configs or refine existing ones.
-
How did you obtain the coefficient values you use in the synthetic benchmark’s safety predicate? It is unclear from the paper.