acceptodds
Under review as a conference paper at ICLR 2027

FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training

Abstract

Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL regularization, verifier allocation, and update budget. Three coupled issues remain unresolved. A future-risk model trained from behavior trajectories need not estimate the risk induced by the controller that will be deployed; a score calibrated on logged state–action pairs can become miscalibrated after selective action choice; and independent per-resource minimum costs do not in general certify a feasible multi-resource continuation. We introduce FSPO, a feedback-state controller for budgeted LLM RL post-training that addresses these issues jointly. FSPO learns a policy-consistent risk-to-go model whose Bellman target follows the same frozen controller used for future decisions, together with a long-horizon utility model. Decision- conditioned trajectory calibration (DCTC) calibrates risk on cross-fitted trajectories generated by actions selected by provisional controllers. A Pareto resource continuation certificate (PRCC) admits an action only when a non-dominated cumulative reservation remains feasible over the residual horizon. The deployed controller selects the action with the largest pessimistic long-horizon utility among candidates passing both gates. Under a matched GRPO resource envelope, FSPO reaches held-out and OOD accuracy, compared with and for PB2, the strongest evaluated adaptive baseline. Its held-out interval-failure rate is , with training GPU-hours and total development GPU-hours per seed. Three paired training seeds give gains of and percentage points over the contextual bandit on held-out and OOD evaluation. Policy-mismatch, selected-action calibration, and multi-resource continuation tests separately examine the three decision mechanisms.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Reinforcement Learning Post-Training, Adaptive Control, Risk-Sensitive Optimization, Policy Optimization, Resource-Constrained Learning, Calibration, Safe Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_YKkFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FSPO, an adaptive controller for LLM RL post-training that aims to improve training performance while maintaining stability and satisfying computational resource constraints. FSPO combines three components: policy-consistent risk estimation for predicting future training instability, decision-conditioned trajectory calibration (DCTC) for calibrating risk on selected actions, and a Pareto resource continuation certificate (PRCC) for ensuring future resource feasibility. The controller selects the highest-utility configuration satisfying the risk and resource constraints. The authors evaluate FSPO on a synthetic benchmark and GRPO post-training tasks, reporting improvements in held-out and OOD accuracy and reduced training-instability rates compared with several baselines. Ablations are provided to support their design choices.

Critical strengths
  1. The paper addresses balancing big performance gains with training stability and resource allocation, which is an important problem especially in the context of LLM post training where training is costly.

  2. The algorithm is mostly reasonable, has a coherent decomposition of the training issues and the existence of components are justified; future risk, calibration, multi-resource feasibility etc. all address specific, well-motivated problems. The paper provides several ablations and baseline comparisons to justify these design choices.

Critical weaknesses
  1. Action space being limited to a small number of fixed, hand-crafted configurations is a significant drawback. For example, the current formulation can learn that an exploratory config is unsafe but since each config has multiple different hyperparameters it says nothing about which hyperparameter/s cause that unsafety (Table 39 does explore this approach, but the deployed controller cannot really exploit this information). Another consequence of this limitation is that if the filtering step filters out all 3 actions then FSPO can do nothing to improve the training. It would be a much more interesting formulation to be able to augment this action space either by expanding the set of configurations over the course of training or by tuning the existing configurations online.

  2. The scalability and impact of the Pareto step are not sufficiently established. The number of non-dominated cumulative resource reservations can grow exponentially with the size of the action catalog and planning horizon, which limits extension to higher-dimensional action spaces (see Weakness 1). Although the paper evaluates catalogs with up to 18 actions, it is unclear how well this approach would scale to substantially larger catalogs. Furthermore, in the original synthetic benchmark, the conservative configuration has the lowest declared costs across all resource dimensions, making the Pareto check equivalent to a much simpler resource-reservation test. Thus it is unclear how much of the observed benefit requires the Pareto formulation rather than prospective budget checking.

Questions for authors
  1. Regarding Weakness 1, why did you use this action space, and how would you propose to handle this drawback? An interesting extension of your work would be combining FSPO with, for example, Bayesian optimization to propose new configs or refine existing ones.

  2. How did you obtain the coefficient values you use in the synthetic benchmark’s safety predicate? It is unclear from the paper.

Reviewer_WwmAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces FSPO, a controller for adapting sampling, optimization, and verifier settings during LLM reinforcement-learning post-training. It addresses three issues: future-risk estimates depend on the controller used for subsequent decisions, calibration can deteriorate after action selection, and independent resource minima may not correspond to a feasible continuation.

FSPO combines policy-consistent utility and risk models, decision-conditioned trajectory calibration, and a Pareto resource continuation check. It selects the action with the highest pessimistic utility among candidates satisfying the risk and resource gates.

The evaluation includes a controlled mechanism benchmark and GRPO post-training, together with component ablations, calibration comparisons, resource accounting, and transfer studies. FSPO reports improved held-out and OOD task accuracy and lower interval-failure rates than the evaluated baselines under the declared training-resource envelope.

Critical strengths
  1. The paper presents a useful integration of future-risk estimation, selected-decision calibration, and resource-aware continuation. The calibration experiments directly address the motivating selection problem rather than measuring only all-action prediction quality. At matched acceptance, Table 35 reports that DCTC reduces selected-decision ECE from 0.039 to 0.022 and trajectory failure from 0.101 to 0.083. This is consequential evidence for the proposed calibration procedure.

  2. The trainer evaluation provides substantial support for the combined controller. FSPO reaches 66.11% held-out and 59.43% OOD accuracy, compared with 64.47% and 57.03% for PB2, at comparable final-training GPU-hours. The inclusion of a same-architecture Safety-Critic Gate, a full component factorial, and feedback-state ablations helps distinguish the proposed design from improvements due only to architecture or static action choices.

Critical weaknesses
  1. The operational description of decision-conditioned calibration is incomplete. Section 3.5 and Appendix A.4 describe replaying provisional-controller actions and collecting complete-trajectory labels. In actual GRPO training, changing an action changes the model and its subsequent trajectory. The manuscript does not clearly explain how the corresponding controller-specific outcomes are obtained. This matters because these labels are central to the claimed correction of policy and selection mismatch. It is a clarification gap rather than a demonstrated error in the reported results.

  2. The development-cost claim depends on an accounting boundary that is not explicit. Table 41 separates fitting, calibration, search, and final training, but does not clearly identify where collection of the fitting and calibration trajectories is charged. Whether those trajectories are newly generated, reused, or amortized affects the interpretation of the reported total development cost. The breakdown is useful, but this boundary should be stated directly.

Questions for authors
  1. How are complete-trajectory labels generated for provisional-controller actions in the GRPO calibration procedure? Please clarify the execution process behind “replay” in Section 3.5 and Appendix A.4.

  2. Where does Table 41 account for collecting the fitting and calibration trajectories? Are any acquisition costs shared or amortized across runs?

  3. Please clarify the secondary reasoning workload, evaluation-set sizes, and construction of the repair-data training and evaluation splits.

  4. If available from the reported runs, please provide the paired seed differences against PB2 and the same-architecture Safety-Critic Gate.

Reviewer_1S5GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary of the paper

This paper proposes FSPO, a feedback-state controller that adapts GRPO training knobs (sampling temperature, group size, clipping, KL coefficient, verifier calls, and escalation) online under a fixed resource budget. The controller consists of three components:

(1) a policy-consistent risk-to-go model, whose Bellman target follows the deployed controller rather than the behavior policy that generated the logged trajectories;

(2) DCTC, an isotonic calibration map fit on cross-fitted, controller-selected decisions rather than on all logged state–action pairs;

(3) PRCC, a Pareto-frontier feasibility certificate over token, verifier, and device-time budgets that admits an action only if a feasible continuation exists for the remaining horizon.

The deployed controller selects, among actions passing both the risk and feasibility gates, the one with the highest pessimistic long-horizon utility. Evaluated on Qwen2.5-3B-Instruct with a structural-repair task, FSPO reports 66.11% held-out and 59.43% OOD accuracy, compared with 64.47%/57.03% for PB2, the strongest baseline, along with the lowest interval-failure rate (0.014) among the evaluated controllers. The paper additionally includes a synthetic mechanism benchmark and a set of component ablations.

Critical strengths

Strengths

Overall, this work has two strengths:

  1. The motivation is clear and impactful. The three failure modes (policy-induced risk shift, selection-induced calibration shift, vector-valued budget feasibility) are clearly articulated and plausible.

  2. Reasonable experimental settings. The ablation design is extensive: a 2×2×2 factorial over the components, matched-budget baselines including a same-architecture Safety-Critic Gate, and paired seed contrasts. And the author has illustrated the results in a clear way.

Critical weaknesses

Nevertheless, I find the following aspects to be critical weaknesses of this paper:

Weaknesses

  1. The role of the mechanism benchmark is unclear, and FSPO is never evaluated on it. Sections 3.3 and 4.2 ("Mechanism benchmark") and Tables 3/18/19/27 compare uniform allocation, a hand rule, a contextual bandit, and a hand-coded "fixed feedback-state controller" (Eq. 14). None of the three claimed contributions (policy-consistent risk, DCTC, PRCC) appears in this benchmark, yet it occupies a large fraction of the main text, which makes it hard for the reader to identify the paper's actual focus. Moreover, the benchmark is a hand-specified simulator: the transition dynamics (Eq. 13, Appendix A.2), the stability predicate (Eq. 1), and the cost model are all author-chosen constants, and its conclusions are not consistent across streams (the rule wins on seed 13, the step-ratio bandit wins on held-out, and the frozen rule wins on OOD; Table 27). Together these points substantially weaken the groundedness of the mechanism-level claims.

  2. The evaluation is too narrow to support the generality of the claims. The main comparison (Table 2) uses a single model (Qwen2.5-3B-Instruct) on a single task family (structural repair), with three training seeds. The only other scale is one Qwen2.5-7B-Instruct row in Table 42, on a "verifier-based reasoning" task that is not defined and against an unspecified baseline. It is therefore possible that the gains are specific to small models or to the Qwen family. Evaluation across additional model families and scales, on clearly defined tasks, is needed to validate the effectiveness of the proposed method.

  3. Budget matching and baseline accounting are internally inconsistent. Appendix C.3 states that population methods are compared "under the same total budget," but Table 41 lists PB2 at 40.73 total GPU-hours versus 10.56 for FSPO. Either PB2 was given roughly 4× the budget (which changes how the 1.6 pp gap should be read) or Table 41 is wrong. In addition, the same result appears under several names with inconsistent costs: the triple 62.77/54.98/0.043 is reported as the "frozen-state learned" baseline (Table 2), "frozen or shuffled telemetry" (Table 38), "frozen initial telemetry" (Table 55), and "without telemetry features" (Table 58), with GPU-hours of 9.28, 10.17, and 9.18 respectively. If these are one run, the baseline count is inflated and the cost columns are inconsistent; if they are separate runs, identical accuracies to two decimal places require explanation.

  4. Presentation obscures the method. The main text is dominated by ledger/contract/receipt vocabulary, while the actual learning procedure (number of fitted policy evaluation iterations, network architecture, DCTC fold count, risk-threshold selection) appears only in Appendix A.4 and Table 5. The headline numbers are repeated verbatim in the abstract, introduction, Section 4.2, and conclusion. Many appendix tables duplicate one another (Table 2 ≈ 16 ≈ 49 ≈ 53; Table 3 = 18; Table 41 ≈ 45), and table numbering is out of order (Table 58 precedes Table 55).

Questions for authors

I would appreciate it if the authors could address the following concerns:

Questions

  1. Beyond the single Qwen2.5-7B-Instruct row in Table 42, could the authors report FSPO's performance on models from other families and scales (e.g., Llama-3-8B-Instruct)? This would clarify whether the gains are specific to small Qwen models or whether the controller transfers more broadly.

  2. The budget matching between FSPO and the population-based baselines is currently inconsistent (Appendix C.3 vs. Table 41). Could the authors propose and run an experimental setting in which the total compute is strictly matched, for example by running PB2 at the same total GPU-hours as FSPO (including fitting, calibration, and search), or by giving FSPO the same total budget as PB2? Without this, it is difficult to interpret the reported gap.

  3. The evaluation is limited to a structural-repair task with a schema verifier. Since FSPO is presented as a general controller for GRPO post-training, could the authors evaluate on widely used verifiable-reward benchmarks such as GSM8K, MATH-500, or IFEval? Performance on standard reasoning or instruction-following benchmarks would provide a more recognizable measure of whether the controller improves the trained policy's general capability rather than only the in-domain repair metric.

Reviewer_zEqROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The main point of the paper is to use the full telemetry from an actual RL post-training run to improve the decision making on the parameters inherent in the run. This is a neat idea and it deserves attention. Kudos to the authors for picking this up.

The paper demonstrates that their algorithm works better on a small problem (a few runs using ~10 GPU hours each) on Qwen 2.5 3B. It relies on a formula with magic coefficients (1). The formula itself isn't derived in a way a human can understand. As such, it doesn't advance the state of the art.

Critical strengths
  1. The paper picks up on the problem that post training depends on many parameters and thus their control deserves a more principled treatment.
  2. The paper shows in one experiment that it works.
Critical weaknesses
  1. The paper is unreadable to humans. Here's an example:

"On the shared 24-case, eight-interval contract, the rule reaches 2.8283 progress units per ledger unit, the fixed controller 2.3440, the η-aligned contextual bandit 1.6744, and uniform allocation 1.4340; rule and controller record zero violations, the bandit 0.0781, and uniform 0.1615 (Appendix D.7)."

  1. A proper technical derivation is absent, in main body and appendix. For instance:

"In the current fixture, κ(at) changes both the accepted-progress penalty and the stability margin: g(st,at) = max{Kt + 0.5 clip(at)−0.08κ(at)−0.26, 0.18−Rt }."

Why 0.08, why 0.26, why 0.18 and will any one of these numbers generalize beyond the single dataset you picked?

  1. Weak experiments (tiny model, few GPUs)
Questions for authors

I would ask you to please use more skill at using generative models for paper writing. There's nothing inherently wrong in doing so (and you can produce great quality this way, when properly used as a tool). I disagree with your claim that AI only helped in the writing. The entire setup and technical content feels thoroughly AI generated and you should have the intellectual honesty of admitting it (again, nothing wrong with it, just please do not claim things that are plainly not the case).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.