acceptodds
Under review as a conference paper at ICLR 2027

Budget Allocation for PPO Text-Game Agents: Long Runs, Candidate Selection, and Successive Halving

Abstract

As reinforcement-learning post-training becomes a routine stage in producing language-model agents, one allocation decision recurs in every training campaign: with a fixed budget for delivering one model, should training go into one long run, or into several shorter runs from the same checkpoint followed by validation selection? Training several candidates creates the chance to select a better model, but under a shared budget each candidate receives less training, so the policies should be compared on the models they deliver. We compare a long run with two- and four-candidate allocations for Qwen2.5-1.5B and SmolLM2-1.7B under one PPO text-game recipe, with every comparison paired by block. A calibrated reference budget charges training together with the operational costs of producing and selecting a model, so all policies are compared at a common accounted budget rather than at equal elapsed time. Selection is sealed before independent evaluation, and every candidate of the two- and four-way splits is evaluated, so selection loss can be measured separately from the delivered gap. At the reference budget, splitting into two or four candidates reduces delivered success on the independent pool in all but one block-level comparison, and the registered reserved-pool check confirms the four-candidate deficit in both instances. On the independent pool, even the empirical best candidate of each four-candidate set falls below its paired long run in every block, so a different pick from those sets would not close the observed gap. With four candidates, the mean deficit exceeds ten percentage points in both instances. One successive-halving allocation improves substantially on four-way fixed splitting in both instances and on both pools. Doubling the budget improves the two-candidate strategy's relative performance in both instances. Its final ordering against the long run nevertheless differs, and the positive contrast in the first instance is sensitive to single-block deletion. These results identify a setting where selecting among more trained candidates fails to offset dividing training, while showing that the comparison depends on the allocation policy, budget, and model instance.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, language model agents, training budget allocation, policy selection, successive halving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VK85OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares delivering one long PPO-trained text-game agent with validation selection among two or four independently trained shorter candidates, holding the supervised start and PPO recipe fixed within each model instance. A calibrated reference budget charges training, validation, and operational costs and determines batch counts in advance. The study uses Qwen2.5-1.5B and SmolLM2-1.7B, six paired blocks, separate validation and independent pools, and a reserved-pool check. Every fixed-split candidate is independently evaluated after the validation choice is sealed. Four-way splitting underperforms the long run by 11.29 and 13.00 percentage points on the independent pool, and even each set's empirical best remains below its paired long run. Two-stem successive halving recovers much of the deficit, while doubling the budget improves the relative performance of two-way splitting without producing a robust universal reversal.

Critical strengths
  1. Evaluating every fixed-split candidate after sealing the validation choice is the strongest part of the design. It distinguishes the observed candidate-set deficit from the empirical selection shortfall rather than attributing every delivered gap to a poor validation pick. The four-candidate empirical maxima remain below the paired long runs in all twelve model-instance/block comparisons, and the held-out delivery comparison supports the same direction. This gives a useful answer for the tested setting that a winner-only evaluation could not provide.

  2. The paper makes its accounting and uncertainty boundaries unusually explicit. The fixed allocations, independent evaluation, paired game-cluster intervals, block-level scatter, incident disclosures, and deletion checks let the reader distinguish observed delivery performance from a claim about all future PPO campaigns. The adaptive arm and doubled-budget comparisons also qualify the result constructively: the conclusion concerns an allocation policy at a budget, not a universal rule that more starts are harmful.

Critical weaknesses
  1. The practical conclusion is closely tied to this pipeline's operational costs and eligible checkpoints. At B, four-way splitting buys only 4×40 versus 270 total batches in M1 and 4×33 versus 242 in M2 (Table 1/Appendix A). Thus candidate breadth changes both training depth and total training purchased, while each arm uses a horizon-specific cosine schedule and final-checkpoint-only selection. This is a valid operational comparison, and the paper acknowledges the confounding, but it does not show how portable the result is to lower-overhead delivery or a different checkpoint-selection policy. The useful contribution is therefore narrower than a general recommendation about PPO budget allocation. An overhead sensitivity map tied to the observed learning curves would make the decision rule more actionable.

  2. The operational delivery comparison conditions on protocol completion. Appendix I.7 says three completed training attempts were replaced after partial validation because of model-output protocol failures; their weights were discarded, and the retries followed different trajectories. The incident-block exclusion supports the main M1 four-way deficit, so this is not evidence that the reported sign is an artifact. However, a deployment policy has to pay for retries or assign outcomes to failed deliveries. Recording those costs in a separate physical ledger does not incorporate them into the fixed reference delivery budget. This limits the interpretation as an end-to-end budget comparison, even though the paper is transparent about the filter.

Questions for authors
  1. Over what range of validation/startup/save/delivery overheads would the qualitative recommendation be expected to persist? Please distinguish an arithmetic reallocation from a performance claim on batch counts that were not trained. Is there evidence near the equal-total-training or low-overhead regime that makes the result useful beyond this pipeline?

  2. For an operational policy that charges retries inside B and treats a protocol failure as a delivery outcome, what conclusions remain supported? I appreciate the incident-block exclusions in Appendix G; a clear statement of the intended completion-conditioned target versus attempted delivery would address part of this concern.

  3. Does the S3 advantage remain when compared with a continuation control of the same delivered horizon and schedule but without validation-based promotion? S3 trains two stems and continues one, so its gain over four-way fixed splitting does not isolate the value of adaptive screening. This would affect the interpretation of the adaptive-arm result, while the main four-way delivery finding can stand as a narrower empirical result.

4 comments
AuthorsOpenReview ↗

Response to Reviewer VK85

Dear Reviewer VK85,

Thank you for your careful review and for recognizing the sealed, evaluate-every-candidate design and the explicit accounting and uncertainty boundaries. Your concerns were the portability of the result to other overheads, the completion-conditioned target, and whether S3's advantage depends on validation promotion. We answer them with the accounting sensitivity and a new intermediate budget, a failures-as-lost rescoring, and a no-promotion control registered before any of its runs.

Q1: Portability to other overheads and checkpoint-selection policies (W1 and Q1).

Response. The trained evidence nearest to the equal-training and low-overhead regimes still favours the long run, except M1 at 1.5B, where no four-way deficit was detected (Table 19, p. 37). Arithmetically (Table 17, p. 35), a four-way candidate receives 13–17% of the long run's batches across margins of 1.0 to 1.5 with or without F, and 25% at zero overhead; no run was trained at these counts. Near equal total training, the parent short candidates trained 88 and 81 batches, more than an equal-training split would give (67.5 and 60.5), yet the expected best of four of them, an optimistic pooled figure, is 83.01% and 55.38% against long-run means of 86.29% and 61.13% (Appendix K, p. 36). The 1.5B control gives the split arm more total training than the B long run (340 versus 270 batches in M1, 296 versus 242 in M2), and its four-way delivery is then −7.42 pp in M2 and −1.67 pp in M1, where the interval covers zero. The abstract and conclusion now state the result at the reference budget, and Limitations notes that selection among intermediate checkpoints remains untested.

Q2: Completion-conditioned target and failures as delivery outcomes (W2 and Q2).

Response. Scoring the failures as delivery outcomes or charging retries inside B both move the comparison further against splitting (Appendix I.7, p. 33). The intended target is delivery conditional on protocol completion, as Section 4.5 (p. 10) states, and all three original failures occurred in split arms. Scoring each re-run four-way candidate as lost, so that its group delivers the validation winner among the other three, enlarges M1's four-way deficit from −11.29 to −12.42 pp; S3's delivery is unchanged because the retried stem was not promoted, and the reserved pool cannot be rescored because the substitute candidate was not evaluated there. Charging the retries inside B would only remove training from the split arms, and the three re-runs of the review-period controls, all with identical input bindings, are listed in Appendix K (p. 38). The attempted-delivery reading you asked for therefore points in the same direction as the completion-conditioned target.

Q3: S3 against a continuation control without validation-based promotion (Q3).

Response. The new control shows that most of S3's advantage over fixed splitting comes from continued training, while validation-based promotion adds a smaller, positive increment (Section 4.3, p. 8; Table 20, p. 39). In every block, the stem that validation did not promote is continued to the same horizon and schedule as S3's continuation, with that stem's seeds; without promotion either stem would be continued with equal chance, so the control is the mean of the two continuations. This control is still 8.48 pp (M1) and 7.73 pp (M2) above k = 4, and 2.81 and 5.27 pp below the long run. Promotion adds a further 1.90 and 1.60 pp, as the promoted continuation beats the unpromoted one by 3.79 and 3.21 pp, in five of six blocks per instance. The revised abstract in the PDF now attributes the adaptive-arm gain mostly to continued training rather than the validation choice, which sharpens how that result should be read.

We hope these new results address your concerns and would appreciate your reassessment in light of the revised evidence. We would be happy to clarify any remaining questions.

Dear Reviewer VK85,

Thank you again for your thoughtful review and for engaging with the revision. We wanted to briefly check whether the new results, the no-promotion continuation control for S3 (Section 4.3) and the failures-as-lost rescoring (Appendix I.7), address your remaining questions, and whether any point would benefit from further clarification before the discussion closes.

Reviewer_VK85OpenReview ↗

Response to authors

Thank you for the detailed response and the new experiments.

Q1: The 1.5B control is helpful. It shows that the four-way deficit is clear in M2 but not detected in M1 at this budget. I agree with stating the main result at the reference budget and noting in Limitations that selection among intermediate checkpoints is untested.

Q2: Scoring the failed runs as lost makes the M1 four-way deficit larger (−11.29 to −12.42 pp). This resolves my concern that conditioning on protocol completion might favor the long run.

Q3: The no-promotion control is a good addition. It shows that most of S3's gain over four-way splitting comes from continued training, and validation promotion adds a smaller gain (1.90 and 1.60 pp). I appreciate that the abstract now says this directly.

My concerns are mostly addressed. The conclusion is still tied to this pipeline and budget, but the paper now states this clearly. I will raise my score to 4.

Dear Reviewer VK85,

Thank you for engaging with the new experiments and for raising your score. In the final version we will keep the scope of the conclusion explicit in the abstract, the conclusion and Limitations, together with the untested setting of selection among intermediate checkpoints. Your questions on the completion-conditioned target and on separating promotion from continued training clearly strengthened the paper, and we are grateful for your careful reading.

Reviewer_Am5WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a practical allocation question in RL post-training of language-model agents: under a fixed delivery budget that charges training, validation, and operational costs, is it better to train one long PPO run or to train several shorter independent candidates and select one by validation? The authors compare a long run (k=1) against fixed splitting (k=2,4) and a successive-halving arm on two LLMs (Qwen2.5-1.5B, SmolLM2-1.7B) playing TextWorld. The main finding is that one long run beats four-way validation selection by more than 10 percentage points, that even each set's empirical best stays below the paired long run (so better selection alone would not close the gap), and that continuing a selected stem or doubling the budget partially mitigates the deficit.

Critical strengths
  1. The question is well-motivated, concrete, and genuinely useful to practitioners who must decide how to spend a training budget on LM agents.
  2. The "sealing + evaluate every candidate" protocol looks a useful methodological contribution.
  3. Reporting is notably honest: limitations (one game family, one recipe, two instances, six blocks), the optimistic nature of the same-pool maximum, protocol failures and retries, and the fact that an interval covering zero is not evidence of equivalence are all stated plainly.
Critical weaknesses
  1. Reference budget B=3600s is small and its choice drives the k=4 result. At B, k=4 gets only 40 (M1) / 33 (M2) batches, barely above the registered 16-batch floor; candidates are severely under-trained, so the k=4 loss is close to a floor effect rather than an interesting allocation finding.
  2. Training length, schedule, and stochastic trajectory are confounded. Fixed-splitting candidates use a cosine schedule spanning their own shorter batch count, while the long run uses a schedule spanning the full count, so the comparison is not "same per-candidate recipe, different depth" -- the schedule shape differs too. The paper does not identify whether the deficit comes from fewer updates, a differently shaped schedule, or seed variance at short horizons. A schedule-matched control (same schedule shape, truncated) would materially strengthen the mechanism claim in Section 4.2.
  3. The operational-cost accounting is heavily calibrated and somewhat arbitrary. u and c1 are set at 1.25x measured duration as a "margin fixed before use," and several components (I, S, R, d, F) are calibrated from 16 training batches. The resulting batch counts (Table 3) are sensitive to these choices. Sensitivity is shown only for accounting A vs B, not for perturbations of the 1.25x margin or of F.
Questions for authors
  1. Can you report a schedule-matched control, where short candidates use a schedule shaped like the long run's and are simply truncated? This would separate "fewer updates" from "different schedule shape" in the k=4 deficit.
  2. Can you run at least one intermediate budget (e.g., 1.5B) or a budget sweep for k=2/k=4, to show the conclusion is not tied to B=3600s and 2B?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Am5W

Dear Reviewer Am5W,

Thank you for your careful review and for recognizing the practical value of the question, the sealing and evaluate-every-candidate protocol, and the honest reporting. Your concerns were the small reference budget, the confound between candidate count and schedule, and the calibrated accounting. You asked for a schedule-matched control and an intermediate budget; we registered both before any of their runs and report them below.

Q1: Budget size and per-candidate training at k = 4 (W1 and Q2).

Response. The new intermediate budget shows that the four-way deficit persists in M2, while in M1 it was not detected at 1.5B (Section 4.4 and Figure 4(c), p. 9). In M2 the deficits at B, 1.5B and 2B are −13.00, −9.54 and −10.08 pp, with all six blocks negative at 1.5B; M2's four-way candidates at B were indeed close to the supervised start (47.16% vs 45.75%), but the deficit persists with 74 and 114 batches per candidate at 1.5B and 2B. In M1 the four-way difference at 1.5B is −0.67 pp [−4.00, +2.46]: its long runs did not improve with budget (86.29%, 85.29% and 85.25% at B, 1.5B and 2B), while four-way delivery gained 9.63 pp. M1's four-way candidates at B were also not at the floor, averaging 65.36% on the independent pool, 23.4 pp above the supervised start (Table 19, p. 37). Your suggestion thus sharpened the paper's claim: the abstract and conclusion now state the main result at the reference budget, and Limitations notes that the intermediate budget covers only the long run and four-way splitting.

Q2: Schedule-matched control (W2 and Q1).

Response. The schedule-matched control you proposed explains part of the four-way deficit, but delivery remains below the long run in both instances (Section 4.2, p. 8; Table 20, p. 39). Four candidates per block train the k = 4 batch counts on the long run's schedule horizon, simply truncated, with the seeds of the paired k = 4 candidates. They improve on k = 4 by 5.17 pp (M1) and 2.38 pp (M2), about half of M1's deficit and a fifth of M2's, but stay 6.13 and 10.63 pp below the long run, and the best of the four is still below it in three (M1) and six (M2) blocks. Seed variance does not close the gap either: even the best of four k = 4 candidates is below the long run in every block (Figure 3, p. 7), and at fixed seeds the long schedule raises each candidate by 7.40 and 2.66 pp on average. Section 4.2 keeps its claim a localisation of the deficit in the candidate sets, as the submitted version stated ("without isolating its cause"), now with the schedule's share measured.

Q3: Sensitivity of the batch counts to the calibrated accounting (W3).

Response. Perturbing the margin and F changes the batch counts, but a four-way candidate still receives only 13–17% of the long run's batches (Table 17 in Appendix K, p. 35). Table 17 recomputes the counts for margins from 1.0 to 1.5 on u and c1, with and without F; even with all overheads removed, the share is only 25%. These are arithmetic reallocations rather than trained results, and the margin mainly rescales the budget, a direction the trained 1.5B and 2B units cover. Measured save and validation times also exceeded the calibrated ones (Appendix K, p. 35), so the registered accounting is, if anything, generous to splitting. The relative allocation behind the main comparison therefore does not hinge on the particular margin or recovery charge.

We hope these new results address your concerns and would appreciate your reassessment in light of the revised evidence. We would be happy to clarify any remaining questions.

Dear Reviewer Am5W,

Thank you again for your suggestions. Following them, we ran both the schedule-matched control and the intermediate 1.5B budget you proposed (Sections 4.2 and 4.4, Table 20). As the discussion period is nearing its end, we would greatly appreciate hearing whether these results address your concerns, and we are happy to answer any further questions.

Reviewer_PGLbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a fixed compute budget for PPO text-game agents is better spent on one long run or on several shorter runs followed by validation-based selection. Its cost accounting covers training, validation, and operational overhead, and every candidate is evaluated independently to separate deficits in the candidate set from failures of selection. On Qwen2.5-1.5B and SmolLM2-1.7B, four-way splitting trails one long run by more than ten percentage points, even when the best candidate is chosen in hindsight. Continuing to train the selected candidate recovers much of this gap, and doubling the budget makes two-way splitting more competitive.

Critical strengths
  1. The paper fixes the validation choice and then evaluates every candidate independently, which directly tests whether better selection could close the gap and is more informative than reporting only the selected model.

  2. The central finding is robust, as the four-way splitting deficit holds across both models, the reserved evaluation pool, and every single-block deletion.

Critical weaknesses
  1. The comparison relies on a calibrated reference budget, but Appendix B does not report the actual wall-clock time and cost of each strategy, including the reused long runs, so the ranking under equal actual expenditure remains unverified.

  2. The number of candidates is confounded with each candidate's training horizon and learning-rate schedule, so the experiments cannot tell whether the four-way deficit stems from shorter training, the schedule, or candidate splitting itself. ​

Questions for authors
  1. Could the authors report the actual training, validation, and delivery costs of each strategy?

  2. How much support remains for M1's two-candidate advantage at twice the budget, and for M2's budget interaction, once uncertainty across training blocks is taken into account?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PGLb

Dear Reviewer PGLb,

Thank you for your careful review and for recognizing that evaluating every candidate after the sealed choice directly tests whether better selection could close the gap. Your concerns were the measured cost of each strategy, the confound between candidate count and learning-rate schedule, and block-level support for the 2B findings. The revised PDF answers them with a measured cost table, a schedule-matched control registered before any of its runs, and block-level intervals.

Q1: Measured wall-clock time and cost of each strategy (W1 and Q1).

Response. Measured costs show that the reference budget does not favour the long run; if anything, it is generous to splitting (Table 18 in Appendix K, p. 36). Table 18 gives, for every policy, the measured time and USD of all candidates of one delivery, including the reused long runs at B, whose ratio is omitted as not comparable because they ran on faster Pods and are timed by their training command. Measured time exceeded the reference share by 3–13% for the long runs at 2B but by 27–60% for four-way splitting, because per-run save and validation took longer than calibrated and are paid by every candidate, and short runs trained more slowly per batch. At 2B, where all policies ran on the same Pods, the split policies took 0.99–1.17 times the long run's time, so equalizing actual expenditure would move budget toward the long run, except for M1's two-way split (0.99 times); we did not train at such expenditure-equalized counts. Appendix B's aggregate cost now comes with a per-policy ledger, so the accounting rule can be compared with actual spending for every arm.

Q2: Training length, learning-rate schedule and candidate splitting (W2).

Response. A schedule-matched control shows that the compressed schedule explains part of the four-way deficit, while the rest remains with candidates that train for fewer batches (Section 4.2, p. 8; Table 20, p. 39). Four candidates per block train the k = 4 batch counts with the paired k = 4 seeds, but on the long run's schedule horizon, so the schedule span is the only design difference (Appendix K, p. 36). They improve on k = 4 by 5.17 pp (M1) and 2.38 pp (M2), about half of M1's deficit and a fifth of M2's, but stay 6.13 and 10.63 pp below the long run; in M1 the remaining gap is uneven across blocks (three below, two tied, one above). Starting several candidates is not by itself what lowers delivery: S3 also trains two stems but continues one, and recovers much of the deficit (+10.38 and +9.33 pp over k = 4, Section 4.3). The confound you identified is now a measured component of the deficit rather than only an acknowledged limitation.

Q3: Block-level support for the 2B findings (Q2).

Response. Neither secondary 2B finding is supported once training-block variation is taken into account, and the revision says so (Appendix I.4, p. 32). The six-block t intervals are [−0.39, +5.14] for M1's two-way minus long run at 2B, and [−1.07, +9.66] and [−2.18, +10.18] for M2's k = 2 budget interaction on the independent and reserved pools. The main claim rests instead on the four-way deficit at B, which is negative in all six blocks of both instances on the independent pool, with t intervals [−16.82, −5.76] and [−16.42, −9.58] (Table 21, p. 40). Table 16 (p. 27) lists the block spread of every headline contrast on both pools; on the reserved pool, M1's four-way deficit is negative in five of six blocks. The revised abstract in the PDF describes the 2B result only as larger budgets narrowing the split-long gap to different degrees, which separates the block-robust main result from the secondary 2B findings.

We hope these new results address your concerns and would appreciate your reassessment in light of the revised evidence. We would be happy to clarify any remaining questions.

Dear Reviewer PGLb,

Thank you again for your review. As the discussion period is nearing its end, we wanted to kindly check whether our response addressed your concerns. In particular, the revision adds the measured per-policy cost you requested (Table 18), a schedule-matched control registered before any of its runs (Section 4.2), and block-level intervals for the 2B findings (Appendix I.4). If any concern remains, we would be glad to clarify before the discussion closes.

Reviewer_PGLbOpenReview ↗

Response to the Authors' Rebuttal

Thank you for your rebuttal and for addressing my concerns. After considering your responses, I have decided to maintain my original score.

Dear Reviewer PGLb,

Thank you for considering our response and for your constructive feedback. Your comments on measured costs and the schedule confound helped us strengthen the paper through the additional cost analysis and schedule-matched control.

If you think any aspect of these additions would benefit from clearer presentation in the final version, we would appreciate your suggestion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.