- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares delivering one long PPO-trained text-game agent with validation selection among two or four independently trained shorter candidates, holding the supervised start and PPO recipe fixed within each model instance. A calibrated reference budget charges training, validation, and operational costs and determines batch counts in advance. The study uses Qwen2.5-1.5B and SmolLM2-1.7B, six paired blocks, separate validation and independent pools, and a reserved-pool check. Every fixed-split candidate is independently evaluated after the validation choice is sealed. Four-way splitting underperforms the long run by 11.29 and 13.00 percentage points on the independent pool, and even each set's empirical best remains below its paired long run. Two-stem successive halving recovers much of the deficit, while doubling the budget improves the relative performance of two-way splitting without producing a robust universal reversal.
Critical strengths
-
Evaluating every fixed-split candidate after sealing the validation choice is the strongest part of the design. It distinguishes the observed candidate-set deficit from the empirical selection shortfall rather than attributing every delivered gap to a poor validation pick. The four-candidate empirical maxima remain below the paired long runs in all twelve model-instance/block comparisons, and the held-out delivery comparison supports the same direction. This gives a useful answer for the tested setting that a winner-only evaluation could not provide.
-
The paper makes its accounting and uncertainty boundaries unusually explicit. The fixed allocations, independent evaluation, paired game-cluster intervals, block-level scatter, incident disclosures, and deletion checks let the reader distinguish observed delivery performance from a claim about all future PPO campaigns. The adaptive arm and doubled-budget comparisons also qualify the result constructively: the conclusion concerns an allocation policy at a budget, not a universal rule that more starts are harmful.
Critical weaknesses
-
The practical conclusion is closely tied to this pipeline's operational costs and eligible checkpoints. At B, four-way splitting buys only 4×40 versus 270 total batches in M1 and 4×33 versus 242 in M2 (Table 1/Appendix A). Thus candidate breadth changes both training depth and total training purchased, while each arm uses a horizon-specific cosine schedule and final-checkpoint-only selection. This is a valid operational comparison, and the paper acknowledges the confounding, but it does not show how portable the result is to lower-overhead delivery or a different checkpoint-selection policy. The useful contribution is therefore narrower than a general recommendation about PPO budget allocation. An overhead sensitivity map tied to the observed learning curves would make the decision rule more actionable.
-
The operational delivery comparison conditions on protocol completion. Appendix I.7 says three completed training attempts were replaced after partial validation because of model-output protocol failures; their weights were discarded, and the retries followed different trajectories. The incident-block exclusion supports the main M1 four-way deficit, so this is not evidence that the reported sign is an artifact. However, a deployment policy has to pay for retries or assign outcomes to failed deliveries. Recording those costs in a separate physical ledger does not incorporate them into the fixed reference delivery budget. This limits the interpretation as an end-to-end budget comparison, even though the paper is transparent about the filter.
Questions for authors
-
Over what range of validation/startup/save/delivery overheads would the qualitative recommendation be expected to persist? Please distinguish an arithmetic reallocation from a performance claim on batch counts that were not trained. Is there evidence near the equal-total-training or low-overhead regime that makes the result useful beyond this pipeline?
-
For an operational policy that charges retries inside B and treats a protocol failure as a delivery outcome, what conclusions remain supported? I appreciate the incident-block exclusions in Appendix G; a clear statement of the intended completion-conditioned target versus attempted delivery would address part of this concern.
-
Does the S3 advantage remain when compared with a continuation control of the same delivered horizon and schedule but without validation-based promotion? S3 trains two stems and continues one, so its gain over four-way fixed splitting does not isolate the value of adaptive screening. This would affect the interpretation of the adaptive-arm result, while the main four-way delivery finding can stand as a narrower empirical result.
4 comments
Dear Reviewer VK85,
Thank you again for your thoughtful review and for engaging with the revision. We wanted to briefly check whether the new results, the no-promotion continuation control for S3 (Section 4.3) and the failures-as-lost rescoring (Appendix I.7), address your remaining questions, and whether any point would benefit from further clarification before the discussion closes.
Response to authors
Thank you for the detailed response and the new experiments.
Q1: The 1.5B control is helpful. It shows that the four-way deficit is clear in M2 but not detected in M1 at this budget. I agree with stating the main result at the reference budget and noting in Limitations that selection among intermediate checkpoints is untested.
Q2: Scoring the failed runs as lost makes the M1 four-way deficit larger (−11.29 to −12.42 pp). This resolves my concern that conditioning on protocol completion might favor the long run.
Q3: The no-promotion control is a good addition. It shows that most of S3's gain over four-way splitting comes from continued training, and validation promotion adds a smaller gain (1.90 and 1.60 pp). I appreciate that the abstract now says this directly.
My concerns are mostly addressed. The conclusion is still tied to this pipeline and budget, but the paper now states this clearly. I will raise my score to 4.
Dear Reviewer VK85,
Thank you for engaging with the new experiments and for raising your score. In the final version we will keep the scope of the conclusion explicit in the abstract, the conclusion and Limitations, together with the untested setting of selection among intermediate checkpoints. Your questions on the completion-conditioned target and on separating promotion from continued training clearly strengthened the paper, and we are grateful for your careful reading.
Response to Reviewer VK85
Dear Reviewer VK85,
Thank you for your careful review and for recognizing the sealed, evaluate-every-candidate design and the explicit accounting and uncertainty boundaries. Your concerns were the portability of the result to other overheads, the completion-conditioned target, and whether S3's advantage depends on validation promotion. We answer them with the accounting sensitivity and a new intermediate budget, a failures-as-lost rescoring, and a no-promotion control registered before any of its runs.
Response. The trained evidence nearest to the equal-training and low-overhead regimes still favours the long run, except M1 at 1.5B, where no four-way deficit was detected (Table 19, p. 37). Arithmetically (Table 17, p. 35), a four-way candidate receives 13–17% of the long run's batches across margins of 1.0 to 1.5 with or without F, and 25% at zero overhead; no run was trained at these counts. Near equal total training, the parent short candidates trained 88 and 81 batches, more than an equal-training split would give (67.5 and 60.5), yet the expected best of four of them, an optimistic pooled figure, is 83.01% and 55.38% against long-run means of 86.29% and 61.13% (Appendix K, p. 36). The 1.5B control gives the split arm more total training than the B long run (340 versus 270 batches in M1, 296 versus 242 in M2), and its four-way delivery is then −7.42 pp in M2 and −1.67 pp in M1, where the interval covers zero. The abstract and conclusion now state the result at the reference budget, and Limitations notes that selection among intermediate checkpoints remains untested.
Response. Scoring the failures as delivery outcomes or charging retries inside B both move the comparison further against splitting (Appendix I.7, p. 33). The intended target is delivery conditional on protocol completion, as Section 4.5 (p. 10) states, and all three original failures occurred in split arms. Scoring each re-run four-way candidate as lost, so that its group delivers the validation winner among the other three, enlarges M1's four-way deficit from −11.29 to −12.42 pp; S3's delivery is unchanged because the retried stem was not promoted, and the reserved pool cannot be rescored because the substitute candidate was not evaluated there. Charging the retries inside B would only remove training from the split arms, and the three re-runs of the review-period controls, all with identical input bindings, are listed in Appendix K (p. 38). The attempted-delivery reading you asked for therefore points in the same direction as the completion-conditioned target.
Response. The new control shows that most of S3's advantage over fixed splitting comes from continued training, while validation-based promotion adds a smaller, positive increment (Section 4.3, p. 8; Table 20, p. 39). In every block, the stem that validation did not promote is continued to the same horizon and schedule as S3's continuation, with that stem's seeds; without promotion either stem would be continued with equal chance, so the control is the mean of the two continuations. This control is still 8.48 pp (M1) and 7.73 pp (M2) above k = 4, and 2.81 and 5.27 pp below the long run. Promotion adds a further 1.90 and 1.60 pp, as the promoted continuation beats the unpromoted one by 3.79 and 3.21 pp, in five of six blocks per instance. The revised abstract in the PDF now attributes the adaptive-arm gain mostly to continued training rather than the validation choice, which sharpens how that result should be read.
We hope these new results address your concerns and would appreciate your reassessment in light of the revised evidence. We would be happy to clarify any remaining questions.