acceptodds
Under review as a conference paper at ICLR 2027

Sharpening Tax in Post-Training

12 views

Abstract

An emerging hypothesis about reinforcement learning (RL) post-training of large language models (LLMs) is that it merely sharpens existing behaviors of a base model, improving single-shot accuracy at the cost of solution coverage. Although this trade-off has been observed in math and coding tasks, it need not extend to agentic tasks, where multi-turn tool use and interaction may require capabilities newly acquired during post-training. Our surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents. Despite far lower accuracy (pass@), they often surpass their post-trained counterparts in solution coverage (pass@) given a sufficient test-time budget. We further analyze the underlying mechanism and show that post-training pushes tasks toward two extremes, always solved or never solved, and thereby improves sampling efficiency and consistency at the cost of solution coverage. To measure this cost, we propose Sharpening Tax, a diagnostic metric that quantifies the loss in test-time scalability after post-training. Across 14 base/post-trained model pairs from four families and three agentic benchmarks (42 cases in total), the tax is prevalent in most settings, can be estimated from a few rollouts, and correlates well with other metrics. Finally, we present posterior-tempered group sampling (PTGS), a simple plug-and-play Bayesian sampler that adapts the sampling temperature per prompt to its estimated difficulty. Applied during RL training in two agentic environments, PTGS pays a smaller tax than the fixed-temperature baseline, solving more tasks under repeated sampling while also improving single-shot accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, post-training, reinforcement learning, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CgeCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends the study of the sharpening phenomenon from coding and mathematical reasoning to agentic tasks. The authors introduce sharpening tax, a metric intended to quantify the difference in test-time scalability between base and post-trained LLMs. They further propose Posterior-Tempered Group Sampling (PTGS), which dynamically adjusts the sampling temperature based on task difficulty. The paper addresses an important problem, and the proposed method appears effective in the reported experiments. However, the presentation needs substantial improvement, and several concepts and results require clarification.

Critical strengths
  1. The paper investigates a novel and important problem.
  2. The proposed PTGS method demonstrates promising empirical effectiveness.
Critical weaknesses
  1. The manuscript does not consistently adhere to the conventions of formal academic writing. Frequent colloquial expressions and awkward phrasing reduce its clarity and readability. Substantial language revision would improve the presentation.
  2. The manuscript would benefit from a more conventional and clearly structured organization. The related work discussion appears only in the appendix, making it difficult to contextualize the contributions when reading the main text. Moreover, the “Preliminaries” section primarily describes the experimental setup rather than providing the background concepts and notation needed to understand the methodology. Much of this material would be better placed in the methodology or experimental setup sections.
  3. The paper attempts to convey several distinct messages without clearly establishing how they contribute to its central argument. This makes the narrative feel fragmented. For example, the connection between the analyses in Figures 2 and 3 and the central concepts of sharpening tax and PTGS is unclear. The paper would benefit from a more coherent narrative that explicitly links these analyses to its main contributions.
Questions for authors
  1. I understand the mathematical definition of A(K) and find it reasonable. However, the motivation for S(K) is less clear. What is its intuitive or operational interpretation? Furthermore, why is it appropriate to quantify sharpening tax using the difference in A(K) or S(K) between a base model and its post-trained counterpart? Given that pass@1 and pass@K vary across models and tasks, it is unclear why these quantities can be compared directly in this manner. In particular, a positive tax does not necessarily seem to imply a loss of test-time scalability. Please clarify the interpretation of this metric and the conditions under which it supports such a conclusion.
  2. Could the authors clarify the conceptual relationship between the proposed diagnostic metric and PTGS? The current presentation does not clearly explain how the sharpening-tax analysis motivates the design of the method. As I understand it, PTGS increases the sampling temperature for difficult tasks to encourage exploration. How does this mechanism specifically address the behavior captured by sharpening tax?
  3. In Figure 1, PTGS decreases pass@1 while increasing pass@K, whereas in Table 2, it improves both. Could the authors explain what accounts for these different effects on pass@1 and clarify whether they arise from differences in the experimental settings?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CgeC (1/3)

Dear Reviewer CgeC, we are really grateful to you for taking the time to provide this careful and constructive review, which was crucial to improving our manuscript. We respond to each weakness (W) and question (Q) below. Please also refer to the revised PDF, which reflects your comments and marks some major edits in red.


W1, W2: Related work appears only in the appendix; "Preliminaries" describes the experimental setup, not the background and notation. Frequent colloquial expressions and awkward phrasing reduce its clarity and readability.

We agree and reorganized the paper into the conventional structure that you describe. First of all, we brought a compact Related Work section into the main text. We also changed the section name "Preliminaries" (Sec. 2) to "Preliminary Experiment Setup" so that the section title matches its content. We also remakrably polished the main text of draft to rewrite colloquial expressions and awkward phrasing in a more formal style. If you have additional suggestions for improving the writing, please point out the specific items. We are happy to reflect them as well.

Q2, W3: How do Figs. 2–3 connect to the sharpening tax and PTGS? How does the sharpening-tax analysis motivate PTGS, and how does heating difficult tasks address the behavior that the tax captures?; The paper would benefit from a more coherent narrative that explicitly links these analyses to its main contributions.

Thanks for pointing this out; we understand your concern. The paper flows through three parts: (1) core observations (Figs. 2-3) and deeper analyses of the underlying mechanism, distribution sharpening (Figs. 4-5), (2) a metric to quantify this sharpening effect in a single number (Sec. 4), and (3) a method to mitigate this tax, inspired by our analyses in Figs. 4-5 (Sec. 5). Specifically,

  • Section 3: We start by observing a surprising phenomenon in Figures 2 and 3: the base model's test-time scaling curve keeps rising while the post-trained one flattens quickly, so the base model eventually catches up with the post-trained one. Then, we go deeper by analyzing the underlying mechanism of this phenomenon, distribution sharpening, in Figures 4 and 5.
  • Section 4: Since conducting this type of qualitative analysis for every case is expensive and not scalable, we wanted to provide a metric that quantifies the effect of post-training on test-time scalability, which connects naturally to our derivation of the Sharpening Tax.
  • Section 5: The observations from Section 3 (Figs. 2, 3) and Section 4 (Fig. 6), which show that post-training improves accuracy at the expense of test-time scalability due to distribution sharpening (Figs. 4, 5), motivated us to counteract this sharpening effect via the per-task adaptive smoothing of PTGS. It heats difficult prompts to incentivize exploratory behaviors, so that they get at least one successful experience during group sampling. Without this adaptive smoothing, those difficult prompts are largely abandoned to the "never-solved zone" (Fig. 4).
    • Our theoretical analysis also characterizes how post-training charges the tax (Theorem 3) and what PTGS could change about this (Theorem 4), connecting the pieces more tightly into a cohesive storyline.

In our original draft, we tried to bridge the sections seamlessly at the beginning of each section (Sec. 4 and Sec. 5). To further strengthen the connections, we not only polished the bridging sentences but also added takeaway blocks at the end of key sections to guide easier reading.

AuthorsOpenReview ↗

Response to Reviewer CgeC (2/3)

Q1: What is the operational meaning of S(K)S(K)? Why can the base–post difference in A(K)A(K) or S(K)S(K) be compared when pass@1 and pass@KK vary across models? Under which conditions does a positive tax imply a loss of test-time scalability?

(1) Operational meaning of S(K)S(K). Given that you already agree with the formulation of A(K)A(K), we want to clarify that the calibrated scalability S(K)S(K) is just a normalized version of A(K)A(K) with respect to the total budget (K−1K-1) and the accuracy headroom (1-pass@11). This measures the average scalability (coverage gain) over all budgets up to K−1K-1, amplified by the value of pass@11. The former operation, i.e., averaging over K−1K-1, allows us to bound the range of the metric to [0,1], whereas the latter operation, i.e., dividing by (1-pass@11), is important for a fair comparison of "effective scalability" across models, i.e., how scalable the model is given its accuracy headroom. Note that the higher the pass@11 of the policy, the less room exists for additional gain from repeated sampling.

(2) Why the base-post difference is comparable. First of all, since A(K)A(K) is just a linear summation over pass@K−pass@k\text{pass}@K - \text{pass}@k, the direct comparison between the A(K)A(K) of base and post is basically the same as the direct comparison of pass@11 or pass@KK between base and post, which is the classic evaluation regime of LLM post-training (Zhao et al. 2025, Yue et al. 2025, Shen et al. 2026). Both policies run on the same tasks with the same budget, the same metric scale, and the same decoding configurations. Therefore, the direct "difference" between base and post across tasks that you mentioned is exactly what we want to measure through our Sharpening Tax formulation.

Meanwhile, ratio-difference-based metrics like our calibrated scalability S(K)S(K) have also been widely adopted in the literature for evaluating diverse models over two quantities simultaneously under a newly proposed criterion, i.e., effective robustness (Taori et al. 2020, Miller et al. 2021, Baek et al. 2022).

(3) When a positive tax implies a loss. By definition, a positive tax always implies a loss of test-time scalability, i.e., the post-trained model gains less from additional rollouts than its base model up to the budget KK. This is our intentional design philosophy, as written in the Abstract and Section 4, rather than a methodological limitation. What a positive tax does not imply by itself is a loss of coverage, i.e., a lower pass@KK at the end-point budget. This distinction is intended, as written in the Abstract and Section 4, and our Corollary 2 makes it explicit: TaxA(K)=K,[ΔC(K)−ΔCˉ(K)]\text{Tax}_A(K)=K\\,\big[\Delta C(K)-\Delta\bar C(K)\big], where the first term denotes the coverage (pass@KK) gap, ΔC(K)=pass@KBase−pass@KPost\Delta C(K)=\text{pass}@K_{\text{Base}}-\text{pass}@K_{\text{Post}}, at a specific end-point budget KK, and the second term, ΔCˉ(K)\Delta\bar C(K), represents the pass@kk difference averaged over all budgets k≤Kk\le K. Since a positive tax means ΔC>ΔCˉ\Delta C>\Delta\bar C, it falls into one of three cases:

CasesInterpretation
ΔC>0,ΔCˉ>0\Delta C>0, \Delta \bar C>0Base achieves higher final coverage and Post gets stuck in a flat scaling curve; additional sampling from Post adds little value.
ΔC>0,ΔCˉ<0\Delta C>0, \Delta \bar C<0Base achieves higher final coverage and Post has a quickly saturating scaling curve; additional sampling from Post adds little value.
ΔC<0,ΔCˉ<0\Delta C<0, \Delta \bar C<0Post achieves higher final coverage, yet its scaling curve saturates quickly, so additional sampling from it adds little value.

Here, we want to clarify that we do not claim that every post-training compromise the coverage of base models. Post-training sharpening either improves pass@K (i.e., ΔC>0\Delta C \gt 0) or not, depending on model size, test-time sampling budget, and dataset. However, the shape of the test-time scaling curve is consistent across model sizes and datasets. Whether post-training ends with a higher coverage or not, its scaling curve always saturates quickly compared to the base model’s scaling curve. That’s why we need a new metric, Sharpening Tax, to quantify this change in the curve's shape.

We revised the Abstract, Introduction, and Section 4 to make these points clear. Thanks again!


Reference

  • Zhao et al. 2025, Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  • Yue et al. 2025, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  • Shen et al. 2026, Understanding Reasoning from Pretraining to Post-Training
  • Taori et al. 2020, Measuring Robustness to Natural Distribution Shifts in Image Classification
  • Miller et al. 2021, Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization
  • Baek et al. 2022, Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
AuthorsOpenReview ↗

Response to Reviewer CgeC (3/3)

Q3: In Fig. 1, PTGS decreases pass@1, whereas in Table 2 it improves both metrics. Do the settings differ?

We would like to clarify that the settings for Fig. 1 and Table 2 are identical and that PTGS actually improves pass@1 in both. In Fig. 1(c), the line style marks the RL algorithm (solid: PPO; dashed: GRPO), and the color shade marks PTGS (lighter: baseline; darker: PTGS). Against the baseline of the same algorithm, PTGS improves pass@1 and pass@128, and it also lowers the tax.

AuthorsOpenReview ↗

Invitation to rebuttal discussion

Dear Reviewer CgeC,

As the discussion period is coming to an end, we wanted to briefly follow up on our responses. Following your suggestions, we:

  1. Substantially revised the writing to remove colloquial expressions and awkward phrasing.
  2. Reorganized the paper by moving a Related Work section into the main text and renaming “Preliminaries” to “Preliminary Experiment Setup”.
  3. Strengthened the links between the analyses, the Sharpening Tax, and PTGS by revising the bridge paragraphs and adding a takeaway block at the end of each core section.
  4. Answered your questions on the operational meaning of S(K) and why base and post-trained models can be compared directly (Q1), and on how the tax motivates PTGS (Q2). For Q3, we clarified that Fig. 1 and Table 2 use the same setup and that PTGS improves pass@1 in both.

We highlighted major edits in red in the revised PDF. If our revisions have addressed your concerns, we would appreciate it if you could consider updating your assessment. Otherwise, we would be happy to provide further clarification!

Thank you again for your time.

Best regards,
 The Authors

Reviewer_qnsTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a metric (Sharpening Tax) to quantify the loss of solution coverage after post training. The proposal is based on an interesting finding that post-training sharpens the policy to improve sampling efficiency and consistency while compromising coverage (pre-trained base models, equipped ith a lightweight tool-use harness, can often still outperform their post-trained counterparts in coverage when given enough attempts). To address this, the authors further propose a training-time sampling method (PTGS) that dynamically changes sampling temperature by drawing success probability estimate from beta distribution.

The main contributions are:

  1. paper provided empirical study of sharpening in agentic tasks, extending previous observations that mainly focused on math and coding tasks.
  2. the proposed Sharpening Tax metric quantifies the loss on test-time scalability after post training, which can be a useful diagnostic tool for test time scaling systems.
  3. the proposed PTGS is effective to help alleviate the tax (demonstrated over Sokoban and FrozenLake)
Critical strengths
  1. comprehensive empirical study of coverage loss in agentic tasks. The paper extends the sharpening discussion beyond math and coding through 14 base/post-trained model pairs across three benchmarks. For example, Gemma-4-31B achieves 87.6% pass@128 on WebShop before post-training but only 56.0% afterward. The harness ablations in table 4 also further strengthen the observation.

  2. the sharpening tax metric is intuitive to understand as its quantity catches the scale of the coverage loss.

  3. the proposed PTGS seems to be effective and simple and demonstrated over 2 tasks (Sokoban and FrozenLake) with both GRPO and PPO.

Critical weaknesses
  1. Limited interpretability of Sharpening Tax as a measure of coverage loss. A positive tax can reflect either fewer tasks being solved within the evaluation budget or improved performance that reaches high coverage with fewer attempts (The authors also explicitly acknowledge this distinction in Section 4 and provide a decomposition in the appendix). This limits the metric’s usefulness as a standalone indicator of coverage loss.

  2. Limited validation of PTGS’s generality. Although the observational study covers multiple models and three benchmarks, PTGS is evaluated only on Qwen2.5-7B-Instruct using Sokoban and FrozenLake that have rather short interaction episodes. These experiments provide an initial demonstration, but leave its effectiveness on more complex tool-use, web-interaction, or reasoning tasks unclear (The authors follow the RAGEN setup without explaining why these two environments provide sufficient validation). Evaluation on richer agentic tasks, with math, and coding as additional domain checks, would help better support the claim.

Questions for authors
  1. has PTGS been applied to math and coding tasks and yielded consistent observation?
  2. Were other models or environments (besides Sokoban and FrozenLake) investigated, and are any additional results available, including unsuccessful cases?
  3. Section 4 acknowledges that positive tax can reflect faster saturation even when performance improves, while the introduction describes the metric as quantifying shrinkage in effective reasoning coverage. How should these statements be reconciled? Which conclusions about coverage loss rely on direct pass@K comparisons, and which rely on the sign of the tax? Clarifying this would help readers to better understand the approach.
5 comments
AuthorsOpenReview ↗

Response to Reviewer qnsT (1/2)

Dear Reviewer qnsT, we really appreciate your positive comments and are happy that you found the empirical study comprehensive, the tax intuitive, and PTGS simple and effective. We respond to each weakness (W) and question (Q) below.


W1, Q3: A positive tax can also reflect faster saturation, so it is limited as a standalone indicator of coverage loss; reconcile Sec. 4 with the introduction; which conclusions rely on direct pass@KK comparisons, and which on the sign of the tax?


Interpretation of Tax

As you pointed out, the tax should not be understood as an indicator of coverage loss, and this is our intentional design philosophy rather than a methodological limitation. We wrote the intended definition in the Abstract and Section 4: the tax quantifies the loss in test-time scalability, i.e., how much less the post-trained model gains from additional rollouts than its base model up to the test-time budget KK. We agree that our initial draft contained some parts that may confuse the interpretation of the Sharpening Tax, so we carefully revised those sentences in the Abstract and Introduction to eliminate the confusion.

Note that our Corollary 2 further provides a decomposition of the tax: TaxA(K)=K,[ΔC(K)−ΔCˉ(K)]\text{Tax}_A(K)=K\\,\big[\Delta C(K)-\Delta\bar C(K)\big], where the first term denotes the coverage (pass@KK) gap, ΔC(K)=pass@KBase−pass@KPost\Delta C(K)=\text{pass}@K_{\text{Base}}-\text{pass}@K_{\text{Post}}, at a specific end-point budget KK, and the second term, ΔCˉ(K)\Delta\bar C(K), represents the pass@kk difference averaged over all budgets k≤Kk\le K, which can capture the overall trend of the scaling curve, not just the end-point measurement. This decomposition allows us to interpret a positive tax from three different angles, as shown below:

CasesInterpretation
ΔC>0,ΔCˉ>0\Delta C>0, \Delta \bar C>0Base achieves higher final coverage and Post gets stuck in a flat scaling curve; additional sampling from Post adds little value.
ΔC>0,ΔCˉ<0\Delta C>0, \Delta \bar C<0Base achieves higher final coverage and Post has a quickly saturating scaling curve; additional sampling from Post adds little value.
ΔC<0,ΔCˉ<0\Delta C<0, \Delta \bar C<0Post achieves higher final coverage, yet its scaling curve saturates quickly, so additional sampling from it adds little value.

Our statement between coverage, pass@K, and tax

We would like to clarify that we do not claim that post-training always lowers the coverage of the base model. As Figure 3 and line 207 suggest, post-training sharpening can still improve both pass@1 and pass@K depending on model size, test-time sampling budget, and dataset. That’s what you can observe from Table 7, showing that some small models achieve higher pass@K after post-training on some datasets. Here, our conclusions about coverage rest on direct pass@KK comparisons.

However, the shape of the test-time scaling curve is consistent across model sizes and datasets, and here, our conclusions about test-time scalability rest on the tax metric. Whether post-training ends with a higher coverage or not, its scaling curve always saturates quickly compared to the base model’s scaling curve (in all 42 cases we explored). End-point metrics such as pass@KK can not reflect this consistent change in curve shape, and that's why we devise the tax to summarize it in one number. Thanks again for raising this point; we polished the draft accordingly.

AuthorsOpenReview ↗

Response to Reviewer qnsT (2/2)

W2, Q1: PTGS is evaluated only on Qwen2.5-7B-Instruct with Sokoban and FrozenLake (why these two?); how about more complex tool-use, web-interaction, math and coding tasks?

In our initial draft, we focused on Sokoban and FrozenLake for three reasons: (1) they are well-cited, representative testbeds for multi-turn agent RL (Wang et al., 2025); (2) these two environments are where RL post-training significantly reduces the coverage of the base model (e.g., by 15.0 and 21.6 points with PPO), so they provide good conditions for measuring the effectiveness of the proposed solution; (3) each run is cheap enough to allow multi-seed repeated experiments for drawing statistically meaningful conclusions, as well as extensive ablation studies. We revised the draft to state this rationale explicitly.

However, we agree that confining the experiments to only these two environments can limit the supporting evidence for PTGS. Following your comment, we additionally evaluated PTGS on tasks that require web interaction, tool use, and math reasoning for this rebuttal, across three model backbones and three RL objectives.

Specifically, we trained on WebShop (Qwen2.5-7B-Instruct, PPO) and SearchQA (Qwen2.5-3B-Instruct, PPO and GRPO) from the RAGEN codebase, and also trained on POLARIS math problems (Qwen3-1.7B-Base, with MaxRL; Tajwar et al., 2026) and tested on the MATH-500 and Minerva datasets from the MaxRL codebase. For all PTGS rows, we adopted the default PPO setting of the paper without hyperparameter tuning, and the training-evaluation protocol follows that of Table 2.

TaskMethodpass@1 ↑pass@128 ↑TaxS(128)\text{Tax}_S(128) ↓
WebShopBase16.062.5–
PPO57.575.50.046
PPO w/ PTGS73.280.70.043
SearchQABase21.475.0–
PPO39.370.30.055
PPO w/ PTGS44.773.40.055
GRPO41.074.50.053
GRPO w/ PTGS43.675.50.053
MATH-500Base55.595.0–
MaxRL64.994.50.004
MaxRL w/ PTGS64.494.9−0.001
MinervaBase20.059.9–
MaxRL27.359.80.000
MaxRL w/ PTGS26.960.2−0.006

As shown in the table above, on the two realistic agentic tasks (WebShop and SearchQA), PTGS consistently improves both pass@1 and pass@128 over the baseline RL methods while paying a matched or lower tax. Meanwhile, in the math domain, we compared with an advanced RL method, MaxRL, which is designed to improve the expectation of pass@k. Here, PTGS still improves pass@128 and achieves an even lower tax than MaxRL, while slightly compromising pass@1. In summary, we demonstrate that PTGS preserves the accuracy hill-climbing benefits of modern post-training while consistently lowering or matching the tax and bringing coverage gains across a variety of tasks, not only logical puzzles but also complex web interaction, multi-hop QA, and challenging mathematical reasoning.

Q2: Were other models or environments investigated, and are additional results available, including unsuccessful cases?

To investigate the working conditions of PTGS more deeply, we also trained Qwen3-1.7B-Base on GSM8K and tested it on GSM8K-Platinum, with the same training configuration as above.

Modelpass@1 ↑pass@128 ↑
Base71.199.8
MaxRL86.799.5
MaxRL w/ PTGS86.698.8

Here, PTGS was not effective in improving either pass@1 or pass@128. We speculate that this is because the tasks are already too easy for the base model, e.g., before training, it already solved about 70% of the tasks in a single shot and 99.8% of the tasks with a sampling budget of 128. Since the majority of tasks are easy to solve, PTGS's per-task temperature scaling collapses into a kind of global cooling operator that incentivizes deterministic behavior of the model, resulting in a 1.0-point drop in coverage. Therefore, we suggest that users adopt PTGS on datasets that are not already saturated by the base model and that span a broad range of task difficulty. We will keep exploring whether this interpretation holds under different hyperparameters, and will add the analysis to the draft.

Reference

  • Wang et al. 2025, RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
  • Tajwar et al. 2026, Maximum Likelihood Reinforcement Learning
AuthorsOpenReview ↗

Invitation to rebuttal discussion

Dear Reviewer qnsT,

As the discussion period is coming to an end, we wanted to briefly follow up on our responses. To address your concerns, we:

  1. Clarified that the tax measures the loss in test-time scalability rather than coverage loss. We also explained which conclusions rely on direct pass@K comparisons and which rely on the tax, and revised the Abstract and Introduction to match Sec. 4.
  2. Extended the PTGS evaluation beyond Sokoban and FrozenLake to web interaction (WebShop), search-based QA (SearchQA), and math reasoning (MATH-500, Minerva). These experiments cover three backbones and three RL objectives (PPO, GRPO, MaxRL), and in all of them PTGS improves pass@128 with a matched or lower tax, while preserving the pass@1.
  3. Reported an unsuccessful case (GSM8K-Platinum), where the base model is already near saturation, to clarify when PTGS is most useful.

We truly value your feedback and would like to know whether our responses have addressed your concerns. If anything remains unclear, we would be happy to provide further clarification or additional results.

Thank you for your time!

Best regards,
 The Authors

Reviewer_jubFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how base and post-trained language models differ in single-attempt success and success achievable through repeated agent rollouts. It evaluates 14 base/instruction-model pairs on three benchmark settings and introduces sharpening-tax statistics derived from finite-budget coverage curves. Theoretical results characterize these statistics and illustrate sharpening under a stylized model. The paper then proposes Posterior Thompson-Guided Sampling (PTGS), which adapts training-rollout temperature using per-task success histories. Experiments with PPO and GRPO on Sokoban and FrozenLake show improvements in single-attempt success and finite-budget coverage relative to the principal training baselines.

Critical strengths
  1. The study examines 14 model pairs across three benchmarks, revealing differences missed by pass@1. Harness and thinking-mode ablations strengthen the evaluation.

  2. PTGS improves success and coverage across two training algorithms, with five-run experiments and fixed-temperature controls in Appendix G. Sokoban PPO rises from 46.5/55.0 to 61.1/69.7 in pass@1/pass@128, without requiring an adaptive test-time procedure.

Critical weaknesses
  1. Positive tax need not imply lost coverage: BFCL Gemma 4 12B improves pass@128 from 0.470 to 0.740 despite positive tax. Although acknowledged in the paper, this distinction should be maintained in its headline claims.

  2. Fixed T=1.5 also improves both Sokoban metrics, so PTGS's specific advantage needs clearer isolation. Its higher point estimates are promising, but paired uncertainty and a control matching the temperature distribution would better establish the value of task-specific adaptation rather than a favorable global temperature choice.

Questions for authors
  1. Can you clarify the coverage-loss claims in light of the positive-tax/improved-coverage cases in Table 7? Please make this distinction explicit in the main summary, using the existing decomposition rather than treating positive tax alone as evidence of lost coverage.

  2. What are the paired confidence intervals for PTGS versus the strongest fixed-temperature baseline? Please also reconcile the claim that only PTGS improves both metrics with the T=1.5 results in Table 9.

  3. Does PTGS still help when task histories are shuffled or Thompson sampling is replaced by the posterior mean? These controls would separate the value of task-specific adaptation and posterior sampling from the benefit of using a favorable temperature distribution.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jubF (1/2)

Dear Reviewer jubF, we sincerely appreciate your positive assessment and are glad that you found the evaluation over 14 model pairs and the five-run PTGS experiments convincing! We respond to each weakness (W) and question (Q) below.


W1, Q1: A positive tax need not imply lost coverage (e.g., BFCL Gemma-4-12B); keep this distinction explicit in the headline claims, using the existing decomposition; Can you clarify the coverage-loss claims in light of the positive-tax/improved-coverage cases?

Our statement on the relationship between coverage, pass@K, and tax

As you pointed out, a positive tax is not by itself evidence of lost coverage, and we would like to clarify that we do not claim that every post-training compromise the coverage of base models. As Figure 3 and line 207 suggest, post-training sharpening can still improve both pass@1 and pass@K depending on model size, test-time sampling budget, and dataset. That’s what you can observe from Table 7, showing that some small models achieve higher pass@K after post-training on some datasets.

However, the shape of the test-time scaling curve is consistent across model sizes and datasets. Whether post-training ends with a higher coverage or not, its scaling curve always saturates quickly compared to the base model’s scaling curve (all 42 cases we investigated). That’s why we devise a new metric beyond the simple end-point metrics like pass@K, and that’s what our tax measure is.

Interpretation of Tax

The tax measures the loss in test-time scalability, i.e., how much less the post-trained model gains from additional rollouts across all budgets up to KK than its base model. Lost coverage is a different quantity, which is represented by the pass@KK gap, ΔC(K)=pass@KBase−pass@KPost\Delta C(K)=\text{pass}@K_{\text{Base}}-\text{pass}@K_{\text{Post}}, at a specific end-point budget KK.

Following your suggestion, we revised the headline claims in our draft to keep these two different quantities distinct and connect them through our decomposition (Corollary 2): TaxA(K)=K [ΔC(K)−ΔCˉ(K)]\text{Tax}_A(K)=K\,\big[\Delta C(K)-\Delta\bar C(K)\big], where ΔCˉ(K)\Delta\bar C(K) is the pass@kk difference averaged over all budgets k≤Kk\le K. Note that this decomposition allows us to interpret a positive tax from three different angles, as shown below:

CasesInterpretation
ΔC>0,ΔCˉ>0\Delta C>0, \Delta \bar C>0Base achieves higher final coverage and Post gets stuck in a flat scaling curve; additional sampling from Post adds little value.
ΔC>0,ΔCˉ<0\Delta C>0, \Delta \bar C<0Base achieves higher final coverage and Post has a quickly saturating scaling curve; additional sampling from Post adds little value.
ΔC<0,ΔCˉ<0\Delta C<0, \Delta \bar C<0Post achieves higher final coverage, yet its scaling curve saturates quickly, so additional sampling from it adds little value.

Therefore, even in the case where the tax is positive, the post-trained model can still have higher coverage (when ΔC<0\Delta C<0), while it shows much worse test-time scalability (i.e., additional sampling changes pass@KK much less than it does for the base model).

Importantly, this is not an unexpected flaw of our Tax; rather, it is exactly the intended design philosophy: to summarize, in a scalar quantity, the overall scalability trend that is missed by end-point metrics alone, such as pass@1 and pass@KK. It can also inspire many interesting applications, such as the dynamic budget allocation that we examined in Appendix F.4. We revised the Abstract, Introduction, and Section 4 to eliminate the confusion. Thanks again!

AuthorsOpenReview ↗

Response to Reviewer jubF (2/2)

W2, Q2: Fixed T=1.5T=1.5 also improves both Sokoban metrics; reconcile the "only PTGS" claim with Table 9; paired CIs against the strongest fixed temperature

While the fixed-temperature (T=1.5T=1.5) baseline also improves both pass@1 and pass@128 on Sokoban, the same temperature gives the lowest pass@128 and the highest tax of the five fixed temperatures on FrozenLake. That is, a global temperature must be tuned per environment, whereas PTGS uses one setting in both environments. Therefore, PTGS is indeed the only method that can improve both metrics in both environments with a single configuration, demonstrating its robustness. We revised the corresponding claim in the Appendix. Thanks for the comment!

Also, following your suggestion, we further present bootstrap 95% confidence intervals (CIs) with 10,000 resamples from the RL runs in both environments (Sokoban and FrozenLake) and the permutation-test pp-values for the performance difference between PTGS and the best fixed-temperature baseline. Since PTGS uses one setting in both environments, we compare it with the best single fixed temperature, T=1.5T=1.5, as well.

Statistics from PTGS minus fixed TTΔ\Deltapass@1 ↑Δ\Deltapass@128 ↑ΔTaxS(128)\Delta\text{Tax}_S(128) ↓
mean [95% CI]+2.9 [−2.2, +7.9]+8.4 [+2.7, +14.2]−0.028 [−0.049, −0.008]
One-sided permutation test pp-value0.160.0120.013

As shown in the table, PTGS achieves statistically significant improvements over the best fixed-temperature baseline in terms of both pass@128 and the tax (their CIs exclude zero and their pp-values are lower than 0.05).

W2, Q3: Does PTGS still help when task histories are shuffled, or when Thompson sampling is replaced by the posterior mean?

Thanks for the productive suggestion! Following your comment, we additionally ran both controlled experiments on Sokoban for this rebuttal (PPO with the RAGEN recipe and the evaluation setup of Table 2). First, the shuffled-temperature setup draws one temperature per prompt from the PTGS posterior and permutes them to break the relationship between prompt difficulty and temperature. Second, the posterior-mean variant of PTGS replaces Thompson sampling with the deterministic posterior mean.

Methodpass@1 ↑pass@128 ↑TaxS(128)\text{Tax}_S(128) ↓
PPO46.555.00.094
PPO w/ shuffled temperatures48.553.60.111
PPO w/ PTGS posterior mean variant45.863.00.077
PPO w/ PTGS Thompson sampling61.169.70.081

As shown, with the temperatures assigned at random, pass@128 falls from 69.7 to 53.6, and the tax rises from 0.081 to 0.111. This indicates that the performance gain and the lower tax come from giving each prompt a temperature that follows its difficulty, not from a favorable temperature distribution. Meanwhile, the PTGS posterior-mean variant fails to raise pass@1, while it meaningfully improves pass@128 and reduces the tax. We speculate that the posterior mean determines each prompt's temperature under a closed feedback loop without exploration, which incurs the risk that cooled prompts keep getting cooler while heated ones keep getting hotter.

AuthorsOpenReview ↗

Invitation to rebuttal discussion

Dear Reviewer jubF,

As the discussion period is coming to an end, we wanted to briefly follow up on our responses. To address your concerns, we:

  1. Clarified that a positive tax means a loss in test-time scalability, not necessarily a loss in coverage. We also revised the headline claims (Abstract, Introduction, Sec. 4) to keep these two quantities distinct.
  2. Reported bootstrap confidence intervals and permutation tests against the best fixed temperature. PTGS improves pass@128 and lowers the tax, both with significant p-values. We also refined the “only PTGS” claim in the Appendix.
  3. Ran two additional experiments you suggested (shuffled temperatures and the posterior mean).

We truly value your feedback and would like to know whether our responses have addressed your concerns. If anything remains unclear, we would be happy to provide further clarification or additional results.

Thank you for your time!

Best regards,
 The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for the constructive feedback. We conducted several new experiments and directly revised and expand the draft (up to 10-page) to reflect the feedback during the rebuttal (See the red-highlight for major revisions). Here, we summarize the recognized strengths, the shared questions, and our brief answers.


Strengths highlighted by reviewers

  1. The paper investigates a novel and important problem (CgeC).
  2. Comprehensive empirical study of coverage loss in agentic tasks that can be missed by pass@1-centric evaluation; ablation studies further strengthen the observation (jubF, qnsT).
  3. PTGS is simple and demonstrates promising empirical effectiveness: two training algorithms over two tasks without requiring an adaptive test-time procedure (jubF, qnsT, CgeC).
  4. The sharpening tax metric is intuitive to understand, as its value captures the scale of the coverage loss (qnsT).

Summary of our responses to the representative questions

(1) Interpretation of the tax (jubF, qnsT, CgeC).

The tax measures the loss in test-time scalability, which indicates how valuable additional sampling is for post-trained model than the base one. It is more about the overall trend of the coverage gain over the computing budget, not solely the coverage gap at the final budget. We polished our draft directly to clear up the confusion about the interpretation of the Sharpening Tax.

(2) What drives the gain of PTGS, and how general it is (jubF, qnsT).

We conducted two additional ablation studies requested by Reviewer jubF (1. shuffled-temperature inference, 2. posterior mean instead of Thompson sampling). We also extended the PTGS validation to other domains that require complex interactions with the web or external databases as well as professional mathematical reasoning and coding (asked by qnsT), with three different model backbones and three RL objectives. We further report an edge cases asked about by qnsT to offer further insight into the working conditions of PTGS.

(3) Presentation (CgeC).

We intensively polished the writing to remove the frequent colloquial expressions and awkward phrasing, brought the Related Work section into the main text, and changed the section name 'Preliminaries'. We also made the connections between sections smoother by adding takeaway blocks at the end of each core section, and polishing the bridge paragraphs so that readers can easily follow the flow from the analyses to the derivation of the Sharpening Tax, and from the Tax to PTGS.


Remark

We believe these revisions substantially strengthen our work, and we warmly welcome further questions; if any reviewer would like an additional targeted analysis, we are glad to run it within the remaining time and report back.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.