acceptodds
Under review as a conference paper at ICLR 2027

Unseen Is Not New: Partial Identification of Capability Expansion in Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) can expose reasoning behaviors that were unseen in a finite base-model sample. However, this observation does not establish capability expansion: a mode with zero probability and one with small positive probability are arbitrarily difficult to distinguish at any finite rollout budget. To address this ambiguity, we develop a probability-spectrum account of what checkpoint evaluations can identify. First, absolute continuity of response laws transports through semantic auditing, yielding support constraints and sharp divergence-to-visibility envelopes. Next, we show that expected discovery has a Hausdorff moment representation: its infinite curve identifies the positive mode-mass spectrum, whereas a finite prefix determines a sharp set of compatible future curves. Consequently, checkpoint rankings can reverse with the rollout budget, and semantic coarsening can conceal opposing probability flows. We connect these results to simultaneous probability-change regions and an access/execution analysis that separates natural conditioning from assigned interventions. Finally, exact certificates and matched neural checkpoints demonstrate identifiable amplification and a Transformer budget crossover. Under the verified full-softmax sampler, architectural support is equal even when observed sets differ. These results support capability claims about probability reallocation and operational visibility without treating finite nonobservation as evidence of structural novelty.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Reinforcement Learning with Verifiable Rewards, Capability Evaluation, Partial Identification, Semantic Coverage, Hausdorff Moment Problem

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ySzaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a probability-spectrum account to analyse what capability changes can be justified by finite rollout evaluations from model checkpoints before and after RLVR. The analysis connects support transport, Hausdorff moment represetation, finite-prefix partial identification, sementic coarsening, simultaneous probability-change regions with a measurd bridge gap。

Critical strengths
  1. This paper separates uniform finite-sample non-testability and finite-prefix partial identification.
  2. This paper theoretically proves that the infinite discovery sequence identifies the positive mode-mass spectrum up to mode labels.
  3. This paper reveals the "budget crossover" phenomenon and theoretically demonstrates that coverage is Schur-concave in mode masses.
Critical weaknesses
  1. The author should explain more clearly the conceptual difference between their proposed finite-prefix partial identification and Hausdorff relaxation, and the existing unseen-species estimation and partial identification literature reviewed in Section 2.
  2. The author should evaluate whether the Hausdorff-relaxation bounds still remain useful when the prefix moments are estimated from finite neural rollouts with sampling noise, rather than being computed from exact analytic prefixes as in Table 3.
  3. The author should clarify the conceptual gap between the within-prompt semantic mode coverage defined in Section 3 and the across-problem level evaluation used for the DeepScaleR budget crossover.
Questions for authors
  1. What does verified full-softmax sampler refer to, the assumption in Appendix C.3 or a particular executable sampler discussed in Appendix M.2?
  2. How does the DeepScaleR budget crossover change when the 2048-token cap is varied, and how many responses from each checkpoint are truncated?
1 comment

Thank you for the positive assessment of the non-testability / partial-identification split, the spectrum identification result, and the budget crossover.

W1. Relation to unseen-species estimation and partial identification.

  • Unseen-species estimation studies one population. It estimates the number or mass of unseen classes from a frequency profile.
  • Our object differs in three ways. (i) The discovery curve itself is a Hausdorff moment sequence of the within-prompt mode spectrum, and its infinite curve identifies that spectrum (Theorem 4.3). (ii) For a finite prefix, we report the sharp set of all compatible futures (Theorem 4.4). (iii) We compare two matched checkpoints, which yields amplification, contraction, signed flow, and budget rank reversal.
  • Partial identification (Manski) supplies the principle. Our contribution is the observable and the set: a truncated moment sequence estimated from rollouts, the exact finite-spectrum program, and its convex outer relaxation.
  • Finite-prefix set vs. Hausdorff relaxation. The finite-spectrum program keeps two facts: each atom's weight equals its probability (weight p at location 1 − p), and the universe has size M. It is sharp but nonconvex. The Hausdorff relaxation drops both links and optimizes over all nonnegative measures with the same moments. It is a linear program and gives valid outer bounds (App. F).

We will add this comparison to Sec. 2.

W2. Bounds when prefix moments come from noisy neural rollouts.

Theorem 4.4 takes moment intervals [ℓ_k, u_k], not exact values. App. F.7 builds these intervals from simultaneous bands on the observed coverage curve, so the bound stays valid at the same confidence level. We ran this on real neural rollouts (MATH-500 for both lineages, and the controlled panel). We estimated the prefix from a random quarter of the rollouts and bounded the curve at the full budget. The bound contained the held-out value in every resample. Its upper extent beyond the sample is governed by rare unseen mass, which Theorem 4.1 shows no finite budget can exclude, and the set reports this directly. Manuscript revision: the real-rollout check now appears next to the exact certificate in App. F.

W3. Within-prompt mode coverage vs. the across-problem DeepScaleR crossover.

Both use the same functional, C_K = Σ_a H_K(p_a) with H_K(p) = 1 − (1 − p)^K, on different atom spaces. Within a prompt, the atoms are successful semantic modes, and C_K counts the modes exposed by K attempts. On MATH-500, each problem has one success cell, and the average first-hit curve is the expected fraction of problems solved by budget K. The concentration argument of Theorem 4.5 applies at both levels: RL can concentrate probability among semantic modes within a prompt, or spread success probability more unevenly across problems, and either can raise one-shot success while lowering large-budget coverage. Propositions 4.6–4.7 describe how merging atoms changes flow and coverage, and the controlled panel reports the fine and coarse levels on the same rollouts (Observation 3). We will state this link in Sec. 3.

Q1. What does "verified full-softmax sampler" refer to?

Both, in two roles. App. C.3 states the condition: finite logits and no vocabulary truncation give every finite token sequence positive probability. App. M.2 explains why an executable sampler with exact zeroing (nucleus, top-k, filters) can break this. "Verified" refers to the sampling configuration that produced our data, recorded in App. P: top-p = 1, no top-k, no repetition penalty, and a shared tokenizer within each pair. This configuration applies no vocabulary truncation, so with finite logits the C.3 condition holds. An explicit witness response within the length cap, for every registered mode and for every MATH problem, then has positive probability at both checkpoints. We will state this configuration and its consequence in one sentence in the main text.

Q2. The 2048-token cap and truncation.

The cap is one coordinate of the registered sampling law, λ_math = (T = 1, top-p = 1, L_max = 2048) (App. P). Both checkpoints use the same cap and the same scoring rule: a response that reaches the cap is scored from any boxed answer it already contains, and otherwise counts as a failure.

The crossover theorem is stated for a fixed law. Changing the cap changes the mass vector, just as changing the temperature does, and the same procedure then reports the curve for that law. This is the paper's claim: capability rankings are indexed by sampler and budget, and the cap is part of the sampler. Under the registered law, the DeepScaleR reversal is certified by the curvewise band, and it is certified again when top-p changes to 0.9 (App. W, Table 20). At this cap the observed truncation rates are 62.0% (base) and 63.9% (RL). Manuscript revision: these per-checkpoint counts are now reported in App. P.

Reviewer_HoqbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Authors propose a framework to determine which claims about RLVR can be supported by a finite number of sampled outputs. Because rare and impossible solutions are indistinguishable from finite number of samples, authors propose reporting probability changes and coverage of distinct solutions across attempt budgets. They argue that a solution that is seen after training but not in a finite base-model sample cannot establish new capability. Experiments on a synthetic arithmetic task, MATH-500 and GSM8K show that apparent new solutions are statistically indistinguishable from rare ones.

Critical strengths

The paper separates set of solutions with non-zero probability, set of solutions observed in a probability of observing a given solution within k attempts, which is an important distinction that prior work often conflates.

Critical weaknesses
  • Most theoretical results stated as theorems or propositions follow directly from standard results and the framing does not make them more clear or useful. For example, Theorem 4.1 is essentially the well-known fact that observing zero events in nn trials is compatible with any rate below roughly 3n\frac{3}{n} at 95% confidence (Hanley & Lippman-Hand, 1983); Theorem 4.5 follows directly from the Schur-concavity of separable sums of concave functions (Marshall et al., 2011, Prop. 3.C.1).
  • The experiments do not test anything new. The headline results (MATH 500) replicate Yue et al. (2025). The main novelty was testing the number of distinct semantic solutions per prompt, and the only experiment that reported this (arithmetic task) used a model not trained in the arithmetic task and had success rate of 2% with paired coverage differences interval including zero. The result is uninformative about the framework. No experiment shows an instance where proposed reporting changes conclusions compared to standard pass@k curves.
Questions for authors
  • Can you add consolidated clear descriptions of the proposed metrics with reporting protocol and guidance for identifying semantic modes to the paper?
  • Can you show with experiments that the proposed metrics are providing more nuanced and different conclusions compared to the standard ones (such as pass@k) in multiple setups, including a natural language one where the RL model was trained on task in question?
2 comments

Thank you for the detailed review, and for noting that separating the support, observed, and within-k visible sets is an important distinction that prior work often conflates.

W1. Theorems follow from standard results.

The contribution is an identification framework for RLVR capability claims. Each result answers one question in it: what finite rollouts can say about support, about probability change, and about visibility.

  • Theorem 4.1 is not the rule of three. The rule of three is a confidence bound for one rate. Theorem 4.1 is about decisions. It gives an exact TV identity, a power bound that holds for every test, and a Le Cam bound for the matched-checkpoint expansion indicator. The last part shows that more RL-side samples cannot remove the base-side ambiguity, because the post-RL law is the same in both worlds. The result extends to adaptive stopping (App. B). Together they turn "the base model never produced it" into a formal non-identification of capability expansion. The rule-of-three bound appears in our paper only as a reported zero-count upper bound.
  • Theorem 4.5 turns concentration into an evaluation statement. A checkpoint with higher one-shot success can have lower large-budget coverage when its success is more concentrated. Answer-level pass@k cannot detect this, because it depends only on total success.
  • The identification results are specific to this problem. The discovery curve is a Hausdorff moment sequence of the mode-mass spectrum, and its infinite curve identifies the spectrum (Theorem 4.3). A finite prefix gives a sharp identified set of future curves (Theorem 4.4). Support transports through semantic auditing with sharp KL-to-visibility envelopes (Proposition 4.2, App. C). Coarsening contracts signed flow (Proposition 4.6).

W2 / Q2. Do the metrics change conclusions compared with pass@k, across setups, including natural language with an RL model trained on the task?

Yes. On existing data:

  1. MATH-500, DeepScaleR. This evaluates a math-RL checkpoint, trained on competition math, on natural-language mathematics from the same task family. Pass@1 says RL is better; pass@64 says the base model is better. Neither single K settles it. Our curvewise band certifies both signs, and Theorem 4.5 explains how the flip can arise (Observation 5).
  2. MATH-500, Qwen3 math-RL. 16 problems are solved only after RL, which a finite observed-only comparison would label as expansion. Under our framework none is identified as new, while the joint region certifies 6 amplified and 34 contracted problems (Table 1, Observation 5).
  3. GSM8K, Qwen3. The RL gain is certified at one rollout, and later budgets are unresolved (App. U). A single large-K pass@k would miss the certified gain.
  4. Controlled task. The outcome-level comparison (what pass@k sees) does not establish a checkpoint difference. The fine-mode audit certifies an amplified mode (5 → 87 successes) and shows that outcome reporting keeps only 13% of the probability movement (Observation 3).

On MATH-500 and Yue et al. Our result is not the curve itself. It is the split between what is identified (named amplified and contracted problems, a certified reversal) and what is not (every RL-only problem), which pass@k curves do not report.

On the controlled task. The estimand is the change between matched checkpoints on a fixed, registered task. It does not require the task to be the RL training task, and capability claims in this debate are made on held-out tasks as well. An exact solver enumerates every valid mode, so rare-mode amplification can be tested directly. Where the aggregate contrast does not separate the checkpoints, the mode-level audit still certifies amplification. This is the more nuanced conclusion the reviewer asks for.

Q1. Consolidated metrics, reporting protocol, and semantic-mode guidance.

Yes. We will add a one-box reporting protocol to the main text (currently App. Q):

  1. Register the semantic auditor and the partition hierarchy (fine → operator → outcome) on a design split, before held-out generation.
  2. Fix and report the sampler: tokenizer, temperature, top-p, length cap, and scoring rule.
  3. Report the full first-hit and coverage curve with the unbiased occupancy estimator and a curvewise band.
  4. Report per-mode probability changes in a simultaneous region, labeled amplified, contracted, or unresolved.
  5. Report signed flow at every level of the hierarchy.
  6. Report zero counts as upper bounds, never as "new".

Mode guidance. A mode must be computed by a deterministic auditor from the prompt and the response, and must be fixed before held-out generation. Each coarser level must be a deterministic merge of the finer one. In our task, the fine mode is the declared first operation, the middle level is its operator, and the outcome level is success (App. K). In MATH-500 and GSM8K, the registered cell is problem-level success.

Reviewer_HoqbOpenReview ↗

Reviewer's reply

Thank you for the response, I maintain my score.

Reviewer_fYzTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines how to interpret claims that RLVR expands the capabilities of a base model. The authors argue that the common practice of treating a solution unseen in a finite base-model sample as evidence of a new capability is not justified, because zero probability and a very small positive probability cannot be distinguished with finite rollouts. They show that under full-softmax sampling, the base and RL checkpoints have the same support, so what RL changes is the probability assigned to each solution mode. They propose reporting budget-indexed quantities, such as first-hit probabilities and expected mode coverage over all rollout budgets, together with simultaneous confidence regions that classify each mode as amplified, contracted, or unresolved. On the theory side, they show that the coverage curve forms a Hausdorff moment sequence, that checkpoint rankings can reverse with the rollout budget, and that coarse evaluation hides probability shifts between modes. Experiments use a controlled arithmetic environment with enumerable solution modes and MATH-500/GSM8K.

Critical strengths
  1. The paper clearly separates three questions that are often mixed up in the RLVR capability debate: whether a model can produce a solution at all, how likely it is to produce it, and whether it is observed within a given number of samples.
  2. The proposed reporting practice is reasonable and easy to adopt: full budget curves instead of a single pass@K value, and simultaneous intervals instead of ad hoc frequency thresholds.
Critical weaknesses
  1. Under the paper’s full-softmax assumptions, both checkpoints assign positive probability to every allowed finite response. Strict support expansion therefore says little about practically useful capability gains. Although the framework also considers finite sampling budgets, it remains unclear how much new insight it adds to the existing debate about whether RL expands model capabilities.

  2. Budget-dependent rank reversals between base and RL models have already been reported in prior pass@K studies, such as Yue et al. (2025). The paper offers a clearer explanation and more careful statistical testing, but the empirical finding itself is not new.

  3. The empirical evaluation is limited in scope. The controlled task has a success rate of about 2%, and only one mode shows a statistically significant increase, leaving most mode-level changes unresolved. The real-task experiments cover two model pairs, and the DeepScaleR crossover is not statistically confirmed under all tested decoding settings. These results provide limited evidence about how broadly the observed patterns hold.

  4. The practical value of the proposed bounds on future mode discovery is not clearly demonstrated. The main example uses a synthetic distribution with known probabilities. Although the paper explains how to account for sampling uncertainty, it does not clearly show whether the resulting bounds are narrow enough to provide useful conclusions for real models beyond the observed sampling budget.

Questions for authors

Given that strict support expansion is ruled out under full-softmax sampling, could you clarify what concrete change the proposed framework would bring to existing conclusions about RLVR capabilities, beyond the budget-dependent ranking behavior that has already been reported?

1 comment

Thank you for the careful review, and for noting that the paper cleanly separates possibility, probability, and visibility, and that the reporting practice is easy to adopt.

Q / W1. What concrete change does the framework bring?

The capability debate often decides "new capability" by finite non-observation: the RL model produced a solution that the base model never produced in n samples. Theorem 4.1 shows that this test cannot decide support at any budget and under any sampler. We replace it with quantities the data do identify. On the same rollouts, this changes the conclusion:

  • "RL-only" problems are not identified as new capabilities. On MATH-500, 16 problems are solved only by the Qwen3 RL checkpoint. A finite observed-only comparison would label them as expansion. Our zero-count bound shows that none of them is identified as new. The joint region instead certifies 6 amplified and 34 contracted problems (Table 1, Observation 5).
  • Flat success can hide real redistribution. In the controlled panel, the aggregate comparison does not establish a checkpoint difference. Yet the mode-level audit certifies an amplified mode (5 → 87 successes) that outcome-level pass@k cannot reveal, and outcome-level reporting keeps only 13% of the fine-mode flow (Proposition 4.6, Observation 3).
  • No single K is the capability boundary. We show when rankings can flip and certify the flip over the whole curve (see W2).

On full softmax. The framework does not rest on it. Theorem 4.1 holds for every sampler, including top-p and top-k, where exact zeros exist. Full softmax and support transport (Proposition 4.2, App. C) add a population fact: an RL update with finite reverse KL cannot create new semantic support. So the useful question is how much probability moved and at which budget it becomes visible. Our sharp KL-to-visibility envelope (App. C) bounds the first. The budget curve with simultaneous regions measures the second. This is the practically useful capability gain, stated in measurable form.

W2. Rank reversals vs. Yue et al. (2025).

Yue et al. observe that pass@k curves cross. We add three things:

  1. Mechanism. Theorem 4.5 separates total success from how it is spread across modes. Higher one-shot success can come with lower coverage at a larger budget. Answer-level pass@k depends only on total success, so it cannot see this.
  2. Scope. The same theorem governs semantic modes inside one prompt, which answer-level pass@k cannot distinguish.
  3. Evidence. The DeepScaleR reversal is certified by a simultaneous band over the whole budget curve: positive at K = 1 and negative from K = 8 onward (Observation 5, Fig. 3). It is a tested statement, not a reading of point curves.

W3. Empirical scope.

  • Scale. 1.07M neural responses: two matched RL lineages, MATH-500 and GSM8K, a decoding sweep, and an enumerable task with 2,623 held-out mode atoms.
  • Exact semantic ground truth. The controlled task enumerates every valid mode before sampling. Each mode-level claim, including claims about rare modes, is therefore directly auditable.
  • Named changes. The procedure labels each change from the data alone. Across the five neural panels it certifies 48 named probability changes (Observation 6).
  • Decoding. The DeepScaleR crossover is certified under two sampling laws (T = 1 and top-p = 0.9), and the point crossover also appears at T = 0.7 (App. W, Table 20). The theory states that rankings depend on the sampling law (App. G), so we index every ranking claim by sampler and budget.

W4. Practical value of the future-discovery bounds.

Theorem 4.4 takes interval constraints [ℓ_k, u_k] on the prefix, not exact values. App. F.7 propagates simultaneous bands on the observed curve into these intervals, so the bound stays valid under sampling noise. We also ran this on real neural rollouts (MATH-500, both lineages, and the controlled panel). We estimated the prefix from a random quarter of the rollouts and bounded the curve at the full budget. The bound contained the held-out value in every resample.

The set reports which future discovery levels the evidence supports. Rare unseen modes cannot be excluded at any finite budget (Theorem 4.1), and the set states this directly instead of hiding it in a point forecast. Within the sampled budget, the occupancy estimator gives the full curve without extrapolation. Manuscript revision: the real-rollout check is now reported in App. F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.