acceptodds
Under review as a conference paper at ICLR 2027

Endogenous Curricula for Verifiable Self-Play: Projected Soft- Coverage–Information Certificates

Abstract

Self-play agents must adapt executable-task curricula without extinguishing valid program paths. We model this as KL-regularized control on a syntax-constrained prefix tree. The same projected soft- iterate yields a completed-program floor and an information-value certificate. A realized-row bound sharpens support control; direct performance identities, quadratic residual penalties, and contrast-aware estimator errors certify information advantage over an IG-ablated comparator without the full-objective optimal-policy oracle. A coherent finite Bayesian specialization gives expected entropy budgets, while identifiable likelihoods and accumulated row floors yield anytime posterior and finite-time identification bounds. Fixed-horizon two-time-scale tracking and epochwise expansion delimit the dynamical scope. In an executable Boolean-DSL study, a matched pure-information curriculum improves target success and semantic coverage over its reference. Shared-belief audits show conditional gains after KL costs, but the original residual-composed information bound is nonpositive throughout the tested configurations. The direct refinements have not been evaluated on those records; we separate mathematical guarantees, observed learning gains, and unresolved finite-time efficacy.

Primary area:
learning theory
Keywords:
Self-Play, Curriculum Learning, KL-Regularized Reinforcement Learning, Information Gain, Verifiable Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_znUcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies endogenous curriculum generation for verifiable self-play through KL-regularized control on a syntax-constrained program tree. The proposed projected soft-Q generator is designed to maintain non-vanishing support over legal programs while favoring tasks with high estimated information gain. The paper derives several theoretical guarantees, including completed-program probability floors, Bellman-residual-based tracking bounds, comparator-relative information certificates, and finite-hypothesis posterior identification results. It also introduces sharper direct residual certificates based on performance identities, quadratic residual penalties, and contrast-aware estimator errors. Experiments are conducted on a finite Boolean DSL with 192 programs and 16 truth-table hypotheses. Under a matched pure-information objective, the proposed curriculum improves target identification and semantic coverage relative to the reference policy. However, the original residual-composed information bound is nonpositive throughout the tested configurations, while the newly introduced direct certificates have not yet been evaluated on the reported experimental trajectories.

Critical strengths

The paper is unusually careful about the scope of its theoretical claims. It explicitly distinguishes support guarantees, information-value certificates, posterior identification results, and empirical learning gains, and repeatedly states where one result does not imply another. This makes the theoretical assumptions and limitations relatively transparent.

The paper does not hide negative or vacuous empirical results. The authors explicitly report that the original residual-composed information lower bound is nonpositive in all tested configurations, rather than presenting only favorable diagnostics. This level of transparency is valuable for assessing what the proposed certificate currently does and does not establish.

Critical weaknesses
  1. The main certificate is empirically vacuous in the reported experiments. The residual-composed lower bound is nonpositive in every tested configuration, and even the exact-TV version is positive only in a subset of rounds. This substantially weakens the practical significance of the paper’s central theoretical construction.

  2. The paper introduces stronger direct certificates after observing this failure, but does not evaluate them on the main experimental trajectories. The manuscript explicitly states that these direct refinements have not been tested on the v2/v3 records. As a result, the strongest theoretical contribution is disconnected from the empirical evidence, and it remains unclear whether it actually resolves the weakness exposed by the original certificate.

Questions for authors

What do the authors consider to be the single primary scientific claim of this paper?

The manuscript develops a sequence of increasingly elaborate results—support guarantees, residual-based information certificates, direct certificates, quadratic refinements, and posterior-identification guarantees—while several of these results address limitations of earlier ones, and the strongest new certificates are not evaluated on the main v2/v3 experiments. At present, I find it difficult to identify which result is intended to constitute the central contribution and what empirical evidence directly validates that claim. Please clarify the main claim of the paper and explain which theoretical and empirical results are necessary to support it.

3 comments

Thank you for asking us to identify one scientific claim and the evidence needed for it. The principal claim is a conditional information-robustness theorem for the actual generator: its advantage over the same-state IG-ablated optimum can be bounded directly by optimization and shared estimation errors, while the same projected iterate preserves program support. This is Theorem 4.14 in the reviewed submission

Its simplest branch says

\ge \frac{\Delta_t^{0,\mathrm{imp}}-\Xi_t}{\lambda_I}-E^\Delta_{t0}.$$ The claim is therefore precise: information advantage is guaranteed when the implemented policy's non-IG objective sacrifice exceeds its certified optimization and estimation penalties. It does not assert that every round has such a margin, or that our selector learns faster than every adaptive alternative. The results have distinct roles rather than forming a succession of replacement claims. Appendix E supplies the exact performance/information identity; Appendices F–G turn it into policy-relevant residual and estimation-error bounds. Appendix H establishes that the positive-margin region is nonempty. The support result ensures task availability under the same decoder. Appendix J separately explains when accumulated support on distinguishing tasks yields finite-model posterior identification. The two-time-scale results concern tracking under additional hypotheses and are not required by the finite-round information theorem. **The empirical gap identified in the review is now addressed by fresh trajectory audits, rather than by the two-query construction alone.** In balanced 192-program experiments, refined direct margins are numerically positive in **1,950/5,120 composite one-update states (38.09%)**, **2,920/5,120 twenty-update states (57.03%)**, and **1,549/5,120 pure-information states (30.25%)**. Direct interval evaluation is stronger on those same records: its corresponding positive counts are **3,631, 3,701, and 3,434**. The original residual-TV-composed absolute-information bound remains nonpositive. Thus the added evidence establishes positive content for the direct sufficient conditions on a subset of audited frozen states, not a claim that the old bound became useful or that every trajectory is certified. The branch breakdown is informative: the local quadratic bound improves the objective-only certificate, while the symmetric branch supplies most positive states in the principal configurations. A separate 2,944-program extension gives lower refined positive fractions, 23.75% and 10.86%; the hypothesis family still has size 16. We retain these weaker results and the negative mean margins in the one-update configurations. This is evidence about finite-model certification, not broad scaling or computational dominance. The adaptive-baseline experiment answers another necessary question. In the Monte Carlo cohort, projected one-step selection succeeds in 195/256 runs versus 144/256 for fixed reference sampling, but greedy IG with 5% reference mixing succeeds in 256/256 and learns faster. Hence our justification is **not** that this decoder is the best active-query rule. It is the quantitative analysis of when a specified regularized curriculum preserves information value despite implementation errors. Eq. (102)'s interval dominance remains part of that analysis, not a result we attempt to overturn. A nonpositive sufficient bound is not a counterexample to its mathematical validity, but it is a real limitation of its usefulness. We address that distinction with proofs, branch-level positive/negative margin distributions, direct-interval comparisons, and separate learning results.

Thank you for the clarification. However, I remain seriously concerned about the overall presentation and coherence of this manuscript.

First, the paper is extremely difficult to follow. The manuscript is 41 pages long, yet it does not contain a single figure. There is no overview of the proposed framework, no visualization of the theoretical results, and not even a learning curve in Section 5.2, which is explicitly titled "Learning Curves and the Objective Trade-off." For a paper combining theoretical analysis, algorithm design, and empirical evaluation, I find this presentation highly unusual and unnecessarily difficult to understand.

Second, the manuscript contains numerous unexplained references to internal experimental versions. The authors repeatedly refer to "v2" and "v3" without ever explaining why these versions are named this way or what happened to "v1." Similarly, what exactly are "Schema 17" and "Schema 18," and why should readers be expected to understand these internal identifiers? The manuscript even discusses inconsistencies across document versions and analyses that have been superseded. This is supposed to be a research paper, not a draft experimental report or an internal development log. I would expect the authors to consolidate their experiments and present a self-contained, coherent account of the research.

Third, I remain concerned about the coherence of the scientific contribution. Throughout the manuscript, the authors repeatedly explain which claims have not been established, which experiments cannot be combined, and which theoretical results have not yet been empirically validated. Acknowledging limitations is important, but the sheer number of qualifications, successive refinements, and disconnected experimental versions makes the paper read more like an accumulation of incremental revisions than a unified scientific investigation. This raises serious concerns about whether the different components have been sufficiently synthesized into a coherent scientific contribution.

Finally, I have concerns about the overall writing style and the degree of independent intellectual synthesis reflected in the manuscript. The writing relies heavily on repetitive, defensive qualifications and carefully hedged statements, which contribute to the impression that the manuscript has been assembled through successive rounds of revision rather than developed around a clear scientific argument. Even in the rebuttal, the authors continue to rely on such defensive qualifications, including:

  • "It does not assert that every round has such a margin..."
  • "This is evidence about finite-model certification, not broad scaling or computational dominance."
  • "A nonpositive sufficient bound is not a counterexample to its mathematical validity..."

While such qualifications may be appropriate individually, their repeated use throughout the paper and rebuttal reinforces my concern that the authors are more focused on defending the limitations of individual claims than on presenting a clear and convincing central scientific contribution. I recognize that such stylistic observations cannot establish whether any content was AI-generated, and I do not make that allegation. I leave any assessment related to the conference's AI-use policies to the Area Chair.

My concerns are ultimately about the clarity, coherence, and scientific maturity of the submission. I therefore maintain my strong rejection recommendation, as I do not believe this manuscript is ready for publication in its current form.

Thank you for identifying these presentation obstacles. We will consolidate the manuscript around one question: when does an implemented executable-task generator preserve program support and certify an information advantage over its same-state, information-ablated comparator? Theorem 4.14 is the principal information result. Theorem 4.12 supplies support and a general distributional route to the same information comparison. This relationship organizes the theory, algorithm, and experiments.

1. The connection from Theorem 4.12 to Theorem 4.14

Both theorems use the same implemented policy, frozen learner state, reference, temperature, clipped scores, and comparator. In Theorem 4.12, projection gives a completed-program probability floor. Its information branch follows Bellman residual → terminal-law TV → information transfer, controlling information through the entire terminal distribution.

Theorem 4.14 retains the additional structure of the objective. Suppressing the frozen state index, write J=J0+λID~J=J_0+\lambda_I\widetilde{\mathcal D}, where J0J_0 includes non-information utility and the KL cost, and let π0\pi^0 maximize J0J_0. The implemented-policy gap is Δt0,imp=J0(π0)−J0(πt)\Delta_t^{0,\mathrm{imp}}=J_0(\pi^0)-J_0(\pi_t). The exact decomposition is

=\Delta_t^{0,\mathrm{imp}}+J(\pi_t)-J(\pi^0).$$ Bounding full-objective suboptimality by $\Xi_t$ and accounting for the estimation error shared by the two policies gives the objective branch of Theorem 4.14: $$\mathcal D(\pi_t)-\mathcal D(\pi^0) \ge\frac{\Delta_t^{0,\mathrm{imp}}-\Xi_t}{\lambda_I}-E^\Delta_{t0}.$$ The performance identity yields a quadratic residual penalty near the optimum; the contrast error charges estimation uncertainty through differences between the policies' terminal weights. Local and symmetric refinements retain more of this structure. The same iterate retains its support floor. **The progression is from general distributional control to direct analysis of the same information comparison.** Subtracting the comparator's value in Theorem 4.12(iii) makes this common target explicit. The earlier tables reported positivity of the absolute information lower bound; the direct audit reports the comparator-relative margin. Retaining the distributional bound makes its conservatism measurable. In the historical baseline, exact TV gives positive information lower bounds in 288/1,600 rounds, whereas residual-to-TV composition gives 0/1,600. Their difference on the same states isolates the loss introduced by that relaxation. The direct analysis quantifies optimization loss and shared estimation error at the objective level. ## 2. The roles of v1, v2, v3, and the schema identifiers The labels record distinct analytical and experimental roles: - **v1:** the preliminary analysis using cross-trajectory local-objective comparisons. C.4 - **v2:** the main study with 17 configurations, 40 seeds per configuration, and 40 rounds per seed. - **v3:** the separate completed-program support audit, with seven reported configurations and additional realized-row floor measurements. Keeping the batches separate preserves measurement provenance: additional support fields belong to the runs that recorded them. The superseded v1 analysis also carries a methodological lesson. Local information values must be compared at the same learner state, because a faster learner can have less uncertainty remaining later in its trajectory. **Shared frozen beliefs evaluate the local information comparison; separate adaptive rollouts evaluate learning progress.** We organize the experiments by purpose: same-state objective evaluation, completed-program support, direct-certificate evaluation, and adaptive-selector comparison. One table will specify objectives, horizons, temperatures, update budgets, information access, seeds, and statistical units. ## 3. Make the appendix follow the mathematical argument We consolidate Appendices D–G as the **derivation and proof of Theorem 4.14**: the performance identity, residual penalties, and contrast-aware information comparison. The opening on page 33 will refer to the model already defined in the main text and introduce only the additional proof notation. ## 4 .Qualifications in the manuscript and our earlier rebuttal During presubmission checking, feedback through ICLR's Paper Assistant Tool program highlighted concerns about rigor and potentially overstated wording. To preserve the exposition, we used LLM-assisted polishing for minimal, local corrections. We often appended scope qualifications rather than integrating them into the argument this is the simplest way to distinguish computational guarantees from claims about learning, generalization, and convergence rates.
Reviewer_zYS5OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies adaptive executable-task curricula using KL-regularized control on a syntax-constrained prefix tree. It derives program-coverage and information-value guarantees for an implemented projected soft-Q generator. Further results provide direct residual-based information certificates, posterior identification guarantees under finite identifiable likelihood models, and asymptotic tracking under two-time-scale assumptions. Experiments on Boolean programs representing 16 truth tables show improvements under a pure-information objective. The original residual-composed information bound remains nonpositive throughout the tested configurations, and the newer direct certificates have not been evaluated on those trajectories.

Critical strengths
  1. The analysis addresses the implemented policy rather than an ideal optimizer. Theorem 4.14 explicitly accounts for both optimization and estimation errors. The direct information comparison, quadratic residual penalty, and treatment of errors shared between policies provide promising technical contributions.
  2. The experiment provides some evidence that information-directed selection can help. The paired comparisons and target-clustered confidence intervals provide more useful evidence than reporting aggregate improvements alone. These results support the effectiveness of the implemented generator in the tested finite setting, although they do not yet validate the new certificates.
  3. The paper distinguishes what is proved from what is observed. It distinguishes information advantage at a fixed learner state from faster learning over an entire trajectory. It also reports unfavorable results, including the nonpositive original bound and cases where the reference sampler performs better on learning metrics.
Critical weaknesses
  1. The strongest new bounds have not been shown to provide useful guarantees for the main learning process. The original residual-composed information bound is nonpositive in every tested configuration. The newer direct bounds improve on it in the two-query construction, but have not been evaluated on the main experimental trajectories. Proposition H.2 establishes positivity near an appropriate optimum for a fixed model and surrogate, without showing how the implemented process reaches or maintains such a region as the learner’s state changes.
  2. The significance of the residual-based certificates needs clearer justification. Equation 102 shows that direct interval comparison is at least as tight when the complete terminal distributions and information intervals are available. However, the paper does not sufficiently demonstrate what additional insight this decomposition provides about the behavior of the learning algorithm, beyond the two-query construction.
  3. The experiment is not sufficient to support broader implications in the context of self-play. The experiment is closer to finite active querying than to training a general-purpose solver through self-generated tasks. The paper acknowledges this limitation, but the implications for curriculum learning remain limited.
  4. Several presentation styles make the paper hard to interpret. The exact definition of "target success" is deferred to the appendix, and the main text introduces numerous technical quantities before explaining a complete generation-and-learning round through a concrete example. Also, the experimental results are presented mainly as aggregate tables, which provide limited visibility into how learning and certificate values evolve over time.
Questions for authors
  1. Which theorems do you regard as the principal original contribution? Can you compare them directly with the closest existing result under matched assumptions, identifying the new conclusion or improved dependence on residual size, horizon, or temperature?
  2. How should the positivity result be interpreted along a changing learning trajectory? Proposition H.2 establishes positivity for a fixed model and surrogate. During learning, both the information advantage over the comparator and the estimation-error terms can change as the posterior concentrates. Does it imply eventual positivity of the direct certificate, or is an additional condition on these changing quantities required?
  3. The paper reports that the original residual-composed bound is nonpositive throughout the tested configurations. Do the existing diagnostics indicate whether this is mainly caused by the residual-to-TV relaxation, information-estimation penalties, or a small comparator advantage? How does this interpretation differ between the baseline and increased-update-budget configurations?
Reviewer_PEdmOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript models adaptive executable-task generation as KL-regularized control over a syntax-constrained prefix tree. It derives two guarantees for the same projected soft-Q iterate: lower bounds on completed-program probabilities and certificates relating information value to optimization and estimation errors. Empirical results provide direct comparator-relative information bounds, posterior identification under likelihood separation, and convergence under specified two-time-scale assumptions. Experiments use 192 Boolean programs representing 16 hypotheses. These results show improvements over reference sampling under a pure-information objective, but the original residual-composed certificate is nonpositive throughout the tested configurations.

Critical strengths
  1. The direct information certificates are a meaningful technical refinement. Theorem 4.14 combines exact performance identities, quadratic residual penalties, and estimation-error cancellation between policies. This exposes how imperfect optimization and information estimation affect the implemented generator’s advantage over its ablated comparator.
  2. The experimental comparisons distinguish several easily conflated claims. Evaluating policies on shared posterior states makes the local objective comparisons interpretable, while separate rollouts assess actual learning. Experimental results provide evidence that adaptive information selection can help in the specified environment.
Critical weaknesses
  1. The added certificates have not demonstrated sufficient usefulness in the setting studied. The original residual-composed information bound is nonpositive in every tested configuration, including the increased optimization-budget settings. Theorem 4.14 could improve this substantially, but its behavior on the main experiments is unknown; the two-query construction establishes possibility rather than effectiveness along learning trajectories. The paper does not establish an information-access or computational setting where its residual certificates offer a compensating advantage. Given that regularized Bellman analysis and error propagation already have established foundations, the significance of the additional certification machinery needs stronger justification.
  2. The empirical evidence is too restricted and insufficiently reproducible to resolve that significance question. The experiment is noisy equality-query identification among 16 known hypotheses, with exact posterior updates and an enumerable model. Full-refresh estimation uses approximately 12.3 million simulated outcomes per round, and no benefit under matched total computation is demonstrated.
Questions for authors
  1. Given the dominance result in Eq. (102), what concrete information-access or computational regime makes the residual certificate preferable to direct interval evaluation?
  2. On the existing v2/v3 records, how often do Theorem 4.14 and its local quadratic refinement certify strictly positive comparator-relative advantage? Please report certificate margins and compare them with Eq. (101), using the same confidence assumptions and including failed coverage events.
  3. Can the complete configuration manifests, trajectory records, and bootstrap specifications supporting the reported tables be supplied? 4. Do the learning improvements persist against a simple adaptive selector, such as direct posterior-IG selection with explicit reference mixing, when observation and computation budgets are both accounted for?
3 comments

Thank you for recognizing the direct certificates and the distinction between shared-belief comparisons and separate learning rollouts. We address computational preference, certificate tightness, reproducibility, and adaptive-baseline performance separately.

Q1. What information-access regime makes the residual certificate preferable to Eq. (101)?

In the fully enumerable setting studied here, we do not establish a tightness or computational advantage over direct interval evaluation. With the same terminal laws and simultaneous information intervals, Eq. (102) gives

\ge \widetilde{\mathcal D}(\pi_q)-\widetilde{\mathcal D}(\pi^0)-E^\Delta_{q0} \ge M_q.$$ Accordingly, Eq. (101) is the appropriate numerical benchmark, and the new audit compares it against the residual certificates on the same states. We do not posit an unproved large-tree or sublinear-query regime to reverse this ordering. The residual result instead answers an algorithm-analysis question: **how accurate must optimization be, relative to the remaining information signal and estimation uncertainty, for information-directed selection to retain an advantage?** The exact identity in Appendix E is $$\lambda_I[\widetilde{\mathcal D}(\pi_q)-\widetilde{\mathcal D}(\pi^0)] =\mathcal K(\pi_q\Vert\pi^0)+\mathcal K(\pi^0\Vert\pi_q) +\mathbb E_{\pi^0}D_q^{\mathrm{res}}-\mathbb E_{\pi_q}D_q^{\mathrm{res}}.$$ Each policy divergence uses the occupancy of its first argument. This identifies the contribution of policy displacement, residual variation under the two curricula, and shared score error. The local quadratic certificate weights action-error spans by actual occupancy; its exact-residual form is invariant to statewise value offsets that leave the decoder unchanged. These dependencies explain which optimization and estimation improvements can strengthen the guarantee, rather than merely testing whether a fully enumerated contrast is positive. Corollary H.1 can be interpreted as an a posteriori acceptance criterion only when its policy-dependent gap and error contrast are themselves certified, with confidence control across repeated checks. We have not demonstrated that using it as a stopping rule saves computation. Likewise, Lemma A.8 assumes target samples at every Bellman coordinate, not arbitrary partial-tree access. The contribution is therefore a structural robustness analysis with explicit sufficient conditions, not a cheaper alternative to interval evaluation in this toy.

Q2. How often are the direct certificates positive, compared with Eq. (101)?

The new experiments answer this on fresh trajectories, not on recovered v2/v3 records. The primary audit has 18 configurations, 128 balanced seeds per configuration—eight per hidden target—and 40 rounds per seed: 92,160 rounds. Seventeen configurations cover the objective, reference, temperature, update-budget, and bias roles. Historical notes disagree on baseline shaping 0.15 versus 0.5; the new core explicitly uses 0.15 and tests 0.5 as an eighteenth configuration. These are declared new configurations, not reconstructed historical random realizations.

The 192-program experiment uses 16 hypotheses, padded horizon 6, γ=0.95\gamma=0.95, baseline temperature parameter 0.4, projection box 4, and relaxation 0.5. The generated policy and all certificates use pre-update qtq_t and the same frozen posterior, reference, temperatures, clipped score vector, discount, and confidence intervals. Pending model-based updates are installed after observation. Each round refreshes all 16 semantic classes with 768,700 nominal IID outcomes per class, simulated using exact multinomial sufficient counts. The information radius is approximately 0.02, augmented by the known injected raw-bias bound in stress arms. This tests robustness to a specified bias envelope, not calibration of an unknown neural estimator.

The branch-level positive counts are:

Configuration; 5,120 rounds eachUniform objectiveLocal quadratic objectiveSymmetricBasic theorem maxRefined maxEq. (101)
Composite, 1 update2641,2851,9501,9501,9503,631
Composite, 20 updates2,4512,8552,9172,9172,9203,701
Pure information, 1 update1389191,5491,5491,5493,434

“Basic theorem max” combines the uniform objective and symmetric branches; “refined max” additionally permits local quadratic and signed bounds. Refined positive fractions are 38.09%, 57.03%, and 30.25%, with exploratory 95% hidden-target-cluster intervals [28.94%, 46.33%], [45.29%, 67.29%], and [22.50%, 37.38%].

The local quadratic bound substantially improves the objective-only branch. In these three arms it adds no positive states beyond the symmetric branch; the signed refinement supplies three additional states in the twenty-update arm. Thus we attribute the overall positive fractions to the direct certificate family, not to the quadratic term alone.

The corresponding margin distributions are:

ConfigurationRefined meanRefined 5th / median / 95th percentileEq. (101) meanEq. (101) 5th / median / 95th percentile
Composite, 1 update-0.002742-0.027995 / -0.000059 / 0.0119230.007728-0.001278 / 0.008285 / 0.019547
Composite, 20 updates0.005131-0.004186 / 0.002776 / 0.0160390.007597-0.000298 / 0.009245 / 0.016977
Pure information, 1 update-0.001261-0.017241 / -0.000166 / 0.0163370.009800-0.000705 / 0.005019 / 0.030494

The negative means in the one-update arms are retained. A positive-state fraction is not a positive average certificate or a guarantee for an entire trajectory. Across all 18 configurations, refined positivity ranges from 0.23% at bias magnitude 0.25 to 57.03% with twenty updates. The original residual-composed bound remains nonpositive throughout these arms.

Confidence and numerical scope. All eligible rounds remain in the denominator, including coverage failures; envelope misses and inequality violations conditional on coverage are separate diagnostics. The reported audit observes zero of either. The 0.05 confidence allocation is per trajectory, not simultaneous across the entire experiment. Positive counts use a margin threshold of 10−1210^{-12}, while inequality checks use tolerance 2×10−102\times10^{-10}; these are numerical audit conventions, not a formal floating-point error certificate. All branch comparisons use the same simultaneous envelope, rather than selecting a favorable confidence event for each bound.

Larger grammar. Two additional configurations use 2,944 programs, horizon 8, and 32 balanced seeds each: 2,560 rounds. Refined positivity is 304/1,280 (23.75%) for composite control and 139/1,280 (10.86%) for pure IG, compared with interval positivity of 895/1,280 and 804/1,280. This expands the syntax/control space while retaining 16 hypotheses; it also changes padding/discount and cost normalization. We report the lower fractions without presenting them as successful epistemic or neural scaling.

Q3. Can the configurations, trajectories, and bootstrap specifications be supplied?

For the new study, the release inventory described in the experimental report includes configurations and source/engine hashes; explicit seed streams and ordered program/reference mappings; pre-update and pending QQ tables; policies and terminal laws; raw Monte Carlo sufficient counts; raw/clipped scores and envelopes; pre/post posteriors; sampled programs and observations; per-seed outcomes; certificate diagnostics; and exact bootstrap resampling indices. The raw arrays are lossless. The designated entry points are README_RELEASE.md for restoring the main archive and all data shards, and RELEASE_MANIFEST.json for the shard inventory and hashes. CERTIFICATE_FINDINGS.md, certificate_summary.csv, validation reports, and figures/ document the new audit and its evolution over time.

These materials support the new results. Availability of that release does not remove the historical reproducibility limitation, and a release inventory alone is not a substitute for accessible complete shards. The new configuration was specified locally before the enlarged runs, informed by an earlier fresh pilot; we do not describe it as public preregistration. Historical results and newly reproducible results remain separately identified.

Q4. Do gains persist against direct posterior-IG selection with reference mixing, accounting for computation?

Not against that adaptive baseline in the tested identification task. We compared one-step and twenty-step projected generators, greedy posterior-IG plus 5% reference mixing, and fixed reference sampling. Exact-IG and full-refresh Monte Carlo cohorts are separate, each with 256 balanced seeds per selector and 40 rounds: 2,048 trajectories and 81,920 rounds overall.

Writing ρ\rho for the original reference terminal law, the mixture is 0.05ρ+0.95gw0.05\rho+0.95g_w, with reference-proportional tie allocation among score maximizers. It guarantees support at least 0.05ρ0.05\rho. All adaptive methods use the same class-score procedure and budget at their own posterior states; support preservation is not exclusive to KL control.

Information accessSelectorTarget successMean capped timeEntropy AUC
Exact IGProjected, 1 update192/25625.4647.42
Exact IGProjected, 20 updates190/25625.9848.29
Exact IGGreedy + 5% reference256/25610.3421.93
Exact IGReference144/25630.2855.18
Full-refresh MCProjected, 1 update195/25625.4647.08
Full-refresh MCProjected, 20 updates187/25625.9848.01
Full-refresh MCGreedy + 5% reference256/25610.3121.97
Full-refresh MCReference144/25630.2855.18

In the MC cohort, projected-one-step minus reference success is +0.1992 [0.1172, 0.2891]; projected minus greedy is -0.2383 [-0.3555, -0.1328]. Projected selection has a mean capped time larger by 15.15 [10.92, 19.11] rounds. Twenty versus one update gives a success difference of -0.0313 [-0.0820, +0.0117] and capped-time difference +0.5156 [-0.5978, +1.6837]. More optimization does not establish faster learning here.

Target success means first reaching true-target posterior mass at least 0.9 with correct MAP within 40 rounds; a failure receives capped time 41. Entropy AUC includes the initial entropy. Paired intervals use 5,000 hidden-target-cluster resamples, retaining all sixteen repetitions per target; they are exploratory and not multiplicity-adjusted. All 30,720 MC adaptive-selector rounds remain in the coverage audit; the reported maximum clipped-score error is 0.00352044, with zero envelope misses at radius approximately 0.02.

Comments on the paper

1 comment

Overall response: the principal claim and the new evidence

We thank the reviewers for identifying both the implemented-policy analysis and the evidence needed to assess its significance. Theorem 4.14 and its supporting proofs were already in that submission; the new work reported here evaluates those certificates and adds the requested adaptive baseline.

Our principal claim is a conditional robustness guarantee for the implemented curriculum: its information advantage over the same-state IG-ablated optimum can be bounded directly by optimization error and shared information-estimation error, while the same projected iterate preserves completed-program support.

For brevity, write Δt0=Δt0,imp=J0,zt(πzt0)−J0,zt(πt)\Delta_t^0=\Delta_t^{0,\mathrm{imp}}=J_{0,z_t}(\pi^0_{z_t})-J_{0,z_t}(\pi_t), retaining the submitted definition. One branch of Theorem 4.14 is

\ge \frac{\Delta_t^0-\Xi_t}{\lambda_I}-E^\Delta_{t0}, \qquad \Xi_t=\min\left\\{2S_Lr_t,\frac{H_Lr_t^2}{2\alpha_{\min}}\right\\}.$$ Here $\Delta_t^0$ is the implemented policy's sacrifice of the non-IG objective, including its KL cost; $\Xi_t$ bounds full-surrogate-objective suboptimality; and $E^\Delta_{t0}$ charges a shared estimation error through the difference between the curricula. Thus $\Delta_t^0>\Xi_t+\lambda_I E^\Delta_{t0}$ is a sufficient condition for information advantage. Appendices F–G provide local quadratic, signed, and symmetric-contrast refinements. These are conditions on the actual policy, not an assertion that every learning round has a positive margin. **The new trajectory audit addresses empirical non-vacuity directly.** On fresh, balanced 192-program runs, the refined certificate is numerically positive in **1,950/5,120 composite one-update states (38.09%), 2,920/5,120 twenty-update states (57.03%), and 1,549/5,120 pure-information states (30.25%)**. Direct interval evaluation under Eq. (101) remains stronger on the same records. The original residual-TV-composed absolute-information bound remains nonpositive. The new results therefore demonstrate positive comparator-relative certification on a subset of evaluated frozen states, not universal positivity or a computational advantage over direct interval evaluation. **The adaptive-baseline comparison limits a different claim.** With the same Monte Carlo information-access budget, one-step projected selection identifies the target in 195/256 runs, versus 144/256 for the fixed reference; greedy posterior-IG selection with 5% reference mixing achieves 256/256. We report this unfavorable comparison rather than infer superiority to adaptive selectors from improvement over a fixed reference. Better certification of a regularized objective and faster identification are distinct outcomes. The evidence now has three roles: the submitted proofs establish the conditional theorem; the new frozen-state audit measures its numerical content; and separate rollouts compare learning behavior. Appendix J supplies an additional identification guarantee under finite, correctly specified, distinguishable likelihoods, not an IG-versus-reference speedup theorem.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.