acceptodds
Under review as a conference paper at ICLR 2027

Reusing Trainable States, Not Just Solutions: Policy-Archive Search at Test Time for Automated Discovery

21 views

Abstract

Test-time discovery can accumulate two coupled histories: the solutions found during search and the trainable policies produced by successive test-time updates. Test-time search with frozen language models reuses discovered solutions, while test-time training additionally updates the model using verifier feedback on newly generated candidates. Yet such methods typically continue training from the most recently updated policy; earlier policies are never again eligible for an update. Solution reuse preserves sampled outcomes, but not the earlier proposal distributions that produced them. Moreover, a policy's proposal behavior depends on the evolving context provided by previously discovered solutions—the Solution Archive. A later Solution Archive can therefore elicit different behavior from a frozen historical policy. Empirically, re-evaluating policies saved at different training stages under the same later Solution Archive reorders their relative performance—earlier states can even outperform the latest one. The starting point of each update is therefore a search decision, not a fixed sequence. We introduce PAST (Policy-Archive Search at Test Time), which extends reuse from solutions to trainable policies. Its Policy Archive allows learning to resume from a historical state under the current Solution Archive, creating a new continuation branch. We instantiate PAST with ProgressEMA, which uses observed capability and delayed parent-to-child progress to choose which archived policy receives the next update. Across three verifier-driven mathematical discovery tasks, PAST–ProgressEMA reaches better frontiers than the Current-only method, which always continues from the latest policy, while maintaining high per-epoch proposal quality even when the cumulative frontier plateaus. These results show that reusing trainable policies, not just solutions, can improve test-time discovery. Code is available at https://anonymous.4open.science/r/submission-artifact-105E/README.md.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time training, AI for science, automated discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eEUnOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAST, which extends TTT-Discover with an archive of historical training states, allowing optimization to resume from earlier checkpoints and create new branches. Its ProgressEMA selector combines historical best generation performance with observed parent-to-child reward improvements to choose the starting point for the next update. The method is evaluated on three automated discovery tasks and shows improvements over continuing exclusively from the latest checkpoint.

Critical strengths

The paper studies test-time training for automated discovery, which is an important research direction, particularly regarding how to allocate adaptation budgets effectively during search.

Critical weaknesses

W1. Limited conceptual novelty beyond checkpoint selection and branching. PAST retains TTT-Discover’s solution-side search and training objective, while introducing an archive of historical checkpoints and a rule for selecting which checkpoint to resume. This is a potentially useful extension, but the methodological advance appears incremental. The paper does not clearly establish what new algorithmic principle is introduced beyond checkpoint reuse, branching, and heuristic allocation.

W2. The presentation obscures a relatively simple underlying procedure. The method introduces numerous overlapping constructs, i.e., continuation states, two archives, pending and active nodes, direct and historical-fresh batches, and several within-round archive versions before providing a compact operational explanation. Although these details specify the implementation, their presentation makes the central contribution unnecessarily difficult to identify. A minimal algorithm first, followed by bookkeeping details, would improve clarity.

W3. The scope of the contribution is insufficiently delineated. The narrative alternates between presenting PAST as a general framework, advocating complete adapter–optimizer state reuse, and proposing ProgressEMA as an allocation strategy. These are distinct claims with different evidential requirements. The paper does not consistently distinguish which component constitutes the main methodological contribution, which components are inherited, and which are implementation choices. This weakens both the positioning and the interpretation of the results.

W4. Resource efficiency and long-horizon scalability remain insufficiently evaluated. The actual-rollout-matched comparisons are useful, but rollout counts do not capture generated-token volume, verifier execution cost, training cost, or end-to-end runtime. Moreover, retaining 50 trainable states requires approximately 127 GB in the reported configuration. Without fixed-storage-budget experiments or archive retention and pruning strategies, the practicality of the approach for longer discovery runs remains unclear.

Questions for authors

Please refer to weakness.

9 comments
AuthorsOpenReview ↗

Response to Reviewer eEUn(1/2)

The review underestimates PAST's algorithmic novelty by narrowly characterizing it as a combination of checkpoint selection, branching, and heuristic allocation, and overlooking the defining feature of PAST—dynamic Solution–Policy coupling—which addresses a fundamentally different search paradigm from the solution-side search employed by TTT-Discover.

  1. Core Algorithmic Contribution (W1 & W3)

The core contribution of PAST lies in the dynamic coupling between the Solution Archive and the Policy Archive, a mechanism that is particularly important for automated scientific discovery. Solutions serve both as discovery outcomes and as contexts for subsequent proposal generation. As new discoveries accumulate, historical policies encounter evolving contexts that may change their proposal behavior and potential for further learning. When reactivating a historical policy, PAST preserves the current Solution Archive, allowing the policy to generate fresh candidates and continue learning under the latest discovery context. In turn, historical policy continuation can produce new discoveries that further advance the Solution Archive. This bidirectional interaction constitutes PAST's central algorithmic innovation: the joint exploration of solution space and historical policy space, rather than merely storing checkpoints or applying a heuristic for checkpoint selection.

To make the contribution boundaries explicit, we distinguish the core method, allocation strategy, state-restoration mechanism, and inherited components:

  • PAST Continuation-Search Framework: The primary methodological contribution, making historical trainable-state selection and branching part of an evolving scientific discovery process.
  • ProgressEMA: One concrete online allocation strategy within the framework, combining observed capability and delayed progress to distribute continuation opportunities among available trainable states.
  • Adapter–Optimizer Restoration: The implementation mechanism for preserving and restoring trainable continuation states.
  • Inherited Components: TTT-Discover's solution-side PUCT reuse mechanism and adaptive entropic training objective.

Furthermore, to verify the effectiveness of PAST's individual components, we compare PAST against Frozen-history and Latest-only-extra to isolate the contribution of historical policy continuation, and also compare ProgressEMA against Capability-only, Progress-only, Uniform, and ScorePUCT to assess the effectiveness of its continuation-allocation strategy. PAST with ProgressEMA achieves the best observed upper bound among all compared methods. The comparisons with Frozen-history and Latest-only-extra demonstrate the additional value of allowing historical policies to resume training, while those with alternative selectors establish the practical benefit of combining capability and delayed progress signals to guide continuation allocation.

Table 1. Historical-continuation controls and selector ablations on AC1 at 3,200 actual rollouts.

Comparison targetMethod / selectorBest observed upper bound ↓
Complete methodPAST–ProgressEMA (ρ = 0.7)1.507896
Historical continuationFrozen-history1.511275
Historical continuationLatest-only-extra1.511275
Continuation allocationCapability-only / Best-observed-checkpoint (ρ = 1)1.510974
Continuation allocationProgress-only (ρ = 0)1.509323
Continuation allocationUniform1.508499
Continuation allocationScorePUCT1.512072

In the experiments above, Frozen-history allows historical policies to generate candidates under the current Solution Archive and update the archive, while training only the latest policy; Latest-only-extra allocates additional sampling to the latest policy. PAST, Frozen-history, and Latest-only-extra follow identical per-round batch-collection and policy-update schedules, with the same number of policy updates and actual rollouts. Frozen-history updates the latest policy using on-policy data generated by that policy. Capability-only (ρ = 1) always resumes the eligible checkpoint with the highest observed on-policy batch-best reward, implementing best-observed-checkpoint continuation within our protocol.

AuthorsOpenReview ↗

Response to Reviewer eEUn(2/2)

2. Presentation Clarity and Method Organization (W2 & W3)

We appreciate the reviewer's suggestion to improve presentation clarity. We agree that introducing detailed execution chronology and state bookkeeping too early can obscure the central algorithmic idea.

In the revised manuscript, we will restructure Section 3 to begin with a compact, four-step operational overview of PAST:

  1. Evaluate: Generate and verify a batch from the current policy and update the shared Solution Archive. This policy is the root initially and the child produced by the preceding round thereafter.
  2. Select: Choose the starting state for the next update from the Policy Archive.
  3. Recontextualize: Reuse the current batch if the current policy is selected; otherwise restore the selected historical state and collect fresh on-policy candidates under the updated Solution Archive.
  4. Continue & Branch: Update the selected policy using its on-policy data and save the resulting child in the Policy Archive as a new continuation option.

Following this high-level workflow, we will present the bookkeeping details—including pending/active nodes, direct/historical-fresh batches, and within-round archive updates—in a dedicated subsection. This separates the core algorithmic concept from implementation mechanics.

3. Computational Costs and Archive Storage (W4)

The resource measurements for nine rollout-matched trajectories across three methods and three tasks, covering 47,616 verifier-evaluated candidates, including prompt and completion token counts, cumulative verifier execution time, and NPU-hours for training and state restoration, are shown in Table 2.

At matched actual-rollout budgets, the results support the effectiveness of PAST's allocation of candidate-generation and policy-update budgets:

(1) PAST performs 50 optimizer updates on each task, compared with 92, 85, and 71 updates for the extended Current-only baselines on CP26, AC1, and Erdős, respectively.

(2) PAST reduces measured training/restoration NPU-hours by 50.2%, 35.1%, and 20.7%, and total prompt-plus-completion tokens by 7.3%, 6.4%, and 3.2%, respectively.

Table 2. Computational Resource Accounting .

Task (Rollouts)MethodTraining roundsTokens (M) Prompt / CompletionTrain + Restore (NPU·h)
CP26 (5,888)Current-only9211.50 / 61.6833.19
UG-TTT9212.54 / 47.7050.05
PAST5011.82 / 56.0116.54
AC1 (5,440)Current-only8522.88 / 46.4426.19
UG-TTT8520.15 / 35.7945.84
PAST5019.74 / 45.1717.01
Erdős (4,544)Current-only717.28 / 42.1519.02
UG-TTT717.56 / 39.4638.81
PAST507.81 / 40.0415.09

Regarding storage, Appendix F reports about 2.537 GB per recoverable state (~126.84 GB for 50 states). We retain complete archives primarily to support detailed retrospective analyses of historical continuation and Policy–Solution interactions. Crucially, a retrospective analysis of historical-state accesses reveals that only 8, 6, and 8 distinct historical states are needed to cover all recorded historical continuation selections on CP26, AC1, and Erdős, respectively. Retaining at most eight historical states plus the current state corresponds to an estimated footprint of approximately 22.83 GB.They indicate substantial potential for selective state retention.

Conclusion

In summary, PAST’s methodological contribution should be assessed as a historical continuation-search protocol coupled to an evolving Solution Archive. The comparisons above separately evaluate the historical-continuation decision, the allocation strategy within that framework, and the optimizer-state restoration choice, providing concrete experimental evidence for these distinct claims.

We respectfully ask the reviewer to reconsider the assessment of PAST's novelty and technical contribution in light of these clarifications and additional evidence.

We sincerely appreciate your valuable comments and constructive suggestions,which have helped us improve our work. We have made every effort to address your concerns in ourresponse. As the author-reviewer discussion period is coming to an end, we would be grateful to hear your thoughts on whether our revisions have sufficiently addressed the points you raised. We look forward to continuing the discussion and welcome any further feedback.

Reviewer_jjHzOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper introduces PAST, which augments test-time discovery with an archive of LoRA adapters and their optimizer states. Instead of always updating the latest policy, PAST can restore an earlier state, generate fresh candidates under the current solution archive, and create a new training branch. Its ProgressEMA selector combines observed proposal quality with delayed parent-to-child improvement. Experiments use Qwen3-8B on three mathematical construction tasks and compare against a local TTT-Discover implementation [1], UG-TTT, and alternative selectors. The complete system improves several reported frontiers, including under matched rollout budgets.

Reference

[1] Yuksekgonul, Mert, et al. "Learning to discover at test time." arXiv preprint arXiv:2601.16175 (2026).

Critical strengths
  1. The end-to-end comparison includes meaningful controls. Current-only and PAST share the backbone, training objective, and solution-reuse mechanism. The paper accounts for additional historical-policy rollouts and includes three-seed comparisons at a common rollout budget. At the 3,200-rollout cap, PAST wins all three CP26 and AC1 comparisons and two of three Erdős comparisons (Table 4). This supports an empirical benefit of the complete procedure.

  2. The diagnostics expose important limitations rather than reporting only favorable results. The paper reports weak checkpoint-ranking repeatability, unsuccessful held-out routing, mixed effects of optimizer restoration, and a CP26 tie at the longer matched-rollout endpoint. These results are informative, even though they weaken the proposed explanation.

Critical weaknesses
  1. The experiments do not establish what historical training adds over simpler checkpoint reuse. Population-based training [1] and its backtracking variants [2] already select and resume trainable checkpoints. PAST's evolving solution archive is a potentially meaningful distinction, but the paper neither develops this comparison nor tests the most relevant simpler controls. There is no control that samples from frozen historical policies without updating them, or uses a simple best-checkpoint rollback rule. PAST also changes the number of batches collected per update, so matching rollout totals does not isolate the effect of training from historical states. The observed gains therefore do not establish the specific contribution claimed by the method.

  2. The selection mechanism is insufficiently validated, and the motivating rankings are largely unstable. Equation (8) compares parent and child batches under different solution archives, mixing parameter-update effects with context changes and sampling noise. Appendix C.4 reports a repeated-parent rank correlation of only 0.0259, with a 95% interval of [-0.1714, 0.1980], and split-half winner agreement of 18.75%, close to the 16.67% chance level. Held-out contextual routing also performs worse than the latest checkpoint (Table 14). Thus, Figure 1's hindsight winners do not establish reliably exploitable checkpoint–context complementarity. No capability-only or progress-only ablation shows that the delayed-progress signal adds value.

References

[1] Jaderberg, Max, et al. "Population based training of neural networks." arXiv preprint arXiv:1711.09846 (2017).

[2] Zhang, Baohe, et al. "On the importance of hyperparameter optimization for model-based reinforcement learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

Questions for authors
  1. Contribution-isolating control: Under the same rollout and update schedule, how does PAST compare with generating candidates from archived policies while leaving those policies frozen? A Current-only control with the same additional-batch schedule would also help separate historical continuation from changes in update frequency.

  2. Selector validation: Does the delayed-progress statistic predict future continuation benefit when the parent, updated child, and a no-update control are evaluated under the same solution archive on fresh rollouts? How does ProgressEMA compare with the capability-only rule, arg⁡max⁡vs(v)\arg\max_v s(v)?

  3. Novelty clarification: What is the substantive algorithmic advance over checkpoint continuation and historical backtracking in [1, 2], beyond applying them to test-time discovery? Can a simple rollback or best-checkpoint continuation baseline establish the value of the proposed archive-allocation mechanism?

References

[1] Jaderberg, Max, et al. "Population based training of neural networks." arXiv preprint arXiv:1711.09846 (2017).

[2] Zhang, Baohe, et al. "On the importance of hyperparameter optimization for model-based reinforcement learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

10 comments
AuthorsOpenReview ↗

Response to Reviewer jjHz(1/2)

The review fundamentally mischaracterizes PAST's core algorithmic contribution. PAST is not simply another checkpoint-reuse or backtracking technique; it addresses a fundamentally different search problem and introduces a dedicated mechanism for automated scientific discovery.

(1) Our PAST is fundamentally different from Population-based training [1] and its backtracking variants [2] mentioned by the reviewer! PBT primarily improves model performance through population-based training, parameter inheritance, and dynamic hyperparameter optimization, while PBT-BT further revisits high-performing historical models and hyperparameter configurations. These methods neither explicitly study how historical policy continuation can be coupled with an evolving archive of verified scientific discoveries, which has proved very effective to advance the discovery frontier in our experiments. PAST explicitly incorporates the bidirectional interaction between historical policies and solution archives into the search process, making historical training-state selection and branching algorithmic decisions in scientific discovery rather than mere checkpoint rollback.

(2) The effectiveness of ProgressEMA as a policy selection mechanism is clearly demonstrated. Inspired by UCT and PUCT, ProgressEMA provides a principled strategy for allocating continuation efforts across historical policies with demonstrated statistical advantages. Furthermore, the experimental results, including the additional results in Tables 1 and 2 below, further validate the effectiveness of this allocation strategy and demonstrate the benefits of resuming training from historical policies.

1. Benefits of Historical Policy Continuation (W1&Q1)

During the rebuttal period, we add two controls to further assess the independent benefits of reopening historical training branches, to address the reviewer's concern about disentangling historical policy continuation from additional sampling and differences in update schedules.

  1. Frozen-history: Allows historical policies to generate candidates and update the Solution Archive, while training only the latest policy.
  2. Latest-only-extra: Collects additional candidates exclusively from the latest policy under the additional-sampling schedule, controlling for the effect of extra sampling.

At the same budget of 3,200 actual rollouts, PAST–ProgressEMA achieves the best observed upper bound (1.507896), outperforming both Frozen-history and Latest-only-extra (1.511275 each) as shown in Table 1. These results provide direct empirical evidence that reopening historical training branches yields additional benefits beyond frozen historical-policy reuse and extra latest-policy sampling under matched rollout and update schedules.

Table 1. Historical-continuation controls on AC1 at 3,200 actual rollouts.

MethodHistorical-policy useBest upper bound ↓
PAST–ProgressEMAHistorical proposals and continuation1.507896
Frozen-historyHistorical proposals only; train the latest policy1.511275
Latest-only-extraAdditional proposals from the latest policy1.511275

It should be noted that all three methods follow identical per-round batch-collection and optimizer-update schedules, with the same number of policy updates and actual rollouts.

AuthorsOpenReview ↗

Response to Reviewer jjHz(2/2)

2. ProgressEMA Provides Effective Online Continuation Allocation and Selection (W2&Q2&Q3)

During the rebuttal period, we perform additional selector ablations to verify the effectiveness of ProgressEMA in online continuation allocation. The ablation results are shown in Table 2. At 3,200 actual rollouts on AC1, ProgressEMA (ρ=0.7) achieves the best observed upper bound (1.507896), outperforming Uniform (1.508499), Progress-only (1.509323), Capability-only (1.510974), and ScorePUCT (1.512072). These results demonstrate that combining capability and delayed-progress signals yields better discovery outcomes than either signal alone or the alternative selection strategies.Capability-only (ρ = 1) always resumes the eligible checkpoint with the highest observed on-policy batch-best reward, implementing best-observed-checkpoint continuation within our protocol.

Table 2. Selector ablations on AC1 at 3,200 actual rollouts.

SelectorSelection signalBest upper bound ↓
ProgressEMA (ρ=0.7)Capability + delayed progress1.507896
UniformUniform selection1.508499
Progress-only (ρ=0)Delayed progress only1.509323
Capability-only/Best-checkpoint (ρ=1)Observed capability only1.510974
ScorePUCTScore-based PUCT1.512072

3. Summary of Contributions

Our new controlled experiments provide stronger evidence for two central contributions of PAST: ProgressEMA effectively guides online continuation allocation across historical policies, while reopening historical training branches achieves better verified frontiers than frozen historical-policy sampling or additional sampling from the latest policy. These findings also reinforce the distinction between the stability of static frozen-policy rankings and the value of dynamic training continuation. Together with PAST's coupled Solution–Policy search mechanism, these results provide stronger support for both its algorithmic novelty and its practical value in scientific discovery.

We respectfully ask the reviewer to reconsider the assessment of our paper's novelty and technical contribution in light of this evidence, and welcome further discussion of any specific concerns.


Thank you for the additional experiments and clarifications. The Frozen-history and Latest-only-extra controls, with matched rollout and update schedules, meaningfully address my concern about attributing the gains to historical continuation. The capability-only/best-checkpoint and progress-only ablations also strengthen the empirical case for ProgressEMA.

However, my question about whether delayed progress predicts continuation benefit under a common solution archive remains unanswered. The endpoint ablations do not separate parameter-update effects from changing contexts and sampling noise. Moreover, the new control and selector comparisons are limited to AC1 without reported replication or uncertainty, so their robustness remains unclear. The coupling with the evolving solution archive clarifies the intended contribution, although I retain reservations about the magnitude of the methodological advance over existing checkpoint-continuation approaches.

In recognition of the stronger evidence, I will raise my rating and maintain my confidence. Please incorporate the new controls and clarify the scope of the mechanism claims in the revision.

Reviewer_kj7uOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAST augments a verifier-driven solution archive with a policy archive containing recoverable LoRA adapters and optimizer states. Instead of always updating the latest policy, it can restore a historical state, generate on-policy candidates under the current solution archive, and create a new training branch. ProgressEMA allocates updates using observed batch-best capability and delayed parent-to-child progress. Experiments use Qwen3-8B on circle packing, an autocorrelation-bound problem, and the Erdős minimum-overlap problem. The paper compares continuation selectors, Current-only training, and a same-stage ensemble, and provides rollout-matched comparisons, seed replications, frozen-checkpoint probes, and optimizer-restoration analyses.

Critical strengths
  1. Making the trainable continuation state an explicit search variable is a useful extension of test-time discovery. The algorithm specifies when a newly trained child is evaluated, when a historical-fresh batch is required, and how adapter and optimizer state are restored. This makes the contribution operationally distinct from simply storing more candidate solutions (Sections 3.2–3.4).

  2. The main empirical advantage survives meaningful budget controls. At the primary trajectories’ matched rollout totals, PAST ties the extended baselines on CP26 and retains better frontiers on AC1 and Erdős (Appendix A.1). Across three sampling seeds at a common 3,200-rollout cap, its mean is better on all three tasks, with lane-wise wins of 3/3, 3/3, and 2/3 (Table 4). The disclosure of negative routing probes and task-dependent optimizer effects also makes the evidence more informative.

Critical weaknesses
  1. The evidence for the specific continuation-allocation mechanism is weaker than the evidence for the overall system. The delayed progress statistic compares parent and child batches evaluated under an evolving solution archive, so it combines update effects, context changes, and sampling noise. Appendix C.4 reports near-chance winner agreement and weak repeatability of frozen-policy rankings; these probes do not invalidate PAST, but they limit the claim that observed historical winners are reliably exploitable. Capability-only and progress-only controls would clarify what ProgressEMA adds beyond historical resampling and branching.

  2. Robustness and resource efficiency are established in a narrow regime. All main results use one 8B backbone and three mathematical discovery tasks; the selector and UG-TTT comparisons have less replication than the Current-only comparison. Matching candidate counts is valuable but does not match generated tokens, verification time, or total training compute. The documented archive footprint of about 126.84 GB for 50 states also matters for longer searches. These limitations constrain generalization, although they do not erase the controlled within-setting gains.

Questions for authors
  1. Mechanism validation: At matched actual-rollout budgets, how do capability-only, progress-only, and a control that resamples archived policies but continues training only the latest policy compare with ProgressEMA? Which evidence specifically supports the value of reopening training branches?

  2. Resource clarification: What are the total generated tokens, verifier time, training GPU time, and end-to-end elapsed time for the matched-rollout runs? The existing checkpoint-I/O measurements do not cover all of these costs.

  3. Robustness clarification: How sensitive are the outcomes to the capability/progress mixture and EMA weight, and are the selector comparisons reproducible across sampling seeds? A compact sensitivity analysis within the current tasks would materially strengthen the recommendation.

4 comments
AuthorsOpenReview ↗

Response to Reviewer kj7u(1/2)

We thank Reviewer kj7u for the thoughtful and constructive review. We particularly appreciate the recognition of our central contribution—treating trainable continuation states as explicit search variables—as well as the positive assessment of our budget-controlled experiments and transparent reporting of diagnostic results.We also acknowledge the reviewer's concerns and address them point by point below:

1. Mechanism Validation: ProgressEMA and Continuation Allocation(W1&Q1)

ProgressEMA combines capability and delayed-progress signals to allocate training opportunities. Our additional ablations (Table 1) show that, at 3,200 actual rollouts on AC1, ProgressEMA achieves the best observed upper bound (1.507896), outperforming Capability-only (1.510974), Progress-only (1.509323), Uniform (1.508499), and ScorePUCT (1.512072). These results support the effectiveness of combining the two signals for online continuation allocation.

Table 1. Selector ablations on AC1 at a matched budget of 3,200 actual rollouts.

SelectorSelection signalBest observed upper bound ↓
ProgressEMA (ρ = 0.7)Capability + delayed progress1.507896
UniformUniform selection1.508499
Progress-only (ρ = 0)Delayed progress only1.509323
Capability-only (ρ = 1)Observed capability only1.510974
ScorePUCTScore-based PUCT1.512072

Regarding the value of reopening historical training branches, Appendix D.2 already provides evidence from actual discovery trajectories. On CP26, a child trained from a historical policy advances the verified frontier from 2.634292 to 2.635977. On AC1, historical-fresh batches and direct batches from historical-continuation children account for 24.03% and 47.98% of the observed frontier improvement, respectively. We further introduce two controls:

  1. Frozen-history: Allows historical policies to generate candidates under the current Solution Archive (adding valid ones to the archive), while strictly training only the latest policy.
  2. Latest-only-extra: Allocates the extra sampling budget exclusively to the latest policy, testing whether increased sampling on the current policy alone can account for PAST's gains.

As shown in Table 2, at 3,200 actual rollouts on AC1, PAST–ProgressEMA achieves the best verified upper bound (1.507896), outperforming both Frozen-history and Latest-only-extra (1.511275 each). This provides direct empirical support for reactivating and continuing training from historical policy states: under an evolving Solution Archive, historical-policy continuation yields better verified discoveries.

Table 2. Historical-continuation controls on AC1 at 3,200 actual rollouts.

MethodHistorical-policy useBest upper bound ↓
PAST–ProgressEMAHistorical proposals and continuation1.507896
Frozen-historyHistorical proposals only; train the latest policy1.511275
Latest-only-extraAdditional proposals from the latest policy1.511275
AuthorsOpenReview ↗

Response to Reviewer kj7u(2/2)

2. Computational and Storage Costs(W2&Q2)

We compile resource statistics across nine matched-rollout trajectories (covering 47,616 verified candidates), and compare our PAST with Current-only and UG-TTT at matched actual-rollout budgets in Table 3. It can be seen that PAST achieves competitive or better discovery outcomes with fewer optimizer updates and substantially lower training/restoration costs.

In particular, compared with Current-only, PAST requires only 50 updates on each task, versus 92, 85, and 71 updates for the extended baselines on CP26, AC1, and Erdős, respectively. This translates into 50.2%, 35.1%, and 20.7% fewer training/restoration NPU-hours, alongside 7.3%, 6.4%, and 3.2% fewer total prompt-plus-completion tokens, respectively. At these matched-budget endpoints, PAST achieves better verified frontiers on AC1 and Erdős while matching the baseline on CP26.

Table 3: Computational resource comparison at matched actual-rollout budgets.

Task (Rollouts)Method (Rounds)Tokens P/C (M)Train+Restore (NPU·h)
CP26 (5,888)Current-only (92)11.50 / 61.6833.19
UG-TTT (92)12.54 / 47.7050.05
PAST (50)11.82 / 56.0116.54
AC1 (5,440)Current-only (85)22.88 / 46.4426.19
UG-TTT (85)20.15 / 35.7945.84
PAST (50)19.74 / 45.1717.01
Erdős (4,544)Current-only (71)7.28 / 42.1519.02
UG-TTT (71)7.56 / 39.4638.81
PAST (50)7.81 / 40.0415.09

Regarding storage, Appendix F reports about 2.537 GB per recoverable state (~126.84 GB for 50 states). We retain complete archives primarily to support detailed retrospective analyses of historical continuation and Policy–Solution interactions. Crucially, a retrospective analysis of historical-state accesses reveals that only 8, 6, and 8 distinct historical states are needed to cover all recorded historical continuation selections on CP26, AC1, and Erdős, respectively. Retaining at most eight historical states plus the current state corresponds to an estimated footprint of approximately 22.83 GB.They indicate substantial potential for selective state retention.

3. Robustness (W2 & Q3)

During the rebuttal period, we further evaluate hyperparameter sensitivity on AC1 across three sampling seeds (Table 4). PAST retains its advantage in mean performance over Current-only under both tested parameter changes: setting ρ to 0.3 while keeping η at 0.95 yields a mean upper bound of 1.510358, whereas setting η to 0.8 while keeping ρ at 0.7 yields 1.508962, the lowest mean among the tested configurations. These results show that the default configuration’s advantage is reproduced across all three sampling seeds and that PAST maintains an advantage in mean performance under both tested hyperparameter changes.

Table 4: Hyperparameter sensitivity on AC1 across three sampling seeds.

Method / configurationS0S1S2Mean ↓
Current-only1.5104041.5155941.5093161.511771
PAST–ProgressEMA, ρ=0.7,η=0.95\rho = 0.7, \eta = 0.951.5078961.5129811.5077991.509559
PAST–ProgressEMA, ρ=0.3,η=0.95\rho = 0.3, \eta = 0.951.5127101.5093311.5090321.510358
PAST–ProgressEMA, ρ=0.7,η=0.8\rho = 0.7, \eta = 0.81.5088881.5085771.5094211.508962

We sincerely appreciate your valuable comments and constructive suggestions,which have helped us improve our work. We have made every effort to address your concerns in ourresponse. As the author-reviewer discussion period is coming to an end, we would be grateful to hear your thoughts on whether our revisions have sufficiently addressed the points you raised. We look forward to continuing the discussion and welcome any further feedback.

Thank you again for your valuable time and constructive guidance throughout the review process. Since the discussion period closes in less than 24 hours, we wanted to gently follow up to see if our responses and new results have addressed your concerns, or if there is anything else we can clarify before the deadline.

Comments on the paper

1 comment

Summary of Revisions

We thank all three reviewers for their careful reading and constructive feedback. We appreciate the recognition of trainable continuation states as explicit search variables (kj7u), the value of budget-controlled comparisons and transparent diagnostic results (kj7u, jjHz), and the suggestions to clarify our methodological contribution and presentation (jjHz, eEUn).

The discussion focus on four questions:

(i) Does historical policy continuation provide benefits beyond frozen historical-policy sampling or additional sampling from the latest policy?

(ii) Does ProgressEMA benefit from combining capability and delayed-progress signals, and is delayed progress informative about future continuation gains?

(iii) What distinguishes PAST's contribution from checkpoint reuse, and can the method be presented more clearly?

(iv) How robust are the results, and what are the computational and storage costs?

We briefly summarize our responses below. The corresponding experiments and clarifications have been incorporated into the revised manuscript.

(i) Historical continuation provides additional benefits under matched budgets. We introduce Frozen-history and Latest-only-extra controls with matched rollout and update schedules. Across two sampling seeds on each of AC1 and Erdős, PAST–ProgressEMA achieves the best mean verified upper bound, supporting the benefit of reopening historical training branches beyond these simpler alternatives. (Sec. 5.2, Table 2; Appendix A.3.)

(ii) The two allocation signals provide complementary empirical value. ProgressEMA achieves better mean results than Capability-only and Progress-only on both AC1 and Erdős. In addition, fixed-Solution Archive experiments at three AC1 branch points show larger one-step continuation gains for parents with higher pre-recorded delayed-progress EMA values. We interpret ProgressEMA as an online allocation heuristic rather than a guaranteed estimator of future continuation value. (Sec. 5.2, Tables 2–3; Appendix A.5.)

(iii) We have substantially rewritten Section 3. The revised method begins with a four-step operational description—Generate & Verify, Select, Recontextualize, and Branch—followed by the coupled archive representations and ProgressEMA selection rule. We also clarify the distinction between PAST's continuation-search framework, ProgressEMA as an allocation strategy, adapter–optimizer restoration as an implementation mechanism, and the components inherited from TTT-Discover. (Secs. 1–3; Fig. 2; Algorithm 1.)

(iv) We have strengthened the robustness and resource analysis. Additional AC1 sensitivity experiments across three seeds preserve the mean advantage over Current-only under the tested changes to the selection mixture and EMA weight. At matched task-level rollout totals, PAST reduces measured training/restoration NPU-hours by 20.7%–50.2% and total generated tokens by 3.2%–7.3% relative to Current-only. We also report retrospective historical-state retention statistics and clarify the limitations of archive storage. (Sec. 5.1; Appendices A.4 and F.)

We thank the reviewers again for their detailed suggestions, which have helped us strengthen the experimental evidence and clarify the scope of our contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.