acceptodds
Under review as a conference paper at ICLR 2027

A Protocol for Auditing Replay-to-Online Transport in Recursive Self-Improvement

Abstract

Historical replay can identify a promising strategy without showing how that strategy will behave when new tasks arrive. We present a registration and reporting protocol for this replay-to-online boundary. The protocol freezes the candidate set and replay winner before revealing a separately registered live block, limits each tested strategy to the input allowed for the current task, and compares the winner with the incumbent on the same tasks and registered repeats. It reports replay gain ΔR, paired live gain ΔL, and their descriptive difference B = ΔR - ΔL. A two-world construction shows why the same replay record can be consistent with opposite live gains when replay and live outcomes are otherwise unrelated. The checker rejects duplicate task identities, incomplete replay coverage, and declared protocol violations; unsupported or missing observations remain UNKNOWN or PARTIAL. We include synthetic checks, a deterministic fixed-policy system check, and a finite model-assisted pilot. These artifacts test the contract and illustrate its reporting boundary; they do not establish predictive validity, population generalization, statistical significance, or online performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
recursive self-improvement, replay protocol, transport audit, decision selection, safe AI, offline-to-online evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pVcvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem in recursive self-improvement that a winner strategy in historical replay may not show improvement in the futures task. To solve this, the authors propose a framework that registering all strategies first, freezing the winner after replay, and then test the winner and original strategy on new tasks. Based on this framework, the authors define replay gain, live gain and their difference. The authors further define regret and the ranking of the replay winner on the new tasks. The authors validate their claim in the experiment with multiple LLMs.

This paper studies a very important problem in recursive self-improvement, how to use the information in replay to update the strategy.

What I concern most is the novelty and significance of this paper. Since the observation that replay winner may not perform good in the future tasks and auditing on additional tasks are relatively standard, it is hard for me to believe the contribution of this paper is enough for ICLR.

  1. Could the authors clarify what contribution the proposed framework provides compared to regular additional auditing or holdout evaluation?
Critical strengths

This paper studies a very important problem in recursive self-improvement, how to use the information in replay to update the strategy.

Critical weaknesses

What I concern most is the novelty and significance of this paper. Since the observation that replay winner may not perform good in the future tasks and auditing on additional tasks are relatively standard, it is hard for me to believe the contribution of this paper is enough for ICLR.

Questions for authors
  1. Could the authors clarify what contribution the proposed framework provides compared to regular additional auditing or holdout evaluation?
Reviewer_is1NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies AI systems in which an improved strategy is selected based on historical data. The objective is to evaluate such proposed improvements. For example, a system can test several candidate strategies by rerunning previous tasks and select the strategy with the highest replay score. The main observation is that a strategy that performs well on historical data does not necessarily outperform the system's current strategy on new tasks. This issue is particularly important for recursive self-improvement of AI systems, where a system repeatedly proposes, evaluates, and adopts changes to its own behavior.

The paper proposes a registration and reporting mechanism for assessing whether improvements observed in historical replay translate into better performance on new tasks. The candidate strategies and the selected winner must be frozen before new tasks are revealed. The winning strategy and the incumbent strategy are then evaluated on the same tasks under similar conditions. The protocol specifies how to identify tasks, aggregate repeated runs, and handle missing results or violations (e.g., information leakage). It also distinguishes between showing that the selected strategy beats the incumbent and showing that it is the best among all candidates. An accompanying software checker validates the submitted records and computes descriptive performance differences. The paper's evaluation includes synthetic examples, experiments involving fixed heuristic strategies, and a small pilot in which language models help select candidates.

Critical strengths

The paper tackles an important question: whether a strategy that looks better on historical data actually improves performance on new tasks. It proposes useful safeguards, including freezing candidates before evaluation and testing under matched conditions.

Critical weaknesses
  • The novelty of the proposed system is not well established. Much of the protocol seems to follow existing principles of held-out evaluation, paired comparisons, and experimental record-keeping. The paper would benefit from a clearer explanation of the new insights or capabilities that its registration contract and checker provide.

  • The validation is not well aligned with the motivating application of the paper. In particular, the pilot selects one fixed strategy rather than evaluating a system that actually modifies itself.

  • The related work discussion misses some relevant work, including Generalization in Adaptive Data Analysis and Holdout Reuse (Dwork et al., NeurIPS 2015), The Ladder: A Reliable Leaderboard for Machine Learning Competitions (Blum and Hardt, ICML 2015), and High Confidence Policy Improvement (Thomas et al., ICML 2015). While these papers have different assumptions and goals, discussing them would help readers better understand the scope and novelty of the proposed approach.

Questions for authors

1- In terms of scientific contributions, what does the protocol offer beyond standard held-out testing and paired comparisons? How does it relate to previous work on adaptive holdout reuse and reliable leaderboards?

2- Have you tried the checker on an actual self-improving AI system? An example where it catches a problem missed by standard evaluation methods would help establish its value.

Reviewer_LK4DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a protocol for comparing a replay-selected strategy with an incumbent on a separately registered block of new tasks. The winner is frozen before reveal, and the protocol reports the replay gain \Delta R, the paired live gain \Delta L, and their difference B. An offline checker recomputes these quantities and assigns explicit states to incomplete or invalid records. The evidence consists of synthetic fixtures, a fixed-heuristic system check, and a pilot in which eight LLMs choose among fixed heuristics.

Critical strengths
  • The paper separates missing, unsupported, and invalid observations instead of imputing them, and withholds selection under incomplete replay coverage. Its counterexamples show concrete cases where naive evaluation gives wrong answers.
Critical weaknesses
  • The protocol largely formalizes established holdout practice. The checker verifies arithmetic and declared fields, but not custody, chronology, or candidate integrity. \Delta_R is subject to winner’s-curse optimism that B cannot separate from genuine shift. PARTIAL results are not bounded, even though scores lie in [0,1].

  • No demonstration in an actual RSI setting. The pilot uses fixed heuristics without candidate generation. Its contrasts mix valid selections with fallback: MiniMax has zero valid responses yet the only positive D. The protocol is never applied to an existing RSI pipeline, and the replay-versus-direct study in Section 7 remains unrun.

Questions for authors

see above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.