acceptodds
Under review as a conference paper at ICLR 2027

Auditing Harness Evolution: Selected Gains, Persistent Edits, and Runtime Behavior

Abstract

An accepted harness edit can earn selected credit without retaining that gain, and can persist without activating its intended behavior. We introduce the RSI Information Ledger to connect disclosure accounting, frozen-state comparisons and execution traces. Across six post-selection 4B coding lineages, the evolved harness gains 1.55 percentage points on the MBPP reserve, with no detectable advantage over an audit-selected single edit at matched candidate-evaluation budget and no detectable positive terminal interaction. An exploratory search under Qwen3.5-4B and a new serving configuration also retains gains; its lineage-minus-single-edit difference is −1.899 points. This comparison matches scheduled candidate slots, with unequal realized scoring and token costs, and establishes neither superiority nor equivalence. Registered eight-seed follow-ups detect neither an acceptance-margin contrast nor a fixed-gate disclosure effect on the audit–reserve gap. In a complementary 27B telecom case, an accepted guard never activates in 3,960 recorded checks. Local wrapper controls yield differing next actions in selected contexts, while synthetic triggers establish conditional blocking and feedback delivery. A subsequent four-condition panel comprises 592 full-task executions on 74 reused tasks. It records zero guard hits and separates content-clearing behavior from the guard; all task-score contrast intervals cross zero, leaving benefit, harm and practical equivalence unresolved. These studies distinguish retained improvement from successive-edit value, and rule persistence from runtime behavior; they do not establish a general causal account of recursive improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, RSI, Harness Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_7shROpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits retained gains and runtime behavior in evolving agent harnesses. Six 4B lineages gain 1.55 points on an MBPP reserve set versus 1.44 for a same-stream single edit (paired difference +0.11; 95% CI [−0.62, +0.83]). Follow-ups establish neither lineage advantage nor equivalence. In a 27B case, the guard has zero activations in 3,960 checks; task-level benefit remains unresolved.

Critical strengths

The same-stream single-edit baseline separates retained gain from lineage value. Wrapper controls help distinguish stored rules from runtime behavior. The authors report uncertainty candidly.

Critical weaknesses

The lineage contrast is inconclusive; follow-ups have cost or service confounds. Telecom tests are diagnostic, not evidence of guard benefit. The Information Ledger is nominal accounting, not a formal leakage bound; no public artifact is supplied to reproduce model runs.

Questions for authors
  1. Which Ledger properties are formal guarantees, and what leakage channels are outside its nominal charges? 2. Can lineage and single-edit results be compared at matched realized cost and fixed serving version? 3. Can code, configurations, and records needed to reproduce the results be released?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7shR (Part 1/2: Formal Guarantees vs. Nominal Accounting and 27B Diagnostic Role)

Thank you for your incisive review. We have uploaded the revised manuscript (inline additions and new headings/captions in blue) and the supplementary material. Part 1 answers Question 1 (formal guarantees, conditional data-processing inequality, nominal counters, and external channels) and clarifies the 27B diagnostic; Part 2 answers Question 2 (35M-token cost-matched and fixed-serving comparisons) and Question 3 (the code and data in the supplementary material).

1. Question 1: Formal Guarantees vs. Nominal Counters and Uncovered Channels in the Information Ledger

In our framework and released implementation, Ledger properties separate strictly into three categories:

A. Conditional Data-Processing Inequality and Cryptographic/Architectural Guarantees

  1. Conditional information-boundary inequality: Under the conditional disclosure boundary Xaud→T→MG∣ZX_{\mathrm{aud}} \to T \to M_G \mid Z (where ZZ is the audit-blind initialization and candidate pool, TT is the disclosed audit transcript across screening rounds, and MGM_G is the selected terminal state), the data-processing inequality formally gives I(MG;Xaud∣Z)≤I(T;Xaud∣Z)≤H(T∣Z).I(M_G; X_{\mathrm{aud}} \mid Z) \le I(T; X_{\mathrm{aud}} \mid Z) \le H(T \mid Z).
  2. Cryptographic state commitment: Every harness state MtM_t (all prompt, example, repair, and decoding fields) is canonically serialized and bound to a SHA-256 digest before evaluation, guaranteeing byte-level state identity across screening, replay, component factorials, and external holdouts.
  3. Architectural non-exposure of lockbox/reserve splits: The search loop loads only the dev and 300-task audit splits; reserve/lockbox prompts, references, and test verdicts are absent from proposer inputs and acceptance predicates.
  4. Deterministic single-edit counterparty (S) and overlap checks: Given a committed candidate stream and audit scores, S and L are deterministic functions, and the transcript audit module deterministically recomputes the saved overlap-match records and summary statistics.

B. Nominal Accounting Quantities (Not Direct Estimates of H(T∣Z)H(T \mid Z) or Performance Bounds)

  • Feedback granularity and query counters: Logging 1-bit commit decisions vs. rich per-item diagnostics across acceptance margins 1 and 5, nominal encoding charges, and query counts provides nominal channel accounting. Nominal charges do not estimate I(T;Xaud∣Z)I(T; X_{\mathrm{aud}} \mid Z) or H(T∣Z)H(T \mid Z) or give a direct finite-sample performance bound; under adaptive text edits, both 1-bit margin-5 and rich margin-1 selection widen the Figure 2 audit-reserve gap by ~2 to 2.5 pp.

C. Leakage and Confounding Channels Not Covered by Nominal Loop Counters (and How We Audit Them)

  1. Analyst post-selection across studies: Tested by freezing all 42 M0/L/S states (18 original 4B + 24 Qwen3.5-4B) and evaluating once on a fresh 358-task external benchmark (Part 2).
  2. Pretraining familiarity in frozen base LLM weights: Addressed via paired within-model contrasts (L - M0, S - M0, L - S) and external validation on a 2026 benchmark.
  3. Inductive bias in candidate-bank construction and schedules: Logged via explicit candidate provenance and tested by comparing the 524-entry fixed bank against the 800-round state-conditioned online proposer pool.
  4. Serving-stack drift across epochs: Tracked via serving-tag digests and configuration hashes, and controlled via unified fixed-serving remeasurement.

In the revised manuscript, Section 3.2 now includes a summary table and explicit discussion distinguishing (A), (B), and (C).

2. Clarification on the 27B Telecom Diagnostic

We agree that the original 27B guard tests are diagnostic and do not show task benefits from the guard rule: although the transfer-guard patch was retained by screening (p = 0.0078), it triggered 0 / 3,960 times in natural episodes, while the outer wrapper (C1, guard disabled) altered prompts and first semantic responses across 110 / 148 trajectory pairs (C1 - C0 and C1 - C3), and all four Table 3 reward contrasts cross zero (C1 - C0 = -4.730 pp, 95% CI [-15.541, +6.757]; C1 - C3 = -2.027 pp, 97.5% CI [-14.189, +10.135]; C2 - C1 = +4.054 pp, 97.5% CI [-2.703, +10.811]; C3 - C0 = -2.703 pp, 95% CI [-8.784, +3.378]). We have sharpened the Section 5 opening and added a Ledger Audit Summary callout so this diagnostic role is unmistakable.

Alongside that single-lineage runtime audit, we also conducted eight separately seeded 27B telecom searches (four rounds per arm; frozen M0/L/S states evaluated across all 74 tasks with four repeats; Appendix H.1, Table 35), yielding L - M0 = +2.58 pp, S - M0 = +2.53 pp, and L - S = +0.04 pp (95% across-seed t CI [-3.25, +3.33]; four lineages retain one edit and four retain M0), corroborating that multi-step search resolves no advantage over single-edit selection.

(Continued in Part 2/2)

AuthorsOpenReview ↗

Response to Reviewer 7shR (Part 2/2: Cost-Matched and Fixed-Serving Comparisons, and Supplementary Material)

3. Question 2: Comparing Lineage (L) vs. Single-Edit (S) Under Matched Actual Cost and Fixed Model-Serving Version

We address both matched actual token expenditure and fixed model-serving version in the supplementary material:

  1. Matched Actual Search Token Budget Under Fixed Serving:

    • Across the full 400-slot schedule (O), multi-step search consumes 514,047,575 realized search tokens vs. 308,649,030 for single-edit selection (+66.55% more tokens in L because retained examples and repair lengthen prompts/completions), yet L establishes no statistically resolved advantage over S (L - S = -1.90 pp, 95% t7t_7 CI [-3.85, +0.05], p=0.0625p = 0.0625). Because L and S consume the identical 524-entry candidate bank and per-seed candidate stream, historical one-time bank-generation cost is an identical additive constant for both routes.
    • To compare L and S at equal realized search token expenditure, using per-round token accounting across all 2,362,318 physical Qwen3.5-4B requests, we selected each arm's longest complete round prefix with cumulative realized search tokens ≤35,000,000\le 35,000,000 (34.55M to 34.91M realized tokens across the 16 arms; paired ∣L−S∣|L - S| budget difference ≤0.93%\le 0.93\%; 50 to 67 lineage rounds vs. 90 to 91 single-edit rounds), and evaluated all 24 resulting frozen M0/L/S panels (11,376 cells, 11,385 calls) on the 474-task MBPP reserve in one unified batch under pinned vLLM 0.18.0 (Table 18):
      • Mean pass rates: M0 = 56.83%, L = 60.94%, S = 62.92%
      • Primary contrast (L - S): -1.98 pp (-1.978 pp, 95% t7t_7 CI [-4.38, +0.43], exact two-sided sign-flip p=0.1172p = 0.1172)
      • Secondary contrasts: L - M0 = +4.11 pp (+4.114 pp, 95% t7t_7 CI [+2.13, +6.10], p=0.0078p = 0.0078); S - M0 = +6.09 pp (+6.092 pp, 95% t7t_7 CI [+5.42, +6.77], p=0.0078p = 0.0078).
    • Thus, under both equal scheduled candidate slots (where L expends +66.55% more tokens) and equal 35M realized search tokens under pinned vLLM 0.18.0, we find no statistically resolved lineage advantage over same-stream single-edit selection.
  2. Additional Fixed-Serving Comparisons Across Evaluation Regimes (Included in the Supplementary Material):

    • Same-serving MBPP reserve: Under a shared serving tag, the 6 original 4B seeds yield L - M0 = +1.55 pp, S - M0 = +1.44 pp, L - S = +0.11 pp (95% CI [-0.62, +0.83], p=0.84375p = 0.84375; Table 2), and same-serving 8-seed remeasurement in the public-check study yields L - S = +0.185 pp (95% CI [-0.770, +1.139]).
    • Fixed-serving Qwen3.5-4B under matched evaluation count (B) and matched candidate count (C): Across all 8 seeds under pinned vLLM 0.18.0, matching completed audit evaluations (L - B = -1.90 pp, 95% t7t_7 CI [-3.85, +0.05], p=0.0625p = 0.0625) or matching each seed's exact lineage candidate count KL∈[317,400]K_L \in [317, 400] (L - C = -1.56 pp, 95% t7t_7 CI [-3.34, +0.23], p=0.1016p = 0.1016) likewise shows no statistically resolved lineage advantage over S.
    • Fixed-serving online pool restoration: Completing all 800 single-edit rounds (756 initial + 44 restored) across the state-conditioned online pool (8 seeds, 3,175 proposals) and remeasuring M0, L, S in one batch yields L - S = -0.13 pp ([-0.46, +0.19]).
    • Fixed-serving 358-task external holdout (Table 19): Evaluating all 42 frozen states once under pinned vLLM 0.18.0 (15,036 cells) yields L - S = -1.40 pp ([-2.87, +0.07], p = 0.0625) on the 6 original 4B seeds and L - S = -0.66 pp ([-2.42, +1.09], p = 0.4063) on the 8 Qwen3.5-4B seeds.

4. Question 3: Release of Code, Configurations, and Records

We have uploaded the supplementary material (documented in the root README.md and EXPERIMENTS.md):

  • Source Code and Configurations: Complete 4B search, scoring, and ledger code; the 524-entry candidate bank and runner; Qwen3.5-4B configurations and runner; and the 27B telecom harness, C0 to C3 condition adapters, and transfer-guard specification.
  • Frozen States and Verdict Records: All 18 original 4B and 24 Qwen3.5-4B M0/L/S full specifications, checkpoints, and Qwen3.5-4B round records; all 15 paper verification modules; the restored 800-round online pool records; the 358-task external holdout dataset and 15,036 cell records; the 35M-token prefix states, per-round search cost logs, and 11,376 cell records; and the eight-seed 27B search states, screening records, paired terminal rewards, and CPU verification scripts.
AuthorsOpenReview ↗

A brief follow-up

Dear Reviewer 7shR,

Thank you again for suggesting a comparison at matched cost. In Table 18, we compare multi-step search with single-edit selection under the same cap of 35 million search tokens per arm. We evaluated the resulting frozen states together under the same serving configuration.

If you’ve had a chance to read our response, we’d appreciate a brief note on whether it addresses your concerns.

Reviewer_7VQcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits recursive harness evolution by separating selected credit, retained endpoint gains, persistent edits, and runtime activation through an RSI Information Ledger. It evaluates frozen-weight coding harnesses and a telecom guard, using matched single-edit comparisons, disclosure controls, frozen checkpoints, and execution traces to characterize what the observed improvements do and do not establish.

Critical strengths

This work separates several claims that are often confused in harness evolution works. The matched lineage versus audit-selected single-edit comparison and the corresponding Qwen3.5 extension, distinguish retained endpoint improvement from additional value of a sequence of edits. The evidence is interpreted as inconclusive about a lineage advantage, rather than being overclaimed into an equivalence/recursive improvement.

The runtime audit is useful because it checks what actually happens, not just an old selection score. It shows the guard never naturally activates, while the wrapper can still change requests and some actions; task-level benefit is still unresolved. So persistence, conditional behavior, and task benefit are clearly separate things.

Critical weaknesses

Section 3.2 and Section 7 state that researchers inspected MBPP reserve aggregates between cells, and Appendix E document extensive reuse across pilots, registered cells, post-selection analyses, and later extensions. The optimization loops themselves did not receive lockbox outcomes, which preserves an important boundary, but researcher access can still affect which cells, analyses, endpoints, and follow-ups are pursued and emphasized. This may be problematic because the central retained-gain result and lineage versus single-edit comparison in Table 2 are both evaluated on this repeatedly accessed reserve. The one-time HumanEval evaluation (Section 4.6) is a useful partial mitigation, but it is only 164 tasks and does not independently validate every main comparison. A genuinely fresh final reserve, evaluated once after freezing all analysis choices, would fix this problem.

Additionally, the reproducibility statement (P.9) says that code and data artifacts are not distributed, while Appendix H explains that the reported reconstruction checks rely on retained internal archives, private score vectors, request records, source snapshots, and infrastructure bindings. The extensive reporting and internal checks increase confidence, but they do not permit an external reader to verify the search histories, audit disclosures, state identities, or terminal score matrices on which the quantitative conclusions rely. I beblieve releasing anonymized verdict matrices, candidate/state hashes and metadata, analysis scripts, and the non-sensitive harness code would resolve much of this concern.

Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7VQc

Thank you for your positive and thoughtful assessment. You identified two concerns and stated what would resolve each: (1) "evaluating once on a truly fresh final holdout after freezing all analysis choices," and (2) "releasing anonymized verdict matrices, candidate/state hashes and metadata, analysis scripts, and non-sensitive harness code." We have implemented both recommendations and uploaded the revised manuscript (inline additions and new headings/captions in blue) and the complete code and data in the supplementary material.

1. One-Shot Evaluation on a Truly Fresh External Final Holdout

Following your recommendation, after freezing all 42 terminal harness states and analysis choices, we executed a single-pass, zero-reselection evaluation on an independent external benchmark:

  • External Benchmark: The Code Complexity Python Benchmark (CC BY 4.0), comprising 358 Python problems (more than double HumanEval's 164 tasks). Prompts display the problem description, function signature, and 5 public example tests; 10 held-out assertions run inside the unchanged sandbox scorer and are never exposed to the model.
  • Frozen States and Protocol (42 states × 358 tasks = 15,036 cell evaluations; 15,048 physical calls including 12 syntax-repair calls total, 6 per cohort):
    1. Original 6-seed cohort (Qwen3-4B-Instruct-2507): All 18 frozen states (M0, L, S across all 6 original seeds, 6,444 cells), verified by SHA-256 against the committed terminal specifications.
    2. Qwen3.5-4B 8-seed cohort: All 24 frozen states (M0, L, S across all 8 registered seeds, 8,592 cells), verified by SHA-256 against the committed terminal specifications. All 15,036 cells were evaluated once under pinned vLLM 0.18.0 (temperature = 0.0, except the two frozen original L states specifying temperature = 0.2), with zero post-evaluation re-selection.
Metric (on 358 Fresh External Tasks)Original 4B (6 Seeds)Qwen3.5-4B (8 Seeds)
Initial baseline pass rate (M0M_0)80.63%76.50%
Multi-step lineage pass rate (LL)80.31%75.94%
Single-edit baseline pass rate (SS)81.70%76.61%
Primary contrast: L−SL - S (95% tt CI, exact pp)-1.40 pp ([-2.87, +0.07], p=0.0625p = 0.0625)-0.66 pp ([-2.42, +1.09], p=0.4063p = 0.4063)
Multi-step gain over initial: L−M0L - M_0 (95% tt CI)-0.33 pp ([-1.52, +0.87], p=0.6563p = 0.6563)-0.56 pp ([-1.79, +0.67])
Single-edit gain over initial: S−M0S - M_0 (95% tt CI)+1.07 pp ([+0.17, +1.97], p=0.0625p = 0.0625)+0.10 pp ([-0.52, +0.73])

Key Findings:

  1. Primary contrast (L - S) on fresh external tasks: On an unseen 358-task external benchmark evaluated once after freezing all choices, multi-step lineage evolution (L) shows no statistically resolved advantage over single-edit selection (S) in either cohort (L - S = -1.40 pp, p = 0.0625 on the 6 original seeds; L - S = -0.66 pp, p = 0.4063 on the 8 Qwen3.5-4B seeds).
  2. Single-edit transfer vs. accumulation: Where prompts already include function signatures and 5 public example tests (higher baseline M0 of 76.50% to 80.63%), multi-component edits selected on the 300-task MBPP audit split do not gain over M0 (-0.33 pp and -0.56 pp), whereas on the original 6 seeds the single instructions edit S transfers a modest gain (+1.07 pp, [+0.17, +1.97], p = 0.0625), corroborating our conclusion on an external holdout (Section 4.6; Appendix E.9, Table 19).

2. Supplementary Material

The supplementary material (documented in README.md) includes all requested verification artifacts:

  • Verdict matrices and score records:
    • All 15 paper verification modules covering Table 2, Figure 2, Section 4.5, completion studies, Qwen3.5-4B replications, positive controls, public-check selection, and the 27B studies.
    • Cell-level verdicts, extracted code, rendered prompt SHA-256 digests, and token usage records for the 358-task external holdout.
    • Cell records and 100-round search token logs for our 35M-token search-budget prefix evaluation across all 8 Qwen3.5-4B seeds (M0 = 56.83%, L = 60.94%, S = 62.92%, L - S = -1.98 pp, [-4.38, +0.43], p = 0.1172).
    • Full 800-round candidate records and 474-task verdict vectors for the restored online candidate-pool single-edit evaluation (L - S = -0.13 pp, [-0.46, +0.19]).
    • Eight-seed 27B telecom search extension (Appendix H.1, Table 35): Frozen M0/L/S states, round-level screening matrices, paired terminal rewards across 74 tasks × 4 repeats (L - S = +0.04 pp, 95% t CI [-3.25, +3.33]), and offline CPU verification.
  • Complete frozen candidate bank and state hashes: All 524 candidate patches, all 18 original 4B M0/L/S specifications and checkpoints, and all 24 Qwen3.5-4B specifications.
  • Non-sensitive harness and experiment source code: Core 4B search, audit gate, and ledger code, Qwen3.5-4B evaluation runner, and the 27B telecom wrapper and C0 to C3 condition adapters.

Thank you for your efforts, including the new external assessment after freezing states and choosing the analyses, and the release of the code and verification materials, which have largely addressed my two major concerns. I feel much more comfortable with my earlier positive review.

Thank you for following up. Your suggestion to evaluate the frozen states on a fresh set of tasks gave us a useful check on the original results. We’re glad that the new evaluation and the verification materials helped address your concerns.

Reviewer_VNpHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper examines whether gains credited during harness evolution survive re-evaluation, and whether accepted edits actually execute their intended behavior. In the coding experiments, evolved lineages improve over the initial harness, but show no clear advantage over a single edit selected from the same candidate stream. The authors also study terminal component interactions and vary the feedback available during search. In a separate telecom case, an accepted guard never fires in the recorded natural trajectories. Further controls show that the surrounding wrapper removes assistant text from subsequent prompts even when the guard is disabled. The paper brings these analyses together as the RSI Information Ledger, with explicit records of disclosures, frozen states, and runtime behavior.

Critical strengths
  1. The single-edit comparison is useful. An improved endpoint is easy to present as evidence for cumulative improvement; selecting a single edit from the same candidate stream provides a much more informative baseline. The original comparison gives gains of 1.55 points for the lineage and 1.44 points for the single edit. I also appreciate the follow-up checks: the Qwen3.5 result is examined under candidate-count and complete-evaluation ceilings, and the original endpoints are remeasured in a common serving environment. These checks give the comparison more substance than a single null result.

  2. The wrapper control is the strongest part of the runtime study. Removing the guard while retaining the nonempty-patch execution path reveals that assistant-content clearing is introduced by the wrapper. This is a concrete failure of attribution that an ordinary patched-versus-unpatched comparison could miss. The authors carefully separate this finding from the guard’s intended function and from task success. The synthetic triggers help establish that the blocking and feedback path works when its predicate is satisfied.

Critical weaknesses
  1. My main concern is the narrow search space. Of 524 candidates, 409 replace one of two text fields, and all six original single-edit endpoints modify task instructions. The results therefore say relatively little about evolution where earlier edits enable later proposals or useful changes require several dependent steps. The Qwen3.5 extension uses the same bank, where exemplar additions become permanently inapplicable once capacity is reached because no removal operation is available. Candidate-count matching addresses unequal scoring budgets but leaves this restriction intact. The authors acknowledge these limits; within them, I find the advance over prior work on budget-controlled evaluation and activation versus benefit modest.

  2. The audit would be more convincing if its supporting materials were available for inspection. Appendix H describes extensive internal checks, but neither the scripts nor the underlying records accompany the submission. Some checks reconstruct summaries: the historical 3,960 guard checks, for example, are recovered from five scope summaries rather than replayed episodes. The later full-task reconstruction is more detailed, but its inputs are also unavailable. Since the contribution depends on linking selection records to runtime behavior, a small release of frozen states, paired verdicts, wrapper code, and representative message traces would help substantially. Calibration also remains incomplete: detected reference-copy proposals are never accepted, while direct injection improves targeted items without producing the required aggregate signal. These diagnostics are useful, but do not validate the full procedure end to end.

Questions for authors
  1. Could you provide an anonymized minimal package for the central results: the frozen states and paired outcomes behind Table 2, the guard/wrapper implementation, and the saved messages used for the example in Appendix F.1? Appendix H suggests that much of this material already exists. Being able to inspect it would substantially address my second concern.

  2. Could you describe the Qwen3.5 terminal edits in the same way as Table 7 describes the original endpoints? In particular, which fields and exemplars survive in the lineage and single-edit solutions? The budget sensitivities are useful, but the endpoint contents would help clarify what the second configuration reproduces and how much its result depends on this candidate bank.

4 comments
AuthorsOpenReview ↗

Response to Reviewer VNpH (Part 1/2: Supplementary Material and Complete Qwen3.5-4B Terminal-State Table)

Thank you for your constructive review and for highlighting our single-edit baseline (S) and 27B wrapper control (C1 vs. C3). We have uploaded the revised manuscript (inline additions and new headings/captions in blue) and the supplementary material. Part 1 documents the supplementary material (Question 1) and the full Table 7 counterpart for all 8 Qwen3.5-4B terminal states (Question 2); Part 2 discusses the search-space implications and reports our 358-task external holdout, 35M-token prefix study, online pool restoration, and calibration results.

1. Response to Question 1: Supplementary Material

The supplementary material (see the root README.md and EXPERIMENTS.md for one-command CPU reproduction) includes every item requested in Question 1:

  1. Frozen states and paired verdicts behind Table 2 and Table 7:

    • Original 4B state module: Contains all 18 complete frozen specifications (M0, L, and S across all 6 original seeds), including exact system prompt, instructions, repair instructions, few-shot examples, repair rounds, temperature, SHA-256 state commitments, per-round checkpoints, and single-edit trajectories.
    • Table 2 verification module: Contains the paired 474-task MBPP reserve verdict vectors for M0, L, S, generations g0 to g5, and the terminal-component ablation vectors, reproducing Table 2 (L - M0 = +1.55 pp, S - M0 = +1.44 pp, L - S = +0.11 pp with 95% CI [-0.62, +0.83], terminal interaction -0.59 pp with 95% CI [-1.55, +0.38]).
  2. Guard and wrapper implementations (Section 5):

    • 27B harness source module: Contains the bound telecom harness implementation, the C0 to C3 condition adapter, and the retained transfer-guard specification, together with offline CPU verification checks.
    • Message-replay module: Standalone replay on task t002 with complete model-visible messages, tool schemas, and saved responses, demonstrating the exact C1 vs. C3 message-conversion divergence.
  3. Saved messages and transformation receipts for Appendix F.1 and Table 3:

    • 27B diagnostic module: Appendix F.1 request projections and transformation receipts, wrapper source excerpt, the 12 natural and 4 synthetic diagnostic states, and historical activation records.
    • Table 3 verification modules: All 592 paired Table 3 episode records, bootstrap indices, the 20-task diagnostic subset (160 episodes), and the 220 paired first-divergence witnesses across C1 - C0 and C1 - C3.
  4. Eight-seed 27B telecom search extension (Appendix H.1, Table 35): Frozen M0/L/S states, round-level screening matrices, and paired terminal rewards across the 74-task, four-repeat panel (L - S = +0.04 pp, 95% t CI [-3.25, +3.33]), together with standalone CPU reconstruction alongside the separate C0 to C3 wrapper controls.

2. Response to Question 2: Complete Table 7 Counterpart for All Eight Qwen3.5-4B Terminal States

The Qwen3.5-4B terminal-state module provides all 24 frozen Qwen3.5-4B terminal states (M0, L, S across all 8 registered seeds, denoted Seed 1 to Seed 8 below) and the SHA-256-verified per-round lineage and checkpoint records against the 524-entry candidate bank (all terminal states have temperature = 0.0, and the 4 cap-hitting seeds account for all 226 capacity-blocked steps during search):

SeedMulti-Step Lineage (L): Edits & Retained Few-Shot ExamplesSingle-Edit (S): Modified Component474-Task Reserve: L vs. S (L - S)
Seed 1system prompt + instructions + 3 examples (hits 3-example cap)instructions only (concise logic comments, code-only)59.70% vs. 64.14% (-4.43 pp)
Seed 2system prompt + instructions + 0 examplesinstructions only (allow multiple/helper functions)62.66% vs. 62.45% (+0.21 pp)
Seed 3system prompt + instructions + 1 repair round + 3 examples (hits cap)instructions only (concise comments, pass assertions)60.55% vs. 62.45% (-1.90 pp)
Seed 4system prompt + instructions + 3 examples (hits 3-example cap)instructions only (concise comments, pass assertions)60.97% vs. 63.92% (-2.95 pp)
Seed 5system prompt + instructions + 2 examplesinstructions only (concise comments, pass assertion)61.81% vs. 62.87% (-1.05 pp)
Seed 6system prompt + instructions + 0 examplesinstructions only (allow multiple/helper functions)63.71% vs. 62.66% (+1.05 pp)
Seed 7system prompt + instructions + 3 examples (hits 3-example cap)instructions only (logic comments in O/B; param. comments in C)58.23% vs. 63.92% (-5.70 pp)
Seed 8system prompt + instructions + 1 exampleinstructions only (allow multiple/helper functions)61.60% vs. 62.03% (-0.42 pp)
8-Seed MeanMulti-component edits (6/8 seeds retain examples; M0 = 56.78%)Single instructions edit (0 examples; M0 = 56.78%)61.16% vs. 63.05% (-1.90 pp)

(Continued in Part 2/2)

AuthorsOpenReview ↗

Response to Reviewer VNpH (Part 2/2: Analysis of Qwen3.5-4B Edits, External Holdout, 35M-Token Prefix and Calibration)

What the 8-seed Qwen3.5-4B terminal table (Part 1/2) and saved round logs reveal:

  1. Structural divergence from Table 7: Whereas 0/6 original 4B lineages retained examples at termination, 6/8 Qwen3.5-4B lineages retain 1 to 3 examples (4 hit the 3-example cap, accounting for all 226 skipped example additions) alongside system prompt and instructions edits (plus 1 repair round in Seed 3), forming multi-component states unreachable by any single patch from M0M_0.
  2. Consistency of S across budgets: Every single-edit endpoint S across the 400-slot stream (O), evaluation ceiling (B), and candidate-count ceiling (C, KL∈[317,400]K_L \in [317, 400]) modifies instructions alone (0 examples, 0 repair, temp = 0.0), and L shows no statistically resolved advantage under any rule (L - S = -1.90 pp, 95% t7t_7 CI [-3.85, +0.05], p=0.0625p = 0.0625 under O and B; L - C = -1.56 pp, [-3.34, +0.23], p=0.1016p = 0.1016 under C).
  3. All 8/8 single-edit winners were scored in L (0/8 blocked by capacity): Because the 3-example cap skips only add-example patches, text patches remain legal in every round; replaying the bundled round records verifies that in all 8/8 seeds (including all 4 cap-hitting seeds), S's winning instructions patch was offered and scored in L. In 5/8 seeds (Seeds 1, 3, 6, 7, 8), that patch failed L's margin-1 gate on the evolved incumbent (-8 to 0 audit net wins vs. incumbent, despite +20 to +23 on M0M_0, e.g., -5 at r53 in Seed 1 and -8 at r87 in Seed 7); in 3/8 seeds (Seed 2 at r57 +7, Seed 4 at r27 +1, Seed 5 at r12 +6), L accepted that patch and later overwrote instructions in subsequent rounds (r88/96/98, r63/70, r38), leaving 0/8 L endpoints retaining S's winning instructions (Section 4.2; Table 14 and 8-seed fate table in Appendix E.9).
  4. Role of the candidate bank and capacity skips: Descriptively, the 4 seeds with zero capacity skips (Seeds 2, 5, 6, 8; all 400 proposals scored) average L - S = -0.05 pp vs. -3.74 pp on the 4 cap-hitting seeds (+0.11 pp vs. -4.06 pp at the 35M-token prefix). Thus, Qwen3.5-4B's negative L - S gap concentrates in trajectories that saturate the 3-example cap (without a removal operator) or overwrite earlier instructions via full-field replacements, whereas the 4 unconstrained seeds match the near-zero L - S pattern of the original 4B cohort and the state-conditioned online proposer below.

3. External Holdout, 35M-Token Prefix Control, Online Pool Restoration, and Calibration

  • Unseen 358-task external benchmark (Table 19): Scoring all 42 frozen M0/L/S states (15,036 cell evaluations) once on the Code Complexity Python Benchmark yields M0 = 80.63%, L = 80.31%, S = 81.70% on the 6 original seeds (L - S = -1.40 pp, [-2.87, +0.07]; S - M0 = +1.07 pp, [+0.17, +1.97]; L - M0 = -0.33 pp, [-1.52, +0.87]) and M0 = 76.50%, L = 75.94%, S = 76.61% on the 8 Qwen3.5-4B seeds (L - S = -0.66 pp, [-2.42, +1.09]; L - M0 = -0.56 pp, [-1.79, +0.67]; S - M0 = +0.10 pp, [-0.52, +0.73]), showing no resolved lineage advantage on fresh external tasks.
  • 35M-token search prefix (Table 18): Truncating each Qwen3.5-4B arm at its longest complete round prefix within 35,000,000 realized search tokens (34.55M to 34.91M tokens/arm; paired ∣L−S∣≤0.93%|L - S| \le 0.93\%) and scoring all 24 states (11,376 cells) under pinned vLLM 0.18.0 yields M0 = 56.83%, L = 60.94%, S = 62.92%, and L - S = -1.98 pp ([-4.38, +0.43], p=0.1172p = 0.1172; L - M0 = +4.11 pp [+2.13, +6.10]; S - M0 = +6.09 pp [+5.42, +6.77]).
  • Online pool single-edit restoration: Completing all 800 single-edit rounds (756 initial + 44 restored) across the state-conditioned online LLM proposer pool (8 seeds, 3,175 proposals) and remeasuring M0, L, S in one serving batch yields L - S = -0.13 pp ([-0.46, +0.19], L - M0 = -0.29 pp, S - M0 = -0.16 pp).
  • Positive-control calibration: In the known-leak chain (6 lineages, 2,400 proposer calls), 27 proposals copied a disclosed audit reference, yet all 27 had negative overall audit net wins (-28 to -8), so 0/27 passed the gate. Direct injection of 3 audit references vs. dev examples yields +30 net solves on the 3 target tasks (+23 vs. M0 across 90 observations) offset by non-target losses across the other 297 audit items (-0.122 pp audit diff; gap contrast -0.530 pp, [-1.723, +0.662], p = 0.34375), showing why aggregate validation gates can dilute sparse item leakage and must be paired with round-level transcript overlap checks (Sections 4.5 and 4.6).

Thank you for the detailed response. The additional results address a substantial part of my original concerns. The 35M-token comparison provides a more direct budget control, and the external benchmark and completed online-pool comparison broaden the evidence beyond the original fixed-bank MBPP study.

I found the Qwen3.5 winner-fate analysis particularly helpful. All eight single-edit winners were evaluated in the corresponding lineages; five were rejected against the current incumbent and three were accepted but later overwritten. This gives a clearer account of where the two strategies diverge.

These additions sufficiently address my main concerns, and I am updating my recommendation to weak acceptance. The remaining limitations concern the scope of the findings.

Thank you for taking the time to read our response and the additional results. Your question about which edits survived in the Qwen3.5 runs helped us explain the comparison more clearly. We’re glad you found that analysis helpful.

Reviewer_rqMmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This manuscript studies recursive self-improvement (RSI) in the restricted regime where the policy is frozen and only a harness specification changes. More specifically, it looks at how much a gain in performance is due to the change in a single edit to the harness. To this end, it introduces the RSI Information Ledger which serves as a principled device to compare across the two arms of experiments (cf. Figure 3 and Table 5). The results appear to be dominated by noise and have narrower scope than what the manuscript shows, demonstrating that with frozen model weights and small edit space, RSI does not yield realiable benefits over the one-shot setting.

Critical strengths
  • The topic is well-motivated and of substantial value to the current AI evaluation landscape as increasingly the focus of the developments is concentrated on the harness rather than the model itself that it wraps around. Moreover, the experiments are framed such that the each harness specification becomes a cheap and reproducible snapshot, enabling the kind of causal questions that the manuscript gestures towards.
  • The experiments appear generally well-engineered and results reported in a disciplined fashion.
Critical weaknesses
  • The experiments represent only proof-of-concept of an audting protocol rather than of acutally deployed workflows. In the 4B-model expeirments, while the model size could be partially justified for applciation in edge devices, the small number of prompt and examplars (3 each) and small hyperparameter space limit its real-world impact. In this regard, the 27B-model expeirments are more realistic, with addition multi-turn tool-use and other layers, but the headline results appear to be non-informative (Table 3) with all intervals covering 0.
  • The prose flow is rather choppy and somewhat difficult to follow. In particular, the RSI Information Ledger, the main artefact, was introduced on p.1 but does not feature prominantly after that until in the appendix.
Questions for authors
  • For 27B, there appears to have been only 1 edit, 1 seed and no replications (intervals in Table 3 were generated via bootstrapping), unlike the 4B experiments, so given the stochastic nature of the LLMs it would be useful to have the same kind of disciplinary with regard to uncertainty.
  • In Fig. 2, why does the gap grow about 2.5 pp in both arms with the 1-bit arm higher?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rqMm

Thank you for recognizing the rigor of our harness-snapshot design. Below we address both questions, the 27B study, and the narrative flow; we have uploaded the revised manuscript (inline additions and new headings/captions in blue) and the supplementary material.

1. Question 1: 27B Search Replication and Runtime Uncertainty

To address search-level variability in 27B telecom, we ran eight separately seeded evolutionary searches from M0 (Section 5; Appendix H.1, Table 35). Each search runs four rounds (two online proposal slots/round) on shared 20-task, four-repeat minibatches: the multi-step lineage (L) updates its incumbent only at >= +8 net wins and exact task-block sign-flip p <= 0.05, whereas single-edit S applies every proposal to M0 and retains the largest positive screening net gain (earliest on ties). Freezing M0, L, and S prior to readout across all 74 telecom tasks with four repeats (5,920 episodes, 7,104 cells) gives mean gains over M0 of +2.58 pp for L (95% t CI [-0.66, +5.81]) and +2.53 pp for S ([-1.94, +7.00]), with L - S = +0.04 pp ([-3.25, +3.33], exact sign-flip p = 1.00; four lineages retain one edit, two tool-use hints and two Python hooks, and four retain M0). Consistent with our 4B cohorts, multi-seed 27B search resolves no multi-step advantage over S; frozen states, screening matrices, paired rewards, and CPU checks are in the supplementary material.

Alongside this 8-seed search, Section 5 and Table 3 audit the runtime mechanism and rollout uncertainty of the historical transfer guard (p = 0.0078):

  • Wrapper mechanism vs. dormant guard: Across 616 natural episodes (3,960 checks), the guard triggered 0 times. Evaluating four nested controls on all 74 tasks twice (C0 native baseline, C1 wrapper with guard off, C2 wrapper with guard on, C3 C1 without assistant-content clearing; 592 episodes, 25,210 requests) shows that even with the guard off (C1), clearing assistant text alongside tool calls alters prompt inputs by the first semantic divergence in 110/148 C1 - C0 and 110/148 C1 - C3 pairs (changing tool-call names or arguments at that turn in 28 and 26 pairs).
  • Task-level uncertainty (Table 3): Repeats disagree on 11 to 16 of 74 tasks per condition; averaging within tasks before paired bootstrap (20,000 resamples) yields intervals spanning zero across all four contrasts (C1 - C0 = -4.73 pp, C1 - C3 = -2.03 pp, C2 - C1 = +4.05 pp, C3 - C0 = -2.70 pp), separating wrapper side effects from dormant rules without claiming task-reward gains.

2. Question 2: Figure 2 Gap Rise (~2.5 pp) and Why Arm B Sits Slightly Higher

  1. Why the gap widens (~2 to 2.5 pp, selection optimism on the 300-task audit split):
    • At g0g_0 (M0M_0), the baseline gap between the 300-task audit split and 474-task MBPP reserve across 8 seeds is +1.18 pp in Arm A (67.87% - 66.69%) and +1.25 pp in Arm B (67.83% - 66.59%).
    • Across g1→g5g_1 \to g_5 (400 slots from the 524-entry bank, 100 rounds × 4 slots), accepting audit-improving edits raises audit scores by +3.00 pp (A: 67.87% -> 70.88%) and +3.17 pp (B: 67.83% -> 71.00%), while reserve scores rise by +1.58 pp (A: 66.69% -> 68.28%) and +0.74 pp (B: 66.59% -> 67.33%), widening the endpoint gap by +1.42 pp (A) and +2.43 pp (B) and the g1…g5g_1 \dots g_5 mean gap by +2.02 pp and +2.23 pp (ending at 2.60 pp and 3.67 pp at g5g_5).
  2. Why 1-bit (Arm B) sits slightly above Arm A:
    • In the frozen-bank study (Section 4.4), proposals are audit-blind, so Arm A (margin 1) and Arm B (margin 5) differ in acceptance threshold: dAUCAB=−0.21 pp\text{dAUC}_{AB} = -0.21\text{ pp} (95% CI [-1.06, +0.64], p=0.73p = 0.73; 3/8 seeds positive; 6-seed same-service replay -0.42 pp, [-1.16, +0.33]), detecting no threshold contrast.
    • Visually, Arm B sits higher mainly at g4→g5g_4 \to g_5 (3.57 vs. 3.95 pp at g4g_4; Arm A narrows by -0.97 pp to 2.60 pp via a -0.33 pp audit dip 71.21% -> 70.88% and +0.63 pp reserve rise 67.64% -> 68.28%). In the online proposer study (Section 4.5), isolating rich diagnostics vs. 1-bit feedback at margin 5 likewise shows no difference (-0.46 pp, [-1.50, +0.59], p=0.31p = 0.31).

3. External Holdout, 35M-Token Prefix and Narrative Flow

  • 358-task external holdout: Scoring all 42 frozen M0/L/S states (15,036 cells) once yields L - S = -1.40 pp ([-2.87, +0.07], p = 0.0625) on the 6 original seeds and -0.66 pp ([-2.42, +1.09], p = 0.4063) on the 8 Qwen3.5-4B seeds.
  • 35M-token search prefix: Under a shared 35M search-token cap (34.55M to 34.91M tokens/arm) and pinned vLLM 0.18.0 (11,376 cells), Qwen3.5-4B yields M0 = 56.83%, L = 60.94%, S = 62.92%, and L - S = -1.98 pp ([-4.38, +0.43], p = 0.1172).
  • Narrative flow: Within the 10-page revised main text, we placed an RSI Information Ledger Summary Table in Section 3.2, unpacked the gate-arm, budget, and Section 5 C0 to C3 definitions (with a task t002 witness), and added Ledger Audit Summary callouts in Sections 4.2 and 5.
AuthorsOpenReview ↗

A brief follow-up

Dear Reviewer rqMm,

Thank you again for your questions. Appendix H.1, Table 35 reports results across eight separately seeded telecom searches. Our response also explains why the audit–reserve gap widened in both arms of Figure 2 and why the evidence for a difference between the arms remains inconclusive.

If you’ve had a chance to read our response, we’d appreciate a brief note on whether it addresses your concerns.

Many thanks for the clarifications. The revised manuscript has now presented the Ledger as an "audit protocol" explicitly in the Abstract. However, the 27B experimental results still have intervals covering 0 which the 8-seed extension did not resolve. Intervals spanning zero cannot by themselves separate wrapper side effects from dormant rules, which analyses in Table 3 are unable to provide. As such, the original ratings have been retained.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions, Supplemental Experiments, and Code Release

We thank four reviewers for their constructive feedback. Compared with the initial PDF-only submission, we have uploaded the revised manuscript (10-page main body; inline additions, new headings and captions in blue) and the supplementary material, summarized below.

1. Supplemental Evaluations and Post-Submission Audits

  1. Untouched 358-task external Python benchmark (Section 4.6; App. E.9, Table 19): Freezing all 42 M0/L/S states across the 6-seed Qwen3-4B (18 states) and 8-seed Qwen3.5-4B (24 states) cohorts plus analysis rules, scoring all 358 Code Complexity tasks once under pinned vLLM 0.18.0 (15,036 cells; zero MBPP/HumanEval overlap; 5 public tests, 10 hidden assertions) gives L - S = -1.40 pp (95% CI [-2.87, +0.07], p = 0.0625; S - M0 = +1.07 pp [+0.17, +1.97]) on the 6 original seeds and L - S = -0.66 pp ([-2.42, +1.09], p = 0.4063) on Qwen3.5-4B.
  2. Matched 35M-token search-budget prefix on Qwen3.5-4B (Section 4.2; App. E.9, Table 18): Truncating L and S in all 8 seeds at the longest complete round prefix <= 35M search tokens (34.55M to 34.91M tokens/arm across all 16 arms; paired |L - S| <= 0.93% of budget) and scoring all 24 M0/L/S panels (11,376 cells) on the 474-task MBPP reserve under pinned vLLM 0.18.0 gives M0 = 56.83%, L = 60.94%, S = 62.92%, and L - S = -1.98 pp ([-4.38, +0.43], p = 0.1172), alongside the 400-slot (O), evaluation-count (B: -1.90 pp, p = 0.0625), and candidate-count (C: -1.56 pp, p = 0.1016) controls.
  3. Eight-seed 27B telecom search extension (Section 5; App. H.1, Table 35): Running eight separately seeded four-round search pairs from M0 (two online slots/round on shared 20-task, four-repeat minibatches; L updates at >= +8 net wins and exact sign-flip p <= 0.05; S applies proposals to M0 and retains the largest positive net gain, earliest on ties) and scoring frozen M0, L, S on all 74 telecom tasks x 4 repeats (5,920 episodes, 7,104 cells) gives L - M0 = +2.58 pp, S - M0 = +2.53 pp, and L - S = +0.04 pp (95% t CI [-3.25, +3.33], p = 1.00; four lineages retain one edit, two tool-use hints and two Python hooks, and four retain M0), complementing the C0 to C3 wrapper audit of the historical guard (0/3,960 natural triggers; 592 episodes, Table 3).
  4. Completed 800-round online pool single-edit restoration (Sections 4.5-4.6; App. E.8): Completing the remaining 44 rounds past the initial 756/800 cutoff (800 rounds, 3,175 slots across 8 seeds) over the state-conditioned online pool under the 3-example cap and scoring M0, L, S in one MBPP batch gives L - S = -0.13 pp ([-0.46, +0.19], p = 0.4531).
  5. Qwen3.5-4B 24-state terminal audit, capacity groups, and S-winner fates in L (Section 4.2; App. E.9, Table 14): Across all 8 seeds, 6/8 lineages retain 1 to 3 examples plus system prompt and instructions edits (4/8 hit the 3-example cap, skipping 226 additions: L - S = -3.74 pp vs. -0.05 pp on the 4 zero-skip seeds), while 8/8 S endpoints modify instructions alone. Round logs verify all 8/8 S-winning instructions patches were scored in L (0/8 cap-blocked: 5/8 failed L's margin-1 gate on the evolved incumbent; 3/8 were accepted and later overwritten).
  6. State verification, Figure 2 decomposition, and task t002 witness (Sections 4.2, 4.4, 5): Verified all 18 original 4B M0/L/S states and checkpoints (Tables 2, 7), added the g0 to g5 decomposition explaining the ~2 to 2.5 pp Figure 2 gap rise and g4 to g5 Arm A narrowing (Section 4.4), and foregrounded the task t002 C1 vs. C3 prompt-clearing witness (Section 5) with standalone CPU replay.

2. Revised Manuscript and Supplementary Material

  • Revised manuscript (10-page main body): Added a three-tier RSI Information Ledger Table in Section 3.2 separating (A) state/split isolation, (B) loop counters under the conditional data-processing boundary, and (C) external channels; unpacked gate-arm (Section 3.3), budget (Section 4.2), two-layer calibration (Section 4.6), C0 to C3 (Section 5), and diagnostic-scope (Section 6) definitions with Ledger Audit Summary callouts; and added Tables 14, 18, 19, 35 plus updated Study Map (Table 23) and Reproducibility sections (App. H).
  • Supplementary material (root README.md and EXPERIMENTS.md): Provides one-command CPU verification across all seven offline suites: all 15 core paper modules (Tables 2-3, Figure 2, 8-seed completion and common-service L - S remeasurement, Qwen3.5-4B O/B/C, two-layer positive controls, public-check selector, and 27B guard/wrapper panels) plus the six supplemental modules above, with the t002 replay module, 524-entry bank, 4B/Qwen3.5-4B/27B harness capsules, and full research source tree (including the 4-round history-value and 27B follow-up studies).

Hello Reviewers,

As the discussion period is approaching its end, please take a moment to review the authors’ responses and share any remaining thoughts (either to authors or privately to other reviewers and me) or updates to your assessment.

Thanks for your thoughtful reviews and discussion!

Bests, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.