- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits retained gains and runtime behavior in evolving agent harnesses. Six 4B lineages gain 1.55 points on an MBPP reserve set versus 1.44 for a same-stream single edit (paired difference +0.11; 95% CI [−0.62, +0.83]). Follow-ups establish neither lineage advantage nor equivalence. In a 27B case, the guard has zero activations in 3,960 checks; task-level benefit remains unresolved.
Critical strengths
The same-stream single-edit baseline separates retained gain from lineage value. Wrapper controls help distinguish stored rules from runtime behavior. The authors report uncertainty candidly.
Critical weaknesses
The lineage contrast is inconclusive; follow-ups have cost or service confounds. Telecom tests are diagnostic, not evidence of guard benefit. The Information Ledger is nominal accounting, not a formal leakage bound; no public artifact is supplied to reproduce model runs.
Questions for authors
- Which Ledger properties are formal guarantees, and what leakage channels are outside its nominal charges? 2. Can lineage and single-edit results be compared at matched realized cost and fixed serving version? 3. Can code, configurations, and records needed to reproduce the results be released?
3 comments
Response to Reviewer 7shR (Part 2/2: Cost-Matched and Fixed-Serving Comparisons, and Supplementary Material)
3. Question 2: Comparing Lineage (L) vs. Single-Edit (S) Under Matched Actual Cost and Fixed Model-Serving Version
We address both matched actual token expenditure and fixed model-serving version in the supplementary material:
-
Matched Actual Search Token Budget Under Fixed Serving:
- Across the full 400-slot schedule (O), multi-step search consumes 514,047,575 realized search tokens vs. 308,649,030 for single-edit selection (+66.55% more tokens in L because retained examples and repair lengthen prompts/completions), yet L establishes no statistically resolved advantage over S (L - S = -1.90 pp, 95% CI [-3.85, +0.05], ). Because L and S consume the identical 524-entry candidate bank and per-seed candidate stream, historical one-time bank-generation cost is an identical additive constant for both routes.
- To compare L and S at equal realized search token expenditure, using per-round token accounting across all 2,362,318 physical Qwen3.5-4B requests, we selected each arm's longest complete round prefix with cumulative realized search tokens (34.55M to 34.91M realized tokens across the 16 arms; paired budget difference ; 50 to 67 lineage rounds vs. 90 to 91 single-edit rounds), and evaluated all 24 resulting frozen M0/L/S panels (11,376 cells, 11,385 calls) on the 474-task MBPP reserve in one unified batch under pinned vLLM 0.18.0 (Table 18):
- Mean pass rates: M0 = 56.83%, L = 60.94%, S = 62.92%
- Primary contrast (L - S): -1.98 pp (-1.978 pp, 95% CI [-4.38, +0.43], exact two-sided sign-flip )
- Secondary contrasts: L - M0 = +4.11 pp (+4.114 pp, 95% CI [+2.13, +6.10], ); S - M0 = +6.09 pp (+6.092 pp, 95% CI [+5.42, +6.77], ).
- Thus, under both equal scheduled candidate slots (where L expends +66.55% more tokens) and equal 35M realized search tokens under pinned vLLM 0.18.0, we find no statistically resolved lineage advantage over same-stream single-edit selection.
-
Additional Fixed-Serving Comparisons Across Evaluation Regimes (Included in the Supplementary Material):
- Same-serving MBPP reserve: Under a shared serving tag, the 6 original 4B seeds yield L - M0 = +1.55 pp, S - M0 = +1.44 pp, L - S = +0.11 pp (95% CI [-0.62, +0.83], ; Table 2), and same-serving 8-seed remeasurement in the public-check study yields L - S = +0.185 pp (95% CI [-0.770, +1.139]).
- Fixed-serving Qwen3.5-4B under matched evaluation count (B) and matched candidate count (C): Across all 8 seeds under pinned vLLM 0.18.0, matching completed audit evaluations (L - B = -1.90 pp, 95% CI [-3.85, +0.05], ) or matching each seed's exact lineage candidate count (L - C = -1.56 pp, 95% CI [-3.34, +0.23], ) likewise shows no statistically resolved lineage advantage over S.
- Fixed-serving online pool restoration: Completing all 800 single-edit rounds (756 initial + 44 restored) across the state-conditioned online pool (8 seeds, 3,175 proposals) and remeasuring M0, L, S in one batch yields L - S = -0.13 pp ([-0.46, +0.19]).
- Fixed-serving 358-task external holdout (Table 19): Evaluating all 42 frozen states once under pinned vLLM 0.18.0 (15,036 cells) yields L - S = -1.40 pp ([-2.87, +0.07], p = 0.0625) on the 6 original 4B seeds and L - S = -0.66 pp ([-2.42, +1.09], p = 0.4063) on the 8 Qwen3.5-4B seeds.
4. Question 3: Release of Code, Configurations, and Records
We have uploaded the supplementary material (documented in the root README.md and EXPERIMENTS.md):
- Source Code and Configurations: Complete 4B search, scoring, and ledger code; the 524-entry candidate bank and runner; Qwen3.5-4B configurations and runner; and the 27B telecom harness, C0 to C3 condition adapters, and transfer-guard specification.
- Frozen States and Verdict Records: All 18 original 4B and 24 Qwen3.5-4B M0/L/S full specifications, checkpoints, and Qwen3.5-4B round records; all 15 paper verification modules; the restored 800-round online pool records; the 358-task external holdout dataset and 15,036 cell records; the 35M-token prefix states, per-round search cost logs, and 11,376 cell records; and the eight-seed 27B search states, screening records, paired terminal rewards, and CPU verification scripts.
A brief follow-up
Dear Reviewer 7shR,
Thank you again for suggesting a comparison at matched cost. In Table 18, we compare multi-step search with single-edit selection under the same cap of 35 million search tokens per arm. We evaluated the resulting frozen states together under the same serving configuration.
If you’ve had a chance to read our response, we’d appreciate a brief note on whether it addresses your concerns.
Response to Reviewer 7shR (Part 1/2: Formal Guarantees vs. Nominal Accounting and 27B Diagnostic Role)
Thank you for your incisive review. We have uploaded the revised manuscript (inline additions and new headings/captions in blue) and the supplementary material. Part 1 answers Question 1 (formal guarantees, conditional data-processing inequality, nominal counters, and external channels) and clarifies the 27B diagnostic; Part 2 answers Question 2 (35M-token cost-matched and fixed-serving comparisons) and Question 3 (the code and data in the supplementary material).
1. Question 1: Formal Guarantees vs. Nominal Counters and Uncovered Channels in the Information Ledger
In our framework and released implementation, Ledger properties separate strictly into three categories:
A. Conditional Data-Processing Inequality and Cryptographic/Architectural Guarantees
B. Nominal Accounting Quantities (Not Direct Estimates of H(T∣Z) or Performance Bounds)
C. Leakage and Confounding Channels Not Covered by Nominal Loop Counters (and How We Audit Them)
In the revised manuscript, Section 3.2 now includes a summary table and explicit discussion distinguishing (A), (B), and (C).
2. Clarification on the 27B Telecom Diagnostic
We agree that the original 27B guard tests are diagnostic and do not show task benefits from the guard rule: although the transfer-guard patch was retained by screening (p = 0.0078), it triggered 0 / 3,960 times in natural episodes, while the outer wrapper (C1, guard disabled) altered prompts and first semantic responses across 110 / 148 trajectory pairs (C1 - C0 and C1 - C3), and all four Table 3 reward contrasts cross zero (C1 - C0 = -4.730 pp, 95% CI [-15.541, +6.757]; C1 - C3 = -2.027 pp, 97.5% CI [-14.189, +10.135]; C2 - C1 = +4.054 pp, 97.5% CI [-2.703, +10.811]; C3 - C0 = -2.703 pp, 95% CI [-8.784, +3.378]). We have sharpened the Section 5 opening and added a Ledger Audit Summary callout so this diagnostic role is unmistakable.
Alongside that single-lineage runtime audit, we also conducted eight separately seeded 27B telecom searches (four rounds per arm; frozen M0/L/S states evaluated across all 74 tasks with four repeats; Appendix H.1, Table 35), yielding L - M0 = +2.58 pp, S - M0 = +2.53 pp, and L - S = +0.04 pp (95% across-seed t CI [-3.25, +3.33]; four lineages retain one edit and four retain M0), corroborating that multi-step search resolves no advantage over single-edit selection.
(Continued in Part 2/2)