- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This position paper argues that evaluations of neural interventions in agents with evolving downstream components must distinguish three quantities: probe prediction, the local intervention effect, and the final-answer effect. For partial records, it derives a sharp interval for the final-answer effect under bounded selector probabilities. For complete records, it gives conditions under which re-executing the updated downstream suffix recovers the final effect. A GSM8K study with Qwen2.5-7B and judge-only prompt optimization illustrates the position. A steering vector with a local effect of +11.8 points yields a final-answer effect of only +0.8 to +1.4 points, and replay agrees with fresh execution.
Critical strengths
The paper identifies a valid distinction between local and final-answer intervention effects in evolving agents, and its analytic example shows clearly that repeated logs cannot resolve the resulting ambiguity.
Critical weaknesses
The main text does not make clear whether the empirical study was executed, and several reported results are implausibly exact. Even taken at face value, the experiments cover a single narrow setting, include no comparison with the practices the paper argues against, and do not test the paper's own identification result.
Questions for authors
- Were the E1 and E2 experiments fully executed? If so, how do replay and fresh execution agree exactly when fresh execution uses new upstream draws, and why is the confidence-interval column empty?
- In the reported setting, how large is the error of the practices the paper argues against, such as reusing the old effect or stitching old continuations?
- Given that replay and fresh execution agree at similar cost, what does the proposed replay procedure offer over re-running the evaluation?
5 comments
Follow-Up on Reviewer Comments
Dear Reviewer qzVWj,
As we near the end of the discussion period, we kindly ask you to review our responses and consider updating your scores if your concerns have been addressed. We're happy to clarify anything further if needed. Thanks
Regards,
Authors
Follow-up on Discussion Phase
Dear Reviewer qzVWj,
We understand you are busy. Since the author-reviewer discussion phase is ending soon, we kindly request that you consider increasing your score if our response has addressed your concerns.
If there are any remaining questions or clarifications needed, we would be happy to respond.
Thank you.
Regards,
Authors
Response to Reviewer qzVW
Thank you for your appreciation and positive feedback. We hope the following clarifications can address your concerns.
Comment 1 of 2: Execution Status, Intervals and Errors of Existing Practices
W1 / Q1: Were E1/E2 Executed? Why Did Replay and Fresh Agree Exactly, and Why Was the CI Column Empty?
Response: The original runs produced records, but our original E2b interpretation was wrong. Greedy upstream decoding meant the new stream identifiers did not create fresh stochastic trajectories, so the zero discrepancy reflects deterministic re-execution (see Appendix F.5; Figure 9). The empty CI column was a reporting gap. In the completed study, every result has a paired 95% interval (see Figures 3-4; Appendices F.1-F.2):
Q2: How Large are the Errors of the Practices the Paper Argues Against?
Response: We now measure them directly (see §6, "Reuse Errors and Practical Value of Replay"; Appendix F.10):
Both reuse errors are small and unresolved in this judge-only setting, and we report them as such. The position calls for checking the relevant claim; it does not predict a large error for every update.
Comment 2 of 2: Practical Value, Identification and Scope
Q3: What Does Replay Offer Over Rerunning?
Response: We added a matched timing comparison on 128 questions with the same target judge (see §6; Appendix F.12, Figure 15). Full regeneration takes 2,576.01 s and persisted replay takes 95.01 s, a 27.11× speedup. Both are cold processes that include loading, validation and I/O, and each performs 256 judge prefills. Replay generates no upstream tokens and reuses no saved judge decisions. The comparison measures repeated downstream assessment when a sufficient boundary already exists. It is one fixed-order pass and excludes acquisition cost. The eight recorded allocations total 12.08 H200-hours, failed work included (see Figure 15(b)).
W2: The Experiments Do Not Test the Identification Result
Response: We now connect the result to the actual candidate pairs (end of §3; Appendix F.9, information-set box). If individual selection probabilities are withheld and per-condition choices may vary independently, the compatible final effect is [−16.02, 25.00] pp. Supplying six aggregate rule effects and assuming one common mixture across conditions narrows it to [−0.88, 9.86] pp, containing the executed J⋆ effect of +5.24 pp. This shows on real records how added information about the judge narrows the interval, which is the point of Eqs. (6)-(8). The general result follows from the proof in Appendix B.5.1.
W3: A Single Narrow Setting
Response: We agree, and the revised paper limits its claims accordingly (see §1, §5, §7 "Scope and Limitations"). The study supports three specific claims: the local and final effects differ; saved inputs permit exact reconstruction; and replay saves computation when a boundary exists. All 116 rescues and 15 harms coincide with validity transitions (see Figure 11), and most critiques reach their token cap (see Appendix F.7), so we do not claim improved reasoning. Other models and generative, tool or memory updates remain untested (see Appendix F.13).
Thanks again for your thoughtful feedback—we hope you'll consider increasing your score.