acceptodds
Under review as a conference paper at ICLR 2027

Position: Revalidating Probes and Steering Vectors in Scaffold-Evolving Agents

Abstract

Neural interventions primarily measure whether an evaluation changes local agent behavior, but not whether that effect persists after an agent updates, or how the resulting answer is used. This position paper argues that intervention validity must distinguish probe prediction, local intervention effect, and final-answer effect as separate evidence levels. For partial records with valid candidate labels and bounded selector probabilities, we characterize a sharp interval of compatible final-answer effects whose width depends on selector uncertainty and candidate disagreement; indistinguishable observations cannot remove the remaining ambiguity through additional logs of the same kind. For complete records, conditional downstream replay reassesses the effect by executing the updated suffix on each intervention arm while preserving a justified unchanged prefix, subject to preserved upstream laws, a sufficient cut without feedback, and faithful conditional execution. To operationalize this position, we specify a controlled GSM8K study with Qwen2.5-7B-Instruct and judge-only prompt optimization to assess intervention effects and reconstruction, independent-fresh agreement, and total cost. The analysis separates sampling uncertainty from ambiguity requiring new information or renewed computation.

Primary area:
interpretability and explainable AI
Keywords:
position, LLM agents, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qzVWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This position paper argues that evaluations of neural interventions in agents with evolving downstream components must distinguish three quantities: probe prediction, the local intervention effect, and the final-answer effect. For partial records, it derives a sharp interval for the final-answer effect under bounded selector probabilities. For complete records, it gives conditions under which re-executing the updated downstream suffix recovers the final effect. A GSM8K study with Qwen2.5-7B and judge-only prompt optimization illustrates the position. A steering vector with a local effect of +11.8 points yields a final-answer effect of only +0.8 to +1.4 points, and replay agrees with fresh execution.

Critical strengths

The paper identifies a valid distinction between local and final-answer intervention effects in evolving agents, and its analytic example shows clearly that repeated logs cannot resolve the resulting ambiguity.

Critical weaknesses

The main text does not make clear whether the empirical study was executed, and several reported results are implausibly exact. Even taken at face value, the experiments cover a single narrow setting, include no comparison with the practices the paper argues against, and do not test the paper's own identification result.

Questions for authors
  1. Were the E1 and E2 experiments fully executed? If so, how do replay and fresh execution agree exactly when fresh execution uses new upstream draws, and why is the confidence-interval column empty?
  2. In the reported setting, how large is the error of the practices the paper argues against, such as reusing the old effect or stitching old continuations?
  3. Given that replay and fresh execution agree at similar cost, what does the proposed replay procedure offer over re-running the evaluation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer qzVW

Thank you for your appreciation and positive feedback. We hope the following clarifications can address your concerns.


Comment 1 of 2: Execution Status, Intervals and Errors of Existing Practices

W1 / Q1: Were E1/E2 Executed? Why Did Replay and Fresh Agree Exactly, and Why Was the CI Column Empty?

Response: The original runs produced records, but our original E2b interpretation was wrong. Greedy upstream decoding meant the new stream identifiers did not create fresh stochastic trajectories, so the zero discrepancy reflects deterministic re-execution (see Appendix F.5; Figure 9). The empty CI column was a reporting gap. In the completed study, every result has a paired 95% interval (see Figures 3-4; Appendices F.1-F.2):

  • E1: Local +9.86 pp [7.75, 11.97]; final +4.83 [3.09, 6.57] (J0J_0) and +5.24 [3.49, 6.99] (J⋆J^\star); update contrast +0.40 [−0.21, 1.02], Holm-adjusted p = .1981.
  • E2a: Exact discrete reconstruction on all 128 same-key questions (see Appendix E.7).
  • E2b (512 questions, distinct keys): Persisted +5.19 pp [2.76, 7.62] versus fresh +2.89 pp [0.55, 5.24]; effect discrepancy +2.30 pp [−0.89, 5.49]; off-condition discrepancy −4.16 pp [−7.98, −0.35] (see Figure 4). We no longer describe this as exact agreement or equivalence.

Q2: How Large are the Errors of the Practices the Paper Argues Against?

Response: We now measure them directly (see §6, "Reuse Errors and Practical Value of Replay"; Appendix F.10):

  • Reusing the Old Effect: Substituting the J0J_0 effect for J⋆J^\star gives an error of −0.40 pp [−1.02, 0.21].
  • Stitching Histories: Combining the on answer with the off critique and revision (256 questions) gives +6.00 pp versus +5.29 pp for the valid history, an error of +0.71 pp [−0.80, 2.22]. This is one deliberately invalid construction, not a prevalence estimate.
  • Using the Local Effect as a Proxy for the Final Effect: It overstates it by +4.63 pp [2.87, 6.39]. This large, resolved error is exactly what the position warns against.

Both reuse errors are small and unresolved in this judge-only setting, and we report them as such. The position calls for checking the relevant claim; it does not predict a large error for every update.


Comment 2 of 2: Practical Value, Identification and Scope

Q3: What Does Replay Offer Over Rerunning?

Response: We added a matched timing comparison on 128 questions with the same target judge (see §6; Appendix F.12, Figure 15). Full regeneration takes 2,576.01 s and persisted replay takes 95.01 s, a 27.11× speedup. Both are cold processes that include loading, validation and I/O, and each performs 256 judge prefills. Replay generates no upstream tokens and reuses no saved judge decisions. The comparison measures repeated downstream assessment when a sufficient boundary already exists. It is one fixed-order pass and excludes acquisition cost. The eight recorded allocations total 12.08 H200-hours, failed work included (see Figure 15(b)).


W2: The Experiments Do Not Test the Identification Result

Response: We now connect the result to the actual candidate pairs (end of §3; Appendix F.9, information-set box). If individual selection probabilities are withheld and per-condition choices may vary independently, the compatible final effect is [−16.02, 25.00] pp. Supplying six aggregate rule effects and assuming one common mixture across conditions narrows it to [−0.88, 9.86] pp, containing the executed J⋆J^\star effect of +5.24 pp. This shows on real records how added information about the judge narrows the interval, which is the point of Eqs. (6)-(8). The general result follows from the proof in Appendix B.5.1.


W3: A Single Narrow Setting

Response: We agree, and the revised paper limits its claims accordingly (see §1, §5, §7 "Scope and Limitations"). The study supports three specific claims: the local and final effects differ; saved inputs permit exact reconstruction; and replay saves computation when a boundary exists. All 116 rescues and 15 harms coincide with validity transitions (see Figure 11), and most critiques reach their token cap (see Appendix F.7), so we do not claim improved reasoning. Other models and generative, tool or memory updates remain untested (see Appendix F.13).


Thanks again for your thoughtful feedback—we hope you'll consider increasing your score.

AuthorsOpenReview ↗

Follow-Up on Reviewer Comments

Dear Reviewer qzVWj,

As we near the end of the discussion period, we kindly ask you to review our responses and consider updating your scores if your concerns have been addressed. We're happy to clarify anything further if needed. Thanks

Regards,

Authors

AuthorsOpenReview ↗

Follow-up on Discussion Phase

Dear Reviewer qzVWj,

We understand you are busy. Since the author-reviewer discussion phase is ending soon, we kindly request that you consider increasing your score if our response has addressed your concerns.

If there are any remaining questions or clarifications needed, we would be happy to respond.

Thank you.

Regards,

Authors

Reviewer_4AGQOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks a question: if you have an AI, and you update its downstream judge, what evidence do you need to know how the AI's answer changes when you steer it with some fixed vector? It says there are three levels of evidence—probe prediction, local intervention effect, final-answer effect—and changing one level does not imply anything about the others. The first result shows that if you can't log everything the judge sees, then all you know is that the final-effect lies within some interval, which gets wider as you're less sure about the judge's decision-making process and as the two possible answers differ more often in correctness. The second result shows that if you can log everything the judge sees, then rerunning only the updated judge on each arm's records gives you a new estimate of the final effect, provided that none of the other parts of the pipeline change and that there isn't any feedback from the judge back into those other steps. Empirically, steering increases the solver's accuracy, and replaying the results versus re-running them from scratch gave identical effects.

Critical strengths

The core interesting part is captured in Table 2 which shows that if we use a steering vector on Qwen2.5-7B-Instruct, it increases its accuracy from 75.49% to 87.30%. But when this same steering vector gets used with the whole pipeline, it only changes the final answer's accuracy from 70.53% unsteered to 71.33% steered, which means less than one-tenth as much local effect makes it through to be reflected in the final answer. This assigns tangible values to previous claims about intervention effects and final-answer effects. In general the work is interesting with actionable recommendations.

Critical weaknesses

(1) Not all of the empirical tables make sense.

  • In Table 3, "replay" and "fresh evaluation" have an overall difference of exactly 0.00, aka no difference at all between replaying and doing the experiment from scratch. Appendix E.8 says that the fresh route uses new upstream random draws, so it looks like the records used in the fresh route weren't actually independent of those used in the replay route after all.
  • Table 4 says there isn't anything implemented yet and Appendix D.3 calls this study "unrun". But Tables 2 and 3 are called measured results? I can't tell whether these numbers are real measurements or not based on the paper alone.

(2) As a position paper there seem to be existing papers arguing close variants: Gonuguntla "The Replay Gap" and Lin & Liu "Mechanistic Interpretability Must Disclose Identification Assumptions For Causal Claims".

(3) The paper's title implies there are scaffold-evolving agents, but only the judge's prompt gets changed. This is the easiest case where you can do replay correctly. Appendix F.7 defers to future work on changing the upstream prompts in a way that makes replay impossible.

Questions for authors
  1. The fresh route uses new random draws from the original solver, so how can it match the replay route exactly? Did the fresh route use the same random seeds as the stored records? If not, why do we expect them to produce matching answers?
  2. In the pipeline under Judge J0, accuracy unsteered is only 70.53%, compared to the solver's own 75.49%. This suggests that the judge is actually making things worse; if so, then part of the observed decrease in effect size might just come from a weaker judge. How does this affect our interpretation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4AGQ (1/2)

Thank you for your appreciation and positive feedback. We hope the following clarifications can address your concerns.


Comment 1 of 3: Exact Replay-Fresh Agreement and Execution Status

W1 / Q1: Why Did Replay and Fresh Evaluation Agree Exactly?

Response: Your suspicion was correct. In the original implementation, ordinary upstream calls used greedy decoding despite the stochastic specification, so the new stream identifiers produced the same trajectories. Because the judge score averages over its final choice, identical records gave identical expected effects. We now describe that historical result as deterministic re-execution, not stochastic agreement (see Appendix F.5; Figure 9). The new study separates two tests (§6, "Reconstruction and Fresh-Draw Comparison"):

  • E2a (same keys): All 128 questions reconstruct exactly under the declared criteria, using the anchor criterion in Eq. (12) (see Appendix E.7).
  • E2b (new keys, 512 questions): Persisted, and fresh effects are +5.19 and +2.89 pp. The discrepancies are −4.16 pp [−7.98, −0.35] (off), −1.87 pp [−5.38, 1.65] (on) and +2.30 pp [−0.89, 5.49] (effect). Figure 4 plots them against the planned ±5-pp equivalence band (see Appendix E.8).

We report this as no established equivalence and no diagnosed route bias, and we highlight the off-condition discrepancy, whose interval excludes zero.


W1: "Unrun" versus Measured Results

Response: The stale "unrun" wording has been removed. The paper now separates three evidence periods, shown in the Three Evidence Periods box in Appendix F.5. The new E1, E2a, E2b, controls, timing and stitching diagnostics were all executed (see §5; Appendices E.6-E.9). The corrections (full role inputs, consumed stochastic keys, randomised candidate mapping, raw judge logits and actual termination metadata) were made jointly, so we do not attribute differences to any single repair.


Comment 2 of 3: Novelty and Scope

W2: Overlap with The Replay Gap and Lin & Liu

Response: We agree these are close, and §7 ("Relationship to Existing Arguments") now cites both directly. We do not claim that rerunning a downstream component, or applying conditional expectation, is new. Our contribution is to organise reassessment around a fixed neural probe or intervention: which of its claims (prediction, local effect, final effect) an update affects, and what records support each. Two results make the information conditions precise. The first is a sharp interval for the final effect when judge information is missing (see §3, Eqs. (4)-(8)). The second is the conditions under which saved inputs recover the updated final effect (see §4, Eqs. (9)-(10). Appendix D.5 and Table 3 compare these with bandit replay, doubly robust OPE, CAR, matched controller replay, The Replay Gap, PROTEA and SEAGym.


W3: The Title Implies Scaffold Evolution, But Only the Judge Prompt Changes

Response: We agree the experiment covers the simplest case. §1 now defines "scaffold-evolving" as any persistent change to the components around fixed model weights. It states that our experiment studies the case most favourable to replay (a judge-instruction update without feedback), and that updates altering earlier inputs, tools or memory require moving the replay boundary earlier (see §4, "Partial Records and Earlier Boundaries"). Table 2 (see Appendix D.4) states the decision for upstream-prompt, feedback and missing-hook updates. §7 states the empirical scope plainly: transfer to other models, generative updates, tools or memory is untested.

AuthorsOpenReview ↗

Response to Reviewer 4AGQ (2/2)

Comment 3 of 3: Weak-Judge Interpretation and Ethics

Q2: Could a Weak Judge Explain the Smaller Final Effect?

Response: This is an important alternative explanation, and we added consumer baselines to test it (see §6, "Validity, Consumers and Probe Prediction"; Appendix F.9, Figure 13). Under the final implementation, solver-copy accuracy is 70.70% off and 80.57% on. J0J_0 reaches 78.44%/83.28% and J⋆J^\star 77.74%/82.98%, so both judges improve on copying the solver in each condition. However, reviser-copy reaches 82.42%/81.54%, which exceeds both judges in the off condition. Equal-probability choice gives +4.49 pp, and the label-informed oracle measures candidate opportunity rather than a deployable consumer. These comparisons support reporting the effect under the actual consumer. They do not show that downstream processing generally suppresses steering, or that judge evolution caused the local-to-final gap. Indeed, the update contrast itself is unresolved (+0.40 pp [−0.21, 1.02]).


E1: Ethics Concern: Reporting of Measurements and Favourable Selection

Response: The original ethics paragraph listed integrity risks; it was not an admission that results were planned or selected on favourable outcomes. Nonetheless, the contradictory execution descriptions warranted correction, and we have reconciled every reported number with the retained evidence. The historical J⋆J^\star was selected on a separate 256-question pool (see Appendix E.5). The study reuses that exact instruction without new optimisation, and its 1,024-question panel is described throughout as previously exposed, not a new confirmation (see §5; Appendix E.1). Both judges and all null-compatible contrasts are reported. The Ethics statement now gives a conventional description; validation and human review are in §Appendix E.9.


Thanks again for your thoughtful feedback—we hope you'll consider increasing your score.

AuthorsOpenReview ↗

Follow-Up on Reviewer Comments

Dear Reviewer 4AGQ,

As we near the end of the discussion period, we kindly ask you to review our responses and consider updating your scores if your concerns have been addressed. We're happy to clarify anything further if needed. Thanks

Regards,

Authors

AuthorsOpenReview ↗

Follow-up on Discussion Phase

Dear Reviewer 4AGQ,

We understand you are busy. Since the author-reviewer discussion phase is ending soon, we kindly request that you consider increasing your score if our response has addressed your concerns.

If there are any remaining questions or clarifications needed, we would be happy to respond.

Thank you.

Regards,

Authors

Reviewer_MRvbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues something along the lines of that we need to study how interventions to models impact downstream applications.

Critical strengths

None

Critical weaknesses

As a position paper, I would expect a clearly articulated position but the current text does not satisfy this. It lacks clarity and concise argumentation. I found the text hardly readable.

  • The text is incredibly dense.
  • The text uses terms that are incoherent in the context and does not explain them. It even highlights such words in color to indicate their importance but then does not elaborate what the phrases mean. For example, after reading the paper, I still don't understand what partial records, selector probabilities, or conditional replay are in the context of the manuscript. I could extend this list arbitrarily.
  • There are many weird sentences in this text that lack context or are generally just confusing.
  • Section 3 introduces a lot of undefined mathematical notation or abstracts lots of content in single variables. The presented rectangular class example is not contextualized, as in it is not explained how the math relates to anything in the position argument. The Figure description is just a list of numbers. This makes it impossible to evaluate whether the section is useful.
  • The experiment descriptions are unclear. It is unclear to me what is being tested, or why it is being tested.

In summary, the position does not present a coherent argument to me. I believe extensive revisions are needed before additional review.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer MRvb

Thank you for your appreciation and positive feedback. We hope the following clarifications can address your concerns.


Comment 1 of 2: Position and Terminology

W1: No Clear Position

Response: The position is now one sentence, boxed at the top of §1: after an agent update, revalidate each neural claim at the outcome it concerns, and justify any reuse of earlier evidence. The next paragraph turns it into three consequences: separate the probe's predictive performance, the steering effect on the solver's own answer, and the effect on the final returned answer; trace what the update can affect; and obtain evidence for the affected outcome. §1 then gives a concrete case. Steering can make the solver correct while the judge returns the revision, so the solver's gain need not reach the final score. §3-§4 develop the two consequences this example raises. §7 turns the position into a reporting rule with a four-step Reassessment Checklist (claim, instrument, change, evidence), and §8 restates it.


W2: Unexplained Terms ("partial records", "selector probabilities", "conditional replay")

Response: Each term is now defined in plain language in §2, through the solver-reviser-judge example, before any formula uses it:

  • A selector is the component that chooses between candidate answers; here, the judge.
  • A selection probability is the probability that the judge chooses the solver's candidate. It is not a probe score and not a probability of correctness.
  • A replay boundary is where saved computation ends and execution resumes. A complete record contains everything the resumed computation needs; a partial record omits some of it.
  • Replay (§4) means executing the updated judge on each condition's own saved record.
  • Identification, estimation and reconstruction are clearly distinguished, and Appendix A.1 collects the notation.

We removed the colored keyword highlighting you noted, shortened dense passages, and consolidated related paragraphs (19 run-in headings in the main text became 11).


Comment 2 of 2: Mathematics, Figure 1 and Experiments

W3: Undefined Notation; The Rectangular Class and Figure 1 Lacked Context

Response: §3 now opens with the question it answers. If the solver and reviser are equally correct, the judge's choice cannot change the score. If only one is correct, uncertainty about the judge becomes uncertainty about the final effect. Every symbol is introduced in words before use, and each displayed equation is numbered and referenced in the text: the selection probability in Eq. (4), the score bounds in Eq. (5), the effect interval and its width in Eqs. (6)-(7), and the two-point bound in Eq. (8). We explain the rectangular class in words before stating the result: each selection probability may lie anywhere in its allowed range, with no link across records or conditions. The interval is sharp within that class. Figure 1 is now a chart rather than a list of numbers. On 40% of questions, steering makes only the solver correct. If the judge's probability p of choosing that solver is unknown, the final gain lies in [0, 40] pp; if p ∈ [.3, .7], in [12, 28]; if p = .5, it is 20. The caption explains that more copies of the same records do not narrow the interval unless they add information about the judge. Proofs and refinements are in Appendix B.5.1.


W4: It Was Unclear What the Experiments Test, or Why

Response: §5 now opens by stating the question and the four parts of the study. E1 compares local and final effects under two judges. E2a checks reconstruction by repeating 128 questions with the original random keys. E2b estimates replay-versus-fresh discrepancies on 512 questions with new draws. A matched timing comparison measures what reusing saved inputs saves. Figure 2 shows the dataflow. §6 reports each part with paired 95% intervals: Figure 3 (E1), Figure 4 (E2b), and the reuse-error and timing results. The headline: steering raises solver accuracy by +9.86 pp, but the final answer by only +4.83 / +5.24 pp, a paired gap of +4.63 pp [2.87, 6.39]. This directly supports the paper's distinction between local and final claims.


Thanks again for your thoughtful feedback—we hope you'll consider increasing your score.

AuthorsOpenReview ↗

Follow-Up on Reviewer Comments

Dear Reviewer MRvbj,

As we near the end of the discussion period, we kindly ask you to review our responses and consider updating your scores if your concerns have been addressed. We're happy to clarify anything further if needed. Thanks

Regards,

Authors

AuthorsOpenReview ↗

Follow-up on Discussion Phase

Dear Reviewer MRvbj,

We understand you are busy. Since the author-reviewer discussion phase is ending soon, we kindly request that you consider increasing your score if our response has addressed your concerns.

If there are any remaining questions or clarifications needed, we would be happy to respond.

Thank you.

Regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and New Evidence

We thank all reviewers for their careful reading. The reviews identified an unclear position, contradictory descriptions of what was executed, an implausible exact replay-fresh agreement, missing confidence intervals, and no evidence on the practices the paper argues against. We agree with these criticisms and have revised the paper substantially. All changes are marked in red in the revised PDF so they can be checked quickly.


1. A Clear Position

Response: The position now appears in a box at the start of §1: after an agent update, revalidate each neural claim at the outcome it concerns, and justify any reuse of earlier evidence. §1 also states that "scaffold-evolving" means any persistent change to the components around fixed weights, and that our experiment studies the simplest such change.


2. Terms Before Mathematics

Response: §2 defines selector, selection probability, replay boundary, complete and partial records, and identification, estimation and reconstruction before §3-§4 use them. The main-text equations are numbered (1)-(10), and each is referenced where it is used.


3. A Corrected Empirical Record

Response: Appendix F.5 now separates three evidence periods: the original submission, a later CPU reanalysis of its unchanged records, and a new fixed-instrument study. We explain the greedy-decoding error behind the original zero replay–fresh discrepancy, and we no longer use the label-informed "probe" AUROCs or the invalid precision justification (see Appendices F.4-F.5, Figure 9).


4. A completed Post-Submission Study with Paired 95% Intervals

Response: §5-§6, using the original probe, vector and judge instructions:

  • Local solver effect +9.86 pp [7.75, 11.97]; final effects +4.83 pp [3.09, 6.57] under J0J_0 and +5.24 pp [3.49, 6.99] under J⋆J^\star (see Figure 3).
  • Judge-update contrast +0.40 pp [−0.21, 1.02], Holm-adjusted p = .1981, which is not resolved as nonzero (see §6; Appendix F.2).
  • E2a: Exact discrete reconstruction on all 128 questions with the original keys (see §6; Appendix E.7).
  • E2b: Fresh-versus-persisted discrepancies with intervals, including an off-condition discrepancy of −4.16 pp [−7.98, −0.35] (see Figure 4; Appendix E.8).
  • The practices the paper argues against, measured directly: Reusing the old effect (−0.40 pp [−1.02, 0.21]) and stitching histories (+0.71 pp [−0.80, 2.22]) (see §6; Appendix F.10).
  • Practical Value: 27.11× faster replay than full regeneration on 128 matched questions (see §6; Appendix F.12, Figure 15).

5. Readability

Response: The result tables are now figures with every value printed (see Figures 3-4 in the main text; Figures 6-15 in the Appendix). §7 adds a four-step reassessment checklist. Every appendix subsection is referenced from the main text. Related paragraphs are consolidated under fewer headings.


What We Do Not Claim

Response: We do not claim fresh-draw equivalence, a resolved judge-update effect, transfer beyond judge-only updates, or improved mathematical reasoning: all 116 rescues and 15 harms coincide with answer-validity transitions. These limits are stated in §7 (Scope and Limitations).


Where Each Concern is Addressed

  • No Clear Position (MRvb): Position box in §1; §8 Conclusion.
  • Undefined Terms and Dense Mathematics (MRvb): Definitions in §2; §3 opening; Eqs. (1)-(10); see Figure 1.
  • Unclear Experiments (MRvb): Study purposes at the start of §5; see Figure 2; §6.
  • Exact Replay-Fresh Agreement (4AGQ, qzVW): §6 "Reconstruction and Fresh-Draw Comparison"; see Figure 4; Appendices E.7-E.8 and F.5.
  • "Unrun" versus Measured (4AGQ, qzVW): §5; Appendix F.5 and its Three Evidence Periods box.
  • Missing Confidence Intervals (qzVW): see Figures 3-4; Appendices F.1–F.2.
  • Overlap with The Replay Gap and Lin & Liu (4AGQ): §7 "Relationship to Existing Arguments"; Appendix D.5, see Table 3.
  • Judge-only Scope (4AGQ, qzVW): §1; §4 "Partial Records and Earlier Boundaries"; §7 "Scope and Limitations"; Appendix D.4.
  • Weak-Judge Explanation (4AGQ): §6 consumer baselines; Appendix F.9, see Figure 13.
  • Errors of Old Practices (qzVW): §6 "Reuse Errors and Practical Value of Replay"; Appendix F.10.
  • Replay versus Rerun (qzVW): §6; Appendix F.12, see Figures 14-15.
  • Identification Result Untested (qzVW): End of §3; Appendix F.9 information-set box.
  • Ethics Flag (4AGQ): Ethics Statement; Appendices E.4-E.5 and E.9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.