- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper tests whether a simple uncertainty rule helps a failed ReAct-style QA agent choose where to regenerate its trajectory. The rule backtracks two steps from the step with the highest perplexity. Across a main HotpotQA study, replications on two models and three datasets, and budget and deadline variants, the rule never beats restart or positional controls. The key methodological contribution is a length-matched swap control: failed trajectories of equal length exchange their uncertainty-chosen origins, so both arms have exactly the same origin distribution. Two confirmatory swap studies, powered on discordant pairs, are also null: −0.17pp at 32B and +1.48pp at 72B.
Critical strengths
- The rigor is exemplary. Protocols are frozen before execution, and cohorts and exclusions are documented. Inference uses paired question-level bootstrap intervals and exact sign-flip tests with Holm correction. Replay, scoring and budgets are fully audited, and a reproduction supplement is provided. Negative results reported this carefully are rare in the agent literature.
- The swap control is clever. Both arms share an identical set of origins, so it separates what the uncertainty signal adds from mere positional preference more cleanly than a position control fitted on development data. It could be reused to evaluate other origin selectors.
- The discussion of statistical resolution is also valuable. The authors point out that an exact sign-flip test over m non-zero blocks cannot report a p-value below 2/2^m, so several earlier cells could never have reached significance. They respond by powering the confirmatory cells on discordant pairs rather than on failures.
Critical weaknesses
- The tested selector is weak, which limits significance. The rule (highest-perplexity step minus two) is the simplest possible one. It is motivated only by the authors’ own uncontrolled, unpublished observations, not by any claim in the literature, so refuting it tells readers little. The authors also say the conclusions do not extend to entropy, action disagreement, verbalized confidence, trained diagnosers or search-based methods.
- There is no headroom analysis. The paper never shows that origin choice matters in this setting. It reports neither an oracle upper bound (the best origin chosen in hindsight) nor how much success varies across origins. Without these, the null cannot separate “perplexity is uninformative” from “origin choice is unimportant in this task.”
- Floor effects and budget limits may drive the result. All policies succeed on about 10% of attempts. The treatment’s effective origin is zero in 62% of rows, so it often turns into a restart. In addition, 168 of 339 treatment attempts stop at a token or step cap without producing an answer. This suggests the recovery budget is too tight. Results under looser budgets are needed; the uncapped runtime study covers only 25 questions.
- Verification is incomplete. The internal audits had not reported by submission. The 32B confirmatory cell finished only 100 of its 131 informative pairs, which deviates from its protocol. The 72B cell should be presented as the primary confirmatory result.
Questions for authors
- Budget sensitivity: Does the swap control give a different result under a 2× or looser recovery budget?
- Equivalence testing: Would you consider an equivalence test against a prespecified smallest effect size of interest? What effect size would you regard as practically meaningful?
- Alternative signals: The swap framework applies directly to other signals such as entropy or action disagreement. Why not test at least one alternative signal within it?
5 comments
Response to Reviewer SuYH - Part 2
W2. Whether origin choice matters
We agree that this analysis was missing. The revision adds two analyses, in the Analysis and Discussion paragraph "Does origin choice matter in this setting?" and in the appendix named in Table R1. The first is a cross-fitted choice among tested origins. For each failed question with at least two tested origins, we chose the origin with the highest mean success on two seeds and scored it on the third. The 95% bootstrap intervals resample questions, or whole pairs in the swap cells.
Table R1. Cross-fitted headroom. Source: appendix "Exploratory Headroom, Precision and Allowance Analyses".
| Cell | Questions | Chosen (%) | Comparator | Comparator (%) | Difference, points [95% interval] |
|---|---|---|---|---|---|
| Main | 113 | 13.27 | Restart | 11.21 | +2.06 [−1.18, +5.90] |
| Main | 113 | 13.27 | Treatment | 9.73 | +3.54 [0.00, +7.67] |
| Diagnosis 1× | 91 | 13.92 | Restart | 11.36 | +2.56 [−0.37, +6.23] |
| Diagnosis 1× | 91 | 13.92 | Treatment | 10.62 | +3.30 [−1.10, +8.06] |
| 72B swap | 248 | 14.78 | Treatment | 12.90 | +1.88 [−0.13, +4.03] |
| 72B swap | 248 | 14.78 | Swap | 11.42 | +3.36 [+0.94, +5.91] |
| 32B swap | 200 | 14.00 | Treatment | 11.50 | +2.50 [0.00, +5.33] |
| 32B swap | 200 | 14.00 | Swap | 11.67 | +2.33 [+0.17, +4.67] |
Chosen origins gain 1.88 to 3.54 points over the policy. The two intervals that lie fully above zero are both against the swap arm, and two more have a lower limit of 0.00. These are cross-fitted references limited to tested origins and chosen from two noisy seeds, not bounds on a hindsight oracle. A deployable selector would lack the other seeds.
The second analysis tests whether origins are consistent across seeds. A positive statistic means that an origin succeeding on one seed tends to succeed on another seed of the same question. The null permutes origin labels within question and seed, 2,000 times. Holm covers the four all-origin tests, and the retained-prefix test drops origin zero.
Table R2. Origin consistency, one-sided permutation p. Source: appendix "Exploratory Headroom, Precision and Allowance Analyses" for the p-values. The counts of questions with any variation were computed for this reply from the packaged trial exports and are not in the paper or the current supplement.
| Cell | Questions with any variation | p, all origins | Holm p | Retained-prefix questions with any variation | p, retained-prefix origins |
|---|---|---|---|---|---|
| Main | 41 of 113 | 0.004 | 0.016 | 18 of 60 | 0.109 |
| Diagnosis 1× | 27 of 91 | 0.007 | 0.022 | 8 of 22 | 0.077 |
| 72B swap | 72 of 248 | 0.029 | 0.059 | 21 of 62 | 0.475 |
| 32B swap | 59 of 200 | 0.037 | 0.059 | 25 of 98 | 0.978 |
The statistic needs variation on at least two seeds. This holds for 19, 12, 35 and 17 questions across all origins but for only 8, 2, 11 and 5 retained-prefix questions, so the retained-prefix tests are too small to tell whether consistency differs among origins that keep a prefix. Origin choice appears to carry some signal across seeds that perplexity did not turn into a gain over restart, which partly separates your two readings.
Edited on Nov 12 after partial internal audit reports: two p-values in Table R2 and the description of the cross-fitted reference.
Response to Reviewer SuYH - Part 3
W3 and Q1. Floors, tight caps and the swap at 2×
We partly agree, and we correct one fact about the runtime study below. The completed swap cells ran only at 1×. To answer this, we have started one new cell, a 72B length-matched swap on fresh HotpotQA questions with twice the token allowance and 16 new steps instead of 8, under a protocol frozen before launch. It keeps the same primary exact sign-flip test and adds an equivalence test with a 3-point margin fixed before launch (corrected after launch, Part 4). Tokens and steps change together, so the cell cannot separate the two limits.
The prespecified diagnosis follow-up ran restart and uncertainty at three allowances on the same 118 failures, with 354 seed attempts per cell. The cap reading below is exploratory.
Table R3. Success and cap stops by allowance. Source: appendix "Exploratory Headroom, Precision and Allowance Analyses" for success and uncertainty any-cap stops, and the "Recovery allowance" paragraph for the uncertainty token-cap stops at 1× and 2×. The other columns were computed for this reply from the packaged trial exports and are not in the paper or the current supplement.
| Allowance | Success, uncertainty (%) | Success, restart (%) | Token-cap stops, uncertainty | Token-cap stops, restart | Step-limit stops, uncertainty | Any-cap stops, uncertainty (%) | Any-cap stops, restart (%) |
|---|---|---|---|---|---|---|---|
| 0.5× | 2.82 | 2.82 | 335/354 | 336/354 | 0/354 | 94.63 | 94.92 |
| 1× | 9.89 | 10.45 | 114/354 | 111/354 | 73/354 | 52.82 | 53.67 |
| 2× | 11.02 | 12.43 | 2/354 | 2/354 | 139/354 | 39.83 | 38.14 |
The token cap binds at 1× but rarely at 2×, where the eight-step limit becomes the binding cap. Uncertainty minus restart is −0.56 points at 1×, 95% interval [−3.67, +2.54], and −1.41 points at 2×, [−4.80, +1.41], with Holm p 1.000 for both. These contrasts are in the appendix "Diagnosis Follow-up: Complete Results and Costs".
At 1× in the 72B cell, 493 of 744 treatment attempts (66.26%) and 512 of 744 swap attempts (68.82%) stopped at a cap, compared with 49.56% (168 of 339) of main-study treatment attempts. The swap cells may therefore be more constrained by the allowance, and the new cell tests this directly. The 62% origin-zero rate, 210 of 339 rows, means that treatment and restart share all three executions on 70 of 113 main questions. On the other 43 questions, treatment minus restart is −3.88 points, 95% interval [−10.85, +2.33], as reported in the appendix "Distinct Intervention, Resources and Metric Sensitivity".
The runtime study removed only the total generated-token cap. It kept the eight-new-step limit and the 512-token per-step limit. Of its 225 attempts on 25 questions, 79 stopped at the step limit and 0 at a token limit. It is therefore a looser token budget, and the step limit still binds.
In the revision, the "Recovery allowance" paragraph and the appendix "Exploratory Headroom, Precision and Allowance Analyses" report the success rates and the uncertainty and swap-cell cap rates. The other Table R3 columns come from the analysis output. The Budget Scope subsection states that the runtime experiment retains the step limits, and future work names the swap under looser token and step limits.
W4. Verification and ranking of the confirmatory evidence
We agree, and the revision follows the suggestion. The complete 72B cell is now primary. It ran all 124 informative pairs, and 51 are discordant, with 30 positive and 21 negative. Uncertainty succeeded on 12.90% and the swapped origin on 11.42%, a difference of +1.48 points, 95% interval [−1.48, +4.44], p=0.397 from the exact pair-block sign-flip test. The completed design has power 0.96 at a discordant-pair win probability of 0.75 and 0.84 at 0.70. These are powers of a sign-only test on the 51 discordant pairs. The weighted sign-flip test used for p has lower power, 0.89 and 0.69, with the observed block magnitudes held fixed. The packaged analysis of the cell reports zero replay, scoring or allowance mismatches over 1,488 trial rows.
The 32B cell is a declared deviation. It stopped at its frozen time limit with 100 of 131 informative pairs run and 31 unrun. Its difference is −0.17 points, 95% interval [−3.17, +2.84], p=1.000. The internal auditors are two co-authors, and the revision says so. Partial audit reports arrived during the discussion period, and the revision corrects what they found. It still calls internal verification incomplete.
In the revision, the Abstract and Conclusion report the 72B cell before the 32B cell, and the section "Replication across Models and Datasets" names it our primary confirmatory result. The Abstract and that section call the 32B cell a declared deviation, the Conclusion calls it truncated, and the appendix "Confirmatory Length-Matched Swap" reports it as a deviation.
Edited on Nov 12 after partial internal audit reports: the power description, the audit status and the equivalence test.
Response to Reviewer SuYH - Part 1
We thank Reviewer SuYH for a careful and constructive review and for three concrete questions. The review notes that the arms of the length-matched swap share one origin set, and that few discordant pairs cap the sign-flip p-value. In response, the revision adds new analyses of the existing runs and makes the complete 72B cell the primary confirmatory result. This reply also adds an agreement analysis for two other uncertainty signals. The new analyses were specified after all outcomes were known and are exploratory. The exception is the allowance success rates, which come from the prespecified diagnosis follow-up. P-values are unadjusted unless marked Holm, and paper pointers refer to the revised PDF by section and appendix name.
W1. Simplicity of the selector
We agree that the rule is the simplest one and that a null on it has limited reach. The rule came from our earlier uncontrolled runs. FLARE (Jiang et al., 2023, EMNLP) and DRAGIN (Su et al., 2024, ACL) use low token confidence to trigger retrieval and then regenerate the affected text within one generation, and the revision now cites both in Related Work. The use of the rule on a finished failed agent trajectory and the two-step backoff are ours.
The claim covers this rule, the two open-weight model families tested and offline QA. Table R5 under Q3 shows how often two other signals pick the same origin. In the revision, the future-work paragraph names the swap control on other signals, such as entropy and action disagreement.