- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies failure attribution in executable language-model reasoning pipelines, separating representation construction, deterministic execution, and answer evaluation. Through oracle-representation substitutions, generation-budget controls, scorer audits, and a failure census across four datasets, the authors show that superficially similar accuracy gaps can have different causes. A prospective study on MATH evaluates verifier-gated repair across two model families and generation budgets. Execution feedback recovers a substantial fraction of execution-invalid outputs, while a matched self-debugging baseline achieves similar results. Executable but semantically incorrect representations remain largely unresolved.
Critical strengths
- The paper provides concrete evidence that evaluation interfaces can substantially distort conclusions about reasoning ability. The TabMWP budget control and TAT-QA percent-conversion audit demonstrate why end-to-end accuracy alone cannot reliably identify representation-construction failures. These examples are useful because they change the interpretation of substantial measured gaps through targeted controls, rather than merely cataloguing errors.
- The study includes valuable controls and reports limitations candidly. The matched self-debugging comparison attributes the recovery to execution feedback rather than verifier-specific machinery. The authors also report unsuccessful predictions, retained oracle-construction errors, and limited semantic-correction performance. Frozen responses, held-out prospective questions, and bootstrap clustering by problem strengthen the empirical analysis and avoid treating repeated question-condition evaluations as independent problems.
Critical weaknesses
- The prospective interaction does not establish differential repairability across diagnosed classes. In Section 7, execution-valid examples receive no repair under the gated policy. Consequently, follows from the intervention design, rather than demonstrating that these examples would not benefit from repair. Execution-invalid outputs also have zero baseline answer accuracy under this pipeline, so their positive improvement primarily measures successful recovery after receiving additional computation. The authors explicitly acknowledge the untreated nature of , but the abstract and surrounding discussion still present as stronger evidence of predictive attribution than this design supports. The experiment establishes useful recovery among flagged failures; it does not independently establish that the diagnosis identifies greater treatment benefit than an alternative allocation of the same repair budget. This distinction matters because prospective predictive validity is a central claimed contribution.
- The breadth and generality of the attribution findings remain limited. The strongest budget and scorer findings concern particular implementation choices in the authors’ pipelines, evaluated on subsets of 30–100 questions. The prospective repair study is confined to MATH and two model families, while the broader failure census contains only 61 cases. These results establish that such confounds can occur, but provide limited evidence about their prevalence in established executable-reasoning systems. Moreover, the TabMWP control appears to change both the token limit and empty-output regeneration policy, complicating attribution specifically to truncation. The oracle comparison also cannot generally provide a formal upper bound on representation-construction loss when oracle errors are retained and some oracle results use smaller subsets. Together, these issues limit the strength of the paper’s broader methodological conclusions.
Questions for authors
- Clarification of the main estimand: Given that receives no repair, should be interpreted as recovery under a gated policy rather than differential treatment response? Is there an existing matched-budget control that establishes the diagnosis’s value beyond detecting an execution failure and applying standard self-debugging?
- Clarification of the budget intervention: In the TabMWP comparison from 0.850 to 0.967, were the regeneration policy and all decoding settings held fixed while only the token limit changed? Appendix G reports 14/60 truncated outputs, whereas the accuracy difference corresponds to seven questions. How do these counts relate?
- Clarification of the oracle-based attribution: What assumptions justify describing as an upper bound on representation-construction loss, given imperfect oracle representations and differing evaluation subsets? Can the authors distinguish a diagnostic reference gap from a guaranteed bound?
3 comments
Response to Reviewer a6H2 (2/2)
Q1: the estimand, and a matched-budget control
"Clarification of the main estimand: Given that receives no repair, should be interpreted as recovery under a gated policy rather than differential treatment response? Is there an existing matched-budget control that establishes the diagnosis's value beyond detecting an execution failure and applying standard self-debugging?"
Yes: is the recovery rate under the gated policy, and Section 7 states this at the definition ("What the gate measures, and what follows from its construction" above). The matched-budget control you ask for exists in two forms: the allocation of the same repair slots under "Spending the same repair budget uniformly", and the forced re-check on the class the gate leaves alone in Reviewer J4cV's response under "Repair attempted on the class the gate leaves alone". Against standard self-debugging, the matched comparison is in Reviewer UiSC's response under "The self-debugging tie and what the verifier contributes". Revision. Section 6 and Section 7.
Q2: the TabMWP budget intervention
"In the TabMWP comparison from 0.850 to 0.967, were the regeneration policy and all decoding settings held fixed while only the token limit changed? Appendix G reports 14/60 truncated outputs, whereas the accuracy difference corresponds to seven questions. How do these counts relate?"
That comparison is Kimi at 800 tokens against the oracle at 1,200 with one regeneration, so two factors move in it, which is why the paper replaces it with a within-model decomposition. On Kimi the parser scores 51/60 under the v2 scorer at 800 tokens and 58/60 at 1,200 with one regeneration; on GLM-5.3-Flash the decomposition is 55/60 at 800, 48/60 at 1,200 without regeneration, and 55/60 with one regeneration (Section 4; Table 10). Budget alone does not close the gap - 55/60 at 800 falls to 48/60 at 1,200 without regeneration - and one regeneration restores the seven. The 14/60 figure was the 500-token pilot; at 800 tokens there are seven empty responses, which is the same seven questions as the accuracy difference (App. F). Revision. Section 4 and Table 10.
The oracle gap is a diagnostic reference, not a bound
"The oracle comparison also cannot generally provide a formal upper bound on representation-construction loss when oracle errors are retained and some oracle results use smaller subsets."
The paper no longer calls it an upper bound. On the oracle-subset of 30 questions the parser reaches 0.800 against the oracle's 0.933, and on all 60 questions the parser is 0.767, so on the matched set (Table 1; Table 2 (+)). is a lower bound on the headroom attributable to representation construction, because the oracle retains its construction errors (App. V), and the paper now calls a diagnostic reference gap (Section 3; App. A). Revision. the matched denominator is printed in Table 1 rather than only in the appendix, and Section 3 and Section 9 use the reference-gap formulation.
Q3: the oracle comparison as a bound
"Clarification of the oracle-based attribution: What assumptions justify describing as an upper bound on representation-construction loss, given imperfect oracle representations and differing evaluation subsets? Can the authors distinguish a diagnostic reference gap from a guaranteed bound?"
The two assumptions you name are the two the paper states - the oracle retains its construction errors, and the MATH comparison runs on the matched subset (Table 1; App. V). The paper now uses your distinction and calls the quantity a diagnostic reference gap (Section 3; App. A). Revision. Section 3, Section 9, Table 1, App. A, App. V.
Follow-up to Reviewer a6H2
We would like to follow up on our rebuttal before the author-reviewer discussion closes on Nov. 18 at 11:59 p.m. AoE. The rebuttal answers each of your comments in detail. In particular, it covers the interpretation of and . Section 7 now defines as the recovery rate under the gated policy and adds the matched-budget control you asked for. Spending the same repair slots uniformly at random gives 17.8 expected recoveries, against 64 under the gate. A forced re-check on the 107 records that the gate leaves alone changes 20 of them and lowers accuracy by 0.13 (Table 5, Table 11). The rebuttal also covers the TabMWP budget comparison, which is now a within-model decomposition (Table 10), and the oracle comparison, which the paper now presents as a diagnostic reference gap with matched denominators (Section 3, Table 1). We hope these additions give a fuller picture of the work.
We would also like to highlight the main result of the paper. Executable reasoning, as in PAL, Program-of-Thoughts and ToRA, is already a standard approach to quantitative reasoning, and these systems are compared by end-task accuracy. Our controls show that two gaps that such a comparison would attribute to the model come from the evaluation interface. The TabMWP gap of 0.117 shrinks to 2/60 at a matched generation budget with one regeneration, and the TAT-QA gap of 0.15 shrinks to 0.016 once the scorer stops converting percent answers twice. These results are, to the best of our knowledge, new in the literature on executable reasoning. Your review notes that these controls change the interpretation of substantial measured gaps, and we hope the new controls complete this picture.
We would be very grateful if you could read the rebuttal before the discussion closes and tell us what you think. If it resolves your concerns, we kindly ask you to consider raising your score. If anything remains open, please let us know and we will answer right away.
Thank you for your time and consideration. We look forward to your reply.
Response to Reviewer a6H2 (1/2)
We thank the reviewer for their comments.
What the gate measures, and what follows from its construction
Section 7 states it at the definition: E0 leaves the loop at its first verifier pass, so Δ0=0 by construction and Γ is the E1 recovery rate, not a treatment effect on valid representations (Section 7; App. A). Two of the four frozen gate conditions hold for the same reason, Δ0≤0.05 and Γ>0, and the paper states this where they are listed (Section 7; App. O). A flagged representation does not execute, so its baseline accuracy is zero and a positive Γ measures recovery after additional computation rather than a gain on records that could already answer. The empirical content is the invalid rate and the size of the recovery: on the gate-A sample the invalid rate is 6% at the natural budget and 21% under stress on Kimi, and 28% and 65% on GLM (Section 7; App. F), against pooled recovery 64/128 = 0.500 [0.397, 0.609], a cluster bootstrap over the 160 problems, 81 of which ever produce an E1 (Section 7; Table 4; App. A).
Spending the same repair budget uniformly
At equal repair slots it does: 64 recoveries under the gate against 17.8 expected under uniform allocation across the 460 evaluations (3.6x; resampling of stored outcomes, p<0.001; Section 7). The comparison re-allocates stored outcomes, so it spends the same calls and answers whether the diagnosis beats an alternative use of that budget rather than offering a second sampling arm. Revision. Section 7 reports the allocation control next to the pooled estimate, and Figure 2 shows the per-condition effects it aggregates.
W1: the interaction does not establish differential repairability
Both halves of your W1 are answered in the two blocks above: the construction zero under "What the gate measures, and what follows from its construction", the allocation under "Spending the same repair budget uniformly". The abstract now reports both arms in the same sentence, and Section 7 gives Γ as a recovery rate before any claim is made from it. Revision. the abstract, Section 7 and Section 9.
Breadth of the evidence
The paper states the scope where the claims are made: 30-100 questions per split, prospective gates on MATH with two models, and a TabMWP prediction tested by controls rather than by a second prospective gate (Section 9; App. B). Within that scope the census counts and their restored version are printed with the per-case labels behind them (Section 5; Table 9; App. H). No dataset was added in the revision; the generality claim is that these confounds can occur and are separable by control, not that their prevalence across systems is known (Section 9; App. B). Revision. Section 9 carries the design-context paragraph and App. B lists every subset with its sample-id hash.
W2: breadth of the attribution findings
The first half is answered in the block above; the TabMWP control is decomposed in Q2 below, and the oracle comparison is answered in "The oracle gap is a diagnostic reference, not a bound" below. The census is printed with its restored version and its per-case labels (Section 5; Table 9; App. H). Revision. Section 4, Section 5, Section 9 and Table 9.