acceptodds
Under review as a conference paper at ICLR 2027

Re-derivability Decides What a Staged Agent Pipeline Recovers After an Upstream Fault

Abstract

One variable sets what an upstream fault costs a staged pipeline of language-model agents: re-derivability, how much of what a stage needs it can rebuild from the original problem. Grounding an inspector agent in that problem is worth to over a blind one on four open-weight backbones served with thinking disabled, and on the two Qwen backbones the blind inspector changes no item at all. The inspector helps only when it can see the problem. That head-to-head is exploratory. We manipulate it directly. One deterministic fault enters the first stage, and we re-expose the original problem to of the downstream stages with agents, items, fault and topology held fixed, on 120 gsm_hard items per arm at temperature zero. Accuracy under fault rises on four of four backbones, from to , the largest Holm-adjusted being . A registered kill test rules out tokens. Blanking every word holds the word slots fixed, and retention tracks the visible fraction on four of four, climbing from 0.221 to 0.692 on the primary. The prompt-token ratio between the ends runs 0.90 to 1.03, above one only on Phi-4, whose masked arm still scores worse. Those two families are confirmatory and everything else here is exploratory. The interaction excludes zero on two of four backbones under the registered pipeline, four of four under a three-stage pipeline, and three of four under full message history, which passes prior stage outputs, the primary at . On Llama-3.1-8B the fault carries no detectable cost at any dose, so the other three carry every claim about what a fault costs. Re-derivability also sets what the architecture costs, and no decomposition we measured reliably beats one direct call. With no fault injected the registered pipeline loses to that call by , and , and on Phi-4 reads at , which the test fails to separate from zero. The repair that works is cheap and front-loaded: the first re-grounded stage buys of matched retention for tokens per item on Qwen3-14B, and the stages after it buy nothing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, multi-agent pipelines, error propagation, fault recovery, re-grounding, re-derivability, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_A2bMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how upstream errors propagate through multi-stage LLM agent pipelines and whether downstream agents can recover from such errors by accessing the original task information. The authors focus on re-derivability, which describes the extent to which downstream agents can reconstruct necessary information from the original problem. To investigate its impact, they construct staged reasoning pipelines, inject incorrect intermediate outputs into the first stage, and progressively reintroduce the original problem to downstream agents. They further conduct controlled experiments, including content occlusion and comparisons between grounded and blind inspectors, to examine the factors contributing to error recovery. Experiments across four LLMs show that re-grounding substantially improves accuracy under upstream faults, with most gains achieved by providing the original problem to the agent immediately following the faulty stage.

Critical strengths

None

Critical weaknesses
  1. Poor writing quality and unclear motivation. The paper is poorly written, focusing excessively on minor experimental details and statistical results without a coherent research narrative. The motivation and significance of the research problem are insufficiently justified.

  2. Artificial experimental setting and questionable significance. Recovering relevant information from misleading outputs is important for long-horizon agent reasoning, but the experiments merely inject errors and reintroduce the original problem to downstream agents. It remains unclear what meaningful insights this provides beyond the intuitive benefit of restoring task information, especially since re-grounding also improves performance without faults.

Questions for authors
  1. I recommend that the authors reconsider under what conditions the studied problem is truly important. The current experimental setting makes it difficult to understand the practical significance and value of this research.

  2. I recommend that the authors reorganize the paper to clearly articulate the research motivation, key challenges, and corresponding technical contributions, rather than focusing excessively on experimental details.

2 comments
AuthorsOpenReview ↗

Response to Reviewer A2bM

Thank you for the review; we answer your points one at a time below.

I recommend that the authors reorganize the paper to clearly articulate the research motivation, key challenges, and corresponding technical contributions

The revision is reorganised along these lines. The new introduction starts from the motivation and one worked example, then states the findings, and leaves the statistics to later sections and the appendix.

I recommend that the authors reconsider under what conditions the studied problem is truly important.

It matters whenever a later agent has to work from what an earlier agent wrote and cannot check it against the original task, as in pipelines that pass each agent only its predecessor's output. Sharing more does not remove the problem: in the pre-registered replication on 480 new items, in the configuration of ours that comes closest to group-chat frameworks (every agent sees the problem and the whole transcript), the same fault still costs 0.14 to 0.27 in accuracy. Where each agent sees only its predecessor's output, one false step and a wrong answer line in the first agent's output cut Qwen3-14B's accuracy to 13.7% of its no-fault level of 0.850 (three-stage pipeline, exploratory; Section 8, Table 9).

the experiments merely inject errors

A single injected fault is a simplification, which we chose because each run can then be compared with the same run without the fault. We have since tried four other fault types, a lower severity and a later injection site on the two Qwen3 models (exploratory; see the comment on the exploratory runs). Re-grounding helps more with the fault than without it in 9 of the 12 model-and-fault cells; the exceptions are a reasoning-error fault (both models) and the lower severity on Qwen3-8B.

It remains unclear what meaningful insights this provides beyond the intuitive benefit of restoring task information, especially since re-grounding also improves performance without faults.

You are right that the submission did not test this directly: it reported the difference (Section 7, Table 6) without making it the main test, and in the main pipeline the difference was clear on only two of the four models. It is now the primary test of a pre-registered replication on 480 new items: re-grounding raises accuracy by about 11 points more with the fault than without it, pooled over the four models, and the difference is clear on three of them (not on Llama-3.1-8B). So the benefit is partly fault-specific and partly just undoes the cost of splitting the task, and we say so in the revision.

The other result we find informative is that the usual remedy, an inspector agent, does not help on any of our four models if it sees only the messages, while the same inspector given the problem helps on all of them. This held on the original 120 items (Table 8, exploratory) and again on 240 new items (see our comment on the inspector comparison).

Thank you for your rebuttal. There remains significant room for improvement in readability and experimental significance. This paper should be polished and include experiments on a real agent benchmark with an up-to-date agent. I suggest that the authors consider evaluating the capabilities of agents based on closed-source models like GPT and Claude on a long-horizon agent benchmark. Without comparison with a SOTA harness, the contribution of this paper is less convincing. Overall, I maintain my original score.

Reviewer_KUj3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether downstream stages' ability to re-derive necessary information from the original problem affects recovery from upstream faults in staged LLM-agent pipelines. The authors inject a fault into the first stage and vary the number of downstream stages receiving the original problem and the fraction of visible words. Across four models, they evaluate accuracy and relative performance against no-fault conditions with matched problem access. They also compare a single direct call, inspectors with different access to the problem, a three-stage pipeline, and a full-message-history configuration.

Critical strengths
  • Analyzing whether each stage can access the evidence it needs, rather than focusing only on agent count or graph structure, is well motivated and practically relevant to agent-system design.
  • Including a matched no-fault condition at each level of re-grounding and using occlusion to distinguish information content from input length are appropriate design choices.
Critical weaknesses

The basis for describing the experiments as confirmatory is insufficient. Section 3 and Appendix A state that the analysis plan was fixed after the point estimates of all four models' dose curves were known. Test sidedness, family membership, thresholds, and the selection of confirmatory contrasts were also unprotected. Fixing the inferential procedure before computing the statistics is meaningful, but it does not provide the same protection as an independent confirmation designed before observing outcomes. A prospectively fixed evaluation on new items is needed. The overall effect of re-grounding does not consistently establish an effect specific to fault recovery. In Table 6, Qwen3-14B improves by +0.392 under fault, but also by +0.300 without fault. The difference-in-differences is +0.092 [−0.025, +0.208], which includes zero. The primary configuration detects a fault-specific effect on only two of four models, while Llama shows no detectable fault cost. The additional three-stage and full-history results partly address this concern, but they are exploratory. The title and central claims should therefore be scoped to the evidence under each configuration. (Sections 5, 7–8) There is a gap between the experimental scaffold and practical agent systems. The primary configuration passes only the predecessor's output, and the full-history condition passes earlier stage outputs rather than the original problem. Section 2 also states that the system prompt continues to say the original problem is unavailable even when it is supplied again. Disclosing this issue is appropriate, but the effect should be verified with consistent prompts and configurations that share the original problem by default. (Sections 2, 8, 10) Generalization is limited. The headline results depend on the first 120 gsm_hard items in file order, one fault type and injection site, and open-weight models with thinking disabled. Additional non-mathematical tasks are included, but the same injector does not impose comparable damage across task families. Further evaluation is needed to establish whether re-derivability explains recovery when tasks involve tool observations or external states that cannot be reconstructed from the original problem alone. (Sections 2, 9)

Questions for authors
  • What justifies calling the two families confirmatory when the analysis plan was fixed after observing all models' point estimates? Can the authors provide results on new items with contrasts, sidedness, and analysis procedures fixed in advance?
  • How do the conclusions change when the fault-by-re-grounding interaction is the primary endpoint? Can the authors distinguish solving the problem again from correcting the faulty upstream information?
  • Do the results persist after removing the system/user prompt contradiction and under actual framework configurations that share the original problem by default?
  • Can the evaluation be extended across fault types, severities, and injection sites, including tasks with tool observations or external states that cannot be reconstructed from the original problem?
3 comments
AuthorsOpenReview ↗

Response to Reviewer KUj3

We thank the reviewer for the detailed and constructive review.

What justifies calling the two families confirmatory when the analysis plan was fixed after observing all models' point estimates? Can the authors provide results on new items with contrasts, sidedness, and analysis procedures fixed in advance?

We have dropped the word, since it was not justified. As you suggested, we posted an analysis plan in this forum on November 12 and only then ran 480 new items. The 480 items were drawn by a fixed hash of their index from GSM-Hard file positions 400 to 1,318 (counting from 0, as in the plan), none of which the submission used, so they are no longer the first items in file order.

How do the conclusions change when the fault-by-re-grounding interaction is the primary endpoint?

It is the primary endpoint of the replication. Pooled over the four models D is 0.110 (95% interval 0.082 to 0.138). For Qwen3-14B, whose interval included zero in the submission, D is 0.092 (by coincidence the submission's point estimate) with an interval of 0.040 to 0.146 once the contradictory system line is removed; with the original prompts it is 0.031 and its interval includes zero (see below). Qwen3-8B and Phi-4 also pass, whereas Llama-3.1-8B does not. In this pipeline, without re-grounding, the fault costs Llama only 0.02 in accuracy (0.154 vs 0.133), as part 2 of our plan anticipated. The fault-specific claim therefore holds for three models, and we scope the title and abstract accordingly. In the three-stage pipeline, which the plan listed as a secondary analysis, D is above zero on all four models, Llama included (0.140 with the system line removed, 0.081 with the original prompts).

Can the authors distinguish solving the problem again from correcting the faulty upstream information?

Only partly; the counts are in our separate comment on correcting versus solving again, which also answers Reviewer WYV8. On the new items, recovery is 0.41 to 0.65 on items a single direct call can solve and 0.07 to 0.15 on the others, and a later stage rarely says anything is inconsistent. That looks like solving again, with the two caveats given there: those items are easier, and a silent correction would not show up.

Do the results persist after removing the system/user prompt contradiction and under actual framework configurations that share the original problem by default?

The primary test above already uses prompts without the contradictory system line. The line turned out to matter for one model: with the original prompts D on Qwen3-14B is 0.031 with an interval that includes zero, and removing the line raises it by 0.060, while the other three models barely change (the numbers are in the results comment). We did not run an actual framework. The closest we have to the default of group-chat frameworks such as AutoGen is a configuration in our own harness: every later agent gets the full transcript and, when re-grounded, the problem as well. There D is above zero on three models (not on Llama), and the fault still costs 0.14 to 0.27 in accuracy with everything shared.

Can the evaluation be extended across fault types, severities, and injection sites, including tasks with tool observations or external states that cannot be reconstructed from the original problem?

We tried four other fault types, a lower severity and a later injection site, on the two Qwen3 models and 240 new items. These runs are exploratory and are reported in the comment on the exploratory runs, together with runs that have thinking enabled. The exception is the reasoning-error fault: on Qwen3-14B re-grounding helps less with it than without it (D = -0.150, interval -0.217 to -0.083), and we do not yet know why. That fault (on both models) and the lower severity on Qwen3-8B are the three cells, out of 12, whose interval of D is not above zero, and we will list both as limitations. On the non-mathematical families you are right that the injector does not do comparable damage; Section 9 also says that state-tracking families "collapse and then recover", which our own Table 5 contradicts (grid navigation regains at most 0.56 of its no-fault accuracy), and we will correct that sentence. We did not build a task in which the needed value exists only in a tool observation, although we agree it is the right test of where re-grounding must stop helping, and the revision names it as the main open question.

AuthorsOpenReview ↗

New items, analysis fixed in advance

You asked for new items with contrasts, sidedness and analysis procedures fixed in advance, and for the fault-by-re-grounding interaction as the primary endpoint. That test is in Section 3 and Appendix B.1 of the revised PDF (Table B1, page 17). Its plan was posted in this forum on November 12, before any of the 480 problems was run. With consistent prompts, as you asked, D averages +0.110 [+0.082, +0.138] over four models and passes after Holm correction on three, the plan's criterion.

I would like to thank the authors for conducting additional experiments, revising the paper, and providing responses. The key weaknesses and questions have been addressed; accordingly, I have updated my rating.

Reviewer_WYV8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on how to recover the upstream errors occured in the linear agentic pipeline. The author design an interesting testing by manually adding errors at first stage to see the result change on math problems. They find giving the original problem to downstream stages, especially the stage right after the error stage. This is a interesting study but writing is too complex and experimenting setting is relatively narrow.

Critical strengths
  • The research question is interesting and reveals information passing patterns between agents.
  • The design is clever: it varies access to the problem step by step, with a matched no-fault control for each condition.
Critical weaknesses
  • The core idea is simple but the paper writing is too complicated and difficult to follow. There is still a long way to be a publish-ready version from paper-writing perspective.
  • The experiments are not comprehensive only evalauted on math questions which is also fixed.
  • The evaluated model and data should be further explored including more model families, various size etc.
  • The finding is not that superising, adding the original question context can improve the performance is aligned with the common sense.
Questions for authors
  • Did you add the length-matched control after seeing the one-line results?
  • Do re-grounded stages correct the errors, or just solve the problem again?
2 comments
AuthorsOpenReview ↗

Response to Reviewer WYV8

Thanks for the review and for the two direct questions, which we answer first.

Did you add the length-matched control after seeing the one-line results?

Yes. The one-line control gave odd readings on Phi-4 and Llama, so we added the length-matched filler to check them, and in the submission it changed the verdict on two models (Qwen3-14B and Llama). The submission marks it as exploratory and we will make this explicit in the text. On the 480 new items both controls were in the posted plan before anything ran, and there the two controls agree in sign on every model (see the results comment), so nothing depends on the late addition any more.

Do re-grounded stages correct the errors, or just solve the problem again?

Mostly they seem to solve it again, though we cannot rule out correction; the counts are in our comment on correcting the error or solving again, which also answers KUj3. On the new items, 0.41 to 0.65 of the items a single direct call can solve are recovered, against 0.07 to 0.15 of the others, and a later agent mentions an inconsistency in only 1 to 7 recovered items per model. Because those items are easier and a silent correction would not show up in our search, this is only suggestive.

adding the original question context can improve the performance is aligned with the common sense

We agree that it is not surprising that re-grounding helps. We were surprised, however, that the usual remedy, an inspector agent, does not help unless it also sees the problem; on Llama it even hurts. This was an exploratory result on the original 120 items (Table 8), and it held again on 240 new items (see our comment on the inspector comparison). We also did not expect so much of the damage to remain. On Qwen3-14B the fully re-grounded faulted pipeline scores 0.600 vs 0.867 without the fault on the original items (Table 6), and 0.575 vs 0.696 on the new ones, with the system line removed.

The evaluated model and data should be further explored including more model families, various size etc.

Section 9 does include six non-math reasoning tasks with 120 items each, but our fault costs little on several of them, so the main claims rest on GSM-Hard. The models cover three families (Qwen3, Phi-4, Llama-3.1) between 8B and 14B parameters. We did not add a larger model or another family during the discussion period, which the revision states as a limitation.

We agree about the writing. In short, an agent that sees only a faulty message has nothing else to work from; given the original problem it can work the answer out again, and a checker that reads only the same messages cannot. The new introduction is built around that idea.

AuthorsOpenReview ↗

What the original question adds

That the original question helps is expected. What Table 1 on page 5 adds is that the later agents fail with the facts in hand. On 480 new problems, the faulted message lists every number of the problem in over 90% of cases, yet passing those agents every earlier message raises accuracy under the fault by at most 0.04. A blanked copy of the problem, nearly as long, adds at most 0.015. A readable copy adds 0.17 to 0.33.

Reviewer_tkzZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates re-derivability, which measures how much of what a stage of language-model agents needs it can rebuild from the original problem. The paper differs kk, which indicates how many stages following the wrong intermediate one are grounded, and presents corresponding recovery of the agent performances across different LLM architectures.

Critical strengths

The paper investigates a problem that can occur in almost any agentic systems using language models. The paper presents confidence intervals of the measured performance metrics across 4 different language model architectures, which addresses potential concerns the reviewers might have on the stochasticity and generalizability of the results.

Critical weaknesses
  1. While the investigation seems to have been done systematically, the results presented in the paper do not seem to be significant enough. There is no interesting theoretical explanation of the confirmed phenomenon or a novel method proposed by the authors that can effectively mitigate the problem.
  2. The paper in general is very hard to read and understand, probably due to a heavy use of AI for writing and lacking sufficient human polishing. The sentences in general are not clear, bearing many terms that are assuming the reader's deep understanding of the project itself. There are trivial writing mistakes as well, such as using past term in one of the non-conclusion sections (line 160, We fixed). I believe this is actually a very serious issue that damages the paper's contribution to the community of human researchers and human-based conferences like ICLR.
Questions for authors

How is disabling thinking mode of each base LM justified in the scenario? Are many of the language model-based agentic systems not relying on reasoning before tool calls, in domains like coding and math?

1 comment
AuthorsOpenReview ↗

Response to Reviewer tkzZ

We thank the reviewer for the comments.

The sentences in general are not clear, bearing many terms that are assuming the reader's deep understanding of the project itself.

We agree. The abstract, introduction and conclusion are new. Terms that only made sense inside our project, such as "post-pin", "headline standard" and "matched retention", are gone from the main text of the revision, and most of the statistical detail has moved to the appendix. At line 160 you pointed to the past tense in "We fixed the analysis plan". Section 3 now opens in the present tense and says that we wrote the plan down after seeing the point estimates and did not change it afterwards.

How is disabling thinking mode of each base LM justified in the scenario?

We kept thinking off so that every stage reliably hands on an answer. With thinking on, a stage can spend its token budget on reasoning and pass on nothing usable: in our new runs, even with 8,192 tokens per stage, Qwen3-8B hits the cap in 0.163 to 0.354 of the items, depending on the condition. Phi-4 and Llama-3.1-8B have no thinking mode in any case. The paper should have said this.

Since many agent systems do reason before they act, we have now run Qwen3-14B and Qwen3-8B with thinking enabled on 240 new items, with up to 8,192 tokens per stage. The fault still costs 0.45 in accuracy on both models, and re-grounding all three later agents still helps more with the fault than without it (D of 0.23 and 0.11). The Qwen3-8B number is the less reliable one because that model often runs into the token cap. The details are in our comment on the exploratory runs.

There is no interesting theoretical explanation of the confirmed phenomenon or a novel method

The paper is an empirical study, not a new method or a formal theory. Its main result now rests on a pre-registered replication on 480 new items: the fault-specific gain from re-grounding is 0.110 pooled (95% interval 0.082 to 0.138) and holds on three of the four models. The paper compares two remedies and gives a simple account of why one of them works: an agent that sees only the faulty message has nothing to check it against, and an inspector that reads the same messages is in the same position. Re-derivability is our name for this, i.e. how much of what an agent needs it can rebuild from its own input.

The inspector result points the same way, on the original 120 items (Table 8, exploratory) and again on 240 new ones (see our comment on the inspector comparison). Reading only the messages, the inspector does not turn a single wrong answer right, or a right one wrong, on Qwen3-14B: accuracy stays at 0.208 on the original items and at 0.200 on the new ones. Given the problem as well, it reaches 0.817 and 0.725. On the original items, re-grounding just the one agent after the fault takes Qwen3-14B from 0.208 to 0.533 for about 60 extra tokens per item (Table 11), and on all 480 new items, with the system line removed, from 0.231 to 0.515.

Comments on the paper

10 comments
AuthorsOpenReview ↗

Pre-registered replication plan (1/3)

The main results in the submission are all on 120 GSM-Hard items. We will rerun the main experiment on 480 new items, and we fix the analysis here before running any of them. In the revision the main claims rest on this replication alone; every analysis on the original 120 items is labelled exploratory. The plan takes three comments because of the length limit. They are posted together.

Some terms first. By a pipeline we mean a chain of LLM stages in which only the first stage sees the problem. Each later stage sees only the previous stage's output. Pipeline A has four stages, B has three. The fault is a corruption that we insert into the first stage's output. Re-grounding means giving the original problem back to k later stages, counting from the stage that receives the faulted output (the second stage). With k = 0 there is no re-grounding; k = 3 covers all later stages of A, and k = 2 all later stages of B. A condition is one such setting run on every item.

acc(F, k) is accuracy with the fault and acc(C, k) accuracy without it, with k stages re-grounded. The quantity we care about is D = [acc(F, k=3) - acc(F, k=0)] - [acc(C, k=3) - acc(C, k=0)], i.e. the gain from re-grounding with the fault minus the same gain without it. D > 0 says that re-grounding helps more when there is a fault to recover from. For pipeline B read k = 2 in place of k = 3.

Items are from GSM-Hard, the PAL file gsmhardv2.jsonl (1,319 items, md5 abb0ee0a37d56d9898c146168e57496c). We drop the first 400 items: they include the submission's 120 and others we have already run. That leaves a pool of 919, at 0-based file positions 400 to 1,318. The item at position i gets the key sha256("ICLR2027-50217|i"), i in decimal, and the sample is the 480 pool items with the smallest keys. As a JSON array in key order the sample list has md5 542b3e0ddaec8d417661609efb5fc7ea.

Same four models and checkpoint revisions as the submission: Qwen3-14B (40c0698), Qwen3-8B (b968826), Phi-4 (2db69c1) and Llama-3.1-8B-Instruct (ungated mirror d10aef7), served with vLLM 0.30.0 on one 32 GB GPU. The 8B models run in bf16. Qwen3-14B and Phi-4 are loaded with vLLM's FP8 quantization. In bf16 Qwen3-14B was left with a KV cache of 9,760 tokens, which is too small for the exploratory thinking runs in part 3 (8,192 tokens per stage plus the prompt). And it was slow: the first condition, with nothing cached, took 146 s in bf16 against 15 s in FP8. Phi-4 is about the same size, so it is served the same way. Part 2 measures what FP8 changes. The launcher refuses to start unless its settings match this line:

SERVING PRECISION: Qwen3-8B bf16, Llama-3.1-8B bf16, Qwen3-14B FP8, Phi-4 FP8.

Decoding follows the submission: temperature 0, at most 1,024 tokens per stage, thinking disabled, answer read from the last ANSWER: line. We use the submission's wrong-intermediate fault. It inserts a fixed false step, saying that the result must be doubled, halfway down the first stage's output (counted in lines), and it forces a wrong number into that output's ANSWER: line, adding the line if there is none.

In the submitted prompts a re-grounded stage is told two things that contradict each other: its system line says the stage does not have the problem, and its user message then supplies the problem. So we run each pipeline both ways: with the submitted prompts verbatim (A-orig, B-orig), and with that system line removed from re-grounded stages (A-consistent, B-consistent).

We add one more variant, which we call the shared transcript. It is A-consistent, except that every later stage is given the full transcript of earlier outputs; at k = 3 every stage then has both the problem and that transcript. This is as close as we get to the default of group-chat frameworks such as AutoGen, though it runs in our own harness.

There are three controls, all from the submission. Normally a re-grounded stage still receives the previous stage's output. The two replacement controls (on A-consistent, k = 3) put a single fixed line in its place, or filler of the same length, to see whether the stage uses that output at all. The occlusion control (on A-orig, k = 3, with the fault) replaces words of the returned problem with blanks, so that 0, 25, 50, 75 or 100% of them stay visible. If the benefit came only from the added tokens, hiding the words would not matter. Apart from the pipelines and controls, each model answers every item once in a single direct call.

AuthorsOpenReview ↗

Pre-registered replication plan (2/3)

Before fixing this plan we re-ran the submission's 120 items on the new server, to see whether the setup reproduces the submitted numbers. That is 11 conditions per model: A-orig at k = 0 to 3 with and without the fault, the two replacement controls (on A-orig here, as in the submission) and the direct call. D is within 0.025 of the submitted value for every model: Qwen3-8B +0.283 (submitted +0.258), Llama-3.1-8B +0.033 (+0.050), Qwen3-14B +0.083 (+0.092), Phi-4 +0.375 (+0.400).

Item by item the agreement is looser. We compared each condition with the submission and counted the items whose correctness changed. The largest count in any one condition was 7 (Qwen3-8B), 10 (Llama-3.1-8B), 21 (Qwen3-14B) and 17 (Phi-4); the last two are the models served in FP8. Part of this is the server's own run-to-run variation. To measure it we ran both 8B models a second time on the same bf16 server with an empty call cache: the two runs differed on up to 9 (Qwen3-8B) and 10 (Llama-3.1-8B) items in one condition, while D moved by at most 0.008. For Qwen3-14B there is a bf16 run to compare with. It completed 10 of the 11 conditions and then stopped during the direct call. Those ten differ from the submission on at most 4 items each, and D = +0.083. So FP8 changes more individual answers of Qwen3-14B than bf16 does (up to 21 items in one condition, against up to 4), but D comes out the same. We have no bf16 run of Phi-4.

We had planned to allow at most 4 changed items per condition (8 under FP8). Two runs on the same bf16 server already disagree on as many as 10 items, so neither limit can serve as a check. We drop both. From here on only D is compared with the submission; the per-condition counts are reported without a limit. On the new items every condition runs on this one server, so any difference between our server and the submission's cannot enter a comparison within the replication. Run-to-run variation is still there, as noise in every condition.

There is one primary test: D on A-consistent, averaged over the four models. We get the uncertainty from an item bootstrap: we resample items with replacement 10,000 times (numpy default_rng(0)), using the same resampled items for all models and conditions. The test is one-sided at level 0.05, with H1: D > 0. The primary hypothesis is supported if the 5th percentile of the resampled average D is above 0.

For each model we report D, its 95% interval and a one-sided bootstrap p-value, namely the share of the same 10,000 resamples in which that model's D is at or below 0. We correct the four p-values by Holm's method; a model passes if its corrected p-value is at most 0.05. We will say that the effect holds per model only if at least 3 of the 4 models pass.

Neither the primary test nor the per-model criterion is evaluated until all four models have finished. With fewer, the analysis script issues no verdict.

We expect a positive average D. Llama-3.1-8B should sit near zero, since the fault costs it little. The interval of Qwen3-14B may again include zero (submitted: +0.092 [-0.025, +0.208]). To check the script we applied both rules to the submitted pipeline-A results on the original 120 items. The average D was +0.200 (one-sided lower bound +0.154), which passes the primary test, but only 2 of the 4 models passed, one short of the per-model criterion.

The rest is secondary and is reported whatever the outcome.

We report D on A-orig, B-orig, B-consistent and the shared transcript. The effect of the contradictory system line is measured as D on A-consistent minus D on A-orig. For the shared transcript there is one more number, the remaining fault cost, acc(C) minus acc(F) at k = 3.

The submission's two families of tests are repeated on the new items. They are k = 0 against k = 3 on A-orig, both with the fault, and 0% against 100% of the problem visible in the occlusion control. Each uses exact one-sided McNemar tests, with Holm's method over the four models.

For each replacement control we give its accuracy minus acc(F, k=3) on A-consistent, with an interval but no test. Direct-call accuracy is reported as it is. All intervals are two-sided 95% percentile intervals from the same resampling.

AuthorsOpenReview ↗

Pre-registered replication plan (3/3)

If the primary test is supported, the per-model criterion sets the scope of the claim. With at least 3 of 4 models passing, we will claim that re-grounding has a fault-specific benefit in a pre-registered replication, on average and for the models that pass, and we will name any model that fails. With fewer than 3 passing, we will claim the fault-specific benefit only on average over these four models. Statements about a single model will then follow that model's own result, and the title will not imply that the effect holds for each model.

If the primary test is not supported, we withdraw the claim that re-grounding is a remedy specific to faults. The revision will then claim only that returning the problem restores information a stage has lost, with or without a fault, and the title will drop the fault-recovery claim.

The two families of tests in part 2 are read per model: a family replicates for a model if that model's Holm-corrected p-value is at most 0.05. Where a family fails for some model we will say so and remove the claims that depend on it.

Until this plan is posted, our driver and launcher refuse to run any set of items that overlaps the pool. The same holds if any file below no longer has the md5 stated here. Paths are relative to the code root.

Driver, analysis, readouts: scripts/run_rebuttal.py md5 f2db5fa2ad13fcf81d62abeff0fbf52e, scripts/prereg_analysis.py md5 187c0020d6ff4c92e5832a98e1c60992, scripts/think_fs_readout.py md5 f44410ccfea53cd14ba655a7fb4063b3

Pipeline, fault, model calls, item sample: fimots/pipeline.py md5 69a5fe7a26a7f5300ebf67f66b9bb297, fimots/fault.py md5 3d82b9092ee48e46ff9c1e5843864d03, fimots/llm.py md5 27a074ec3f91300ccedc82c3323165ba, rebuttal_items.py md5 c6029a7884cb08ecf254626e2020dbbc

Answer parsing, scoring, result files: fimots/executor.py md5 1a3b87790950ed15c8b444cdb477fb3b, resultio.py md5 5f50a893bdfa3f3e53e2b8a56b7b7245, fimots/tasks/__init__.py md5 97cb9dbcc3cf2f3d81f673cc46a3bc68, fimots/tasks/base.py md5 7c48cdf43799a15ceb0db787e638654e, fimots/tasks/cascade.py md5 b1466b19efe12cfc290cf02849371675, fimots/tasks/gsm8k.py md5 365387e0383278a402ad7315fdca6018, fimots/tasks/lastletter.py md5 1c5aef8654ca989e0ab16ca8a4cdc4ef, fimots/tasks/math500.py md5 3672bf84de13ef5d1265f38a8d6a30d1, fimots/tasks/mmlu.py md5 80647dba1b0a6a6490312a241d1ec513

We will post the hashes of the result files together with the results. The per-item results and the code will be released in an anonymized repository.

Two more sets of runs on the new items are exploratory and outside this plan. Both use only the two Qwen3 models and the first 240 sampled items; neither starts before this plan is posted. We will report them in full but rest no main claim on them.

One enables thinking, with at most 8,192 tokens per stage. If a stage is cut off at that cap, the item counts as wrong in that condition. The same analysis is repeated on the items where no condition was cut off, and we give the cut-off rate per condition.

The other is a fault sweep: four further fault types (reasoning error, message truncation, role drift, malicious noise), one lower severity and one later injection site. Where a fault type makes random choices, its seed is the item's file position. The lower severity inserts the same false step at the same place but forces no wrong answer, so its edits are a strict subset of the full fault's. The later site puts the fault into the second stage's output, which leaves two later stages; k = 2 then takes the place of k = 3, and the site has its own no-fault conditions. For the other variants we report D with the no-fault conditions of A-consistent on the same items.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.