- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how upstream errors propagate through multi-stage LLM agent pipelines and whether downstream agents can recover from such errors by accessing the original task information. The authors focus on re-derivability, which describes the extent to which downstream agents can reconstruct necessary information from the original problem. To investigate its impact, they construct staged reasoning pipelines, inject incorrect intermediate outputs into the first stage, and progressively reintroduce the original problem to downstream agents. They further conduct controlled experiments, including content occlusion and comparisons between grounded and blind inspectors, to examine the factors contributing to error recovery. Experiments across four LLMs show that re-grounding substantially improves accuracy under upstream faults, with most gains achieved by providing the original problem to the agent immediately following the faulty stage.
Critical strengths
None
Critical weaknesses
-
Poor writing quality and unclear motivation. The paper is poorly written, focusing excessively on minor experimental details and statistical results without a coherent research narrative. The motivation and significance of the research problem are insufficiently justified.
-
Artificial experimental setting and questionable significance. Recovering relevant information from misleading outputs is important for long-horizon agent reasoning, but the experiments merely inject errors and reintroduce the original problem to downstream agents. It remains unclear what meaningful insights this provides beyond the intuitive benefit of restoring task information, especially since re-grounding also improves performance without faults.
Questions for authors
-
I recommend that the authors reconsider under what conditions the studied problem is truly important. The current experimental setting makes it difficult to understand the practical significance and value of this research.
-
I recommend that the authors reorganize the paper to clearly articulate the research motivation, key challenges, and corresponding technical contributions, rather than focusing excessively on experimental details.
2 comments
Thank you for your rebuttal. There remains significant room for improvement in readability and experimental significance. This paper should be polished and include experiments on a real agent benchmark with an up-to-date agent. I suggest that the authors consider evaluating the capabilities of agents based on closed-source models like GPT and Claude on a long-horizon agent benchmark. Without comparison with a SOTA harness, the contribution of this paper is less convincing. Overall, I maintain my original score.
Response to Reviewer A2bM
Thank you for the review; we answer your points one at a time below.
The revision is reorganised along these lines. The new introduction starts from the motivation and one worked example, then states the findings, and leaves the statistics to later sections and the appendix.
It matters whenever a later agent has to work from what an earlier agent wrote and cannot check it against the original task, as in pipelines that pass each agent only its predecessor's output. Sharing more does not remove the problem: in the pre-registered replication on 480 new items, in the configuration of ours that comes closest to group-chat frameworks (every agent sees the problem and the whole transcript), the same fault still costs 0.14 to 0.27 in accuracy. Where each agent sees only its predecessor's output, one false step and a wrong answer line in the first agent's output cut Qwen3-14B's accuracy to 13.7% of its no-fault level of 0.850 (three-stage pipeline, exploratory; Section 8, Table 9).
A single injected fault is a simplification, which we chose because each run can then be compared with the same run without the fault. We have since tried four other fault types, a lower severity and a later injection site on the two Qwen3 models (exploratory; see the comment on the exploratory runs). Re-grounding helps more with the fault than without it in 9 of the 12 model-and-fault cells; the exceptions are a reasoning-error fault (both models) and the lower severity on Qwen3-8B.
You are right that the submission did not test this directly: it reported the difference (Section 7, Table 6) without making it the main test, and in the main pipeline the difference was clear on only two of the four models. It is now the primary test of a pre-registered replication on 480 new items: re-grounding raises accuracy by about 11 points more with the fault than without it, pooled over the four models, and the difference is clear on three of them (not on Llama-3.1-8B). So the benefit is partly fault-specific and partly just undoes the cost of splitting the task, and we say so in the revision.
The other result we find informative is that the usual remedy, an inspector agent, does not help on any of our four models if it sees only the messages, while the same inspector given the problem helps on all of them. This held on the original 120 items (Table 8, exploratory) and again on 240 new items (see our comment on the inspector comparison).