acceptodds
Under review as a conference paper at ICLR 2027

The Irreversibility Threshold: When Error in LLM Agent Pipelines Outruns Human Correction

Abstract

Large language model (LLM) agents are increasingly composed into pipelines, in which each agent conditions on the output of the one before it. We show that this decomposition pattern carries a structural failure mode: as a multi-agent pipeline deepens, semantic error does not merely accumulate, it compounds. We develop a formal model of error propagation in linear LLM agent pipelines and give the accumulated error a closed form, the cascade equation, from which we derive a geometric lower bound with cascade ratio . Two thresholds follow in closed form: a critical depth , the deepest pipeline that stays within an error tolerance without intervention, and an irreversibility threshold , beyond which no bounded human validator, modelled as a gate that combines domain expertise and provenance traceability with achievable capacity , can restore the output to within . These partition pipeline depth into safe, recoverable, and irreversible regimes. The model reframes reliability, rather than per-agent capability, as the binding constraint on deep agentic workflows, offers a candidate, testable structural account of why many multi-agent generative-AI deployments fail to deliver value, and proposes as a candidate pre-deployment audit trigger for human-oversight requirements such as the EU AI Act. Empirically, two exploratory pilots and two pre-registered, replicated mechanism experiments map the regimes. Relaying facts under a behavioural token budget degrades output by erasure with negligible alteration in all three models tested. Processing steps inject distortion net of a gold-state floor, whereas the same models' relay distortion is indistinguishable from zero in one model and one to two orders of magnitude smaller in the other. A pre-registered coupling assumption, that compression injects distortion, is not supported in relay; in processing it is supported in one of two models, whose fitted parameters place the worst-case handoff at an interior compression, and not in the other. We report the failed calibration gates and an instrument-precision sensitivity that lowers the processing estimates without changing their ordering, except for one model at the value-only extreme of the measured precision interval. Following emerging usage, we call this failure mode a hallucination cascade, and give it, to our knowledge, its first closed-form account.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, multi-agent systems, hallucination, error propagation, reliability, human-in-the-loop oversight, irreversibility threshold, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Pk4mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper models semantic error propagation in linear LLM agent pipelines using the recursion E_k = E_{k−1}(1 + δ_k κ_{k−1}) + δ_k, derives a geometric lower bound E_m ≥ δ_1 r^{m−1} and a critical depth m*, and introduces an "irreversibility threshold" m† beyond which a bounded validator with capacity h̄ < 1 (with h = αβ and E^H = E(1 − h)) cannot reduce error below a tolerance θ.

Two preregistered experiments examine relay and processing pipelines. The results suggest that budget-constrained relays primarily erase rather than alter information, while processing steps can introduce alteration above a gold-state floor. The proposed compression–distortion coupling (A9) is supported in one of two processing models but not in the relay setting. The paper connects this framework to poor enterprise GenAI ROI and proposes m† as a potential EU AI Act audit trigger.

Critical strengths
  • An important, practically motivated problem. Linear and multi-agent LLM pipelines are now a common deployment pattern, and how error propagates across handoffs, and when human review can still recover it, is a real and under-formalized question. Framing reliability in terms of pipeline depth and handoff compression, rather than per-agent quality alone, and connecting it to human oversight, is a useful perspective for both practitioners and the agent-reliability community.
  • Auditable and transparent empirical claims. Hypotheses and gates are preregistered, amendments are dated, failed gates (G2/G3) are reported rather than rerun, and one command regenerates every Section 4 number from logged traces. The authors are also candid about mixed support for A9 and instrument limitations. This materially increases confidence in the reported observations, even where their interpretation is debatable.
  • Useful empirical observation. Fictitious entities and a counterfactual world reduce parametric-memory shortcuts, and gold-state and no-notes arms provide reference floors. The finding that budgeted relays erase while processing steps can introduce alterations suggests that reliability may depend more on substantive processing steps than on agent count, a useful direction for further study.
Critical weaknesses
  • The reported thresholds substantially overstate the safe depth of the paper’s own recursion.
    • The lower bound is extremely loose for the illustrated regimes. Theorem 3.7 is valid conditional on Eq. (2) but discards every subsequent additive injection term. In Fig. 1’s δ_min = 0.08 configuration, Eq. (2) stays within θ only through m = 5 and within θ/(1 − h̄) only through m = 13, versus the reported m* = 30 and m† = 46. For δ_min = 0.04, the exact depths are 10/24 rather than 59/90. Thus, in the δ_min = 0.08 configuration, depths 6–30 labeled “safe” in Fig. 1 are already above θ under Eq. (2) itself, and the “recoverable” regime label is likewise not established by the lower bound. This could affect Fig. 1, App. J, Sec. 4.4, and the central safe/recoverable/irreversible framing. The paper acknowledges that m* is an upper estimate of the safe depth, but never quantifies the gap and still labels these depths "safe."
    • Amplification is assumed, not established. The inherited-error coefficient (1 + δ_k κ_{k−1}) ≥ 1 is built into Eq. (2), so inherited error cannot be attenuated, although a more general propagation coefficient could permit attenuation. The paper characterizes the full-context refinement setting of [7], which empirically exhibits attenuation, as a different information-flow regime; however, such attenuation lies outside Eq. (2)’s dynamics, and in the uncoupled model κ = 1 instead maximizes inherited-error amplification.
    • The error scale is inconsistent with the quantitative use made of it. E is described as ordinal-only, yet the recursion and thresholds require cardinal operations that are not invariant under monotone reparameterization.
  • The experiments do not test the theory’s load-bearing claims, and the oversight and policy conclusions exceed the evidence.
    • Theory and experiments are disconnected. The experiments show erasure under constrained relays and some alteration under processing, but none directly tests inherited-error amplification, geometric growth, m*, m†, or the human-validation quantities α, β, and h̄. Estimates obtained conditional on Eq. (2) cannot independently validate it, and the BIC form contest runs on recall trajectories even though the empirical operationalization assigns recall loss to erasure/compression and separates it from the alteration distortion δ that Eq. (2) tracks.
    • The empirical contrast is confounded and weakly measured. Relay and processing differ in task, data, scoring instrument, and realized compression regime, and Exp B counts reversions to real-world answers as alteration, introducing a knowledge-conflict effect absent from the fictitious-entity relay task. Judge κ = 0.59, altered-label precision is 1/8 in Exp M and 8/16 for entity-kind labels in Exp B, G2/G3 failed, and Haiku's processing-distortion result is indistinguishable from its gold floor at the value-only extreme (p = 0) of the post-hoc correction.
    • Repeated validation is not analyzed. The model allows a validator to be inserted at an intermediate node, but the irreversibility threshold is derived by applying a bounded correction to the accumulated error at the depth being evaluated. It does not analyze a pipeline with repeated corrections throughout the chain. With validators every s steps, the resulting affine recurrence can remain bounded when (1 − h̄) · r^s < 1, so the current m† result does not by itself establish irreversibility under repeated human correction.
    • The ROI and EU AI Act claims are unsupported at their current strength. The model contains no review-cost or ROI mechanism, so it does not establish the proposed explanation for poor enterprise GenAI ROI. The human-validation quantities needed to estimate m† are not measured, and their deployment estimation is only sketched (e.g., a seeded-error audit for β), leaving the proposed EU AI Act audit trigger unsupported empirically.
Questions for authors
  1. Threshold interpretation. For Fig. 1's δ_min = 0.08 configuration, direct iteration of Eq. (2) stays within θ only up to m = 5 and within θ/(1 − h̄) only up to m = 13, whereas the reported values are m* = 30 and m† = 46. Since the bound discards every subsequent additive injection term, how large is this gap in general, and is the "safe" label for depths 6–30 in Fig. 1 justified? Does the same discrepancy affect Sec. 4.4 (m* = 59, m† = 90; exact depths 10 and 24) and the App. J uncertainty interval for m† ([37, 64]), which propagates the same bound?

  2. Amplification assumption. What independently justifies the inherited-error coefficient (1 + δ_k κ_{k−1}) ≥ 1 in Eq. (2), rather than one that could be below, equal to, or above one? Which experiment tests this term directly, rather than estimating quantities conditional on Eq. (2)?

  3. Form contest. How does the BIC contest on recall trajectories test Eq. (2), given that recall loss is attributed to erasure (κ) and separated from the alteration distortion δ that Eq. (2) tracks?

  4. Repeated validation. How does m† extend to pipelines with repeated validators? With validators every s steps, the resulting affine recurrence can have multiplier (1 − h̄) · r^s and remain bounded when this is below 1. Does the current irreversibility result extend to this setting, and if so, what is the corresponding threshold?

  5. Operational audit metric. How would δ, κ, h̄ (or α, β), and θ be estimated in a real deployment reliably enough for m† to serve as an EU AI Act audit trigger, given that no experiment estimates the human-validation quantities?

1 comment
AuthorsOpenReview ↗

Re: thresholds, amplification, repeated validation

Your recomputation is correct, and it is the most useful point in these reviews. Answers in the order of your questions:

Q1 (thresholds). Yes. Under Eq. (2), Figure 1's configuration stays within θ only up to m = 5, not 30 (closed form in our general comment). On the size of the gap in general: the loose threshold overstates m∗m^* by roughly a factor of log⁡(θ/δ1)/log⁡((θ+c)/(δ1+c))\log(\theta/\delta_1)/\log((\theta+c)/(\delta_1+c)) with c=1/κmin⁡c = 1/\kappa_{\min}, about 6 to 7 at κmin⁡=1\kappa_{\min} = 1 and about 12 at κmin⁡=0.5\kappa_{\min} = 0.5. The tight threshold barely depends on κmin⁡\kappa_{\min}, which changes how compression enters the argument. Section 4.4 and Appendix J inherit the same gap.

Q2 (amplification). No experiment of ours tests the inherited-error coefficient directly. The case for it being at least one is a modelling argument, and [7] shows attenuation when agents see the full context. We plan to free the coefficient, Ek=akEk−1+δkE_k = a_k E_{k-1} + \delta_k, and estimate aka_k by injecting a controlled error at an early step.

Q3 (form contest). Agreed. It is fitted on recall, which our own estimators assign to erasure, so it says nothing about the alteration channel of Eq. (2). It should be reported as descriptive only.

Q4 (repeated validation). Correct. With a review every s steps, the recurrence over one cycle has multiplier (1−hˉ) rs(1-\bar h)\,r^s, so the error stays bounded when s<log⁡(1/(1−hˉ))/log⁡rs < \log(1/(1-\bar h))/\log r; staying below θ is stricter and also depends on the injection terms. The design quantity becomes a maximum review spacing, and m†m^\dagger only covers a single final review. This needs its own section.

Q5 (audit metric). As it stands, it could not serve as one: α, β and hˉ\bar h are unmeasured and the seeded-error audit is only a sketch. We will drop the EU AI Act framing until a measurement backs it.

Two smaller points we accept: E has to be read as cardinal under a fixed similarity proxy, since the recursion is not invariant to monotone rescaling; and the relay/processing contrast is confounded as you describe. A matched design (same facts and scorer, relay vs. one processing step) would isolate it.

Reviewer_5HigOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a model of error propagation in LLM-agent chains and derives depth thresholds for a given error tolerance and for when a bounded human reviewer can no longer correct the chain. The experiments show that relay steps mainly lose information by omitting facts, while processing steps introduce factual errors, which seem to increase with compression.

Critical strengths

The paper addresses an important potential reliability problem. The experiments are carefully designed, with useful controls and transparent reporting of negative results and failed calibration gates. Providing a Lean formalization of the theoretical results is also good practice (though I didn't verify whether the statements are correct in Lean).

Critical weaknesses

Overall, although the problem setting of error accumulation in MAS is highly relevant, the paper starts from strong assumptions that it does not sufficiently justify or test against empirical results. In practice, many MAS have access to a common environment, shared memory, or to the original documents directly, rather than depending exclusively on the previous agent’s output. So I am unsure how representative the specific linear-pipeline setting is, especially at large depths.

Experiments do not clearly validate the thesis (Thm. 3.7). The authors find information loss during relay, but it is less clear whether their proposed model of factual error accumulation is also empirically relevant (and these are different processes). Moreover, the processing experiments only evaluate chains of up to 4 agents, which seems insufficient to establish geometric error growth. The alteration measurements also seem noisy, and although the authors account for this and the results for GPT-5.4-mini remain robust, I find that overall the experimental results don’t support convincingly enough that their assumptions and claims apply to real settings.

The theory follows immediately from an assumption that the paper does not empirically validate. The main concern is that Eq. (2) makes the strong assumption that error gets amplified by 1+δkκk−11+\delta_k\kappa_{k-1}, and geometric growth (and most of the paper) is a somewhat straightforward “corollary” of this assumption/modeling choice.

Other minor weaknesses:

Although the authors acknowledge that the lower bound cannot certify safety or recoverability, the paper still labels the corresponding regions as “safe” and “recoverable” (e.g., in Figure 1), which seems misleading given the necessity but not sufficiency that the bound establishes.

The human correction ceiling is assumed, and its justification seems insufficient. In particular, it is unclear why information lost during handoffs would necessarily be inaccessible to a human, who might inspect the original documents and execution logs. The limitation therefore seems insufficiently grounded in practical settings.

Questions for authors
  1. Can you provide experimental evidence that errors compound across agents, rather than reaching a steady state or accumulating additively?
  2. In your main equation, why do you ignore the terms (+δk+\delta_k) when deriving the lower bound for the thresholds? Including these terms could yield tighter bounds.
  3. Does the irreversibility threshold also apply when humans repeatedly review intermediate steps, rather than performing a single correction? What information are reviewers assumed to have access to, and why?
1 comment
AuthorsOpenReview ↗

Re: the amplification assumption and the human ceiling

Your main concern is fair. Eq. (2) builds amplification in, and the geometric growth follows from that choice. Our experiments show erasure in relay and alteration in processing, but at four steps they cannot tell compounding apart from additive or saturating growth.

On your questions. (1) We do not have that evidence yet. What we have in mind is a longer processing chain with a known error planted early, to see what later steps do with it. (2) There was no good reason to drop the +δ_k terms. Keeping them still gives a closed form, exact for constant parameters, and the thresholds come down a lot (numbers in our general comment). It also bears on your point about the "safe" label: the loose bound did not justify it, and even the tight one only marks depths not yet ruled out, so the zones need renaming. (3) As written, the threshold assumes one review at the end. If someone reviews every few steps, the error can stay bounded, and the useful quantity is how far apart those reviews can be.

On access to sources: we assumed reviewers only see what survives the chain. Many real systems keep the original documents, logs or a shared memory, and then traceability goes up and the threshold moves out. Access should be a parameter of the model, not an assumption, and the scope should say plainly that it covers linear handoffs without shared state.

Reviewer_qMvBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a mathematical model to explain how linear LLM agent piples accumulated increasing. It derives two threshold including a safe depth without human intervation and another one indicates the depth which can not be corrected with the human verification. Some experiments are conducted to demonstreate partial of the machnisms.

Critical strengths
  • The research question is interesting and has practical meaning. It attributes the multi-agent failure reason from individual agent to linear agentic workflow architecture and points out with the depth increasing, human verification is difficult to repair.
  • The paper gives a theoretical framework to descreb the error propagation to derive the depth for better human evaluation before deployment.
Critical weaknesses
  • The paper is not well-written containing too many formulas, symbols and numbers which can be provided in appendix. The author need to think about how to improve the readaibility for human.
  • The theoretical assumption is not be sufficiently demonstrated. Only partial of the mechanisms are demonstrated while the derived depth are not. Only maximum four steps are analysed which is much less than derived depth. In addition, lack of human reviewing or verification which is highly related to main claim.
  • The model only evalauted on limited number of models and majority of them are small size. Lack of comparison between various size, model families, types.

The idea is interesting, the paper writing and experiment designing still need a large room of improvement.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Re: readability and scope

Thanks for sticking with a dense paper; the readability complaint is fair, and most of the notation belongs in the appendix. We agree with your substantive points too. The derived depths are never tested (processing stops at four steps, and the thresholds themselves needed correcting, see the general comment), no human reviewer is measured, and a wider range of model sizes and families is needed. All three are on the list for the next version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Two points raised in several reviews

Thanks to all three reviewers. Two issues came up in more than one review, so we answer them once here.

  1. The thresholds are too loose. Reviewer Pk4m is right, and this is also Reviewer 5Hig's Q2. Theorem 3.7 holds, but the thresholds we derived from it drop every injection term after the first. Keeping them, the bound is still closed-form, Em≥δ1rm−1+δmin⁡(rm−1−1)/(r−1)E_m \ge \delta_1 r^{m-1} + \delta_{\min}(r^{m-1}-1)/(r-1) with r=1+δmin⁡κmin⁡r = 1+\delta_{\min}\kappa_{\min}, which gives m∗=1+⌊log⁡((θ+c)/(δ1+c))/log⁡r⌋m^* = 1 + \lfloor \log((\theta + c)/(\delta_1 + c)) / \log r \rfloor with c=1/κmin⁡c = 1/\kappa_{\min}, and m†m^\dagger by replacing θ\theta with θ/(1−hˉ)\theta/(1-\bar h). It is exact when the parameters are constant. For Figure 1 this means m∗=5m^* = 5, m†=13m^\dagger = 13 at δmin⁡=0.08\delta_{\min} = 0.08, and m∗=10m^* = 10, m†=24m^\dagger = 24 at δmin⁡=0.04\delta_{\min} = 0.04, the same values the reviewer obtained by iterating Eq. (2). Figure 1, Section 4.4 and Appendix J all change, and the regimes start much earlier.

  2. Compounding is assumed, not shown. Our processing chains stop at four steps, which cannot separate compounding from additive or saturating growth. The next version needs an experiment that injects a known error early in the chain and tracks whether later steps amplify it, carry it or repair it.

We would rather fix both properly than patch the PDF in the last days of the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.