acceptodds
Under review as a conference paper at ICLR 2027

Intervened partial replay for multi-turn agent trajectory repair

Abstract

The growing presence of AI agents in real-world applications is contrasted by their very limited reliability in task execution, resulting in agent failures that are intrinsically context-dependent and laborious to fix on a case-by-case basis. Failures in multi-turn interactions often occur at long horizons that are hard to foresee at task initiation. To understand and mitigate these failures, we consider measuring agent reliability in multi-turn interactions as life testing and construct local insertion fixes through agent-agent interactions. Inspired by repair mechanisms in conversations and programs, we propose an offline replay method to produce quality instruction interventions using best-of-N sampling, which induce behavioral changes in the agent trajectory leading to repair. Our replay-based approach delegates trajectory intervention to a separate agent, thereby eliminating the cognitive capacity needed for self-initiated repair. Using a single localized feedback around failure within the agent trajectory, we reduce the inference cost and demonstrate significant performance improvement on existing long-horizon agent tasks without any model capability enhancement, as quantified by the success rate and turn-indexed survival metrics. The results offer insights to designing verbalized controls to shape multiagent interactions that underlie many existing AI agent operations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time scaling, multiagent systems, agent failures, reliable AI, failure mitigation, instruction following

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_NTKqOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Intervened Partial Replay (IPR), an offline method for repairing failed multi-turn agent trajectories without updating model weights. An external intervenor analyzes a completed trajectory, identifies potential failure points, and generates instructions that are inserted before replaying the remaining interactions. A best-of-N procedure selects a successful repair using outcome feedback. The paper also introduces a turn-indexed reliability analysis based on Kaplan–Meier estimation and mean turn to failure (MTTF). Experiments on τ-bench and two CRAFT-MD subsets compare repair success rates, with additional rerolling baselines on τ-bench and qualitative examples in the appendix.

Critical strengths
  1. The paper studies a useful and concrete problem: recovering from a failed interaction while retaining an existing trajectory prefix. Separating the intervenor from the worker makes the approach modular and potentially useful for offline repair in environments that support rollback. The examples show actionable instructions, such as checking availability before modifying a reservation, rather than only generic requests to reconsider an answer.

  2. The partial-reroll baseline is a meaningful control because it uses the identified intervention locations without the corrective instructions. Some results suggest that the instructions contribute beyond replay alone. For example, with GPT-4.1-mini on airline tasks, IPR achieves 70% success, compared with 62% for full rerolling and 58% for partial rerolling. This provides encouraging, although not consistent, evidence for the approach.

Critical weaknesses
  1. The reliability analysis does not adequately justify what its metrics measure. Section 4.3 records failure at trajectory termination, whereas Section 3.2 discusses failure points occurring earlier within the interaction. A method that simply takes longer to terminate unsuccessfully could therefore receive a higher MTTF without becoming more reliable. Treating successful completion as right censoring also requires a clearer definition of the underlying failure-time quantity and the assumptions supporting that treatment. Moreover, Definition 1 uses a population average exceeding the maximum task length to characterize reliability for all tasks, which does not follow from an average alone. Since reliability measurement is a stated contribution, these issues substantially weaken the interpretation of Figures 4–5.

  2. The experiments do not yet establish a consistent benefit over rerolling or substantiate the efficiency claim. In Table 1, several retail improvements over the strongest reroll baseline are only 1.7–2.6 percentage points, without uncertainty estimates for these differences. With GPT-5-mini as both worker and intervenor on airline, IPR reaches 54%, compared with 72% for full rerolling. CRAFT-MD lacks corresponding reroll baselines, so the contribution of intervention cannot be separated from additional attempts. The paper also does not report end-to-end costs including the initial failed trajectory, transcript analysis, intervention generation, replay, and evaluation. Equation (4) substitutes MTTF for the replay location and assumes that Nα can be held constant; this does not establish a general scaling advantage over full rerolling. These gaps leave the main performance–cost claim insufficiently supported.

Questions for authors

The following are clarification requests that could change my assessment, looking forward to receiving your rebuttals:

  1. What precise quantity does the Kaplan–Meier curve estimate when failures are recorded at termination and successful trajectories are censored? How is the curve handled beyond the largest observed trajectory length, and how sensitive are the MTTF comparisons to the chosen upper limit?

  2. Does the reported trial budget include the initial failed run? What are the total token or monetary costs of IPR and the reroll baselines, including all analysis and evaluation calls? Are repeated-run results or paired uncertainty estimates available for the success-rate differences?

  3. In Appendix C.2, the intervention refers to normal imaging and reflexes, but these findings do not appear in the displayed prefix or original failed continuation. Is this a general conditional suggestion, or does it use additional case information? Please specify whether the intervenor can access the underlying patient description or reference answer.

  4. In Appendix C.3, both branches terminate when the user supplies confirmation together with ###STOP###, and the original worker already provides the warning requested by the intervention. What evidence supports attributing this failure to intervention timing, rather than an ineffective instruction or the interaction’s termination rule?

Reviewer_TZ1rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Intervened Partial Replay (IPR) to repair failed multi-turn agent trajectories without updating model parameters. A separate Intervenor identifies potential failure points, generates corrective instructions, and replays the remaining trajectory from each intervention point. A Best-of-N procedure selects successful interventions based on task outcomes. The paper also introduces turn-indexed survival analysis to measure agent reliability. Experiments on τ-bench and CRAFT-MD show improvements in task success across different agent configurations.

Critical strengths
  1. The paper draws on life-testing theory from reliability engineering to assess multi-turn agent reliability, and takes inspiration from conversational and program repair to design localized trajectory interventions. This provides an interesting perspective on improving agent reliability without modifying model weights.

  2. The experiments cover different task domains and agent configurations. Comparisons with self-reflection, full reroll, and partial reroll baselines on τ-bench help evaluate the contribution of instruction interventions beyond simply resampling trajectories.

Critical weaknesses
  1. In L98–99, the author considers tasks with more than 10–20 interaction turns as long-horizon. This definition seems rather conservative, as many modern agent tasks involve hundreds of steps. It remains unclear whether the method would be effective on substantially longer trajectories.

  2. The validity of Survival Function and MTTF as reliability metrics is unclear. Failures are recorded at trajectory termination rather than at the first error, so an agent that fails early but continues interacting may appear more reliable. Treating successful trajectories as right-censored also raises questions about the assumptions of Kaplan–Meier estimation.

  3. The reliability criterion in Definition 1 needs further justification. TiT_i is described as both the observed trajectory length and the turns required by a task. Since MTTF≤T+−T−\mathrm{MTTF}\leq T^+-T^-, any trajectory reaching T+T^+ makes MTTF>Tmax⁡\mathrm{MTTF}>T_{\max} impossible under the observed-length definition. The choice of T+T^+ and the estimation of R(T)R(T) beyond observed trajectories also remain unclear.

  4. IPR requires multiple partial rollouts, error localization, and intervention generation, which appear to incur substantial computational cost. The efficiency claim in Section 3.3 relies on the assumption that NαN\alpha remains constant, which may not hold in practice.

  5. Table 1 should include token consumption and wall-clock time for each method. Comparing methods with the same number of trials does not necessarily reflect their actual computational costs.

  6. The experiments rely on retrospective repair, where the complete failed trajectory and final outcome feedback are available. This may overestimate the method's practical effectiveness. Could the authors evaluate a prospective setting where the Intervenor identifies and repairs potential errors without knowing whether the task has failed?

Questions for authors

See Weaknesses

Reviewer_C9DsOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose Intervened Partial Replay (IPR) for offline repair of unsuccessful multi-turn agent interactions without updating weights. An independent intervenor inspects a completed failed trajectory to find potential failure sites and generates short corrective instructions, which are injected into the trajectory at each potential failure site, and the remainder of the trajectory is regenerated. A successful repair is chosen by best-of-N search guided by verification of the task outcome.

The authors also propose a reliability analysis indexed by turn using Kaplan--Meier survival curves and a bounded mean-turn-to-failure measure, along with an inference-cost motivation for partial replay. Experiments are conducted for airline and retail tasks from τ-bench and two subsets of CRAFT-MD medical dialogue tasks, using various GPT mini models. The results show improved outcomes relative to the original unaided trajectories. Compared to full and partial rerolling for τ-bench, further improvement was observed in some cases, little improvement in others, and a large regression for GPT-5-mini airline tasks. No equivalent controls for retry are presented for CRAFT-MD.

Critical strengths
  1. A concrete and modular approach to post-hoc repair of trajectories in the offline setting.

The proposed approach is decoupled between diagnosing and providing a correction to the trajectory, and executing the task, and re-uses the prefix of the trajectory as a new local instruction to the agent without retraining any of the weights. This is a very useful structure in replayable environments to debug agent behavior. The simplicity of the approach makes the intervention mechanism quite easy to reason about, and possibly to integrate in a pre-existing pipeline. This is the strongest element of the paper.

  1. The τ\tau-bench experiment is conducted with an interesting ablation and presents some promising results in control conditions.

The partial reroll approach using the same failure points but without the corrective instruction in the context (§5) is a good control to check if there is additional value provided by the content of the instruction, beyond re-starting from a point in the trajectory. For instance, we see in Table 1 that 70% is achieved for GPT-4.1-mini on airline tasks using IPR versus 62% with full rerolling and 58% with partial rerolling, providing first glimpses that the intervention can add something beyond more samples, while not fully clarifying how consistent this additional value is (and with what confidence).

Critical weaknesses

What's the reason that the suggested survival analysis cannot be interpreted as task reliability?

In 4.3 it's mentioned that failure is indicated by the end of a failing interaction, and the risk set is defined in terms of unterminated trajectories. In this framing successful interactions get censored out of the risk set. But successful interactions are terminal competing events -- once a task is successfully completed it's no longer at risk for subsequent failure during that interaction. In general, censoring in that way (as described in the competing risks literature, PubMed) doesn't make Kaplan-Meier estimator an estimate of marginal probability of avoiding task failure as claimed.

So if at turn 1 9 out of 10 succeed, and one fails at turn 2 then the estimator will end up with n2=d2=1n_2=d_2=1, and so R^(2)=0\hat{R}(2)=0. Now suppose those same 9 that succeeded finish at turn 3. Then R^(2)=0.9\hat{R}(2)=0.9. That means that just by having successful ones finish sooner you can end up with a lot less of a "reliability" measure reported, even if the final outcomes (and failure time of the unsuccessful task) are the same.

Also, the time of unsuccessful termination in 4.3 seems not to match the onset of error/location of insertion previously referred to in 3.2. Those are not the same thing. So MTTF is not generally able to be used as a proxy for the reusable prefix length in Eq. (4). And the integration in Eq. (1) past the max length trajectory seen, and the fact that Definition 1 is decided by MTTF>Tmax⁡\mathrm{MTTF}>T_{\max} need to be justified, too. (Restricted mean large enough doesn't mean reliable completion across tasks.) So a big claim there falls through.

Questions for authors
  1. Reliability definition and estimation

Is the survival function calculated using the Kaplan-Meier approach with censoring at the successful cases' stop points? If so, what exactly is the intended meaning of the estimator in this case, and how should the counterexample above be understood? Please give the experimental T+ values, and clarify whether the result holds when the successful completion is considered as an absorbing competing event. In Eq. (4), what does the MTTF stand for, and how is it related to the actual stop points of the intervention?

  1. Relative saving by avoiding the re-try

Please give multiple head-to-head comparisons of full and partial rerolling, with confidence intervals for their differences. Especially, the best-of-three settings for CRAFT-MD are important. Please specify the number of repetitions, the aggregation method, and whether the failed initial rollout is included in N for each method.

  1. End-to-end efficiency -- evidence needed for current claim.

Please report the actual token and API costs (including the failed initial rollout) for Docent/intervenor calls, all attempted continuations, and for the judging, and specify how they add up to the saving of roughly a factor 10. Does IPR only increase success for a fixed cost, or does it also reduce the number of needed candidate rollouts (thereby reducing the total computational or monetary cost)?

  1. Information available for the intervenor and the replay implementation

What information is available for the intervenor to formulate the instructions: the full transcript up to the stop point, a failure indication, the correct answer, the hidden user/patient information, the expected database state? Here, verification with a known outcome is explicitly part of the offline scenario -- it is not clear what information is used for the instruction generation, and what for candidate generation. Please clarify the exact procedure for restoring the environment state and the database state at a replay point. Give the cited appendices and the exact Reflexion baseline scenario.

  1. Relation to very similar replay methods

What is the substantial difference of IPR compared to previous diagnosis-guided prefix-preserving replay? In particular, REFLECT already performs diagnosis from complete traces, generates corresponding instructions, replays from close rollback points with injected guidance and outcome verification (but with the main objective of error attribution). So what is the new methodological contribution of IPR beyond application to task-oriented dialogue and the proposed reliability analysis?

Reviewer_LuaeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to improve the reliability of multi-turn agent interactions. It adapts "life testing" to interaction turns, intending to measure the distribution of the number of turns until a failure in the interaction. In addition, it proposes intervened partial replay (IPR), a repair procedure in which a separate intervenor agent examines a completed unsuccessful interaction, identifies a candidate error location, and generates corrective insertions. For each candidate, the procedure retains the conversation prefix preceding the selected intervention point and regenerates the rest of the interaction with the intervenor's instruction inserted. Multiple candidate interventions are tested and a successful repair is retained if one is found. The authors use the reliability metric they introduce to measure the efficacy of their proposed IPR procedure. They test their method on several benchmarks for measuring multi-turn interaction success and compare to several baselines and prior methods.

Critical strengths

Adapting life testing from reliability engineering and lifetime statistics to agent interactions is a potentially useful perspective, as it aims to characterize how failures develop over interaction turns rather than reporting only aggregate task success rates. Furthermore, the IPR procedure shows some evidence of efficacy on certain benchmarks and in certain settings, for example increasing success rate by 8 percentage points on the τ\tau-bench airline benchmark compared to the best baseline method when GPT-4.1-mini models the worker and user agents, GPT-5-mini models the intervenor agent, and best-of-5 sampling is used for picking the intervention.

Critical weaknesses

There are several critical weaknesses to the paper. The examples below illustrate recurring clarity problems rather than an exhaustive list. I encourage the authors to generalize the provided feedback and address these issues consistently throughout the manuscript.

The first weakness is the lack of clarity in the writing, presentation, definitions, notation, and formulas. Several specialized terms (e.g. "life test") are introduced before being adequately explained. The writing is overly complicated and convoluted in certain places, where a much simpler explanation would suffice. For example: 1) the authors could briefly explain that by "replay method" they mean a method that continues to generate the agent-agent interaction after the inserted intervention, 2) they could clarify that "single localized feedback" means that each replay attempt inserts a single corrective instruction at a single selected location in the interaction, 3) they define life tests as assessing "the distribution of persistent system operation before failure occurs," when "the distribution of the number of turns until failure" (after providing a suitable definition for "failure") would suffice, 4) their definition 2 defines an "intervention strategy" as "the task-dependent policy for an agent to interrupt and alter the trajectory of another's course of interaction towards a goal" which could be simplified for example to "a rule for when and how one agent supplies instructions to change another agent's behavior toward a goal." Furthermore, the figures, especially figures 1 and 2 are opaque and it is hard to decipher what they are trying to communicate. There are inconsistencies in the definitions e.g. in line 155 TiT_i is described as the total number of turns in the interaction, but later in definition 1 it is defined as the number of turns a task requires for completion, which may differ. Moreover, notation such as TϵT_\epsilon is introduced without an explicit definition and, although explained verbally, a formal definition for R(t)R(t) is missing. The notation used in some formulas could also be substantially simplified and these formulas could also be explained in words to improve readability, for example equation (2).

Some claims are too strong. For example, the authors state that their definition of a "reliable multi-turn system" (definition 1), which requires the mean turn-to-failure to exceed the maximum number of interaction turns, characterizes the system as reliable "for all tasks." However, the mean could exceed this maximum even while only a small fraction of the tasks truly succeed. Furthermore, the authors state that they "reduce the inference cost" however, to the best of my knowledge, they do not report actual token usage etc.

It is not clear what the failure event actually studied is. Line 154 defines the data as the total interaction length and the final outcome for each task, while lines 184-187 describe the turn at which failure actually occurs (failure onset). These measure different quantities.

Although touched on in the limitations section, it should be made clear at their introduction whether assumptions A1-A3 are needed, what role they play, why they motivate localized repair, and what their limitations are. The language used to describe these assumptions could also be substantially simplified. For example, A1 provides a plausible restart/intervention location, A2 appears to motivate considering distinct failure sources, and A3 explains why earlier intervention can improve later outcome. It is also not clear if "independent" in A2 refers to statistical independence, causally unrelated events, or separately repairable failures.

Questions for authors
  1. Is the failure event defined as the first failure in the interaction, the first detected failure, failure at the end of the interaction, etc.?
  2. For TiT_i, do you consider the number of turns until the end of the interaction, even if unsuccessful, or the number of turns required for successful task completion?
  3. Do you have data comparing different measurements of end-to-end inference time cost such as number of tokens, monetary cost, etc.?
2 comments
AuthorsOpenReview ↗

Responses 1/3

In Weaknesses:

1). Replay means rerun generally, and reroll (roll as in trajectory rollout) specifically in our context. These three terms are interchangeable in our manuscript but the choices of wording are eventually based on context. In the machine learning community, "replay" was originally popularized in "experience replay" but it has since been used in various related scenarios. In our manuscript, when we discussed replay with full or partial trajectory rollout in our paper (the system and task prompts are fixed as the original run in both cases), we specifically used reroll because it is more specific.

2). Yes. We only insert a single instruction as intervention at an inferred failure point for all examples we have evaluated with our IPR (intervened partial replay) approach.

4). The definition for intervention strategy is simplified: "The procedure to interrupt and alter the trajectory of an agent’s course of actions towards a goal is an intervention strategy".

Confusion about figures: We can understand that Figure 1 and 2 may seem more abstract since they are less about data, but standard conference papers tend to have overall schematics that represent an abstraction of the paper content. Our figures are well-intentioned in their design and they are surrounded by explanatory context.

  • Figure 1 illustrates the idea that designing agent trajectory repair pipeline can take inspirations from the benefits of conversational repair and program repair (second paragraph in Introduction) because it contains the characteristics of both. Conversational repair benefits from the turn-taking process (along with other cognitive features of the communicators) to improve its efficiency [1]. While in program repair, fault localization via rerunning modified code across many iterations is possible. This is not possible for conversational repair, but can be absorbed into an agent trajectory repair process to some extent. This is why we put the agent trajectory repair somewhere between the other two.
  • In Figure 2a, we adapted a standard illustration of the LLM self-refinement community [2] for agentic system. Figure 2b uses a concrete example in our data to illustrate the effect of the intervened partial replay framework proposed in our manuscript. Figure 2c illustrates life test across multiple tasks. We have updated the caption of Figure 2 to improve clarity.

[1] Albert & de Ruiter, Repair: The Interface Between Interaction and Cognition. Top. Cogn. Sci. (2018)
[2] Kamoi et al., When Can LLMs Actually Correct Their Own Mistakes? TACL (2024)

Definition of TiT_i: We apologize for the confusion. There is no change of definition for TiT_i on Line 155 and in Definition 1. The two different ways of describing it mean the same. We unified them as "the total number of turns which task ii requires for completion".

AuthorsOpenReview ↗

Responses 2/3

In Weaknesses:

3). Our definition of life test is meant to be general. The "persistent system operation" is quantified at the turn level in this study, but it could also be task runtime, calendar dates, tokens counts (before a failure action), etc. Using the interaction turn to index the test process has practical values in our work.

"Strong claims" in the definition: Our description of a reliable multi-turn system is a definition, not a claim. It can be made more restrictive or more relaxed, but using the mean of TTF (turn-to-failure) data is generally accepted in engineering practices ([1] already cited in our original manuscript). We adapted this convention to agent-agent interactions.

[1] Ascher & Feingold, Repairable systems reliability: modeling, inference, misconceptions and their causes. Marcel Dekker (1984)

Failure and failure points: We feel the distinction between failure and failure points should be emphasized thanks to the reviewer's comment. In common sense, stating that a system involves failure doesn't indicate the full visibility of that failure, which also requires information about its whereabouts, type, and cause, etc. We follow a general definition of failure based on the outcome of the task the worker agent completes (i.e., an agent fails if the task is unsuccessful). We don't use a specific failure taxonomy since it may not be transferrable between datasets. Instead, the type of failure is inferred through the failure point reasoning process (described in Section 4.2).

What the reviewer refers to as "failure events" are better aligned with our failure points. In our setting, the failure points are the locations (in turn indices) in an agent trajectory after which the agent can no longer succeed in the task by itself. The simplest example of a failure point is the turn index at failure onset (i.e., the earliest location in an agent trajectory). An analogous concept exists in the program repair community called fault location [2], which we take inspiration from (Fig. 1). The fault locations are also inferred through the outcome.

We hope these explanations can resolve the reviewer's confusion about line 154 vs. lines 184-187.

[2] Mechtaev et al., DirectFix: Looking for Simple Program Repairs. ICSE (2015)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.