- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Intervened Partial Replay (IPR), an offline method for repairing failed multi-turn agent trajectories without updating model weights. An external intervenor analyzes a completed trajectory, identifies potential failure points, and generates instructions that are inserted before replaying the remaining interactions. A best-of-N procedure selects a successful repair using outcome feedback. The paper also introduces a turn-indexed reliability analysis based on Kaplan–Meier estimation and mean turn to failure (MTTF). Experiments on τ-bench and two CRAFT-MD subsets compare repair success rates, with additional rerolling baselines on τ-bench and qualitative examples in the appendix.
Critical strengths
-
The paper studies a useful and concrete problem: recovering from a failed interaction while retaining an existing trajectory prefix. Separating the intervenor from the worker makes the approach modular and potentially useful for offline repair in environments that support rollback. The examples show actionable instructions, such as checking availability before modifying a reservation, rather than only generic requests to reconsider an answer.
-
The partial-reroll baseline is a meaningful control because it uses the identified intervention locations without the corrective instructions. Some results suggest that the instructions contribute beyond replay alone. For example, with GPT-4.1-mini on airline tasks, IPR achieves 70% success, compared with 62% for full rerolling and 58% for partial rerolling. This provides encouraging, although not consistent, evidence for the approach.
Critical weaknesses
-
The reliability analysis does not adequately justify what its metrics measure. Section 4.3 records failure at trajectory termination, whereas Section 3.2 discusses failure points occurring earlier within the interaction. A method that simply takes longer to terminate unsuccessfully could therefore receive a higher MTTF without becoming more reliable. Treating successful completion as right censoring also requires a clearer definition of the underlying failure-time quantity and the assumptions supporting that treatment. Moreover, Definition 1 uses a population average exceeding the maximum task length to characterize reliability for all tasks, which does not follow from an average alone. Since reliability measurement is a stated contribution, these issues substantially weaken the interpretation of Figures 4–5.
-
The experiments do not yet establish a consistent benefit over rerolling or substantiate the efficiency claim. In Table 1, several retail improvements over the strongest reroll baseline are only 1.7–2.6 percentage points, without uncertainty estimates for these differences. With GPT-5-mini as both worker and intervenor on airline, IPR reaches 54%, compared with 72% for full rerolling. CRAFT-MD lacks corresponding reroll baselines, so the contribution of intervention cannot be separated from additional attempts. The paper also does not report end-to-end costs including the initial failed trajectory, transcript analysis, intervention generation, replay, and evaluation. Equation (4) substitutes MTTF for the replay location and assumes that Nα can be held constant; this does not establish a general scaling advantage over full rerolling. These gaps leave the main performance–cost claim insufficiently supported.
Questions for authors
The following are clarification requests that could change my assessment, looking forward to receiving your rebuttals:
-
What precise quantity does the Kaplan–Meier curve estimate when failures are recorded at termination and successful trajectories are censored? How is the curve handled beyond the largest observed trajectory length, and how sensitive are the MTTF comparisons to the chosen upper limit?
-
Does the reported trial budget include the initial failed run? What are the total token or monetary costs of IPR and the reroll baselines, including all analysis and evaluation calls? Are repeated-run results or paired uncertainty estimates available for the success-rate differences?
-
In Appendix C.2, the intervention refers to normal imaging and reflexes, but these findings do not appear in the displayed prefix or original failed continuation. Is this a general conditional suggestion, or does it use additional case information? Please specify whether the intervenor can access the underlying patient description or reference answer.
-
In Appendix C.3, both branches terminate when the user supplies confirmation together with ###STOP###, and the original worker already provides the warning requested by the intervention. What evidence supports attributing this failure to intervention timing, rather than an ineffective instruction or the interaction’s termination rule?
Responses 1/3
In Weaknesses:
1). Replay means rerun generally, and reroll (roll as in trajectory rollout) specifically in our context. These three terms are interchangeable in our manuscript but the choices of wording are eventually based on context. In the machine learning community, "replay" was originally popularized in "experience replay" but it has since been used in various related scenarios. In our manuscript, when we discussed replay with full or partial trajectory rollout in our paper (the system and task prompts are fixed as the original run in both cases), we specifically used reroll because it is more specific.
2). Yes. We only insert a single instruction as intervention at an inferred failure point for all examples we have evaluated with our IPR (intervened partial replay) approach.
4). The definition for intervention strategy is simplified: "The procedure to interrupt and alter the trajectory of an agent’s course of actions towards a goal is an intervention strategy".
Confusion about figures: We can understand that Figure 1 and 2 may seem more abstract since they are less about data, but standard conference papers tend to have overall schematics that represent an abstraction of the paper content. Our figures are well-intentioned in their design and they are surrounded by explanatory context.
[1] Albert & de Ruiter, Repair: The Interface Between Interaction and Cognition. Top. Cogn. Sci. (2018)
[2] Kamoi et al., When Can LLMs Actually Correct Their Own Mistakes? TACL (2024)
Definition of Ti: We apologize for the confusion. There is no change of definition for Ti on Line 155 and in Definition 1. The two different ways of describing it mean the same. We unified them as "the total number of turns which task i requires for completion".