- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces typed retry trajectories, a structured representation of the failed trajectory logs, and uses these typed retry trajectories to fine-tune a model to predict the whole list of correct tool-calls. They divide the failed trajectory into four parts (instruction, data, requirement and execution), each holding one or more retry-specific tags. All arms are fine-tuned on the same base model, an LLMON-pretrained model, which takes the failed trajectories as input and outputs the correct tool-call chain; the typed versions are written in LLMON markup.
The motivation for this work comes from the way failed trajectories are fed to the agents currently. The entire failed trajectory history is fed into the agent as a single unstructured text: the original task, results, errors, feedback and earlier actions all together. This means that the model (driving the agent) needs to "infer the role and reliability of each part" of the trajectory and decide how to proceed.
The trajectories they experiment on come from 460 recorded runs over 115 tasks (with 182 failed trajectories), from the retail domain of τ-bench. The repair task is to take the failed run and output a correct list of tool calls that would have accomplished the task. The paper trains the model on different arms: (1) plain JSON, (2) the same content as prose logs, (3) channels only, with no new syntax, (4) tags only, without the channels, and (5) tagged channels. The full version with tagged channels gives +0.043 and the channel assignment arm gives +0.088 improvement in tool-name F1 score when compared to the plain JSON. The paper also tests markup pretraining, prompt injection and a retry loop against a replay environment.
Critical strengths
-
Typed retry trajectories for agents are an interesting idea and they may help in certain scenarios (e.g., choosing the right tool or resisting prompt injection).
-
Cross-validation (avoids relying on one small test split) and matched pairs (compares otherwise identical models) are careful choices in the experimental design.
-
The authors are honest about the results. The paper says that there was no improvement from the tagged-channels arm over the channel-assignment arm. They also report in detail the pipeline artifacts that led to their earlier inaccurate results (Section 6). These lessons are very valuable and can be helpful to others using similar pipelines to keep in mind.
-
The idea is worth pursuing with a stronger evaluation or a different experimental design.
Critical weaknesses
-
The model may be ignoring the failed history and just associating the original task (goal) with the tool list. An ablation that trains with only the goal as input, and the tool list as the label, without the trajectory history, would show how much the failed trajectory contributes.
-
The gain is modest (+0.088 tool-name F1 over a plain-JSON baseline of 0.497, Table 2) and only in tool names, not in argument accuracy. This suggests the typed representation has limited benefit in practical terms.
-
There are inconsistencies in the paper: the abstract talks about next tool-call, but the results only deal with the tool-call chain. There are also inconsistencies in how the arms are divided. The paper refers to four arms (line 199), but there appear to be 5 arms. There are references to sub-sections that don't exist (e.g., 4.1, 5.1 and so on). The abstract also says argument accuracy improves, while Table 2 and line 307 say it doesn't.
-
Lack of details:
a. The paper does not describe how the original failed runs were generated - which model was used?
b. verifier_feedback is mentioned but no details of what it entails (or references) are provided.
c. The paper does not name the base model that is fine-tuned to output the tool-call chain, or the models in the matched pairs ("micro (MoE-hybrid)" and "3B (dense)", Table 6). The LLMON work covers several models of different sizes, so this is not obvious from the citation.
d. It would be helpful to have an example prompt per arm.
-
The paper does not take into account related prior work on input reformulation and channel separation defenses. A comparison would clarify what the retry-specific schema adds.
Redesigning the experiments and revising the presentation of the paper to address these points would strengthen the contributions of the method.
Questions for authors
- Please share any experiments you have already run that show the models trained on the goal alone, without the failed trajectories, don't do as well as the models trained on the failed trajectories.
- Beyond the replay-environment loop mentioned in the paper (near Table 6), have you run any experiments on live τ-bench?
- Did you retrain any arm with different random seeds? If so, how much did F1 vary?
- Could you correct the inconsistencies in the paper (as discussed in the Weaknesses section)?
- Could you provide details on which model generated the failed trajectories and what produces the verifier_feedback? Could you also name the models used for fine-tuning and, if possible, include an example prompt for each arm?