- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose Agent-Reflex, a training framework for helping language agents recover from tool failures and respond to user corrections. The framework first extracts environment-specific recovery skills from successful and failed trajectories. These skills are then used to generate verified examples of local repair, restarting a failed approach, and user-guided correction. Finally, the framework combines RL with skill-conditioned on-policy distillation. Experiments with Qwen3-8B report improvements across five tool-use benchmarks and AgentNoiseBench.
Critical strengths
(1) The paper addresses an important practical problem. Agents operating with external tools need to handle failures, incomplete feedback, and user corrections. Training explicitly for these situations is well motivated and relevant to deployment.
(2) Agent-Reflex reports improvements across general tool-use and noisy-interaction settings. Its overall score in Table 1 is 57.6, compared with 53.6 for Agent-World and 50.9 for EnvScaler, suggesting that the complete training recipe has practical value.
Critical weaknesses
(1) The contribution of the mined reflection skills is not sufficiently isolated. The framework introduces several changes together, including a broad synthetic environment collection, a diverse model pool, a stronger analyzer, verified correction examples, and skill-guided distillation. The results show that the complete pipeline helps, but do not establish whether the particular skill-mining mechanism is necessary.
(2) The baseline comparisons are not documented consistently enough. Section 4.1 describes a shared experimental setting, whereas Appendix C says most headline baseline results are taken from the original papers. Table 1 marks NoisyAgent as reproduced, while Appendix C.2 describes its table results as quoted. Figure 4 describes training from Qwen3-8B, while Appendix D.4 states that ORPO starts from the ReSFT checkpoint. These ambiguities make it difficult to determine whether the comparisons isolate the proposed method.
(3) The ablations do not clearly separate supervision quality from supervision quantity. Removing SR, KR, or UR reduces performance, but it is unclear whether the removed examples are replaced to preserve the number of training tokens and updates. Without this control, the drops could partly reflect less training data. There is also no direct ablation of the diverse-model skill-mining process itself.
(4) The evidence for improved reflection is less conclusive than the performance evidence. Task success and trajectory validity are useful outcomes, but they do not by themselves establish that the agent diagnoses failures and corrects its behavior for the intended reason.
(5) Uncertainty and training cost are incompletely reported for the main comparisons. The appendix states that locally measured scores average three runs and provides bootstrap analysis for some diagnostic results. However, the headline tables and component ablations do not display uncertainty. The substantial offline work used to construct the environments and skills is also not clearly accounted for in comparisons of training efficiency.
Questions for authors
Initialization clarification: Did every method in Figure 4 start from the identical ReSFT checkpoint? Please specify the initialization, training data, noise exposure, and budget accounting for each method.
Baseline clarification: Which entries in Tables 1 and 2 were measured locally, and which were quoted? How were differences in evaluation protocols, decoding settings, and tool-call budgets handled?
(1) When SR, KR, or UR is removed, are its examples replaced to preserve training tokens and updates? If not, how much of the observed drop can be attributed to reduced supervision? (2) What evidence distinguishes the benefit of mined environment-specific skills from generic recovery guidance, direct trajectory corrections, or alternative privileged context under the same data and compute budget (3) Equation 11 assigns a positive likelihood weight even to sampled tokens that become less probable under skill conditioning, although their weight is smaller. What evidence shows that this weighting supplies useful corrective supervision, particularly on failed trajectories? (4) Can you provide variability or paired confidence intervals for the main comparisons and component ablations?
3 comments
Response 2/2
W4. Evidence that the model identifies specific failures and repairs for the intended reason.
Thank you for raising this important point. The submission already uses Stability-Gated Success and 13 diagnostic axes (Appendix E.1, Figure 5): Agent-Reflex leads on all axes, with derailment 28.5% (EnvScaler 44.8%, backbone 56.1%). These results connect gains to resisting specific noise, but do not directly establish type-matched diagnosis.
We therefore add three metrics independent of final task success, measured after the first injected error:
| Metric | Qwen3-8B | EnvScaler-8B | Agent-Reflex-8B |
|---|---|---|---|
| Recovery after the first error | 18.4% | 26.7% | 41.5% |
| Blind-retry rate | 47.2% | 38.9% | 19.6% |
| Recovery action matching the failure type | 22.1% | 31.4% | 48.9% |
| Trajectories derailed by noise (Figure 5) | 56.1% | 44.8% | 28.5% |
Blind retries fall from 38.9% to 19.6%, type-match rises from 31.4% to 48.9%, and post-error recovery rises from 26.7% to 41.5%. These results indicate type-matched diagnosis and recovery at the point of failure, rather than lucky final answers after derailment. We will add these diagnostics to §4.4.
W5 & Q4. Uncertainty for main results/ablations, and full cost including environment and skill construction.
Thank you for the suggestion. Appendix D.5 already states that local results are three-run means, and Appendix E.1 reports bootstrap intervals for the 13 diagnostic axes. We further perform 1,000 task-paired bootstrap resamples:
| Setting | Method/variant | Score | 95% CI |
|---|---|---|---|
| Five-benchmark Overall | Agent-Reflex | 57.6 | [56.4, 58.7] |
| Five-benchmark Overall | Agent-World | 53.6 | [52.3, 54.8] |
| AgentNoiseBench Overall | Agent-Reflex | 29.1 | [27.4, 30.9] |
| AgentNoiseBench Overall | EnvScaler | 18.2 | [16.9, 19.6] |
| ANB-τ² ablation | Full model | 45.9 | [44.5, 47.2] |
| ANB-τ² ablation | w/o SR | 41.3 | [39.8, 42.9] |
Main-comparison and full-versus-w/o-SR intervals do not overlap. Revised Tables 1–3 will report means and CIs.
Offline cost: environment construction ~180 GPU hours and sampling/analysis/verification ~260, totaling about 440 one-time GPU hours; most model calls use self-hosted GPU inference, with limited external API usage. ReSFT ~480 and ORPO ~208 per run; ORPO adds about 18% over equal-step plain RL. Appendix D will separate one-time, per-run, and inference costs.
Q3. Why does Equation 11 retain Δ < 0 tokens, and can soft weighting provide useful supervision from failed trajectories?
Thank you for the careful reading of Equation 11. The weight γ = σ(β·Δ) lies in (0, 1), where Δ is the gradient-detached log-likelihood difference for the same sampled token with versus without the skill prompt. When Δ < 0, γ < 0.5: the OPD term still increases the sampled token's probability, but more weakly. Thus γ controls relative emphasis rather than changing the gradient direction.
We prefer soft weighting because most tokens in a failed trajectory are not themselves incorrect; a hard gate 1[Δ > 0] would discard remaining signal and increase estimator variance, and the sign of Δ is itself noisy under sampling.
We compared soft and hard gating: replacing γ with a hard gate reduces ANB-τ² from 45.9 to 45.2 and increases training-reward variance by about 23%. We will add this result to the appendix to clarify the design motivation.
Thank you again for the constructive suggestions. We sincerely hope these clarifications and additional experiments address your main concerns, and we respectfully ask you to reconsider your score.
A Kind Reminder for Reading the Response
Dear Reviewer WXX1,
We have revised the paper and added additional results to address your comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!
Thank you, The authors of Agent-Reflex
Response 1/2
We sincerely thank the reviewer for the thorough and constructive feedback. The reviewer's careful examination of our equations, tables, and appendix documentation has been invaluable in helping us clarify and strengthen our contributions. Below, we address each concern systematically and comprehensively.
W1 & Q2. Is skill mining necessary after controlling environment scale, model pool, analyzer, data, and compute?
Thank you for the suggestion. Submitted Table 3 shows that removing SR/KR/UR/OPD hurts performance, but does not by itself establish the necessity of environment-level skill mining. We therefore add controls over skill source and use.
First, with data volume and budget fixed, we vary only the skill source:
A mismatched random skill reaches only 52.7, below the matched skill at 55.1, ruling out arbitrary extra context; generic guidance helps but remains below environment-level skills.
Second, we vary SFT supervision and ORPO privileged context under a fixed budget:
These controls separate more data, the presence of privileged context, and its content. The key advantage is reusability across turns and tasks; a per-trajectory correction is tied to one episode.
Finally, with total samples fixed at 20K, we vary only the number of environments covered:
Broader coverage helps with diminishing returns, but matched random/generic/direct-correction controls show that environment count alone cannot replace skill mining.
W2 & Initialization/Baseline Clarification. Clarify baseline provenance, NoisyAgent labeling, and Figure 4 initialization.
Thank you for identifying these documentation issues:
W3 & Q1. Quantity-matched backfilling to separate supervision quality from quantity; direct diverse-pool ablation.
Thank you for the suggestion. Submitted Table 3 removes samples outright. We add quantity-matched variants that backfill removed supervision with the same number of tokens from verified correct trajectories.
After backfilling, removing SR/KR/UR still reduces ANB-τ² by 2.8/1.9/1.6; about 55%–61% of the original drops remain attributable to supervision content. The UR drop concentrates on User-Noise. The diverse-pool ablation appears under W1 & Q2 (53.5 vs 55.1). Revised Table 3 will show both columns.