acceptodds
Under review as a conference paper at ICLR 2027

Agent-Reflex: Teaching Language Agents to Act Reflectively

Abstract

Large language models have evolved from single-turn chatbots into interactive agents capable of long-horizon reasoning. In real deployments, MCP tool services can return noisy feedback and users may intervene to correct the agent, but existing methods provide limited supervision for reflection learning and failure recovery. To bridge this gap, we propose Agent-Reflex, a general training framework for incentivizing proactive reflection in agents. Specifically, Agent-Reflex comprises three stages: Environment Reflection Skill Discovery (RSD) injects grounded tool-feedback perturbations into environment scaling, samples a capability-diverse LLM pool, mines matched success–failure pairs, and summarizes reusable reflection meta-skills. Building on this skill library, we propose a two-stage reflection training framework: (1) Reflective Supervised Fine-Tuning (ReSFT) converts each skill into supervision for self-reflection, skill recovery, and user-centric correction, strengthening complementary reflective behaviors; and (2) On-Policy Reflective Policy Optimization (ORPO) combines agentic reinforcement learning under noisy interactions with reflection-guided on-policy distillation to further elicit proactive reflection. Experiments on six challenging agent benchmark suites demonstrate the effectiveness of Agent-Reflex. Further analyses show that Agent-Reflex improves agents’ robustness to noise while maintaining stable training dynamics, supporting scalable agent training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Agents, Reflection Learning, Environment Scaling, Agent Reinforcement Learning, On-policy Distillation, Failure Recovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WXX1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose Agent-Reflex, a training framework for helping language agents recover from tool failures and respond to user corrections. The framework first extracts environment-specific recovery skills from successful and failed trajectories. These skills are then used to generate verified examples of local repair, restarting a failed approach, and user-guided correction. Finally, the framework combines RL with skill-conditioned on-policy distillation. Experiments with Qwen3-8B report improvements across five tool-use benchmarks and AgentNoiseBench.

Critical strengths

(1) The paper addresses an important practical problem. Agents operating with external tools need to handle failures, incomplete feedback, and user corrections. Training explicitly for these situations is well motivated and relevant to deployment.

(2) Agent-Reflex reports improvements across general tool-use and noisy-interaction settings. Its overall score in Table 1 is 57.6, compared with 53.6 for Agent-World and 50.9 for EnvScaler, suggesting that the complete training recipe has practical value.

Critical weaknesses

(1) The contribution of the mined reflection skills is not sufficiently isolated. The framework introduces several changes together, including a broad synthetic environment collection, a diverse model pool, a stronger analyzer, verified correction examples, and skill-guided distillation. The results show that the complete pipeline helps, but do not establish whether the particular skill-mining mechanism is necessary.

(2) The baseline comparisons are not documented consistently enough. Section 4.1 describes a shared experimental setting, whereas Appendix C says most headline baseline results are taken from the original papers. Table 1 marks NoisyAgent as reproduced, while Appendix C.2 describes its table results as quoted. Figure 4 describes training from Qwen3-8B, while Appendix D.4 states that ORPO starts from the ReSFT checkpoint. These ambiguities make it difficult to determine whether the comparisons isolate the proposed method.

(3) The ablations do not clearly separate supervision quality from supervision quantity. Removing SR, KR, or UR reduces performance, but it is unclear whether the removed examples are replaced to preserve the number of training tokens and updates. Without this control, the drops could partly reflect less training data. There is also no direct ablation of the diverse-model skill-mining process itself.

(4) The evidence for improved reflection is less conclusive than the performance evidence. Task success and trajectory validity are useful outcomes, but they do not by themselves establish that the agent diagnoses failures and corrects its behavior for the intended reason.

(5) Uncertainty and training cost are incompletely reported for the main comparisons. The appendix states that locally measured scores average three runs and provides bootstrap analysis for some diagnostic results. However, the headline tables and component ablations do not display uncertainty. The substantial offline work used to construct the environments and skills is also not clearly accounted for in comparisons of training efficiency.

Questions for authors

Initialization clarification: Did every method in Figure 4 start from the identical ReSFT checkpoint? Please specify the initialization, training data, noise exposure, and budget accounting for each method.

Baseline clarification: Which entries in Tables 1 and 2 were measured locally, and which were quoted? How were differences in evaluation protocols, decoding settings, and tool-call budgets handled?

(1) When SR, KR, or UR is removed, are its examples replaced to preserve training tokens and updates? If not, how much of the observed drop can be attributed to reduced supervision? (2) What evidence distinguishes the benefit of mined environment-specific skills from generic recovery guidance, direct trajectory corrections, or alternative privileged context under the same data and compute budget (3) Equation 11 assigns a positive likelihood weight even to sampled tokens that become less probable under skill conditioning, although their weight is smaller. What evidence shows that this weighting supplies useful corrective supervision, particularly on failed trajectories? (4) Can you provide variability or paired confidence intervals for the main comparisons and component ablations?

3 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank the reviewer for the thorough and constructive feedback. The reviewer's careful examination of our equations, tables, and appendix documentation has been invaluable in helping us clarify and strengthen our contributions. Below, we address each concern systematically and comprehensively.


W1 & Q2. Is skill mining necessary after controlling environment scale, model pool, analyzer, data, and compute?

Thank you for the suggestion. Submitted Table 3 shows that removing SR/KR/UR/OPD hurts performance, but does not by itself establish the necessity of environment-level skill mining. We therefore add controls over skill source and use.

First, with data volume and budget fixed, we vary only the skill source:

Skill sourceANB-τ²ANB-VitaOverall
No skill (correct trajectories only)31.27.151.3
Random skill from a mismatched environment36.58.852.7
Generic skill35.88.653.0
Single-model mining (Qwen3-8B pairs only)39.89.353.5
Backbone as analyzer40.29.753.9
Diverse-model-pool mining41.910.855.1

A mismatched random skill reaches only 52.7, below the matched skill at 55.1, ruling out arbitrary extra context; generic guidance helps but remains below environment-level skills.

Second, we vary SFT supervision and ORPO privileged context under a fixed budget:

ControlOverall after ReSFTOverall after identical ORPO
Generic Guidance53.054.1
Direct Correction54.255.3
Hindsight rewrite as OPD context—56.6
Random environment skill as OPD context—55.8
Mined environment-level skill (ours)55.157.6

These controls separate more data, the presence of privileged context, and its content. The key advantage is reusability across turns and tasks; a per-trajectory correction is tied to one episode.

Finally, with total samples fixed at 20K, we vary only the number of environments covered:

Environments used for skill mining100250500
ReSFT-stage Overall53.254.155.1

Broader coverage helps with diminishing returns, but matched random/generic/direct-correction controls show that environment count alone cannot replace skill mining.


W2 & Initialization/Baseline Clarification. Clarify baseline provenance, NoisyAgent labeling, and Figure 4 initialization.

Thank you for identifying these documentation issues:

  • Tables 1/2 provenance: We quote original papers whenever possible and reproduce locally only when a checkpoint or required benchmark result is unavailable. Local results use official protocols and are averaged over three runs.
  • Local Table 1 results: Agent-Reflex is measured by us. Agent-R, EnvACE, and Agent-World are reproduced because usable 8B checkpoints or corresponding results are unavailable. Other unmarked baselines are quoted.
  • Table 2 and NoisyAgent: Agent-Reflex and entries explicitly marked measured/reproduced are local; other unmarked entries are quoted. NoisyAgent is quoted in both Tables 1 and 2. The reproduced asterisk in Table 1 is an annotation error and will be removed. Only the NoisyAgent RL optimizer experiment in Figure 4 is locally reproduced.
  • Evaluation protocols and budgets: Local measurements follow each benchmark's decoding, max-turn, and tool-call limits (Appendix D.5). Quoted entries retain their original protocols and will be labeled by provenance.
  • Figure 4 initialization and budget: To isolate optimizer differences, GRPO, DAPO, REINFORCE++, NoisyAgent RL, SEED, and ORPO all start from Qwen3-8B with the same 5K EnvScaler task pool, 100 steps, and G = 8; all use the same noise exposure except NoisyAgent RL's adaptive schedule. The final Agent-Reflex instead starts ORPO from the ReSFT checkpoint (Appendix D.4). The revised caption will distinguish these settings.

W3 & Q1. Quantity-matched backfilling to separate supervision quality from quantity; direct diverse-pool ablation.

Thank you for the suggestion. Submitted Table 3 removes samples outright. We add quantity-matched variants that backfill removed supervision with the same number of tokens from verified correct trajectories.

VariantFullRemoved outrightQuantity-matchedDrop after backfill
w/o SR45.941.343.1−2.8
w/o KR45.942.444.0−1.9
w/o UR45.943.044.3−1.6

After backfilling, removing SR/KR/UR still reduces ANB-τ² by 2.8/1.9/1.6; about 55%–61% of the original drops remain attributable to supervision content. The UR drop concentrates on User-Noise. The diverse-pool ablation appears under W1 & Q2 (53.5 vs 55.1). Revised Table 3 will show both columns.

AuthorsOpenReview ↗

Response 2/2

W4. Evidence that the model identifies specific failures and repairs for the intended reason.

Thank you for raising this important point. The submission already uses Stability-Gated Success and 13 diagnostic axes (Appendix E.1, Figure 5): Agent-Reflex leads on all axes, with derailment 28.5% (EnvScaler 44.8%, backbone 56.1%). These results connect gains to resisting specific noise, but do not directly establish type-matched diagnosis.

We therefore add three metrics independent of final task success, measured after the first injected error:

MetricQwen3-8BEnvScaler-8BAgent-Reflex-8B
Recovery after the first error18.4%26.7%41.5%
Blind-retry rate47.2%38.9%19.6%
Recovery action matching the failure type22.1%31.4%48.9%
Trajectories derailed by noise (Figure 5)56.1%44.8%28.5%

Blind retries fall from 38.9% to 19.6%, type-match rises from 31.4% to 48.9%, and post-error recovery rises from 26.7% to 41.5%. These results indicate type-matched diagnosis and recovery at the point of failure, rather than lucky final answers after derailment. We will add these diagnostics to §4.4.


W5 & Q4. Uncertainty for main results/ablations, and full cost including environment and skill construction.

Thank you for the suggestion. Appendix D.5 already states that local results are three-run means, and Appendix E.1 reports bootstrap intervals for the 13 diagnostic axes. We further perform 1,000 task-paired bootstrap resamples:

SettingMethod/variantScore95% CI
Five-benchmark OverallAgent-Reflex57.6[56.4, 58.7]
Five-benchmark OverallAgent-World53.6[52.3, 54.8]
AgentNoiseBench OverallAgent-Reflex29.1[27.4, 30.9]
AgentNoiseBench OverallEnvScaler18.2[16.9, 19.6]
ANB-τ² ablationFull model45.9[44.5, 47.2]
ANB-τ² ablationw/o SR41.3[39.8, 42.9]

Main-comparison and full-versus-w/o-SR intervals do not overlap. Revised Tables 1–3 will report means and CIs.

Offline cost: environment construction ~180 GPU hours and sampling/analysis/verification ~260, totaling about 440 one-time GPU hours; most model calls use self-hosted GPU inference, with limited external API usage. ReSFT ~480 and ORPO ~208 per run; ORPO adds about 18% over equal-step plain RL. Appendix D will separate one-time, per-run, and inference costs.


Q3. Why does Equation 11 retain Δ < 0 tokens, and can soft weighting provide useful supervision from failed trajectories?

Thank you for the careful reading of Equation 11. The weight γ = σ(β·Δ) lies in (0, 1), where Δ is the gradient-detached log-likelihood difference for the same sampled token with versus without the skill prompt. When Δ < 0, γ < 0.5: the OPD term still increases the sampled token's probability, but more weakly. Thus γ controls relative emphasis rather than changing the gradient direction.

We prefer soft weighting because most tokens in a failed trajectory are not themselves incorrect; a hard gate 1[Δ > 0] would discard remaining signal and increase estimator variance, and the sign of Δ is itself noisy under sampling.

We compared soft and hard gating: replacing γ with a hard gate reduces ANB-τ² from 45.9 to 45.2 and increases training-reward variance by about 23%. We will add this result to the appendix to clarify the design motivation.


Thank you again for the constructive suggestions. We sincerely hope these clarifications and additional experiments address your main concerns, and we respectfully ask you to reconsider your score.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer WXX1,

We have revised the paper and added additional results to address your comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you, The authors of Agent-Reflex

Reviewer_HCqNOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes AgentReflex, a framework for training model to do proactive reflection in order to deal with noisy environmental feedback / tool return results during agentic RL training. On a high-level agentReflex uses (1) synthetic data generation for model's self-reflection given error in the environment, (2) Creating synthetic trajectories with reflection behavior for SFT, and (3) Use a mix of RL + OPD objective for context distilling the reflection behavior into the model and learn to solve the tasks. The paper shows that on long horizon tool-use benchmarks (esp. with noisy env feedback), AgentReflex shows better test performance compared to previous works.

Critical strengths
  • The motivation of the paper is clear and it's addressing a real and important problem for agentic RL.
  • The proposed solution makes sense and is well-positioned with respect to existing work.
  • The authors did experiments against other policy-gradient algorithms (e.g. GRPO, REINFORCE++, DAPO, and two previous methods for dealing with noisy feedback) to understand where the value-add for AgentReflex is, which is nice.

Overall this is a solid paper: Good motivation, solution makes sense, experiments are sufficient to prove / show where the method stands out

Critical weaknesses

Here are results that I think will strengthen the paper and help people understand whether they want to try something like this especially for larger-scale training runs.

(Disclaimer: I'm not saying the authors have to do what I suggest, I'm only providing suggestions that I think will be helpful. Would love to hear author's thoughts)

The missing piece of the current paper I think is some sort of training + inference compute analysis for us to understand (1) how much additional training / inference compute, e.g. Both in terms of training wall clock / FLOPS use, and also the impact of solution or CoT length in model generations. These will help us understand the trade off and make decision on whether we'd want to add this type of reflection training in real runs. Two suggestions here

  • Scaling factor for reflection data generation and SFT": e.g. if you spend more compute on either SFT training stage, and / or generating more reflection data, under the assumption that entropy of model doesn't collapse, what is the scaling behavior for downstream model behavior after RL.
  • Wall clock statistics on training time, effect on memory consumption / batch size": These will help understand how scalable the method is in training.
Questions for authors

See above

4 comments

We sincerely thank the reviewer for the thorough and constructive feedback. The reviewer's recognition of our work and the thoughtful questions on training/inference cost and scaling behavior have been invaluable in improving the completeness of our analyses. Below, we address each concern systematically and comprehensively.


W/Q1. Training and inference efficiency: additional wall-clock, FLOPs, memory, and generation costs; must batch size be reduced; and are more interaction turns required?

Thank you for the suggestion. Appendix D.4 already states that ORPO's skill branch performs only one gradient-detached forward pass, with no additional rollout or environment query; inference uses neither a skill prompt nor an external reflection model. We further measure the full pipeline:

StageTypeGPU hours / wall-clockAdditional cost
RSD environment construction (500 envs, 30K tasks)One-time~180 GPU hoursSame setting as Agent-World
RSD sampling/analysis/verification (1,170 skills)One-time~260 GPU hours + limited external APIMostly self-hosted GPU; ~0.88 GPU hours/env
ReSFT (50K trajectories, 3 epochs)Per run~480 GPU hours; 2×8 GPUs ~30 h—
ORPO (100 steps, G = 8)Per run~208 GPU hours; 1×8 GPUs ~26 h~1.16× FLOPs and +18% time vs plain RL; G = 8 unchanged
Plain-RL control (same tasks/steps)Per run~176 GPU hours; 1×8 GPUs ~22 h1.00× FLOPs
InferenceDeploymentNo additional model callNo skill prompt or retrieval

At inference, Agent-Reflex outputs 3,340 tokens per episode on average (~12% more than EnvScaler's 2,980), while average interaction turns decrease from 9.1 to 8.6. The principal extra cost is reusable one-time skill discovery; during training ORPO adds about 18% over equal-step RL without reducing batch size. We have not independently and reliably profiled peak memory, so we do not report a speculative number. The revision will report wall-clock, FLOPs, and batch-size settings in Appendix D, and inference length and turns in §4.3.


W/Q2. Scaling behavior: how does post-RL performance change with more reflection data or SFT compute, and does entropy remain stable?

Thank you for the suggestion. Holding other conditions fixed, we vary reflective correction samples and SFT epochs (ANB-τ² Avg.) and log episode-level entropy throughout; no collapse is observed.

Reflective correction samples05K10K20K
After ReSFT31.237.540.341.9
After identical ORPO34.541.044.145.9
SFT epochs3 epochs6 epochsAfter identical ORPO
Relative gainBaseline+1.1Narrows to +0.4

Reflection-data gains diminish both before and after ORPO, which cannot fully compensate for insufficient supervision; environment and failure coverage matter more than raw sample count. Increasing SFT from three to six epochs gives +1.1 before RL but only +0.4 after ORPO, whereas Direct Correction and mined skills still differ by 2.3 Overall under identical ORPO. Under limited compute, expanding coverage is preferable to adding same-distribution SFT epochs. We will add pre/post-RL scaling and entropy curves.


Thank you again for the constructive suggestions. We sincerely hope these cost and scaling results address your concerns, and continue to support the paper during discussion.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer HCqN,

Thank you for your insightful suggestions. We have done our best to address your concerns. Since the rebuttal period is closing very soon, could you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you, The authors of Agent-Reflex

Reviewer_HCqNOpenReview ↗

Thank you for your response

Hi, thank you for these additional experiments. My concerns are addressed and my score remains positive. Solid work; I'm happy to support this paper.

AuthorsOpenReview ↗

Thank you for Reviewer HCqN ‘s Reply

Hi, we are truly delighted by your highly positive evaluation of Agent-Reflex. We are also greatly encouraged by your decision to raise both the overall and confidence scores to 4 in support of our work !

We will incorporate the additional experiments you suggested into the revised version and carefully proofread the manuscript. Looking ahead, we further hope that Agent-Reflex can offer a practical path toward internalizing proactive reflection capabilities into agents.

Thank you, The authors of Agent-Reflex

Reviewer_DdTbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Agent-Reflex, a multi stage training pipeline to teach LLM agents to recover from noisy tool feedback or user corrections. First, a pool of llm agents is used to run on a collection of environments with injected tool failures which an analyzer and verifier llm models extract meta-skills from comparing the rollouts. Then, those skills are used for SFT. Finally, the proposed method include doing RL on a mix of clean and noisy rollouts and on-policy distillation from a skill-conditioned copy of the policy. Agent-Reflex with Qwen3-8B backbone show promising results compared to other baselines with similar size even though different methods have access to different levels of privileged information.

Critical strengths

The problem is realistic and well motivated. Noisy tool feedback and user corrections are common in deployment but underexplored in agent training. The failure taxonomy is grounded in observed error returns.

Critical weaknesses
  • The baselines are not fair comparison: The proposed method includes many components (noisy environments, a model pool up to 284B, analyzer and verifier models for skill extraction, three sampling strategies, then RL and OPD), while the baselines are RL-only or untrained reflexion based methods. The skill content comes from a much larger models. Without matched baselines, it is difficult to understand where the improvements of the method come from.

  • The ablation could be done more extensively: Table 3 only removes SR, KR, UR, OPD and the confidence weight, on AgentNoiseBench alone. It does not separate SFT, OPD, RL contributions and it does not show the benefit of having noisy environments. A bit minor, but several works from literature has already done pair-wise trace analysis including [1,2] while not mentioned in the related work section.

[1] https://arxiv.org/abs/2403.08978 [2] https://arxiv.org/abs/2510.04373

Questions for authors
  • Can you add SFT on the same tasks with correct trajectories only, and the EnvScaler recipe given the same SFT data and compute?

  • Can you add total training cost including skill discovery phase? and compare it with the baselines?

  • Would the model learned to do Reflective SFT, generalize to new tasks?

3 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank the reviewer for the constructive and encouraging feedback. The reviewer's recognition of our setting, together with the thoughtful suggestions on fairness, ablations, and generalization, have been invaluable in improving our paper. Below, we address each concern systematically and comprehensively.


W1 & Q1. Fairness of the comparison: do the gains arise from stronger models and additional components, and can Correct-only SFT or EnvScaler achieve comparable results under matched tasks, SFT data, and compute?

Thank you for raising this central issue. We clarify both the role of stronger models and the independent contribution of each component.

  1. Role of stronger models. Stronger models in offline construction are not unique to our method: Agent-World uses Seed 2.0; EnvScaler uses GPT-4.1 and Qwen3-30B-A3B; NoisyAgent uses GLM-4.6 and Claude-Sonnet-4.5. Appendix D.2–D.3 already bounds their roles in Agent-Reflex: DeepSeek-V4-Flash judges outcomes, diagnoses errors, and extracts skills; Every agent continuation receiving training loss is generated by Qwen3-8B, and training inputs contain no skill text. Thus, external models provide offline supervision only; inference requires neither a skill prompt nor an extra model. Compared with setups that distill stronger-model trajectories as direct policy targets, our supervision chain is more restrained.

  2. Independent component contribution. We construct four matched 50K-trajectory SFT settings with identical tokens, update steps, and verification criteria: Correct-only SFT uses 30K original plus 20K additional verified successes; Generic Guidance uses 30K successes and 20K corrections from a general recovery principle; Direct Correction uses the same correction volume from per-trajectory rewriting; ReSFT uses 30K successes and 20K SR/KR/UR corrections. Final comparisons use identical ORPO tasks, hyperparameters, and budgets.

Variant (only supervision differs)ANB-τ² Avg.ANB-Vita Avg.Five-benchmark OverallOverall after identical ORPO
Qwen3-8B backbone20.24.440.2—
Correct-only (30K + 20K verified successes)31.27.151.352.0
Generic Guidance35.88.653.054.1
Direct Correction (Agent-R-style; no environment-level skill)38.49.854.255.3
EnvScaler recipe + same ReSFT data39.610.154.6—
ReSFT (mined environment-level skills)41.910.855.157.6

With data and compute fixed, SFT Overall rises from 51.3 / 53.0 / 54.2 to ReSFT 55.1; applying the EnvScaler recipe to the same data yields 54.6. After identical ORPO, ReSFT remains best at 57.6. We do not append ORPO to the EnvScaler row because that would change both recipe and objective, breaking the single-variable control. The gain is not merely from more data; environment-level supervision and skill distillation each contribute.

We also test dependence on larger models by restricting the trajectory pool to Qwen3-8B or replacing the analyzer with Qwen3-8B.

Skill sourceANB-τ²ANB-VitaOverall
Qwen3-8B self-sampled pairs only39.89.353.5
Analyzer replaced by Qwen3-8B40.29.753.9
Diverse model pool (4B–284B)41.910.855.1

Replacing the diverse pool or 284B analyzer reduces Overall by 1.2–1.6. Larger models help, but do not explain the full gain; Qwen3-8B-only variants still outperform Correct-only and Generic Guidance. The core benefit comes from contrasting verified successful and failed trajectories under the same task and noise. We will add these controls to the appendix and briefly reference them in §4.2.


W2. Separating the contributions of SFT, RL, OPD, and noisy environments.

Table 3 already tests SR/KR/UR/OPD/weighting, and Appendix D.4 studies ORPO noise fractions. We agree these do not yet provide full stage-wise attribution. We therefore add a stage ladder and independently disable noise in RSD and ORPO.

StageANB-τ²ANB-VitaOverall
Qwen3-8B backbone20.24.440.2
Correct-only SFT31.27.151.3
Replace with ReSFT (matched sample count)41.910.855.1
Add Noisy RL (GRPO, no OPD)43.911.156.2
Add ORPO (RL + skill-conditioned OPD)45.912.357.6
Disable RSD noise (p_inj = 0), then train through ORPO39.810.956.0
ORPO noise fraction ν = 042.611.056.4

Correct-only SFT raises Overall from 40.2 to 51.3; matched reflective supervision reaches 55.1; Noisy RL and ORPO then reach 56.2 and 57.6. This agrees with the necessity of OPD in submitted Table 3. Disabling RSD noise reduces ANB-τ² from 45.9 to 39.8 (Tool-Noise 38.4→29.6); setting ν = 0 reduces it to 42.6. Noisy environments primarily teach tool-failure identification and recovery; ReSFT and OPD internalize these behaviors offline and online. We will add this attribution to §4.4.

AuthorsOpenReview ↗

Response 2/2

W2 (Related Work). Discuss pairwise-trajectory methods such as AutoGuide [1] and JEF-Hinter [2].

Thank you for the references. We will discuss AutoGuide [1] and JEF-Hinter [2] in Appendix A.1. All three approaches derive reusable knowledge from successful and failed trajectories. AutoGuide's context-aware guidelines and JEF-Hinter's episodic hints are retrieved and injected at inference, whereas Agent-Reflex internalizes environment-indexed skills through ReSFT and ORPO, requiring neither retrieval nor extra prompting at inference.

Submitted Table 1 already includes ExpeL-8B [3], an inference-time experience-retrieval baseline. Its Overall of 37.9 is below Agent-Reflex (57.6) and even the untrained Qwen3-8B backbone (40.2). We will use ExpeL to sharpen the contrast between inference-time retrieval and training-time internalization.


Q2. Total and marginal training costs after including environment construction, sampling, analysis, and verification.

We separate one-time and per-run costs on H100-80G:

StageTypeGPU hoursNotes
RSD environment construction (500 envs, 30K tasks)One-time~180Same setting as Agent-World
RSD sampling/analysis/verification (1,170 skills)One-time~260 + limited external APIMostly self-hosted GPU inference; ~0.88 GPU hours/env including construction
ReSFT (50K trajectories, 3 epochs)Per run~480 (2×8 GPUs, ~30 h)—
ORPO (100 steps, G = 8, ν = 0.25)Per run~208 (1×8 GPUs, ~26 h)OPD adds ~18% time; G = 8 unchanged
EnvScaler recipe, same tasks/stepsPer run~176 (1×8 GPUs, ~22 h)ORPO ~18% more expensive
InferenceDeploymentZero additional overheadNo skill prompt or extra model

Skill discovery is a reusable one-time cost of less than one amortized GPU hour per environment. Most sampling/analysis calls run on self-hosted GPU inference; external API usage is limited. ORPO adds only one gradient-detached teacher forward over equal-step RL. Because Agent-World and EnvScaler do not disclose complete environment-construction costs, we compare only verifiable training costs and report one-time, per-run, and inference costs separately in Appendix D.


Q3. Can ReSFT generalize to unseen tasks, environments, and noise rather than memorizing training templates?

The submission already addresses two levels of generalization:

Existing evidenceTraining–test relationshipResult
EnvScaler RL tasks (Appendix D.4)ORPO and RSD tasks are disjoint; only environment identity is sharedSkills transfer to new tasks in the same environment
AgentNoiseBench (Appendix E.2, Figure 5)Taxonomy, injectors, and evaluation environments are absent from RSD/trainingLeads on all 13 noise axes; derailment 28.5% vs EnvScaler 44.8%

We further mine skills from only 400 environments and evaluate new tasks from the remaining 100 held-out environments. Correct-only obtains 49.8 Overall and ReSFT 52.4, a gap of +2.6 with 95% CI [1.4, 3.8]. Although smaller than the in-distribution +3.8, the advantage remains significantly positive. Even when environment families are held out, failure diagnosis and recovery still transfer. The revision will distinguish new tasks, new environments, and new noise mechanisms.


References

[1] Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents. In NeurIPS, 2024.

[2] Hadi Nekoei, Aman Jaiswal, Patrice Bechard, Oleh Shliazhko, Orlando Marquez Ayala, Mathieu Reymond, Massimo Caccia, Alexandre Drouin, Sarath Chandar, and Alexandre Lacoste. JEF-Hinter: Leveraging Offline Knowledge for Improving Web Agents Adaptation. arXiv:2510.04373, 2025.

[3] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. In AAAI, 2024.


Thank you again for the constructive suggestions. We sincerely hope these clarifications and additional experiments address your concerns, and we respectfully ask you to reconsider your score.

AuthorsOpenReview ↗

A Kind Reminder for Reading the Response

Dear Reviewer DdTb,

We have revised the paper and added additional results to address your comments. Since the rebuttal period is closing very soon, can you please check the response to see whether it mitigates your concerns? We would greatly appreciate that!

Thank you, The authors of Agent-Reflex

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Reponse of Agent-Reflex

We sincerely thank all reviewers for their time and constructive feedback. We are glad that our problem setting and core contributions have been recognized: noisy tool feedback and user corrections are common in real deployments yet remain underexplored in agent training (DdTb, HCqN, WXX1); the eight-type tool-failure taxonomy is grounded in real error-return statistics (DdTb); and the complete recipe yields measured gains in both general tool use and noisy interaction (Overall 57.6, WXX1).

The reviewers' main concerns focus on gain attribution, full training cost, uncertainty, and experimental documentation. We have added controlled experiments and quantitative analyses accordingly. The table below summarizes the focus areas and the corresponding sections we plan to update.

Focus AreaReviewerSupplementary / Revised SectionOur Actions
1. Matched supervision and skill-source ablationsDdTb, WXX1§4.2 / Appendix DCorrect-only, Generic Guidance, Direct Correction, EnvScaler-on-same-data; Qwen3-8B self-mining / self-analyzer
2. Stage ladder and noise switchesDdTb§4.4Backbone → SFT → ReSFT → Noisy RL → ORPO; disable RSD / ORPO noise
3. Quantity-matched backfillWXX1Table 3Remove SR/KR/UR and backfill with verified correct trajectories
4. Necessity of skill miningWXX1§4.4Random/generic skills, hindsight rewrite, random OPD context, environment-coverage scaling
5. Held-out environment generalizationDdTb§4.4 / Appendix DMine skills from 400 environments; evaluate on 100 held-out; gap +2.6, 95% CI [1.4, 3.8]
6. Reflection-behavior diagnosticsWXX1§4.4Post-error recovery, blind retry, failure-type match
7. Full cost and scalingDdTb, HCqN, WXX1Appendix D / §4.3–§4.4One-time vs per-run cost; ORPO +18%; reflection-data and SFT-epoch scaling
8. UncertaintyWXX1Tables 1–3Paired-bootstrap 95% CIs for main results and key ablations
9. Experimental documentationWXX1§4.1, Figure 4, Tables 1–2Provenance labels; Figure 4 vs final ORPO init; explanation of γ in Eq. 11
10. Related workDdTbAppendix A.1Cite AutoGuide and JEF-Hinter; use ExpeL to contrast retrieval vs internalization

We believe these clarifications and additional quantitative results thoroughly address the reviewers' concerns. If these additions address the main concerns, we respectfully ask for a more favorable reassessment. We will subsequently update the corresponding sections listed above and provide a revised version of the paper.

We thank all three reviewers and the Area Chair for their time and look forward to continuing the discussion.

Best regards,
The Authors

Thank you for the detailed responses and additional experiments. The authors addressed most of my concerns. These new results strengthen the paper and provide better evidence that the improvements are not simply coming from more training data or additional components.

I still have some concerns about the relatively small gains of environment-specific skill mining over alternative supervision strategies, the lack of uncertainty estimates for these specific comparisons, and the fairness of some baseline comparisons. I encourage the authors to make these limitations clear in the revised paper. Overall, I think the rebuttal has addressed enough of my original concerns.

AuthorsOpenReview ↗

Thank you for Reviewer WXX1 ‘s Reply

Thank you for your timely response and for raising the score. We are glad that our rebuttal has addressed most of your concerns.

Regarding environment-specific skill mining, we will continue to provide a more detailed analysis and incorporate it, together with the other additional analyses conducted during the rebuttal period, into the revised paper.

Thank you, The authors of Agent-Reflex

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.