acceptodds
Under review as a conference paper at ICLR 2027

Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks

Abstract

Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional agent-side warming up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to environment-side adaptation by constructing Feedback-EnrichedEnvironments (FEEs). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs (1) stabilizes training dynamics by reducing entropy volatility, (2) facilitates proactive state-space exploration in difficult tasks, (3) ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and (4) identifies intra-group feedback consistency as a critical boundary for stable optimization.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Environment Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qXueOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Feedback-Enriched Environments (FEEs) to address sparse rewards in long-horizon agentic RL. It augments environment observations with Action Guidance (AG) or Observation Enrichment (OE), applying AG earlier and OE later in both interaction trajectories and training. Experiments on SciWorld and BFCL-V3 Multi-Turn with Qwen3-4B/8B and GRPO, DAPO, and GSPO report improvements when the trained agents are evaluated in standard environments. Further analyses examine exploration, training stability, feedback internalization, and intra-group feedback consistency.

Critical strengths

The paper targets an important bottleneck in long-horizon agentic RL with a clear and appealing idea: enrich feedback from the training environment rather than introduce another complicated optimization pipeline. The method is simple, interpretable, and compatible with existing RL algorithms. The evaluation covers two benchmarks, two model scales, and three RL algorithms, and tests the learned policies without enriched feedback at inference time. The public implementation is another useful contribution.

Critical weaknesses

Insufficient rollout-level evidence and component attribution. The paper's central causal claim is that enriched feedback increases the chance of positive-reward rollouts, yielding more informative RL updates and ultimately better policies. Although Figure 4 reports BFCL training-time avg@8 success rates, it does not systematically quantify the improvement over standard rollouts at matched policy checkpoints, particularly early in training, or the proportion of non-degenerate groups with non-zero reward variance. Nor does it isolate the contributions of AG, OE, and their scheduling to these intermediate outcomes. Without controlled, matched-budget rollout-level ablations linking these effects to final performance, the evidence does not fully establish the proposed mechanism for alleviating reward sparsity.

Insufficient justification of feedback scheduling and general applicability. Figure 2 compares feedback types and timing, but the early and late interaction windows have unequal lengths (steps 1–3 versus 6–10), while the switch at training step 100 and the 0.5 enrichment probability lack sufficient justification. The authors should compare AG→OE against AG-only, OE-only, OE→AG, and a fixed AG/OE mixture under matched feedback budgets, and examine how performance changes with these design choices. More fundamentally, the FEE implementations rely on environment-specific information: SciWorld AG uses ground-truth expert trajectories, while OE exploits internal task-progress or tool-state information. It remains unclear how AG/OE can be constructed in environments where such information is unavailable. Furthermore, without comparisons against alternatives using the same expert information, such as SFT warm-up, it is difficult to determine whether FEE genuinely addresses the original limitations or merely changes how expert guidance is delivered.

Questions for authors
  1. At matched checkpoints and rollout budgets, how much do FEEs improve positive-reward rollout rates and the proportion of non-degenerate groups, especially early in training?
  2. Which components drive these rollout-level gains? Can the authors compare AG, OE, and alternative schedules, and report how performance changes when the switch step, enrichment probability, and interaction windows are varied?
  3. How can AG/OE be constructed in environments without expert trajectories or accessible internal progress signals? Under matched expert information, how does FEE compare with SFT warm-up?
  4. How do the feedback-format probes demonstrate useful policy internalization? Why does Section 5.2 claim improvements across difficulty tiers when Figure 4 shows declines on medium and easy tasks?
5 comments
AuthorsOpenReview ↗

Rebuttal 1 / 3

We thank the reviewer for the detailed comments. They identify real gaps and one error in our text, and we address each.

Q1 / W1: Rollout-level evidence. We ran new runs with full rollout dumps, standard versus FEE, on SciWorld with Qwen3-4B and GRPO, the setting where we had complete per-rollout logging. Over 27 training steps each condition has 432 rollout groups of 8 trajectories (3,456 trajectories per condition). A group is non-degenerate when it contains both successful and failed trajectories, which is exactly the condition for a non-zero group-normalized advantage.

StandardFEEDelta
Positive-reward trajectories19.73%22.63%+2.89
Non-degenerate groups (0<k<8)40.97%45.60%+4.63
All-zero groups (k=0)56.71%51.85%-4.86
All-success groups (k=8)2.31%2.55%+0.23

Two observations.

First, the standard environment wastes more than half of its rollout budget: 56.71% of groups contain no successful trajectory and contribute exactly zero gradient. This quantifies the sparsity the paper motivates, which we had only described qualitatively.

Second, and this is what speaks to the mechanism rather than to difficulty: of the 4.86 points by which FEE reduces all-zero groups, 95.2% reappear as non-degenerate groups and 4.8% as all-success groups. If enriched feedback were simply making tasks easier, the all-success band would absorb a large share of the shift, and those groups produce no gradient either. It does not move: 2.31% to 2.55%. FEE converts groups that carry no learning signal into groups that do, without pushing them through into saturation. The same holds further up the distribution: groups with k>=5 rise from 15.51% to 18.75%, and the mean success count per group from 1.579 to 1.810.

Q2 / W2: Components, schedule, and the unequal windows. Figure 2 is already a 2x2 over what is enriched (AG or OE) and when it is delivered (steps 1-3 or 6-10), with AG-only and OE-only runs in it.

On the unequal windows specifically: the 6-10 window is 5 steps against 3, so it delivers about 67% more hints. If the outcome were driven by how much feedback is given, the longer window should win. For AG it loses, and loses badly: AG-Late stays near zero success for the first 50 training steps, below even the standard environment, while AG-Early is already climbing. The budget asymmetry runs against the conclusion we draw from that comparison rather than in its favour. Reading the four runs across both factors, the type of information separates them more than the window does: AG leads OE by 0.083 at steps 1-3 and by 0.043 at steps 6-10, while moving the window within a fixed type changes the result by 0.023 for AG and 0.017 for OE.

We are asking what properties a training environment needs, not searching for the configuration that maximizes the score. The switch step and the enrichment probability were fixed once in the pilot and reused unchanged across all 6 configurations in Table 1, two benchmarks, two model scales, and three algorithms; a schedule that was never tuned on the main experiments still improves every configuration's average. The corollary is that our numbers are very likely not the ceiling of this design space, and we do not claim AG-then-OE is its optimum. What we claim is the pair of properties that govern the outcome: what kind of information the environment returns, and whether it arrives early or late, both within an episode and across training. Those two are what Figure 2 separates, and they are what carries to a new environment. A specific switch step or window boundary does not carry: in a different environment it would have to be found again regardless of what we reported here.

AuthorsOpenReview ↗

Rebuttal 2 / 3

Q3: Constructing AG/OE without expert trajectories; FEE versus SFT under matched expert information.

On the SFT comparison, setting it up exposes an asymmetry before any training happens. SciWorld ships gold action sequences, not reasoning traces, and supervising on bare actions breaks the think/action format. An SFT warm-up therefore needs a rationale synthesized for each of the 8,463 gold actions; we generated these with a separate teacher model (gpt-6-astra). They are post-hoc explanations fitted to actions already chosen, and leakage can be constrained by prompting but not verified. The teacher prompt was:

You are creating synthetic ScienceWorld SFT action rationales. This is a text transformation only: do not call tools, read files, or browse. Return ONLY the requested JSON. Each input record is an independent decision. For each numbered turn, write a short English rationale for the supplied action using the task and observations from that turn or EARLIER turns ONLY. Never borrow information from a later record, even though the records are batched for efficiency. Do not predict or assert the result of the action before executing it. Do not invent temperatures, properties, measurements, success, locations, object observations or verifier progress. The action is supplied supervision, NOT evidence of an object's hidden property. If supporting evidence is absent, phrase the action as a next step to try/check; do not fabricate certainty. No references to gold paths, oracle, ground truth, annotations, future turns, hidden state, reward or supplied answers. Use 1-3 concise sentences (typically 20-65 words). Tie the current subgoal to the next action; avoid repetitive deliberation, 'wait actually', and generic filler. An apparently redundant look/wait/open may be acknowledged honestly as a check; never invent a benefit for an invalid action. Keep action strings EXACTLY unchanged. Do not include XML tags in rationale: the exporter adds <think> and <action>. These are post-hoc synthetic explanations, not a claim about a model's internal thought. Before returning, check every rationale against the chronology and remove hindsight. Output {"turns":[{"turn":1,"action":"exact input action","rationale":"..."},...]}. Include exactly one output for every supplied turn, in order.

With that data built, SciWorld with Qwen3-8B, one protocol applied identically to both rows (128 episodes, greedy decoding):

Step budgetBaseSFT warm-upDelta
1516.413.91-12.50
3026.564.69-21.88

SFT warm-up removes 76% of the base model's task success at a 15-step budget and 82% at 30 steps. A warm-up exists to raise the probability of obtaining an initial positive reward, and this one removes four fifths of them.

The gold data explains the direction. 8.1% of supervised actions return "already in that state", 673 of them already-open doors, and 267 rationales call their own action redundant. Only 71 of 200 gold paths finish within 15 steps; the median is 23 and 52.3% of supervision comes from beyond step 30, past the budget the agent is evaluated under. Recovery is undemonstrated: 0.6% of prompts contain an invalid-action message. At evaluation the SFT model's formatting is better than base, but rejected actions rise from 24.4% to 49.4% between the halves of an episode.

On whether FEE merely changes how expert guidance is delivered, the quantities involved are not comparable. SFT consumes all 8,463 actions, each with a rationale attached. Action guidance reads the first 3 actions of each of the 200 training tasks, 600 in total, about 7% of the SFT supervision, directly from the gold path with no processing, and only within steps 1-3 of an episode, with probability 0.5, during the first half of training. It is not a cheaper demonstration pipeline; it is a decision about where in an episode a small amount of hand-authored signal is worth placing. The gold path's own defects make the distinction concrete: the redundancy and the 773-step tails that sink SFT are concentrated in exactly the parts of the trajectory that action guidance never touches.

For environments without expert trajectories, two parts of our own setup already use none. Observation enrichment comes from the environment's built-in goal-progress checker in SciWorld and from simulator state in BFCL, both of which are information the environment already computes and simply does not show the agent; the analogues in open-ended settings are test logs, compiler output, and DOM differences. And in BFCL one class of action guidance is authored from the task category alone: for tasks with missing parameters the hint reads "Essential information is missing from the user's request. Do not make any assumptions; explicitly ask the user to clarify the missing parameters", which requires no knowledge of the correct answer.

AuthorsOpenReview ↗

Rebuttal 3 / 3

Q4a: What the probes show. We may not have made the design clear enough, so let us restate what the probe controls for.

The question is whether enriched feedback is absorbed into the weights or only acts while present in the context. To separate these, the probe is asked with no enriched feedback anywhere in the context: the model gets a plain multiple-choice question about what a tool call returns, in the standard format, and we read off the probability it assigns to the enriched return versus the original one. Because nothing in the prompt mentions the enrichment, any difference between an FEE-trained and a standard-trained model can only come from the weights. The construction is not measuring whether the model can use a hint it can see; it is measuring whether it retained the environment dynamics after the hint was gone.

On what the result shows, we should be more careful than the paper is. The margin P(B) - P(A) stays negative for every probe at every checkpoint, so the model always prefers the original option; only the degree varies. Apart from the cd probe, which rises from about -0.83 to -0.67 at step 150, the enriched curves stay between -0.95 and -1.0 and by step 200 have returned to about -0.98. The standard-trained curves are flat at -1.0 throughout, so the two conditions do differ in the predicted direction.

On the "useful" part of the question, the probes are not where that evidence comes from, and we should not have leaned on them for it. It comes from the evaluation protocol: every number we report, in Table 1 and in Figures 2, 4 and 6, is measured in the standard environment with no enrichment present. FEE-trained agents are better there. Whatever they gained during training, they kept after the scaffold was removed, which is the property that matters for the claim that the guidance is not merely an inference-time prior.

Q4b: Section 5.2 contradicts Figure 4. The reviewer is right that the sentence is wrong, and we will remove it. But the tiers also deserve to be read in absolute terms, because they are very different sizes.

TierNStandardFEEDeltaEnvironment-equivalents
Hard2356.510.9+4.3+10.1
Medium7350.849.8-1.0-0.7
Easy9287.687.0-0.6-0.6
All40033.235.5+2.3+8.8

The hard tier holds 235 of the 400 environments, and the gain there is worth about 10 environments. The declines on medium and easy are each worth less than one: 1.0 point spread over 73 environments is 0.73 of an environment, and 0.6 points over 92 is 0.55. These sit below the granularity at which a single environment flips, so we would not read them as degradation. The environment-weighted average moves from 33.2 to 35.5.

The revised text will say: the gain is concentrated on hard environments, which are also the majority of this benchmark, while medium and easy are unchanged to within the resolution of a single environment.

Reviewer_h2yROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FEEs to improve RL for LLM agents by enriching environment feedback through two mechanisms: action guidance and observation enrichment. Experiments on SciWorld and BFCL demonstrate consistent average performance improvements across different model sizes and RL algorithms.

Critical strengths
  1. The paper shifts attention from agent-side optimization to environment-side feedback design, offering a novel direction for improving agentic RL.

  2. The proposed strategy is straightforward and can be integrated with multiple RL algorithms.

  3. Besides the main performance comparisons, the paper investigates training stability, state-space exploration, feedback internalization. These analyses provide useful insights into the role of environment feedback.

Critical weaknesses
  1. The scalability and construction cost of AG and OE remain unclear. In more complex, open-ended environments, such as software engineering, web search, deep research, obtaining AG and OE would become expensive.

  2. There is no ablation study to isolate the contribution only from AG and OE, respectively.

  3. The reported performance improvement is 2.8% and lacks statistical significant analysis, making it difficult to assess whether these improvements are statistically reliable.

Questions for authors

see weaknesses.

3 comments
AuthorsOpenReview ↗

Rebuttal 1 / 2

We thank the reviewer for recognizing the environment-side direction and the analyses.

Since the review records that little here was new, we want to start by naming the two findings we think are not obvious, because they may not have come through in the writing.

The first is that feedback type and timing interact in a direction that contradicts the natural default. One would expect hints to be most useful when the agent is stuck, that is, late in an episode and late in training. For observation enrichment that holds. For action guidance it is strongly false: AG-Late stays near zero success for the first 50 training steps, below even the standard environment, while AG-Early is already climbing, and this is with AG-Late receiving about 67% more hints because its window is longer. Delivering the same information later does not merely stop helping.

The second is quantitative and we had not reported it in the submission. Measuring every rollout group in a standard SciWorld GRPO run, 56.71% of groups contain no successful trajectory and therefore contribute exactly zero gradient. Reward sparsity in long-horizon agentic RL is usually described qualitatively; this says that in this setting more than half of all rollout compute produces no learning signal at all. It also gives a precise way to check whether an environment design helps for the right reason rather than by making tasks easier. In the matching FEE run that figure falls to 51.85%, and 95.2% of the shift moves into groups containing both successes and failures rather than into all-success groups, which stay at 2.31% versus 2.55%. Had the enriched feedback simply lowered difficulty, the saturated band would have absorbed much of it, and those groups produce no gradient either.

W1: Construction cost of AG and OE. In our setup the cost is low. Both are rule-based wrappers: SciWorld OE reads the environment's built-in goal-progress checker, and BFCL OE exposes simulator state the original tool output hides. Neither calls an LLM at training time, so the overhead is negligible next to rollout. On the action-guidance side, what is read from ground truth is small and bounded: in SciWorld the first 3 actions of each of the 200 training tasks, 600 actions in total, taken directly from the gold paths that ship with the benchmark, and in BFCL the step hints come from the benchmark's ground-truth function calls. In both cases the signal is read, not annotated, and it covers only the opening of an episode.

The agent-side alternative scales worse on this axis. Building an SFT warm-up for the same environment required synthesizing a rationale for each of 8,463 gold actions with a separate teacher model, because the benchmark ships actions but not reasoning and supervising on bare actions breaks the think/action format. That is roughly 14x the expert information, plus a generation step for every item. In more open-ended domains the gap widens: a gold patch or a recorded click trace is harder to turn into faithful step-level reasoning than a SciWorld action list, while the observation-side signals get richer.

  • OE is often nearly free, because the information already exists inside the environment and benchmarks simply do not surface it: test logs, compiler and linter messages, and the repository diff after each edit in software engineering; the DOM or accessibility-tree difference after an action in web navigation; search metadata such as hit counts and already-visited sources in deep research.
  • AG is the costlier one without ground truth, but it does not require full demonstrations. In BFCL, one class of our action guidance is written from the task category alone: for tasks with missing parameters the hint asks the agent to seek clarification, with no knowledge of the correct answer.

We have not run FEE on software-engineering or web environments and will state this as a limitation.

W2: Isolating AG and OE. Figure 2 already contains AG-only runs (AG-Early, AG-Late) and OE-only runs (OE-Early, OE-Late) alongside the combination, crossed over what is enriched and when it is delivered.

Two readings follow from it. First, the type of information matters more than the window it arrives in: AG leads OE at both window positions, by 0.083 at steps 1-3 and 0.043 at steps 6-10, while moving the window within a fixed type changes the result by 0.023 for AG and 0.017 for OE. Second, OE alone underperforms the standard environment throughout the pilot, so OE is not a standalone alternative to AG but the second stage of a schedule, applied once AG has brought the policy to a level where the extra state information is usable. AG-Early peaks near step 125 and then declines, which is what switching is meant to avoid.

We did not run OE-then-AG or a fixed AG/OE mixture, so we do not claim AG-then-OE is optimal over the schedule space. The claim is narrower: feedback type is the dominant factor, and AG and OE are most useful at opposite ends of training.

AuthorsOpenReview ↗

Rebuttal 2 / 2

W3: Statistical significance. Three things.

(1) Consistency across configurations. The average improvement is positive in all 6 model x algorithm configurations (+0.69 to +4.61; one-sided sign test p = 1/64 = 0.016). On SciWorld all 6 are positive, mean +4.95. Across the 30 cells, 20 improve, 6 degrade, 4 tie. The 4 BFCL subsets share a trained model and are not independent, so we read this as a consistent direction rather than a precise p-value.

(2) Single-cell noise. With 128 SciWorld tasks at success rates near 40%, the standard error of the difference between two independent evaluations is about 6 points. A 5-point single-cell gain is about one standard error, and we do not claim individual cells are significant. The 2.8% figure is an average over configurations and should be read as one.

(3) A larger-sample view of the mechanism. Separately from Table 1, we instrumented a SciWorld GRPO run and its FEE counterpart at the rollout level, covering 432 rollout groups of 8 trajectories per condition. All-zero groups fall from 56.71% to 51.85%, non-degenerate groups rise from 40.97% to 45.60%, and all-success groups are unchanged at 2.31% versus 2.55%. Comparing the mean success count per group across the 27 paired training steps, FEE is higher in 21, lower in 4, tied in 2 (Wilcoxon signed-rank p = 0.0029). We pair by training step because the 16 groups within a step share a policy and because the success count rises over training; pooling all 432 groups and discarding the pairing gives p = 0.131, which we also report. These are one run per condition, so we read them as evidence about training dynamics rather than as independent replicates.

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_jvEpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Feedback-Enriched Environments (FEEs), an approach to improving reinforcement learning (RL) for long-horizon LLM agents by enriching environmental feedback. The authors first investigate two types of feedback: action guidance, which suggests next actions, and observation enrichment, which provides additional information about the environment state and task progress. Based on a preliminary study, the authors propose a feedback design strategy that uses action guidance in the early stages of exploration and training, followed by observation enrichment in later stages. Experiments across SciWorld and BFCL, using different model scales and RL algorithms, demonstrate improvements over training in standard environments. Further analyses investigate the effects of FEEs on training stability, exploration, feedback internalization, and intra-group feedback consistency.

Critical strengths

The paper addresses a well-motivated and practically relevant problem of improving RL training for long-horizon LLM agents under sparse rewards. The idea of enriching environmental feedback rather than modifying the RL algorithm itself is interesting. The paper is also well-written and easy to follow. The empirical evaluation demonstrates the effectiveness of proposed algorithm compared to the baselines.

Critical weaknesses

The proposed FEE method mainly follows a simple feedback curriculum, using action guidance in the early stages of training and observation enrichment in later stages. While this strategy is intuitive, its methodological novelty seems relatively limited.

Moreover, FEE relies on ground-truth expert trajectories to provide action guidance in SciWorld. This raises concerns about its dependence on expert supervision, as one of the main motivations presented in the introduction is to offer an alternative to SFT warm-up, which also relies on expert trajectories.

Questions for authors
  1. Dependence on expert supervision. Since FEE uses ground-truth expert trajectories for action guidance in SciWorld, how does it compare with an SFT warm-up baseline using the same trajectories? Can FEE still be effective without expert demonstrations?

  2. Feedback curriculum. What motivates the specific transition from action guidance to observation enrichment? How sensitive is the performance to the switching schedule?

4 comments
AuthorsOpenReview ↗

Rebuttal 1 / 2

We thank the reviewer for the careful summary.

W1: Methodological novelty. The final recipe is simple by design: it only changes what the environment returns, so it plugs into GRPO, DAPO, and GSPO without touching the optimizer. The contribution is in three places.

(1) What to enrich and when to enrich interact, and not in the direction one would guess. The natural default is to give hints when the agent is stuck, that is, late in an episode and late in training. Figure 2 shows this is right for OE and wrong for AG. AG-Late stays near zero success for the first 50 training steps, below the standard environment, even though its window (steps 6-10) is longer than AG-Early's (steps 1-3) and delivers about 67% more hints. The same information delivered later does not stop helping; it hurts.

(2) The schedule transfers. Fixed from the SciWorld pilot without re-tuning, it improves the average in all 6 model x algorithm configurations across two benchmarks.

(3) A constraint specific to environment-side methods: feedback must be consistent within a rollout group (Section 5.4). Group-based RL computes advantages by comparing rollouts of the same task, so varying feedback across a group confounds that comparison. Agent-side warm-up has no counterpart to this.

W2 / Q1: Expert trajectories, SFT warm-up, and whether FEE needs demonstrations.

We ran the SFT comparison. Setting it up already exposes an asymmetry: SciWorld ships gold actions, not reasoning traces, and supervising on bare actions breaks the think/action format, so SFT first requires a synthesized rationale for each of the 8,463 gold actions. We produced these with a separate teacher model (gpt-6-astra); they are post-hoc explanations fitted to actions already chosen. FEE requires none of it, because guidance arrives as an environment token masked out of the policy gradient (Section 2) and never has to be a well-formed model output.

On SciWorld with Qwen3-8B, one protocol applied identically to both rows (128 episodes, greedy decoding):

Step budgetBaseSFT warm-upDelta
1516.413.91-12.50
3026.564.69-21.88

SFT warm-up removes 76% of the base model's task success at a 15-step budget and 82% at 30 steps. A warm-up exists to raise the probability of obtaining an initial positive reward, and this one removes four fifths of them.

The gold data accounts for the direction. 8.1% of supervised actions return an "already in that state" response, 673 of them opening an already-open door, and 267 rationales describe their own action as redundant. Only 71 of the 200 gold paths finish within 15 steps; the median is 23 and 52.3% of supervision comes from beyond step 30, past the interaction budget the agent is evaluated under. Gold paths succeed by construction, so only 0.6% of training prompts contain an invalid-action message. The behaviour matches: the SFT model's formatting is better than base, but explicitly rejected actions rise from 24.4% in the first half of an episode to 49.4% in the second.

On whether FEE needs demonstrations at all, this is where we should have been clearer. Action guidance is not a cheaper way to collect expert supervision; it is a piece of environment design, and the amount of expert information it consumes reflects that. SFT uses all 8,463 actions with a rationale attached to each. Action guidance reads the first 3 actions of each of the 200 training tasks, 600 in total, about 7% of the SFT supervision, straight from the gold path with no processing, and only within steps 1-3 of an episode with probability 0.5 during the first half of training. The design question is where in an episode a small amount of hand-authored signal does the most good, not how to scale up demonstrations. The gold path's own defects make the distinction concrete: the redundancy and the 773-step tails that sink SFT sit in exactly the parts of the trajectory that action guidance never touches.

Two parts of our setup already use no ground truth. Observation enrichment comes from the environment's built-in goal-progress checker in SciWorld and from simulator state in BFCL. And in BFCL one class of action guidance is authored from the task category alone: for tasks with missing parameters the hint reads "Essential information is missing from the user's request. Do not make any assumptions; explicitly ask the user to clarify the missing parameters", which requires no knowledge of the correct answer.

AuthorsOpenReview ↗

Rebuttal 2 / 2

Q2: Motivation and sensitivity of the schedule. The motivation is the pilot in Figure 2: AG-Early leads early, OE-Late rises later and overtakes OE-Early. Our interpretation, offered as a hypothesis rather than an established mechanism, is that AG narrows the search space while the policy cannot yet reach reward, while OE supplies information only usable once it can. Figure 2 also shows AG-Early peaking near step 125 and then declining, which is what switching avoids.

On sensitivity, we did not sweep the switch step or the enrichment probability. Both values were fixed once in the pilot and reused unchanged for every configuration in Table 1, across two benchmarks, two model scales, and three RL algorithms. A schedule that was never tuned on the main experiments still improves the average in all 6 configurations, which is the evidence that matters for a claim about environment design, and it also means our numbers are unlikely to be the best this design space offers.

We can say something about one end of that range from a rollout-level measurement on SciWorld. A rollout group contributes a non-zero group-normalized advantage only when it contains both successful and failed trajectories. FEE reduces all-zero groups from 56.71% to 51.85%, and 95.2% of that shift moves into mixed groups rather than into all-success groups, which stay flat at 2.31% versus 2.55%. Since all-success groups produce no gradient either, an enrichment probability approaching 1.0 should hurt by saturating groups. We did not run that sweep, so this is a prediction the mechanism makes rather than a result.

On the ethics flag. The reviewer flagged this submission without stating a concern. We have posted a separate comment setting out the data sources, the sandboxed nature of all agent actions, and the dual-use considerations in full, and we would welcome a specific statement so we can address it directly.

I noticed that you flagged this submission for ethics review, but no specific ethical concern is mentioned in your review. Could you please confirm whether the flag was intended or selected by mistake?

In addition, the authors have posted their responses to your questions. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Thank you for the rebuttal and additional experiments.

  • Expert supervision: FEE still relies on expert trajectories in SciWorld, making it less fair to compare to the standard RL baselines (GRPO, GSPO, ect). Moreover, the degraded performance in the SFT experiments is a bit unexpected. In the simplest case if we take Qwen3.8-27B to SFT Qwen3-8B, I'd believe there'd be a clear performance improvement.

  • General applicability: The proposed expert-free feedback mechanisms relies on the specific feature of the environment, which might not be avaiable to other environments.

  • Feedback schedule: In my opinion, a fixed switching schedule may not generalize to environments with different exploration and learning dynamics.

On enthic flag. I appologize that the ethic flag in my initial review came from a misclick and I confirm that I do not have any ethical concerns for this paper.

Comments on the paper

1 comment

The authors have posted their rebuttal, and the discussion period will end soon. Could you please read it and reply at your earliest convenience, stating whether your concerns have been addressed and whether your rating changes? Thank you so much for your active participation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.