- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Feedback-Enriched Environments (FEEs) to address sparse rewards in long-horizon agentic RL. It augments environment observations with Action Guidance (AG) or Observation Enrichment (OE), applying AG earlier and OE later in both interaction trajectories and training. Experiments on SciWorld and BFCL-V3 Multi-Turn with Qwen3-4B/8B and GRPO, DAPO, and GSPO report improvements when the trained agents are evaluated in standard environments. Further analyses examine exploration, training stability, feedback internalization, and intra-group feedback consistency.
Critical strengths
The paper targets an important bottleneck in long-horizon agentic RL with a clear and appealing idea: enrich feedback from the training environment rather than introduce another complicated optimization pipeline. The method is simple, interpretable, and compatible with existing RL algorithms. The evaluation covers two benchmarks, two model scales, and three RL algorithms, and tests the learned policies without enriched feedback at inference time. The public implementation is another useful contribution.
Critical weaknesses
Insufficient rollout-level evidence and component attribution. The paper's central causal claim is that enriched feedback increases the chance of positive-reward rollouts, yielding more informative RL updates and ultimately better policies. Although Figure 4 reports BFCL training-time avg@8 success rates, it does not systematically quantify the improvement over standard rollouts at matched policy checkpoints, particularly early in training, or the proportion of non-degenerate groups with non-zero reward variance. Nor does it isolate the contributions of AG, OE, and their scheduling to these intermediate outcomes. Without controlled, matched-budget rollout-level ablations linking these effects to final performance, the evidence does not fully establish the proposed mechanism for alleviating reward sparsity.
Insufficient justification of feedback scheduling and general applicability. Figure 2 compares feedback types and timing, but the early and late interaction windows have unequal lengths (steps 1–3 versus 6–10), while the switch at training step 100 and the 0.5 enrichment probability lack sufficient justification. The authors should compare AG→OE against AG-only, OE-only, OE→AG, and a fixed AG/OE mixture under matched feedback budgets, and examine how performance changes with these design choices. More fundamentally, the FEE implementations rely on environment-specific information: SciWorld AG uses ground-truth expert trajectories, while OE exploits internal task-progress or tool-state information. It remains unclear how AG/OE can be constructed in environments where such information is unavailable. Furthermore, without comparisons against alternatives using the same expert information, such as SFT warm-up, it is difficult to determine whether FEE genuinely addresses the original limitations or merely changes how expert guidance is delivered.
Questions for authors
- At matched checkpoints and rollout budgets, how much do FEEs improve positive-reward rollout rates and the proportion of non-degenerate groups, especially early in training?
- Which components drive these rollout-level gains? Can the authors compare AG, OE, and alternative schedules, and report how performance changes when the switch step, enrichment probability, and interaction windows are varied?
- How can AG/OE be constructed in environments without expert trajectories or accessible internal progress signals? Under matched expert information, how does FEE compare with SFT warm-up?
- How do the feedback-format probes demonstrate useful policy internalization? Why does Section 5.2 claim improvements across difficulty tiers when Figure 4 shows declines on medium and easy tasks?
5 comments
Rebuttal 2 / 3
Q3: Constructing AG/OE without expert trajectories; FEE versus SFT under matched expert information.
On the SFT comparison, setting it up exposes an asymmetry before any training happens. SciWorld ships gold action sequences, not reasoning traces, and supervising on bare actions breaks the think/action format. An SFT warm-up therefore needs a rationale synthesized for each of the 8,463 gold actions; we generated these with a separate teacher model (gpt-6-astra). They are post-hoc explanations fitted to actions already chosen, and leakage can be constrained by prompting but not verified. The teacher prompt was:
You are creating synthetic ScienceWorld SFT action rationales. This is a text transformation only: do not call tools, read files, or browse. Return ONLY the requested JSON. Each input record is an independent decision. For each numbered turn, write a short English rationale for the supplied action using the task and observations from that turn or EARLIER turns ONLY. Never borrow information from a later record, even though the records are batched for efficiency. Do not predict or assert the result of the action before executing it. Do not invent temperatures, properties, measurements, success, locations, object observations or verifier progress. The action is supplied supervision, NOT evidence of an object's hidden property. If supporting evidence is absent, phrase the action as a next step to try/check; do not fabricate certainty. No references to gold paths, oracle, ground truth, annotations, future turns, hidden state, reward or supplied answers. Use 1-3 concise sentences (typically 20-65 words). Tie the current subgoal to the next action; avoid repetitive deliberation, 'wait actually', and generic filler. An apparently redundant look/wait/open may be acknowledged honestly as a check; never invent a benefit for an invalid action. Keep action strings EXACTLY unchanged. Do not include XML tags in rationale: the exporter adds <think> and <action>. These are post-hoc synthetic explanations, not a claim about a model's internal thought. Before returning, check every rationale against the chronology and remove hindsight. Output {"turns":[{"turn":1,"action":"exact input action","rationale":"..."},...]}. Include exactly one output for every supplied turn, in order.
With that data built, SciWorld with Qwen3-8B, one protocol applied identically to both rows (128 episodes, greedy decoding):
| Step budget | Base | SFT warm-up | Delta |
|---|---|---|---|
| 15 | 16.41 | 3.91 | -12.50 |
| 30 | 26.56 | 4.69 | -21.88 |
SFT warm-up removes 76% of the base model's task success at a 15-step budget and 82% at 30 steps. A warm-up exists to raise the probability of obtaining an initial positive reward, and this one removes four fifths of them.
The gold data explains the direction. 8.1% of supervised actions return "already in that state", 673 of them already-open doors, and 267 rationales call their own action redundant. Only 71 of 200 gold paths finish within 15 steps; the median is 23 and 52.3% of supervision comes from beyond step 30, past the budget the agent is evaluated under. Recovery is undemonstrated: 0.6% of prompts contain an invalid-action message. At evaluation the SFT model's formatting is better than base, but rejected actions rise from 24.4% to 49.4% between the halves of an episode.
On whether FEE merely changes how expert guidance is delivered, the quantities involved are not comparable. SFT consumes all 8,463 actions, each with a rationale attached. Action guidance reads the first 3 actions of each of the 200 training tasks, 600 in total, about 7% of the SFT supervision, directly from the gold path with no processing, and only within steps 1-3 of an episode, with probability 0.5, during the first half of training. It is not a cheaper demonstration pipeline; it is a decision about where in an episode a small amount of hand-authored signal is worth placing. The gold path's own defects make the distinction concrete: the redundancy and the 773-step tails that sink SFT are concentrated in exactly the parts of the trajectory that action guidance never touches.
For environments without expert trajectories, two parts of our own setup already use none. Observation enrichment comes from the environment's built-in goal-progress checker in SciWorld and from simulator state in BFCL, both of which are information the environment already computes and simply does not show the agent; the analogues in open-ended settings are test logs, compiler output, and DOM differences. And in BFCL one class of action guidance is authored from the task category alone: for tasks with missing parameters the hint reads "Essential information is missing from the user's request. Do not make any assumptions; explicitly ask the user to clarify the missing parameters", which requires no knowledge of the correct answer.
Rebuttal 3 / 3
Q4a: What the probes show. We may not have made the design clear enough, so let us restate what the probe controls for.
The question is whether enriched feedback is absorbed into the weights or only acts while present in the context. To separate these, the probe is asked with no enriched feedback anywhere in the context: the model gets a plain multiple-choice question about what a tool call returns, in the standard format, and we read off the probability it assigns to the enriched return versus the original one. Because nothing in the prompt mentions the enrichment, any difference between an FEE-trained and a standard-trained model can only come from the weights. The construction is not measuring whether the model can use a hint it can see; it is measuring whether it retained the environment dynamics after the hint was gone.
On what the result shows, we should be more careful than the paper is. The margin P(B) - P(A) stays negative for every probe at every checkpoint, so the model always prefers the original option; only the degree varies. Apart from the cd probe, which rises from about -0.83 to -0.67 at step 150, the enriched curves stay between -0.95 and -1.0 and by step 200 have returned to about -0.98. The standard-trained curves are flat at -1.0 throughout, so the two conditions do differ in the predicted direction.
On the "useful" part of the question, the probes are not where that evidence comes from, and we should not have leaned on them for it. It comes from the evaluation protocol: every number we report, in Table 1 and in Figures 2, 4 and 6, is measured in the standard environment with no enrichment present. FEE-trained agents are better there. Whatever they gained during training, they kept after the scaffold was removed, which is the property that matters for the claim that the guidance is not merely an inference-time prior.
Q4b: Section 5.2 contradicts Figure 4. The reviewer is right that the sentence is wrong, and we will remove it. But the tiers also deserve to be read in absolute terms, because they are very different sizes.
| Tier | N | Standard | FEE | Delta | Environment-equivalents |
|---|---|---|---|---|---|
| Hard | 235 | 6.5 | 10.9 | +4.3 | +10.1 |
| Medium | 73 | 50.8 | 49.8 | -1.0 | -0.7 |
| Easy | 92 | 87.6 | 87.0 | -0.6 | -0.6 |
| All | 400 | 33.2 | 35.5 | +2.3 | +8.8 |
The hard tier holds 235 of the 400 environments, and the gain there is worth about 10 environments. The declines on medium and easy are each worth less than one: 1.0 point spread over 73 environments is 0.73 of an environment, and 0.6 points over 92 is 0.55. These sit below the granularity at which a single environment flips, so we would not read them as degradation. The environment-weighted average moves from 33.2 to 35.5.
The revised text will say: the gain is concentrated on hard environments, which are also the majority of this benchmark, while medium and easy are unchanged to within the resolution of a single environment.
Rebuttal 1 / 3
We thank the reviewer for the detailed comments. They identify real gaps and one error in our text, and we address each.
Q1 / W1: Rollout-level evidence. We ran new runs with full rollout dumps, standard versus FEE, on SciWorld with Qwen3-4B and GRPO, the setting where we had complete per-rollout logging. Over 27 training steps each condition has 432 rollout groups of 8 trajectories (3,456 trajectories per condition). A group is non-degenerate when it contains both successful and failed trajectories, which is exactly the condition for a non-zero group-normalized advantage.
Two observations.
First, the standard environment wastes more than half of its rollout budget: 56.71% of groups contain no successful trajectory and contribute exactly zero gradient. This quantifies the sparsity the paper motivates, which we had only described qualitatively.
Second, and this is what speaks to the mechanism rather than to difficulty: of the 4.86 points by which FEE reduces all-zero groups, 95.2% reappear as non-degenerate groups and 4.8% as all-success groups. If enriched feedback were simply making tasks easier, the all-success band would absorb a large share of the shift, and those groups produce no gradient either. It does not move: 2.31% to 2.55%. FEE converts groups that carry no learning signal into groups that do, without pushing them through into saturation. The same holds further up the distribution: groups with k>=5 rise from 15.51% to 18.75%, and the mean success count per group from 1.579 to 1.810.
Q2 / W2: Components, schedule, and the unequal windows. Figure 2 is already a 2x2 over what is enriched (AG or OE) and when it is delivered (steps 1-3 or 6-10), with AG-only and OE-only runs in it.
On the unequal windows specifically: the 6-10 window is 5 steps against 3, so it delivers about 67% more hints. If the outcome were driven by how much feedback is given, the longer window should win. For AG it loses, and loses badly: AG-Late stays near zero success for the first 50 training steps, below even the standard environment, while AG-Early is already climbing. The budget asymmetry runs against the conclusion we draw from that comparison rather than in its favour. Reading the four runs across both factors, the type of information separates them more than the window does: AG leads OE by 0.083 at steps 1-3 and by 0.043 at steps 6-10, while moving the window within a fixed type changes the result by 0.023 for AG and 0.017 for OE.
We are asking what properties a training environment needs, not searching for the configuration that maximizes the score. The switch step and the enrichment probability were fixed once in the pilot and reused unchanged across all 6 configurations in Table 1, two benchmarks, two model scales, and three algorithms; a schedule that was never tuned on the main experiments still improves every configuration's average. The corollary is that our numbers are very likely not the ceiling of this design space, and we do not claim AG-then-OE is its optimum. What we claim is the pair of properties that govern the outcome: what kind of information the environment returns, and whether it arrives early or late, both within an episode and across training. Those two are what Figure 2 separates, and they are what carries to a new environment. A specific switch step or window boundary does not carry: in a different environment it would have to be found again regardless of what we reported here.