acceptodds
Withdrawn from ICLR 2027

No Do-Overs: Observation-Grounded Credit for Language-Model Agents Without Re-Execution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Improving a long-horizon language-model (LM) agent is a problem of credit assignment: localizing a terminal outcome to the steps that earned it. We study it in the regime a deployed agent faces (credit assignment in deployed agents), where a text-action LM agent must learn from what it logged. A deployed agent updates its policy off-peak, while it is not serving user traffic, on the trajectories that serving already produced, and by update time cannot rewind a session, re-execute a tool call, or consult an oracle: its actions are irreversible. Yet the dominant tools for per-step credit in agentic RL rely on exactly the counterfactual access a deployed agent lacks, so credit must come from the logged trajectory alone. One signal the rule leaves readable is the observation itself, which in a text-action setting states in words what each action did. Prior methods consume observations too, but only through a learned model or a monolithic self-belief; we instead read credit directly off the observation, with no trained reward or value model on the credit path. We call this observation-grounded step credit. As a GRPO reward it improves long-horizon agents across three environments using only logged information. On ALFWorld, two rounds raise the base from to , the highest among same-backbone methods and on par with a privileged comparator that uses strictly more information; the same operator improves over its terminal-credit ablation and strong baselines on ScienceWorld.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-evolving agents

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_d4VJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper should be desk-rejected. There are two hallucinated references in this paper.

On page 12, the reference titled “Understanding the performance gap between online and offline alignment algorithms” (Line 645-647) is attributed to ICML 2024. However, its author list—including Mark Rowland, Pierre Harvey Richemond, and Bilal Piot—matches “Generalized Preference Optimization: A Unified Approach to Offline Alignment,” which appears in the official ICML 2024 proceedings. The paper bearing the cited title has a different author list in its original arXiv record, 2405.08448, and I could not locate that title in the ICML 2024 main-conference proceedings. This appears to conflate the title of one paper with the authors and publication venue of another. The mismatch is present in the manuscript itself.

There is also a separate venue error on page 13: “Free Process Rewards without Process Labels” (line 700-701) is listed as an ICLR 2025 paper, whereas its formal publication record is ICML 2025.

Critical strengths

The author honestly discloses the attempts of their failure work in Appendix F WHAT WE TRIED THAT DID NOT WORK.

Critical weaknesses

This paper should be desk-rejected. There are two hallucinated references in this paper.

On page 12, the reference titled “Understanding the performance gap between online and offline alignment algorithms” (Line 645-647) is attributed to ICML 2024. However, its author list—including Mark Rowland, Pierre Harvey Richemond, and Bilal Piot—matches “Generalized Preference Optimization: A Unified Approach to Offline Alignment,” which appears in the official ICML 2024 proceedings. The paper bearing the cited title has a different author list in its original arXiv record, 2405.08448, and I could not locate that title in the ICML 2024 main-conference proceedings. This appears to conflate the title of one paper with the authors and publication venue of another. The mismatch is present in the manuscript itself.

There is also a separate venue error on page 13: “Free Process Rewards without Process Labels” (line 700-701) is listed as an ICLR 2025 paper, whereas its formal publication record is ICML 2025.

Questions for authors

NA

Reviewer_tDwHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies step-level credit assignment for LM agents when environment re-execution is unavailable. It proposes OG-Gen to synthesize observations for unexecuted candidate actions and an observation-grounded reward combining execution validity, subgoal progress, and compliance. Experiments on ALFWorld, ScienceWorld, and Deep Search report improvements over terminal-reward baselines.

Critical strengths
  1. The paper addresses a practically relevant problem: improving agents using logged trajectories without environment re-execution.

  2. The empirical analysis of pre-action judgments, post-action observations, and reward degeneration provides useful insights. The reported improvements in agent performance are encouraging.

Critical weaknesses
  1. Insufficient justification for the central contribution of OG-Gen. A central claim of the paper is that generating synthetic observations with an LLM-based world model enables more reliable step-level credit assignment than directly judging candidate actions. However, the experiments lack a direct comparison between OG-Gen and an LLM-based process judge under matched computational budgets. Since synthetic observations are themselves generated by an LLM, it remains unclear whether this additional generation step provides meaningful benefits over direct action evaluation. Such a comparison is important to establish the effectiveness and incremental contribution of OG-Gen.
  2. Limited evidence for the proposed reward decomposition. Although the paper proposes a unified E/V/O credit-assignment framework, Appendix C.1 reveals that its effective reward signals differ substantially across environments. Without component-wise ablations, it remains unclear whether the proposed decomposition offers meaningful benefits beyond environment-specific step-level reward shaping.
  3. Reference accuracy. The bibliography contains at least one incorrectly attributed reference. Specifically, Understanding the Performance Gap between Online and Offline Alignment Algorithms (Line 645-647) is cited with the authors and ICML 2024 venue of a different paper. The authors should carefully verify the accuracy of their references.
Questions for authors

Please refer to the concerns raised in the Weaknesses.

Reviewer_D7XoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies credit assignment for language-model agents under an irreversible environment access constraint, where policy updates can only use previously logged interactions without resetting or re-executing the environment. The authors propose OG-Gen, which uses a frozen language model to synthesize post-action observations for unexecuted candidate actions, together with an observation-grounded step-credit mechanism combining execution validity, subgoal progress, and action compliance signals. The resulting rewards are used for GRPO training without a separately trained reward or value model. Experiments on ALFWorld, ScienceWorld, and Deep Search report improvements over terminal-reward baselines and several existing methods.

Critical strengths
  1. The paper addresses the important problem of improving deployed language-model agents from logged interactions without environment re-execution. Using textual observations to construct step-level learning signals is a practically relevant direction, particularly for long-horizon tasks where terminal rewards provide limited information.

  2. The proposed method improves ALFWorld success rate, outperforming the terminal-credit GRPO baseline. It also reports improvements on ScienceWorld and Deep Search, along with reduced environment actions and token usage.

Critical weaknesses
  1. A central component of OG-Gen is using a frozen language model to predict observations for unexecuted candidate actions. However, these observations are not necessarily consistent with the true environment dynamics. The paper does not sufficiently establish the fidelity of generated observations or analyze how simulation errors affect reward quality and policy optimization. This is particularly concerning for irreversible or partially observable environments.

  2. The paper argues that pre-action language-model judgments are unreliable and that actual post-action observations provide more trustworthy feedback. However, OG-Gen generates hypothetical post-action observations using another language model, which can inherit similar prediction errors and biases. It is unclear why generating an observation before evaluating it should be more reliable than directly predicting action success.

  3. The proposed reward combines execution validity, subgoal progress, and format compliance using a weighted sum. These are intuitive reward-shaping signals commonly used in agent training. The paper's distinctive contribution is primarily the application of this decomposition under a logged-data constraint, but the advantage over simpler observation-based reward heuristics is not sufficiently isolated.

Questions for authors
  1. How accurately does OG-Gen predict real environment observations? Can the authors compare synthetic observations against held-out real transitions, reporting execution-success prediction accuracy, subgoal-verification accuracy, and reward-ranking agreement? How sensitive is policy performance to these prediction errors?

  2. What is the benefit of the two-stage pipeline (generate an observation, then verify it) compared with directly asking the same frozen model to predict execution success or subgoal progress? A controlled comparison would help isolate the value of synthetic observation generation.

Reviewer_tWKwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors study “How AI Assistants Learn in Deployment Environments.”

The paper first proves that none of the three common routes work - the model itself judges whether this step will be successful, and 75% to 81% of the failed steps are confidently guessed wrong; training a "reward distributor" is almost irrelevant to the real right and wrong; the reward of just looking at the probability of action will completely fail in GRPO.

Then propose a scheme: OG-Gen (simulate offline with a frozen AI model what the environment will respond to after performing an action) + no-trained operator that scores the observation text directly (whether the execution is effective E/whether to advance the sub-goal V/whether the format is compliant O). As a reward for GRPO, the two rounds of training increased the success rate of ALFWorld from 67.2% to 78.4%. On ScienceWorld and Deep Search, it also exceeded the baseline of only seeing the final success or failure, and the number of actions and tokens was less, and there was no ability to forget. However, there are hidden dangers in its key links: OG-Gen simulates "observation without performing actions" with a frozen model, and whether the simulation is accurate is never directly verified - and this directly determines whether the scoring and training signals are credible, which is the core and weakest part of this method.

Critical strengths

Complete logic chain with solid evidence: statistics first rule out self-judgment and learned redistribution, then a solution that exactly avoids both pitfalls, with failures and boundaries reported honestly.

Critical weaknesses
  1. Claims and evidence are inconsistent: the title of the paper says "read credit from three signals: execution, sub-goal, and compliance", but the author himself admits that only "execution effectiveness" (E) of the most important ALFWorld results really provides a learning signal - only 0.1% of the 800 decision points of the "sub-goal" signal (V) have a degree of discrimination, and "compliance" (O) is a constant in the group. In other words, the actual function is "execution signal + OG-Gen". The decomposition of the three signals has never been verified separately. The weights are still set manually and have not been ablated.

  2. The strongest results cannot withstand detailed questions: all are single seeds, different models are mixed with "leveling with GiGPO", and the gap is not opened at all on BFCL.

Questions for authors
  1. Does the subgoal signal V actually help? Please ablate V, or show a cell where V alone provides learning signal; if not, would you agree to restate “three signals” as “execution-grounded”?

  2. Can you run GiGPO on the same backbone (Qwen3.5-9B)? The “matching a privileged baseline” claim is confounded by model differences.

  3. Is the round-over-round improvement just noise? (R1→R2 +2.3%, R2→R3 +2.6%, both inside the ±4% band.) Is there multi-seed evidence? Does the 12% ScienceWorld gap survive fixed training steps or averaged checkpoints?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.