acceptodds
Under review as a conference paper at ICLR 2027

Falsify Before You Amplify: Conservative Cross-Fitted Credit for Self-Evolving Language Agents

Abstract

Hindsight skills provide dense supervision for language agents, but explaining a completed trajectory does not establish which actions deserve credit. When the same trajectory supplies both a skill and its supporting evidence, reconstruction can be mistaken for guidance. We introduce EvoCred, a reinforcement learning method that separates the authority to establish credit from the ability to allocate it. A frozen policy evaluates each target's fixed actions under a skill generated from another rollout of the same task. Evidence strength and compatibility with verifier-derived advantages govern the skill's influence. Admitted evidence then redistributes token advantages through a positive, bounded transformation that preserves their signs and each trajectory's total absolute credit. The verifier establishes the credit budget; hindsight only reallocates it. Across ALFWorld, Knowledge QA, and WebShop with three Qwen backbones, EvoCred ranks first on ten of twelve aggregate comparisons and outperforms GRPO, SEED, and AgentOPSD on all twelve. On Qwen3-1.7B, it improves ALFWorld macro success from 42.1% to 91.2% and WebShop success from 38.3% to 76.5% over GRPO, without additional environment rollouts or inference-time skill access.

Primary area:
reinforcement learning
Keywords:
Agentic Reinforcement Learning, Temporal Credit Assignment, Self-Evolving Language Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_18ZNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Core problem: Generating a skill retrospectively from a trajectory and then using that skill to increase the likelihood of actions in the same trajectory may mistake the reconstruction of past behavior for evidence of transferable learning. EvoCred investigates when such hindsight should influence credit assignment, on what grounds, and to what extent. Method: We first derive the standard GRPO advantage from within-group task rewards. For each target trajectory, we use a skill generated from another trajectory for the same task, and compare the likelihoods with and without the skill while holding the original actions, history, and prefix fixed. The resulting change in recursive support is filtered through gates for raw strength and directional compatibility, then standardized, passed through a tanh transformation, and centered using absolute base-credit weighting to produce a positive, bounded advantage multiplier. Skills may only redistribute credit within each trajectory’s existing budget; they cannot alter the sign of any nonzero advantage, create support where the base advantage is zero, or change the total absolute credit. All skill-related operations are used only during training. Novelty: The main contributions lie in the source of the skills, the admission criteria governing their influence, and exact per-trajectory budget conservation. The insight that the environment determines the update direction while the teacher modulates token-level magnitude has precedent in RLSD. Recursive support, sign alignment, and positive bounded multipliers are also consistent with AgentOPSD, while the reuse of trajectories within the same group has precedent in SDPO. The paper should therefore be positioned as a principled combination of existing components with novel constraints, rather than claiming every foundational component as original. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9) when distinguishing token-importance weighting from provenance-aware credit redistribution. A broader conceptual comparison with Ning Yang et al., Proactive Constrained Policy Optimization with Preemptive Penalty (AAAI 2026; https://doi.org/10.1609/aaai.v40i32.39978) could clarify the difference between penalty-based constraint handling and the exact sign and credit-budget invariants used by EvoCred; PCPO does not address hindsight provenance and is not a directly interchangeable experimental baseline. Evaluation: The experiments cover three Qwen backbones and three task suites—ALFWorld, Search-QA, and WebShop—and include two environment-specific component ablations, skill diagnostics, and interventions using erroneous skills.

Critical strengths

SEED, Skill-SD, OPID, and AgentOPSD all treat an increase in the likelihood of a trajectory’s own actions under a trajectory-derived privileged teacher as a valid learning signal. Yet a skill is itself retrospectively inferred from the success label of that same trajectory. Its ability to increase the likelihood of the source trajectory’s actions may therefore amount to little more than reproducing source-specific observations and actions, rather than providing genuine evidence that those actions deserve credit.The paper’s value lies in showing that this problem cannot be solved through temporal aggregation or bounded reweighting. It must be addressed at the level of evidence provenance. The proposed framework is also conceptually clean: it separates the authority that determines credit from the mechanism that distributes credit, and elevates provenance to an explicit constraint. Specifically, the teacher trajectory must differ from the target trajectory, and the pairing must be fixed before any conditional scores are observed. This makes the framework broadly transferable.

Critical weaknesses

Classical debiasing through cross-fitting or sample splitting relies on samples being independent and identically distributed across folds. By contrast, the k sibling trajectories are correlated samples generated by the same policy on the same task. As policy entropy decreases, sibling trajectories are increasingly likely to exhibit similar behavioral patterns and failure modes. A skill derived from trajectory A is therefore not independent of trajectory B. The paper directly treats this insufficiently characterized correlated process as a falsification mechanism, yet provides no analysis of how sibling similarity affects debiasing effectiveness and includes no experiments varying the number of sibling trajectories.The empirical evidence likewise does not support a clean causal attribution. The primary evidence for the proposed mechanism is the ALFWorld self-source ablation, where performance increases from 84.3 percent to 89.6 percent. However, this comparison changes several factors simultaneously. Under the self-source condition, the target receives a restatement of its own experience. Under cross-source, outcome-aware pairing with reusable donors, it instead receives a skill derived from another trajectory. This changes the skill’s textual content, quality, and alignment with the target’s specific actions. The observed gap therefore cannot be attributed solely to the removal of resubstitution bias; it may simply result from replacing the original skill with one that contains a different amount or type of information.More concerningly, outcome-aware pairing selects the donor distribution according to trajectory outcomes before any scoring takes place. This may introduce a new form of selection bias, but the paper provides no analysis of this possibility.

Questions for authors

1.In the actual implementation underlying Table 2, does the “without centering” variant use q divided by two or u divided by two? Please provide the configuration or implementation corresponding to this row and reconcile the descriptions in Section 3.5 and Appendix C.1. If the implementation uses q divided by two, could the authors add a comparison that removes only the final weighted-centering step while retaining all other transformations? This distinction directly determines whether the independent empirical contribution of credit conservation is supported. 2.What is the complete outcome-aware donor-selection and fallback procedure? How is the corruption in Figure 4 generated, what proportion of skills are corrupted, and which skills receive the corruption? Which specific operators are disabled in the “No Hard Constraints” condition? Please report the task, backbone, number of repetitions, and meaning of the shaded regions, and explain the setup of the 240-step experiment. These details are necessary to determine whether the diagnostic and intervention results support the paper’s proposed mechanism. 3.Does the advantage of source exclusion and adaptive admission persist when donor success status is matched as closely as possible, or when the gate is fixed to produce an average influence magnitude similar to that of the full method? If additional training is required, a limited multi-seed evaluation on one representative task against the strongest neighboring baseline would be sufficient.

2 comments

Thank you again for your careful review and helpful suggestions. As we are now past the halfway point of the rebuttal period, we would appreciate hearing whether our responses and additional experimental results have fully addressed your concerns. If so, we would be grateful if you would consider raising your score in light of the new evidence. If any questions remain, please let us know so we can address them during the discussion period.

We will incorporate the new results and the clarifications from this discussion into the revised manuscript.

Area_Chair_tMTeOpenReview ↗

Please follow up before the discussion period ends

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Thank you for your contribution!

Best,

Your AC

Reviewer_NEvuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EvoCred, a method for using hindsight skills to redistribute credit during reinforcement learning of language agents. Its motivation is that rescoring a trajectory using a skill extracted from that same trajectory can reward retrospective reconstruction rather than transferable guidance. EvoCred instead assigns each target trajectory a skill generated from another rollout of the same task. A frozen policy measures skill-induced likelihood changes on the target’s recorded actions. An admission gate controls influence according to evidence strength and agreement with return-derived advantages. Positive, bounded multipliers and credit-weighted centering preserve advantage signs and each trajectory’s total absolute credit.

Critical strengths
  1. The identification of source-trajectory resubstitution as a source of ambiguity in assigning credit from trajectory-derived hindsight skills is good. And the authors utilize the skill knowledge of sibling trajectories and verifier-aligned admission to better conduct credit allocation.

  2. The performance is good, that evaluation on ALFWorld, Search-QA, and WebShop demonstrates leading performance in ten of twelve aggregate comparisons, alongside component ablations, evidence diagnostics, a corrupted-skill experiment, and a separate compute study.

Critical weaknesses
  1. Tables 1 and 2 provide point estimates without uncertainty estimates or a clear account of independent training repetitions. It is better to give the average and std of multiple training trials.

  2. How's the performance on the other backbones, such as distilled llama?

Questions for authors

See the Critical Weaknesses.

2 comments

Thank you again for your careful review and helpful suggestions. As we are now past the halfway point of the rebuttal period, we would appreciate hearing whether our responses and additional experimental results have fully addressed your concerns. If any questions remain, please let us know so we can address them during the discussion period.

We will incorporate the new results and the clarifications from this discussion into the revised manuscript.

Area_Chair_tMTeOpenReview ↗

Please follow up before the discussion period ends

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Thank you for your contribution!

Best,

Your AC

Reviewer_WHwzOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

EvoCred incorporates three layers of safety mechanisms for "hindsight skill supervision": experiences are prohibited from self-verification on the trajectories that generate them, evidence must pass intensity and direction audits, and credits can only be redistributed within a fixed budget, which transforms the auxiliary signal from a "potentially self-indulgent teacher" to a "supervised teaching assistant".

Critical strengths

Strength 1: Exceptional Problem Clarity and a Well-Motivated Research Gap

The paper converts the vague suspicion that "a skill's ability to explain a trajectory does not imply its ability to guide learning" into a precisely defined phenomenon: source-trajectory resubstitution. It further grounds this problem in two classic theoretical perspectives:

  • Adaptive data analysis (Dwork et al., 2015): reusing the same data for both construction and evaluation can undermine generalization.
  • Double dipping (Kriegeskorte et al., 2009): selecting and validating hypotheses on the same data introduces systematic bias.

This framing gives the paper a clear conceptual foundation and motivates the need to separate skill construction from credit evaluation.

Strength 2: A Clean and Generalizable Separation-of-Authority Principle

The central design principle can be summarized in one sentence:

Task returns determine the sign and total budget of credit; hindsight only redistributes credit within that budget.

This principle captures the paper's key methodological contribution: it "separates the evidence establishing credit from the evidence redistributing it." By restricting hindsight-derived skills to redistribution rather than allowing them to independently determine credit, the method establishes a clear boundary between evidence authority and allocation flexibility.

Strength 3: Rigorous Theoretical Guarantees with Clearly Stated Boundaries

The appendix provides a rigorous derivation of several properties:

  • Sign preservation: positive redistribution multipliers preserve the signs of the original token-level advantages.
  • Exact mass conservation: credit-weighted centering ensures that the total absolute credit assigned to each trajectory remains unchanged.
  • Bounded perturbation: the relative change of each coefficient and the total variation distance are bounded.

Importantly, the paper explicitly distinguishes conservation of the scalar credit coefficient from conservation of the policy gradient. This distinction demonstrates methodological rigor and avoids overstating what the theoretical results establish.

Strength 4: Hierarchically Designed Experiments and Informative Negative Controls

The experimental evaluation provides three complementary layers of evidence:

  1. Broad empirical comparisons: the main results cover three backbone models and three benchmark suites, with EvoCred winning all 12 aggregate comparisons reported by the authors.
  2. Diagnostic ablations: the shuffled-allocation variant incurs the largest performance degradation, suggesting that the temporal placement of evidence matters and that arbitrary bounded reweighting alone is insufficient.
  3. Corruption robustness: under skill corruption, unconstrained training deteriorates while the constrained variant remains comparatively stable, supporting the claim that conservative credit redistribution can mitigate the impact of unreliable skills.

Together, these experiments evaluate not only whether EvoCred improves performance, but also whether its proposed design choices contribute to those improvements.

Strength 5: Honest Methodological Self-Limitation

The paper explicitly acknowledges that cross-fitting is not equivalent to independent validation or causal identification. This limitation is stated in multiple places, indicating that the authors distinguish the evidence their method provides from stronger claims it cannot establish.

Such explicit boundaries strengthen the paper's methodological credibility and make its contribution easier to assess without conflating reduced resubstitution bias with full statistical independence or causal attribution.

Critical weaknesses

Weakness 1: The Core Claim of Cross-Trajectory Evidence Transferability Is Not Directly Validated

The core claim—that cross-trajectory evidence is transferable and can reliably identify useful hindsight skills—is never directly validated.

The admission gate in Eq. 1 is heuristic: ρ\rho is a continuous function of evidence strength mim_i and compatibility CiC_i. However, the paper provides no calibration analysis linking ρ\rho to independent future returns. In particular, it reports no AUROC, bucketed success rates, or rank correlations that would establish whether the gate's confidence scores meaningfully predict the quality of transferred evidence.

The supporting evidence is indirect. Removing the gate (the w/o admission variant in Table 2) reduces performance by 3.4 and 2.4 percentage points on ALFWorld and WebShop, respectively. However, this ablation conflates two distinct mechanisms:

  1. Evidence selection: the gate identifies and retains genuinely useful evidence while rejecting unreliable evidence.
  2. Update-magnitude control: the gate simply limits the magnitude of credit redistribution, regardless of whether it correctly identifies useful evidence.

The observed performance gap does not distinguish between these explanations.

Similarly, the corrupted-skill study in Section 3.6 introduces a single type of corruption at a single training step (step 80). This experiment demonstrates robustness to a narrow failure mode, but it does not establish that the admission gate reliably rejects bad evidence across different corruption types, severities, or training stages.

Overall, the paper establishes that constrained training can outperform unconstrained training, but it does not demonstrate that the constraint is applied selectively to the right evidence. Direct calibration analyses, such as correlating ρ\rho with independently measured future returns, together with controls that separate evidence selection from update-magnitude restriction, would substantially strengthen the central claim.

Questions for authors

Q1. Is the Benefit of Admission Due to Selective Filtering or Merely Magnitude Control?

The w/o admission variant sets ρ=1\rho = 1, which simultaneously removes selectivity and increases the redistribution amplitude to its maximum. Could the authors run a control that keeps ρ=1\rho = 1 but scales down λ\lambda so that the average redistribution magnitude matches that of the full method (approximately E[ρ]≈0.7\mathbb{E}[\rho] \approx 0.7)?

If the performance degradation disappears under this control, the gate's contribution may be primarily magnitude control rather than selective evidence filtering. This would substantially weaken the interpretation of the compatibility mechanism in Eq. 1 as a means of identifying reliable evidence.

Q2. Is the Gate Blind to Within-Trajectory Misallocation, Where Credit Assignment Matters Most?

Since A0A^0 is broadcast across tokens,

Aikℓ0=mikℓ⋅Ai0,A^0_{ik\ell} = m_{ik\ell} \cdot A^0_i,

all tokens within a trajectory share the same underlying advantage sign, with token-level differences introduced only through mikℓm_{ik\ell}. Consequently, CiC_i appears to measure whether turn-level evidence agrees with the trajectory-level outcome, rather than whether credit is correctly assigned among individual actions.

A skill that is directionally correct at the trajectory level but misallocates credit among actions—for example, reducing the credit assigned to the genuinely critical step—could therefore pass the gate undetected.

Do the authors have any within-trajectory measure of evidence quality, or is the gate structurally limited to detecting trajectory-level errors? An analysis of whether admitted skills correctly identify critical actions would help clarify this limitation.

Q3. How Much of the Gain Comes from Successful-Trajectory Reweighting Rather Than Skill Evidence?

The asymmetric admission rates—84% for positive-advantage trajectories versus 19% for negative-advantage trajectories in Figure 5b—suggest that ρ\rho is strongly correlated with trajectory success. However, GRPO already assigns different relative advantages to successful and unsuccessful trajectories.

How much of EvoCred's performance gain, particularly the reported +49.1 percentage-point improvement on Qwen3-1.7B, can be attributed to effectively upweighting successful trajectories rather than exploiting the information contained in hindsight skills?

A relatively inexpensive control would amplify the group-relative credit assigned to successful trajectories without using any skill scoring or evidence-admission mechanism. If this control reproduces a substantial fraction of EvoCred's gains, the improvement may be driven primarily by credit reweighting rather than evidence admission itself.

Q4. Can the Transferability of Admitted Skills Be Tested Through Zero-Shot Conditioned Inference?

Although skills are not available at deployment, the authors could directly test their behavioral utility during evaluation by using admitted and rejected training-time skills as zero-shot conditioning prompts on held-out tasks.

Specifically, could the authors compare three conditions: (1) admitted skills, (2) rejected skills, and (3) no skill conditioning?

If admitted skills consistently improve zero-shot success relative to both rejected skills and the no-skill baseline, this would provide direct behavioral evidence supporting the claim that the admission mechanism identifies transferable, useful guidance. Conversely, if admitted skills provide no measurable benefit, the training gains might instead arise from an implicit regularization effect or other aspects of the training procedure, rather than from the claimed transferability of cross-trajectory evidence.

Such a diagnostic would help distinguish the proposed evidence-transfer mechanism from alternative explanations for EvoCred's empirical gains.

2 comments

Thank you again for your careful review and helpful suggestions. As we are now past the halfway point of the rebuttal period, we would appreciate hearing whether our responses and additional experimental results have fully addressed your concerns. If any questions remain, please let us know so we can address them during the discussion period.

We will incorporate the new results and the clarifications from this discussion into the revised manuscript.

Area_Chair_tMTeOpenReview ↗

Please follow up before the discussion period ends & AI use clarification

Dear reviewer,

We are less than a day away from the end of the discussion period. Please read the authors' response and any revisions, and post a brief follow-up indicating which concerns remain and whether your assessment has changed. Please raise any outstanding questions promptly so the authors have time to respond.

Given the limited time, please focus on clarifications and existing evidence. You may identify gaps in the experimental support, but please do not expect the authors to conduct new experiments before the discussion period closes.

Additionally, your review was flagged for AI use while the corresponding field in your review does not mention how it was used. Please clarify this for me.

Thank you for your contribution!

Best,

Your AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Update

Thank you all reviewers for your thoughtful feedback. As the rebuttal period draws to a close and we have received no further comments since our responses, we have completed our revisions and updated the appendix with the additional results and experimental details. All changes in the appendix are marked in blue for ease of review. If you feel that the new evidence and clarifications have addressed your concerns, we would be grateful if you would consider raising your score. Thank you again for your time and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.