acceptodds
Withdrawn from ICLR 2027

From Trajectory Memory to Action Selection: Trajectory-Context Action Credit for Language Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language agents increasingly reuse past interaction trajectories as examples, memories, abstractions, and decision signals. However, retrieving a relevant trajectory does not directly tell an agent which action it should prefer at the current Decision State. A trajectory contains temporally extended experience, while action selection requires choosing among the Candidate Actions available now. We formalize this missing link as Trajectory-Context Action Credit, a candidate-level signal that makes the contribution of trajectory memory to action selection explicit. We introduce TrajCredit, a training-free instantiation that converts Retrieved Slices into candidate-level credit using Local State-Action Support, Procedural-Stage Consistency, and Outcome-Calibrated Evidence, and combines the resulting credit with an existing Base Selector. Across controlled candidate-action ranking, retrieval and memory controls, credit-to-progress alignment, and bounded environment intervention, TrajCredit consistently improves action ranking, provides information beyond raw retrieval support, aligns with independently measured short-horizon value, and improves executed behavior. These results support Trajectory-Context Action Credit as a useful interface between reusable agent experience and candidate-action selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, trajectory memory, action selection, experience reuse, sequential decision making, action credit

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Aa89OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how historical interaction trajectories can inform the selection of current candidate actions in language agents. It introduces the Trajectory-Context Action Credit interface and a training-free implementation, TrajCredit. The method retrieves trajectory slices relevant to the current task, observation, and interaction history, then combines local action similarity, procedural-stage consistency, and subsequent trajectory progress to score candidate actions. These credit scores are subsequently fused with those of a base selector. The authors evaluate TrajCredit on ScienceWorld and ALFWorld through controlled action-ranking experiments, retrieval-confusing candidates, component ablations, short-horizon value alignment, and bounded environment interventions. The reported results show improvements in candidate-action ranking across the evaluated configurations and gains in short-horizon environment progress on ScienceWorld.

Critical strengths
  1. The paper addresses a practically relevant problem with a clear motivation.
  2. The proposed implementation is lightweight and modular.
  3. The experiments provide encouraging evidence of utility.
Critical weaknesses
  1. The conceptual novelty of the proposed interface is insufficiently established.\textbf{The conceptual novelty of the proposed interface is insufficiently established.} Converting past experience into action-specific scores and using those scores to influence action selection is already central to related research. For example, Retrospex [1] combines action values learned from historical experience with language-model action likelihoods, while Neural Episodic Control [2] uses experience memory to support action-value estimation. The paper acknowledges these connections but primarily distinguishes its contribution through an explicit candidate-level interface and a training-free implementation. The definition of �0��0� in Equation (4) is sufficiently general to encompass existing experience-based action-scoring approaches. It does not establish distinctive properties, constraints, or analytical results that would make the interface a substantial conceptual contribution. As currently presented, the formalization mainly provides an organizing perspective on an existing problem.
  2. The algorithmic contribution is largely a combination of established scoring components, and the substantive technical advance remains unclear.\textbf{The algorithmic contribution is largely a combination of established scoring components, and the substantive technical advance remains unclear.} Combining Equations (14)–(17) gives �1��1� Thus, TrajCredit can be understood as a neighborhood-based action-support score whose weights are modulated by stage and outcome information. Its components consist of text retrieval, string similarity, a scalar stage kernel, outcome weighting, and standardized linear fusion. Although this construction is reasonable, the paper does not sufficiently establish what new decision-making mechanism or generalizable scientific insight emerges from this particular combination. This concern is especially apparent on ALFWorld, where �2�implies�3��2� implies �3�. The three-term formulation therefore reduces to a weighted combination of two distinct signals. Results on this environment cannot directly establish independent contributions from all three evidence views.
  3. The experiments do not adequately isolate the contribution of the proposed design.\textbf{The experiments do not adequately isolate the contribution of the proposed design.} The main comparisons evaluate whether adding the complete TrajCredit module improves a base selector. However, two particularly informative controls are missing: Base + local support �4��4� under the same retrieval and tuning protocol, and a TrajCredit-only selector. In Table 2, outperforming the base selector under retrieval-confusing candidates does not, by itself, rule out the possibility that a simpler retrieval-score fusion would achieve comparable gains. Furthermore, Table 8 shows continued improvement as the fusion coefficient �5��5� increases up to 16. Without a credit-only comparison, it remains unclear whether the gains reflect complementary information from the base selector and trajectory credit, or predominantly the influence of the memory-based score. Consequently, the current evidence does not sufficiently distinguish the benefit of the proposed multi-view construction from the benefit of introducing a strong memory-based scoring signal. This distinction is central to assessing the paper’s methodological contribution.
  4. The selection procedure and final values of the three component weights are insufficiently documented.\textbf{The selection procedure and final values of the three component weights are insufficiently documented.} Equations (17)–(18) specify that �6��6� are nonnegative and sum to one. Section 5.1 further states that all coefficients are selected on an independent development split and frozen before testing. However, the paper does not report the actual component weights, their search space, the specific selection criterion, or whether they are shared across environments, backbones, and base selectors. The sensitivity analysis in Table 8 concerns the external fusion coefficient �7��7�, which controls the combination of the base selector and TrajCredit. It does not assess sensitivity to the three internal component weights. Since this weighted combination constitutes the core of the method, these omissions limit reproducibility and make it difficult to determine whether the reported gains depend on a particular weight configuration.

[1] Xiang Y, Shen Y, Zhang Y, et al. Retrospex: Language agent meets offline reinforcement learning critic. EMNLP2024.

[2] Pritzel A, Uria B, Srinivasan S, et al. Neural episodic control. ICML2017.

Questions for authors
  1. What is the central technical contribution that distinguishes TrajCredit from existing experience-based action-scoring methods?\textbf{What is the central technical contribution that distinguishes TrajCredit from existing experience-based action-scoring methods?} Beyond a training-free implementation and an explicit interface, what substantive change does the method introduce in its mechanism or conclusions? Please clarify its relationship to experience-derived action values and candidate-action rescoring.
  2. $\textbf{Can the authors provide Base + �0��0� and TrajCredit-only comparisons within the existing evaluation settings?}$ Please use the same development-set tuning principles and report paired differences against the full method. These comparisons would help isolate the contribution of the multi-view construction.
  3. How are the three component weights determined in practice?\textbf{How are the three component weights determined in practice?} Please report the final values of �1��1�, their search space and selection criterion, whether they are selected jointly with �2��2�, and whether they are shared across experimental settings. How much improvement does the selected configuration provide over equal weighting, and do the main conclusions remain stable under reasonable changes to these weights?
Reviewer_tMXmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TrajCredit, a training-free mechanism that converts retrieved trajectory experience into candidate-action scores. It combines local action similarity, procedural-stage consistency, and historical outcome evidence, then fuses these scores with a base selector. Experiments evaluate controlled action ranking on ScienceWorld and ALFWorld, supplemented by short-horizon environment interventions.

Critical strengths
  1. The method is lightweight and interpretable, requiring neither additional training nor additional LM forward passes beyond the base selector.
  2. The ranking experiments show consistent improvements across multiple selectors and backbones, with useful ablations, randomized-memory controls, and paired statistical analyses.
  3. The environment interventions provide evidence that the proposed scores can improve short-horizon progress beyond agreement with reference actions.
Critical weaknesses
  1. My primary concern is that candidate-action setup substantially limits practical relevance. In the main ScienceWorld evaluation, each candidate set contains the reference action and 15 sampled distractors, guaranteeing reference-action availability. ALFWorld uses environment-provided admissible actions. These settings bypass action generation, whereas open-ended agents must construct tool calls and arguments themselves, which means that this method is difficult to extend to the pratical LLM agents.
  2. ReAct, SayCan, Reflexion, ExpeL, and CLIN are converted into a common ranking interface, but their prompts, scoring procedures, and retention of their original reasoning or reflection mechanisms are not adequately documented. Consequently, improvements over these implementations do not necessarily imply improvements over the corresponding complete agent systems.
  3. Experiments use Qwen2.5 and Llama3.1 backbones, leaving the incremental benefit over stronger reasoning-based agents uncertain.
Questions for authors
  1. Can the authors provide complete baseline prompts and scoring procedures, together with memory-only and base-plus-local-support comparisons?
  2. How are intervention states selected, and why do sample counts differ across horizons?
  3. Can this method generate the arguments for the tool use?
Reviewer_mDiGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that retrieving a relevant past trajectory does not by itself tell a language agent which of the currently available actions to prefer, and proposes to make this link explicit as a per-candidate scalar, "Trajectory-Context Action Credit" C(a | x_t, M). The instantiation, TrajCredit, is training-free: it retrieves the top-N stored steps by similarity over (task, observation, history), and scores each candidate action by a retrieval-weighted sum of its character n-gram similarity to the retrieved actions. This local score is optionally modulated by (i) a stage kernel exp(−|p_t − p_i|/σ_p), where p is the normalized cumulative task score in ScienceWorld, and (ii) an outcome weight η based on the clipped immediate score change and undiscounted return-to-go of the stored step. The three resulting scores are mixed with fixed weights, z-normalized over the candidate set, and added to an existing base selector's score with a coefficient λ.

Critical strengths
  1. Credit assignment is an important and neglected topic in the community. I would see the work done in the paper as a thoughtful attempt towards improving the body of work around this. Not many have addressed it from the lens of retrieval and similarity.
  2. Separating "is this trajectory relevant" from "which current action does it support" is a reasonable framing, and keeping the base selector, candidate set, and decision state fixed gives a clean way to measure the effect of memory on action ranking.
Critical weaknesses
  1. Missing related work: Although the contribution is framed as action credit, the paper does not engage with the RL credit-assignment literature at all. In particular:
  • RUDDER (Arjona-Medina et al., NeurIPS 2019) assigns step-level credit within trajectories by return decomposition and reward redistribution.
  • Align-RUDDER (Patil et al., ICML 2022) aligns a small number of demonstrations to identify shared stages and redistributes reward to steps matching those stages. This is conceptually very close to the combination of Procedural-Stage Consistency and Outcome-Calibrated Evidence proposed here.
  • Model-Free Episodic Control (Blundell et al., 2016) is a direct precursor of non-parametric, retrieval-based action valuation from stored episodes (only its successor, NEC, is cited).
  • Hindsight Credit Assignment (Harutyunyan et al., 2019), Temporal Value Transport (Hung et al., 2019), Synthetic Returns (Raposo et al., 2021), and eligibility traces / TD(λ) are also relevant.
  1. Method is simple, but presented as complex

Writing is a major weakness of this paper and in my opinion needs to be completely re-written. TrajCredit is a similarity-weighted k-NN vote over retrieved actions with two heuristic multipliers. There are roughly dozen new terms in this work: Decision State, Candidate Set, Base Selector, Retrieved Slices, Action-Centered Trajectory Memory, Local State-Action Support, Procedural-Stage Consistency, Outcome-Calibrated Evidence, Candidate-Level Credit Aggregation, Short-Horizon Environment Value, Bounded Environment Intervention. This is my first interacting with these terms and it makes the paper considerably harder to read. Most of these terms could have been avoided and simpler existing terms could have been used.

  1. Credit is driven by similarity, not by return

The fundamental problem in credit assignment is to attribute observed return to the actions that caused it, often in hindsight. TrajCredit instead assigns credit mainly by matching the current context to stored contexts and the current candidates to stored actions. In every term of the credit (Eqs. 14–16), the factor w_i·κ(a, a_i) determines which candidates receive support. Return enters only through η, which multiplies this similarity in the O term. It does not appear at all in L or P, which together make up the remaining weight. As a result: Return can only rescale support that similarity already assigned. A candidate gets no credit unless it lexically resembles a retrieved action, however valuable it is.

  1. Figure 1 is largely decorative and contributes little to the understanding of the paper

To the authors: The score/rating I have given to your work is largely a function of the writing of the paper. The paper in my opinion needs to be rewritten completely. In case the authors decide to do this, please consider using terms which already existing in the literature, this helps communicate your work better and also makes it easy for the reader to understand.

Questions for authors
  1. Where do the memory trajectories come from (expert, scripted, base-agent rollouts)? What fraction is successful? Are reference trajectories for test states generated by the same policy?
  2. What are the values of θ_L, θ_P, θ_O, and how sensitive are results to them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.