- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Trajectory-to-Step Policy Optimization (T2SPO), which converts historical successful trajectories into auxiliary step-level feedback for training LLM agents. Each visited state is labeled with the number of turns remaining in its successful source trajectory. A frozen text encoder and TabPFN regressor use these examples to predict remaining completion distance for current rollout states. Differences between consecutive predictions are transformed into auxiliary advantages for GRPO or auxiliary rewards for PPO. Experiments evaluate Qwen2.5-1.5B and Qwen2.5-7B on ALFWorld and WebShop, alongside offline distance-prediction analyses and runtime measurements.
Critical strengths
- The paper addresses an important problem in agentic reinforcement learning: sparse outcome rewards provide limited guidance for identifying which intermediate actions contribute to task success.
- The paper presents a practical approach to deriving step-level supervision from historical successful trajectories without requiring manual process annotations.
Critical weaknesses
- The target measures the remaining length of a particular successful continuation and therefore depends on the collection policy, including its redundant actions and detours. These labels decrease by exactly one at every step of a successful source trajectory, even when an action makes no useful progress. Since the estimator explicitly receives the step index, accurate prediction may partly reflect temporal regularities rather than meaningful differences in action quality. A step-index-only baseline and an ablation removing temporal metadata would help clarify this issue. Furthermore, the offline evaluation covers only successful-trajectory states, whereas online shaping also scores states from unsuccessful trajectories. Low prediction error on successful continuations does not establish reliable feedback for mistakes, recovery decisions, or potentially unrecoverable states.
- T2SPO combines historical trajectory reuse, semantic representations, TabPFN regression, context refresh, and several shaping transformations, but the experiments do not sufficiently isolate their contributions to policy improvement. The comparison with kNN concerns offline prediction error; it does not demonstrate that TabPFN produces better policies when used for online training. Similarly, a fixed-context comparison is needed to assess the contribution of continually incorporating new experience.
- On ALFWorld, T2SPO underperforms GiGPO at both model scales and exhibits substantially higher variance. Its improvements over GRPO also require stronger statistical evidence to establish their reliability.
- Since the central contribution is improved step-level credit assignment, the evaluation should more thoroughly position T2SPO against methods that explicitly address this problem. Beyond GiGPO and the limited Turn-PPO experiment in Appendix D, other relevant approaches are discussed without empirical comparison. For example, ARPO [1] combines adaptive step-level sampling with advantage attribution, while ArCHer [2] uses hierarchical value estimation to support learning under delayed rewards.
[1] Dong G, Mao H, Ma K, et al. Agentic reinforced policy optimization. ICLR2026.
[2] ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. ICML2024.
Questions for authors
- Could the authors provide a concrete example showing how an ALFWorld observation and interaction history are converted into the structured state representation used by the estimator? Please clarify which information is retained and how it is organized.
- Which model and training algorithm produced the initial trajectory archives? In particular, which policy generated the WebShop archive shared by the 1.5B and 7B models?
2 comments
Q1: Concrete state representation example (ALFWorld). We illustrate with an actual ALFWorld replay (pick_and_place_simple-AlarmClock-None-Desk-307/trial_T20190907_072303_146844; task: put a alarmclock in desk.). At zero-based step , after go to sidetable 1 → take alarmclock 2 from sidetable 1 → go to desk 1, the raw observation is:
You arrive at desk 1. On the desk 1, you see a book 2, a bowl 1, a cd 3, a pencil 2, a pencil 1, and a remotecontrol 1.
The state representation keeps this verbatim and appends tracked location and held-object status (updated from prior actions/observations):
Location: desk 1. Items in hand (status): alarmclock 2(unprocessed).
History is not paraphrased by an LLM; each prior pair is serialized as past_observation_i and past_action_i. The full text passed to the frozen encoder is:
history: past_observation_0: -= Welcome to TextWorld, ALFRED! =-
You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 1, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a shelf 2, a shelf 1, and a sidetable 1.
Your task is to: put a alarmclock in desk.
past_action_0: go to sidetable 1
past_observation_1: You arrive at sidetable 1. On the sidetable 1, you see a alarmclock 3, a alarmclock 2, a alarmclock 1, a creditcard 1, a desklamp 1, a keychain 1, a pen 3, a pen 2, and a pencil 3.
past_action_1: take alarmclock 2 from sidetable 1
past_observation_2: You pick up the alarmclock 2 from the sidetable 1.
past_action_2: go to desk 1
policy_id: official-walkthrough-cpu-replay
policy_capability: {"collector":"scripted walkthrough replay"}
collection_round: 0
step_index: 3
task_id: alfworld:27402622b2e835db71528522a54f2ce27ab2c81050cfc4ed90ef467ba37a968a
task: put a alarmclock in desk.
state: You arrive at desk 1. On the desk 1, you see a book 2, a bowl 1, a cd 3, a pencil 2, a pencil 1, and a remotecontrol 1. Location: desk 1. Items in hand (status): alarmclock 2(unprocessed).
(In training, collector metadata is populated by the actual collector; task_id is the task-description hash.) Thus, the input retains the task, current observation, location/holding status, past observations/actions (steps only), step index, and metadata—excluding the current action, future observations, terminal outcome, remaining-distance label, and admissible-action list. The encoded text is concatenated with ( here) before context-fitted standardization and SVD. Full-prefix estimator history is separate from the policy's two-turn window.
Q2: Source of initial archives. Collected by GRPO-trained checkpoints at update 100: on WebShop, the 1.5B policy generated the shared archive (~16.7 GPU-hours to train); on ALFWorld, 1.5B and 7B collected theirs separately. Cold-start T2SPO still reaches 68.0% on WebShop (1.5B) vs. 56.8% GRPO and 65.0% GiGPO w/ std (General Response (2)).
We thank the reviewer for the thoughtful review and for recognizing the problem's importance and the practicality of annotation-free step-level supervision. We will also add a worked state-representation example (Q1) and an end-to-end pseudo-code figure.
W1: Policy-dependence of (T−t), step index, and failed trajectories. While each individual label depends on the collection policy, the estimator regresses over many successful trajectories across diverse tasks and improving policies, approximating the expected remaining distance among successful continuations (averaging out redundant actions via more direct paths). Our online ablations (General Response (3)) show removing the step index retains 69.5% success, while step-index-only gives 70.6%. We agree that low error on successful continuations does not establish reliable feedback on mistakes, recovery decisions, or unrecoverable states, and will state this limitation explicitly.
W2: Isolating component contributions. General Response (2)–(3) reports both controls on WebShop (1.5B): (i) online estimators from the same archive—TabPFN 73.7%, kNN-64 71.4%, step-only 70.6%, semantic-only 69.5%, all well above GRPO's 56.8% (full model best, especially in score: 88.5 vs. 82.5–85.8); (ii) cold-start T2SPO reaches 68.0% vs. 73.7% with the archive. In cold start, the context is built solely by refresh, showing continual incorporation of self-experience suffices for most of the gain.
W3: ALFWorld vs. GiGPO, variance, statistical evidence. We acknowledge T2SPO does not outperform GiGPO on ALFWorld and will state this plainly rather than emphasizing average-rank summaries. The two are complementary: GiGPO groups steps via exact in-batch anchor-state matches, whereas T2SPO transfers semantic similarity from historical successes across tasks and time—most helpful where exact recurrence is rare (e.g., WebShop). We will qualify ALFWorld claims accordingly. On WebShop, all ablations (69.5–73.7%, 3 seeds each) stay well above GRPO (56.8%).
W4: Positioning against other credit-assignment methods. We will expand Related Work as follows: