acceptodds
Under review as a conference paper at ICLR 2027

T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning

Abstract

Reinforcement learning enables large language model (LLM) agents to learn multi-step behaviors through interaction with their environments. However, rewards in many interactive tasks reflect only the final outcome, providing limited guidance on which intermediate decisions advance the task. Successful training trajectories contain intermediate states that can provide supervision for subsequent interactions. We introduce , a method that uses past interaction trajectories to provide step-level feedback for policy learning. T2SPO derives remaining-distance targets from successful trajectories and pairs them with representations of the states visited along the way. Conditioned on these examples, a pretrained TabPFN regressor estimates the remaining distance to success at each state of a new rollout. Changes in this distance estimate across consecutive states yield auxiliary credit for agent steps alongside task-level supervision. As training proceeds, newly completed trajectories refresh the estimator's context, incorporating new experience without updating its parameters. Experiments with 1.5B and 7B language models on ALFWorld and WebShop show that T2SPO consistently improves overall task success over GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agents, reinforcement learning, temporal credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Qvi2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Trajectory-to-Step Policy Optimization (T2SPO), which converts historical successful trajectories into auxiliary step-level feedback for training LLM agents. Each visited state is labeled with the number of turns remaining in its successful source trajectory. A frozen text encoder and TabPFN regressor use these examples to predict remaining completion distance for current rollout states. Differences between consecutive predictions are transformed into auxiliary advantages for GRPO or auxiliary rewards for PPO. Experiments evaluate Qwen2.5-1.5B and Qwen2.5-7B on ALFWorld and WebShop, alongside offline distance-prediction analyses and runtime measurements.

Critical strengths
  1. The paper addresses an important problem in agentic reinforcement learning: sparse outcome rewards provide limited guidance for identifying which intermediate actions contribute to task success.
  2. The paper presents a practical approach to deriving step-level supervision from historical successful trajectories without requiring manual process annotations.
Critical weaknesses
  1. Remaining successful-trajectory length is insufficiently validated as a signal of action quality.\textbf{Remaining successful-trajectory length is insufficiently validated as a signal of action quality.} The target �0��0� measures the remaining length of a particular successful continuation and therefore depends on the collection policy, including its redundant actions and detours. These labels decrease by exactly one at every step of a successful source trajectory, even when an action makes no useful progress. Since the estimator explicitly receives the step index, accurate prediction may partly reflect temporal regularities rather than meaningful differences in action quality. A step-index-only baseline and an ablation removing temporal metadata would help clarify this issue. Furthermore, the offline evaluation covers only successful-trajectory states, whereas online shaping also scores states from unsuccessful trajectories. Low prediction error on successful continuations does not establish reliable feedback for mistakes, recovery decisions, or potentially unrecoverable states.
  2. The experiments do not establish which components are responsible for the gains.\textbf{The experiments do not establish which components are responsible for the gains.} T2SPO combines historical trajectory reuse, semantic representations, TabPFN regression, context refresh, and several shaping transformations, but the experiments do not sufficiently isolate their contributions to policy improvement. The comparison with kNN concerns offline prediction error; it does not demonstrate that TabPFN produces better policies when used for online training. Similarly, a fixed-context comparison is needed to assess the contribution of continually incorporating new experience.
  3. The current evidence does not establish a consistent and reliable advantage over existing approaches to step-level credit assignment.\textbf{The current evidence does not establish a consistent and reliable advantage over existing approaches to step-level credit assignment.} On ALFWorld, T2SPO underperforms GiGPO at both model scales and exhibits substantially higher variance. Its improvements over GRPO also require stronger statistical evidence to establish their reliability.
  4. Comparisons with alternative step-level RL methods are insufficient.\textbf{Comparisons with alternative step-level RL methods are insufficient.} Since the central contribution is improved step-level credit assignment, the evaluation should more thoroughly position T2SPO against methods that explicitly address this problem. Beyond GiGPO and the limited Turn-PPO experiment in Appendix D, other relevant approaches are discussed without empirical comparison. For example, ARPO [1] combines adaptive step-level sampling with advantage attribution, while ArCHer [2] uses hierarchical value estimation to support learning under delayed rewards.

[1] Dong G, Mao H, Ma K, et al. Agentic reinforced policy optimization. ICLR2026.

[2] ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL. ICML2024.

Questions for authors
  1. Could the authors provide a concrete example showing how an ALFWorld observation and interaction history are converted into the structured state representation used by the estimator? Please clarify which information is retained and how it is organized.
  2. Which model and training algorithm produced the initial trajectory archives? In particular, which policy generated the WebShop archive shared by the 1.5B and 7B models?
2 comments

We thank the reviewer for the thoughtful review and for recognizing the problem's importance and the practicality of annotation-free step-level supervision. We will also add a worked state-representation example (Q1) and an end-to-end pseudo-code figure.

W1: Policy-dependence of (T−t), step index, and failed trajectories. While each individual label depends on the collection policy, the estimator regresses over many successful trajectories across diverse tasks and improving policies, approximating the expected remaining distance among successful continuations (averaging out redundant actions via more direct paths). Our online ablations (General Response (3)) show removing the step index retains 69.5% success, while step-index-only gives 70.6%. We agree that low error on successful continuations does not establish reliable feedback on mistakes, recovery decisions, or unrecoverable states, and will state this limitation explicitly.

W2: Isolating component contributions. General Response (2)–(3) reports both controls on WebShop (1.5B): (i) online estimators from the same archive—TabPFN 73.7%, kNN-64 71.4%, step-only 70.6%, semantic-only 69.5%, all well above GRPO's 56.8% (full model best, especially in score: 88.5 vs. 82.5–85.8); (ii) cold-start T2SPO reaches 68.0% vs. 73.7% with the archive. In cold start, the context is built solely by refresh, showing continual incorporation of self-experience suffices for most of the gain.

W3: ALFWorld vs. GiGPO, variance, statistical evidence. We acknowledge T2SPO does not outperform GiGPO on ALFWorld and will state this plainly rather than emphasizing average-rank summaries. The two are complementary: GiGPO groups steps via exact in-batch anchor-state matches, whereas T2SPO transfers semantic similarity from historical successes across tasks and time—most helpful where exact recurrence is rare (e.g., WebShop). We will qualify ALFWorld claims accordingly. On WebShop, all ablations (69.5–73.7%, 3 seeds each) stay well above GRPO (56.8%).

W4: Positioning against other credit-assignment methods. We will expand Related Work as follows:

  • ArCHer trains a separate off-policy TD utterance-level critic and token-level actor, relying on bootstrapping under sparse rewards. T2SPO trains no value model (frozen estimator adapted via in-context examples) and uses realized successful trajectories (Monte-Carlo-style distance), avoiding bootstrapping error.
  • ARPO focuses on where to branch (entropy-based rollouts after tool calls) with credit from the current rollout tree. T2SPO injects cross-episode, historical information and is orthogonal to rollout strategy (both could be combined).

Q1: Concrete state representation example (ALFWorld). We illustrate with an actual ALFWorld replay (pick_and_place_simple-AlarmClock-None-Desk-307/trial_T20190907_072303_146844; task: put a alarmclock in desk.). At zero-based step t=3t=3, after go to sidetable 1 → take alarmclock 2 from sidetable 1 → go to desk 1, the raw observation is:

You arrive at desk 1. On the desk 1, you see a book 2, a bowl 1, a cd 3, a pencil 2, a pencil 1, and a remotecontrol 1.

The state representation keeps this verbatim and appends tracked location and held-object status (updated from prior actions/observations):

Location: desk 1. Items in hand (status): alarmclock 2(unprocessed).

History is not paraphrased by an LLM; each prior pair is serialized as past_observation_i and past_action_i. The full text passed to the frozen encoder is:

history: past_observation_0: -= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 1, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a shelf 2, a shelf 1, and a sidetable 1.

Your task is to: put a alarmclock in desk.
past_action_0: go to sidetable 1
past_observation_1: You arrive at sidetable 1. On the sidetable 1, you see a alarmclock 3, a alarmclock 2, a alarmclock 1, a creditcard 1, a desklamp 1, a keychain 1, a pen 3, a pen 2, and a pencil 3.
past_action_1: take alarmclock 2 from sidetable 1
past_observation_2: You pick up the alarmclock 2 from the sidetable 1.
past_action_2: go to desk 1
policy_id: official-walkthrough-cpu-replay
policy_capability: {"collector":"scripted walkthrough replay"}
collection_round: 0
step_index: 3
task_id: alfworld:27402622b2e835db71528522a54f2ce27ab2c81050cfc4ed90ef467ba37a968a
task: put a alarmclock in desk.
state: You arrive at desk 1. On the desk 1, you see a book 2, a bowl 1, a cd 3, a pencil 2, a pencil 1, and a remotecontrol 1. Location: desk 1. Items in hand (status): alarmclock 2(unprocessed).

(In training, collector metadata is populated by the actual collector; task_id is the task-description hash.) Thus, the input retains the task, current observation, location/holding status, past observations/actions (steps 0,…,t−10,\ldots,t-1 only), step index, and metadata—excluding the current action, future observations, terminal outcome, remaining-distance label, and admissible-action list. The encoded text is concatenated with [t,log⁡(1+t),log⁡(1+collection round)][t, \log(1+t), \log(1+\text{collection round})] ([3,1.386294,0][3, 1.386294, 0] here) before context-fitted standardization and SVD. Full-prefix estimator history is separate from the policy's two-turn window.

Q2: Source of initial archives. Collected by GRPO-trained checkpoints at update 100: on WebShop, the 1.5B policy generated the shared archive (~16.7 GPU-hours to train); on ALFWorld, 1.5B and 7B collected theirs separately. Cold-start T2SPO still reaches 68.0% on WebShop (1.5B) vs. 56.8% GRPO and 65.0% GiGPO w/ std (General Response (2)).

Reviewer_hLJMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes T2SPO for step-level credit assignment in agentic reinforcement learning. Successful historical trajectories provide remaining-length labels for intermediate states. A frozen text encoder and TabPFN regressor estimate this distance for new states, and differences between adjacent predictions provide auxiliary step credit. Experiments use Qwen2.5-1.5B and 7B on ALFWorld and WebShop, with additional offline prediction and Turn-PPO experiments.

Critical strengths
  1. The method uses successful interaction histories to obtain additional supervision without human step annotations. The estimator adapts through context updates without training its parameters.

  2. The reported WebShop improvements are substantial. Success increases from 56.8% to 73.7% at 1.5B and from 66.1% to 75.8% at 7B compared with the published GRPO results. The paper reports three-seed means and standard deviations.

  3. The offline prediction comparison uses disjoint task identities and the same features for TabPFN and kNN. TabPFN achieves lower MAE at all tested context budgets, supporting its ability to predict the trajectory-derived targets.

Critical weaknesses
  1. The main GRPO comparison uses results from prior work. A control with the process term disabled in the same implementation is needed to isolate T2SPO’s contribution. An ablation without the update-100 trajectory archive would also clarify its dependence on additional historical experience.

  2. The target T−tT-t decreases even during detours within a successful trajectory, and the predictor receives the step index. Offline prediction is evaluated only on successful trajectories. Step-only and no-step-feature controls, together with checks on failed or backtracking states, would help validate the progress signal.

  3. TabPFN is compared with kNN only through offline prediction error. The full state-estimation pipeline accounts for about 21% of training-step time. Online comparisons with simpler predictors under matched wall-clock budgets would clarify whether TabPFN provides sufficient practical benefit.

Questions for authors
  1. Can you provide a GRPO control with the process term disabled in the same implementation? How do the results change without the initial update-100 trajectory archive?

  2. How do step-only and no-step-feature predictors perform? Can the predicted distance differences distinguish useful progress from detours or actions leading to failure?

  3. How does online policy performance change when TabPFN is replaced with a simpler predictor, such as kNN, under the same wall-clock budget?

1 comment

We thank the reviewer for the positive assessment of the annotation-free supervision, WebShop gains, and controlled offline comparison.

W1/Q1: Same-implementation control and archive dependence. T2SPO builds on verl-agent with identical hyperparameters and evaluation; disabling the process term recovers GRPO (General Response (1)). The only addition is the initial archive (WebShop: 1.5B update-100 checkpoint, ~16.7 GPU-hours, shared with 7B; ALFWorld: collected separately). Without it, cold-start T2SPO (WebShop 1.5B, 3 seeds) reaches 68.0 ± 4.9% vs. 56.8% GRPO and 65.0% GiGPO w/ std (General Response (2)), showing the archive adds 5.7 points but is not the main source of gain.

W2/Q2: Step-only and no-step predictors; detours and failures. From the same archive (General Response (3)), removing the step feature gives 69.5% success, confirming the signal does not rely on step index. Step-only gives 70.6%, reflecting only global step statistics rather than state-dependent credit. Combining both is best. We will note the absence of state-level diagnostics on detour, backtracking, and failure states as a limitation.

W3/Q3: Online comparison with simpler predictors. Online kNN-64 reaches 71.4 ± 2.3% success (score 85.8 ± 2.5), vs. 73.7 ± 3.3% (88.5 ± 0.3) for TabPFN and 56.8% for GRPO (General Response (3)). Thus, the gain comes mainly from trajectory-to-step supervision, with TabPFN adding an improvement (clearer in score, matching its lower offline MAE). kNN trades a small performance drop for lower cost, reducing total training time on WebShop (1.5B) to ~88% of full T2SPO.

Reviewer_J9hJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

T2SPO uses historical successful trajectories to provide step-level feedback for multi-turn LLM agents. A frozen TabPFN regressor predicts the remaining number of interactions until success, and changes in this prediction supplement the policy’s outcome-based training signal. Experiments with two Qwen2.5 model sizes on ALFWorld and WebShop report improvements over published GRPO results, particularly on WebShop.

Critical strengths

Thank you for submitting your paper. Overall, the concept is clear, and I appreciated the detailed explanation of the method and its implementation. Below are the main strengths from my point of view.

  • Clever use of historical trajectories and in-context regression: I particularly liked the idea of adapting a frozen TabPFN regressor through its context to estimate intermediate task progress. Successful trajectories already contain completion-length information, making them a useful source of supervision without requiring additional human annotations. Using semantic state representations also allows the estimator to transfer information between states without requiring exact matches. The prediction-before-insertion procedure is a sensible design choice, since it prevents a rollout’s own outcome from entering the context used to evaluate that rollout.
  • Useful empirical analysis of the progress estimator: The offline experiments provide meaningful initial evidence that TabPFN can predict the proposed targets on tasks excluded from its context. I appreciate that TabPFN and kNN use the same examples, features, and query states. At the main context-256/SVD32 setting, TabPFN achieves an MAE of approximately 0.82 turns, compared with 1.00 for the best tested kNN configuration. The context-size study also shows that additional examples can improve prediction. These results support the feasibility of the estimator, although its usefulness for policy optimization needs further validation.

I also appreciated the detailed scope of the theory. You explicitly acknowledge that normalization, gating, and terminal exceptions prevent directly applying the standard policy-invariance guarantee.

Critical weaknesses

Despite the aforementioned strengths, there are two concerns that limit how confidently I can attribute the reported improvements to the proposed method.

  • The main GRPO comparison lacks a matched control: All baseline results in Table 1 are imported from Feng et al. (2025), including the GRPO baseline against which the main improvements are calculated. These references provide useful context, but a local GRPO run with T2SPO disabled is necessary to isolate the contribution of the auxiliary signal. This is particularly important because Appendix A.4 describes outcome normalization over expanded turn rows, giving longer trajectories more weight than the standard trajectory-level formulation in Equation 1. The method also starts with an archive collected from a separate trained checkpoint. Without a matched control, the reported differences cannot establish how much improvement comes from the proposed credit signal under the implemented training pipeline. I would suggest rerunning GRPO and GiGPO with shared training and evaluation settings, and explicitly accounting for the initial archive.
  • Predicting remaining trajectory length does not yet establish useful action credit: The offline evaluation measures prediction error against (T-t) on successful trajectories. However, this target decreases by exactly one between consecutive states of a successful trajectory, including steps that are redundant or part of a detour. Since the estimator also receives the step index, good prediction accuracy could partly reflect temporal regularities rather than an ability to distinguish helpful from unhelpful decisions. Moreover, the evaluation excludes failed-trajectory states, although these are important inputs during training. This does not invalidate the approach, but it leaves the central credit-assignment claim insufficiently tested. I would like to see diagnostics on detours, repeated actions, and failed trajectories, together with online controls using a step-index-only predictor and kNN. These would help establish whether the semantic distance estimates provide useful information beyond simpler signals.

There are some additional points that I would like to phrase as questions below.

Questions for authors

As mentioned above, there are some further important points that I would like to phrase as questions to the authors.

1) Initial trajectory archive: Appendix A.2 states that the initial archive contains 128 trajectories collected from a separate policy checkpoint at update 100. Which algorithm and model produced these checkpoints, particularly the shared WebShop archive? Could you report performance when starting with an empty archive, as described in Section 4.2? This would clarify how much the method depends on successful experience supplied before its own training begins. Please also account for the resources used to produce that experience.

2) Practical computation cost: I appreciate the detailed accounting. State estimation and context preparation together consume approximately 21.14% of measured training-step time, so their cost is consequential despite the frozen parameters. Could you report performance against elapsed training time or GPU-hours for a matched baseline? The current component breakdown does not establish whether the additional computation is offset by faster learning.

3) Where does the method reliably help?: The ALFWorld improvement over the published 1.5B GRPO reference is 1.9 percentage points, while T2SPO’s standard deviation is 10.0 points. The matched Turn-PPO experiment also shows no mean improvement at 1.5B and only 1.2 points at 7B, with one seed improving and the other declining at each scale. Could you discuss whether the strongest evidence is currently specific to WebShop and the GRPO integration? Per-seed learning curves would help clarify the variability.

4) Offline versus online estimator settings: Appendix B uses a two-turn history for the offline prediction study, whereas the main online estimator uses the full interaction prefix. Could you explain this difference and evaluate prediction quality using the actual online configuration? It would also be useful to know whether the lower offline error with 512 context examples translates into better policy learning, given the additional computational cost.

I would appreciate it if you could address these points!

3 comments

We thank the reviewer for the constructive review and for recognizing the in-context regression design, prediction-before-insertion procedure, and careful scoping of our theory.

W1: Matched control. T2SPO builds on verl-agent (paper_version) with identical shared hyperparameters (including 150 updates) and evaluation; disabling the auxiliary term recovers GRPO exactly. The expanded turn-row normalization (Appendix A.4) is inherited from paper_version and shared by the published GRPO baseline, so the discrepancy is between Eq. (1) and the shared code rather than between methods—we will revise Eq. (1) to match. Regarding the initial archive, cold-start T2SPO (empty archive) still reaches 68.0% on WebShop (1.5B), 11.2 points above GRPO (General Response (1)–(2)).

W2: Remaining length as a credit signal. We agree that raw (T−t) drops by 1 even on detours. Our claim concerns the estimator, fit across many successful trajectories, tasks, and policies: a state is labeled with different remaining lengths across trajectories, so the regression target approximates the expected remaining distance among successful continuations (detour states tend to appear with larger remaining distance than their predecessors). Our online controls (General Response (3)) show that removing the step index retains 69.5% success, while step-index-only (70.6%) acts only as a global step/length bias (E[T−t | t]) and online kNN reaches 71.4%. We will discuss the lack of state-level diagnostics on detour, repeated-action, and failed states as a limitation.

Q1: Initial archive. As detailed in General Response (2), the initial 128 trajectories were collected at update 100 (shared 1.5B archive on WebShop: 16.7 GPU-hours to train, minutes to collect; collected separately by 1.5B and 7B on ALFWorld). Cold-start T2SPO reaches 68.0 ± 4.9% (score 83.7 ± 1.9) vs. GRPO 56.8%, GiGPO w/ std 65.0%, and full T2SPO 73.7%, showing most of the gain comes from the policy's own successes.

Q2: Computation cost. We summarize the computation overhead as follows: (1) a GRPO step costs ~79% of a T2SPO step under our pipeline; (2) on WebShop (1.5B), replacing TabPFN with kNN-64 reduces total training time to ~88% of full T2SPO (while reaching 71.4%); (3) archive state embeddings can be cached across refreshes to avoid re-encoding unreplaced entries. We will add this to the cost section.

Q3: Where the method reliably helps. We agree and will state this explicitly: the strongest evidence is on WebShop + GRPO (+16.9 / +9.7 pp at 1.5B / 7B). On ALFWorld at 1.5B, T2SPO shows higher seed variance than GRPO, and its mean gain (+1.9 points) is within that variance; at 7B the gain is larger (+6.0 points), and Turn-PPO is mixed. We will state this in the limitations and temper claims in the abstract and introduction accordingly.

Q4: Offline vs. online estimator settings. The offline study used a two-turn history purely as a feasibility diagnostic on held-out tasks, whereas the online estimator uses the full interaction prefix (see Qvi2 Q1). We will state this explicitly so offline numbers are read as feasibility evidence rather than exact online prediction quality; General Response (3) evaluates the actual online configuration. We kept 256 context examples instead of 512, expecting diminishing returns given the extra cost and modest TabPFN–kNN gap.

Dear authors,

thank you so much for the reply. The points seem to make sense, but I would need a clearer explanation for each one of them. I would have loved to see an updated PDF with these points. Hence, I keep my current score.

Thank you for the follow-up. We have uploaded a revised PDF that incorporates each point of our response:

  • Matched control and Eq. (1): Eq. (1) now uses the expanded turn-row normalization of the shared verl-agent implementation (Section 3.1, Appendix A.4), and Section 5.1 states that disabling the auxiliary term recovers this GRPO configuration.
  • Initial archive: its source and cost are in Section 5.1 and Appendix A.2; cold-start results are in Section 5.3 (Table 3).
  • Step-index, semantic-only, and kNN controls: Section 5.3 (Table 3), with the remaining-distance interpretation in Section 4.1.
  • Computation cost: Section 5.6.
  • Where the method reliably helps: Section 5.2 and the limitations in Section 6, with the abstract and introduction tempered accordingly.
  • Offline vs. online settings: Section 5.4.

We hope the revised version makes each point clearer.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback and for recognizing the clarity of converting successful trajectories into step-level supervision via in-context regression (J9hJ, hLJM, Qvi2), the prediction-before-insertion design (J9hJ), the controlled offline analysis (J9hJ, hLJM), and the WebShop gains (hLJM). We summarize our response to the three shared concerns below.

(1) On the GRPO comparison. Our implementation builds directly on verl-agent (https://github.com/langfengQ/verl-agent/tree/paper_version), the official codebase of Feng et al. (2025), using identical model, rollout group size, max turns, optimizer settings, update count (150), and evaluation protocol. The expanded turn-row formulation noted by Reviewer J9hJ (Appendix A.4) is the native implementation of the paper_version codebase, so the published GRPO baseline uses the exact same normalization as T2SPO. T2SPO only adds the auxiliary step-level term.

(2) Dependence on the initial archive. The initial archive (128 trajectories) was collected by a GRPO-trained checkpoint at update 100 using the same verl-agent configuration: on WebShop, the 1.5B archive is shared by the 1.5B and 7B runs (training the 1.5B checkpoint took ~8.3 h / 16.7 GPU-hours; collection took a few minutes), whereas on ALFWorld, the 1.5B and 7B models collected their archives separately. To isolate its effect, we ran cold-start T2SPO (empty initial archive, populated only by the policy's own successes) on WebShop (Qwen2.5-1.5B-Instruct, 150 steps, 128 evaluation episodes, 3 seeds):

MethodArchiveSuccess (%)Score
GRPO (Feng et al., 2025)—56.8 ± 3.875.8 ± 3.5
GiGPO w/ std (Feng et al., 2025)—65.0 ± 3.283.1 ± 1.6
T2SPOsame as paper73.7 ± 3.388.5 ± 0.3
T2SPOcold start (empty)68.0 ± 4.983.7 ± 1.9

Without pre-collected experience, T2SPO still beats GRPO by 11.2 points and performs on par with or above GiGPO w/ std (65.0%), showing the gain does not depend on the external archive. The archive adds 5.7 points, which we attribute to stronger initial trajectories helping the policy succeed on more tasks and providing richer states early in training, when the policy itself produces few successes.

(3) Whether the estimator carries information beyond the step index. We ran the requested controls (WebShop, Qwen2.5-1.5B-Instruct, 150 steps, 128 episodes, 3 seeds) from the same initial archive as full T2SPO (73.7%):

MethodArchiveSuccess (%)Score
GRPO (Feng et al., 2025)—56.8 ± 3.875.8 ± 3.5
T2SPO (TabPFN, step + semantic)same as paper73.7 ± 3.388.5 ± 0.3
kNN-64 instead of TabPFNsame as paper71.4 ± 2.385.8 ± 2.5
Step index onlysame as paper70.6 ± 0.984.6 ± 2.8
Step index removed (semantic only)same as paper69.5 ± 1.682.5 ± 2.7
T2SPO (TabPFN, step + semantic)cold start68.0 ± 4.983.7 ± 1.9

Three findings:

  1. Robustness: Every variant retains most of the gain (69.5–71.4% success, +12.7 to +14.6 points over GRPO). Full T2SPO remains best, particularly in score (88.5 ± 0.3 vs. 82.5–85.8).
  2. Semantic signal vs. step index: Removing the step index still yields 69.5% success (score 82.5), showing semantic state information alone provides a useful progress signal. Step-index-only reaches 70.6% (score 84.6), but depends solely on step count: conditioned on success, E[T−t | t] acts as a global step/length bias rather than state-dependent credit. Combining both features performs best, and we will report these findings transparently. Why such a simple step-only bias already yields notable gains is an interesting question that we leave for future work.
  3. Tabular regressor: Replacing TabPFN with kNN-64 gives 71.4% success (score 85.8). The framework works with a simple regressor, while TabPFN adds a further gain (mainly in score, consistent with its lower offline MAE). As a modular component, kNN removes TabPFN inference cost at a small performance trade-off.

All new results and clarifications will be incorporated into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.