acceptodds
Under review as a conference paper at ICLR 2027

Align as Act: Step-level Credit Assignment for LLM Agents via Sequential Subspace Residuals

Abstract

Reinforcement learning (RL) enables LLM agents to improve their policies by learning from environmental feedback. However, designing reliable intermediate rewards is challenging, and reliance on sparse terminal rewards can hinder efficient training. To obtain step-level signals without a separately trained evaluator, we first examine the policy's own contextual representations in trajectories on ALFWorld. We find that diverse action-feedback sequences exhibit more broadly distributed hidden-state variation than repetitive sequences. Connecting the geometry of contextual representations with classical linear innovations, we propose Align as Act (AAA), a representation-based method for constructing step-level learning signals. Processing post-action contextual representations in trajectory order, AAA projects each representation onto a subspace of directions retained from earlier interactions within the same trajectory. This separates the history-aligned component from an orthogonal innovation. The innovation energy is then transformed into an intrinsic reward signal, normalized within successful trajectories, and used to construct step advantages. We instantiate this construction as GRPO+AAA and extend it with a GiGPO-style anchor-group correction to obtain GiGPO+AAA. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct show improvements over the corresponding GRPO and GiGPO baselines. With the 1.5B policy, GRPO+AAA improves the ALFWorld success rate by 11.9 percentage points and increases the WebShop score and success rate by 8.8 points and 14.9 percentage points, respectively. Additional experiments and analyses address AAA's performance with an alternative policy backbone, the use of NNLS for failure-side credit assignment, and the role of the history basis in step-reward construction. These results suggest that trajectory-relative representation geometry provides a practical basis for step-level credit assignment in outcome-supervised LLM agent training.

Primary area:
reinforcement learning
Keywords:
LLM agents, Reinforcement learning, Credit assignment, Intrinsic rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4GmyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Align as Act (AAA), a step-level credit assignment method for LLM agents. For each rollout, AAA reads the policy's final-layer hidden state after each action's feedback. It projects that state with a fixed Gaussian random matrix and sequentially orthogonalizes it (Gram–Schmidt) against a basis of earlier residuals from the same trajectory. The residual component is mapped to a score st=0.5log⁡(1+∥et∥2)s_t = 0.5 \log(1 + \|e_t\|^2), motivated by the innovations decomposition of Gaussian entropy. Scores are boundary-corrected and normalized within each successful trajectory. Failed rollouts receive default GRPO reward.

Critical strengths
  • The method is simple, critic-free, and needs no learned reward model or step-level annotation. It is easy to add to an existing verl-agent pipeline.
  • Gains over GRPO are consistent across two environments, two Qwens, and a Llama backbone.
  • The diagnostics are informative: advantage dispersion in all-success groups, the advantage-composition analysis, which shows that the innovation term carries most of the advantage mass at δ=0.1\delta = 0.1.
  • The authors report unfavourable or mixed results openly (HGPO) and document clearly algorithms and hyperparameters.
Critical weaknesses

W1. Positioning relative to elliptical and information-gain novelty scores.

Consider the regularized Gram matrix and its log-det increment:

Ct−1=I+∑k<tvkvk⊤,12log⁡det⁡Ct−12log⁡det⁡Ct−1=12log⁡(1+vt⊤Ct−1−1vt).C_{t-1} = I + \sum_{k<t} v_k v_k^\top, \qquad \tfrac{1}{2}\log\det C_t - \tfrac{1}{2}\log\det C_{t-1} = \tfrac{1}{2}\log\big(1 + v_t^\top C_{t-1}^{-1} v_t\big).

This is the per-step information gain of GP-UCB with a linear kernel (Srinivas et al., 2010) and the log form of the episodic elliptical bonus vt⊤Ct−1−1vtv_t^\top C_{t-1}^{-1} v_t used by E3B (Henaff et al., NeurIPS 2022). RepExp (Tuyls et al., ICLR 2026) uses elliptical bonuses on the LLM's own hidden states as an RL reward bonus, computed across samples rather than across steps. AAA's score can be read as a hard-thresholded Gram–Schmidt analogue of this increment.

W2. The "new information" interpretation.

The geometric reading (log-volume) holds for any vectors, and the paper gives it. The information-theoretic reading, however, relies on random variables with a covariance inner product ⟨yi,yj⟩=E[yiyj]\langle y_i, y_j\rangle = \mathbb{E}[y_i y_j] and on Gaussianity, whereas hth_t are deterministic, uncentered vectors compared with the Euclidean inner product. The threshold κ\kappa also breaks the exact product decomposition det⁡G=∏t∥et∥\sqrt{\det G} = \prod_t \|e_t\|.

W3. The objective

Eq. (8) is not the advantage of any single fixed reward. The reward is also non-stationary (it depends on πθold\pi_{\theta_{\text{old}}}) and trajectory-dependent (through SiS_i), and the results are sensitive to δ\delta.

W4. The contribution of the innovation scores is not isolated from length and position effects.

Sum-to-one normalization gives rt≈1/Tr_t \approx 1/T. Under "Uniform rewards" (rt=1/Tr_t = 1/T), the step advantage depends only on trajectory length, so that ablation is essentially a test of a short-trajectory preference. It already reaches 67.7% WebShop success, against 71.1% for full AAA and 56.8% for GRPO in Table 1.

Position is a second confound. Residuals may shrink as the basis grows. Neither the training runs nor the enrichment analysis include a position-only baseline st=f(t)s_t = f(t).

W5. hth_t and the AAA hyperparameters are under-ablated.

App. B.3 analyses readout position and layer only geometrically, on the untrained model. There is no downstream RL ablation of position, layer, mm, κ\kappa, the seed of PP. Value hth_t also encodes a truncated history window, so the residual may reflect old history dropping out of the prompt rather than new information in oto_t.

W6. GiGPO+AAA.

The paper notes that it does not reproduce GiGPO's estimator. Its final advantage is Ai,tAAA+λdBi,tA^{AAA}_{i,t} + \lambda_d B_{i,t}.

Still Bi,tB_{i,t} uses the immediate rtir^i_t rather than returns from the anchor state, so GiGPO's comparison semantics is lost. The name overstates the relation to GiGPO. A variant with GiGPO's original advantage plus AAA, discounted AAA returns inside Bi,tB_{i,t}, and a λd\lambda_d sweep would clarify this.

W7. Short horizons only.

ALFWorld is limited at 50 steps and WebShop at 15. Jointly solved episodes average about 8–12 steps (Fig. 5). AAA may have specific long-horizon risks. At least one longer setting would be valuable, e.g. ScienceWorld, AppWorld, SWE environments, or MiniHack/Crafter.

W8. The score ablations are diagnostic only.

NoQ (12log⁡(1+∥vt∥2)\tfrac{1}{2}\log(1 + \|v_t\|^2)) and RAW (∥ht−ht−1∥\|h_t - h_{t-1}\|) appear only in the enrichment analysis (Fig. 4), not as training rewards.

W9. statistical support.

  • Table 7 reports GRPO+AAA on ALFWorld at 74.5 ± 3.6, versus 84.7 ± 0.7 in Table 1; GiGPO+AAA is at 86.5 versus 88.9. The model size and number of seeds for Table 7 are not stated. Please explain it and rerun GRPO under identical conditions.
  • Most ablations use a single training seed: Tables 3 and 6, Figs. 6/15, E.7, and F.2.
  • The two least favourable results (Table 7 and HGPO in F.1) are not referenced from the main text.

W10. Comparisons.

HGPO outperforms both AAA variants (Table 8), but only at 1.5B; HGPO+AAA is anticipated but not run, and there is no analysis of why HGPO is stronger. A discussion of other step-level credit methods would help:

  • GraphGPO (ICML 2026);
  • HiPER (ICML 2026), which reports higher ALFWorld success but changes the action structure;
  • iStar (ICLR 2026);
  • arXiv works: G2PO (already cited), HCAPO, and the concurrent SRPO (arXiv 2608.23493).
Questions for authors
  1. What fraction of steps extend the basis under κ=0.03\kappa = 0.03, and how strongly does sts_t correlate with the log-det increment 12log⁡(1+vt⊤Ct−1−1vt)\tfrac{1}{2}\log(1 + v_t^\top C_{t-1}^{-1} v_t)?
  2. What objective does GRPO+AAA optimize? How does it relate to reward-weighted fine-tuning on successful trajectories?
  3. How do Uniform rewards and a position-only score compare to full AAA?
  4. What model size and how many seeds were used in Table 7, and why does ALFWorld drop by about 10 pp there?
  5. How is std computed in Table 1?
  6. How sensitive are results to readout position, layer, pooling, mm, κ\kappa, and the seed of PP?
  7. Why are immediate rewards rather than returns used in Bi,tB_{i,t}, and why is Bi,tB_{i,t} not clipped for successful steps?
  8. How does AAA behave on longer horizons?
14 comments
AuthorsOpenReview ↗

Response to Reviewer 4Gmy (1/9)

We thank the reviewer for their insightful comments and careful assessment. Our point-by-point responses are provided below.

 

W1/W2, Q1. Relationship to elliptical bonuses, theoretical motivation, and empirical agreement

W2. We address W2 first. As mentioned by the reviewer, the log-det formula arises in Gaussian entropy, where jointly Gaussian observation random variables X1,X2,⋯ ,XnX_1,X_2,\cdots,X_n have covariance matrix Σ\Sigma, i.e., (X1,X2,⋯ ,Xn)∼N(μ,Σ)(X_1,X_2,\cdots,X_n)\sim \mathcal{N}(\mu,\Sigma). Note that the entropy does not depend on the mean μ\mu but only on Σ\Sigma. For positive-definite Σ\Sigma, the entropy is given by 12log⁡((2πe)n∣Σ∣)\frac{1}{2}\log((2\pi e)^n|\Sigma|). By the maximum entropy principle, Gaussian variables achieve the maximum differential entropy under a fixed covariance constraint.

The classical geometric interpretation of this entropy formula is based on the volume of the uncertainty ellipsoid. This ellipsoid volume can be obtained by eigendecomposition of Σ=UΛU⊤\Sigma=U\Lambda U^\top, where each column vector in UU represents an ellipsoid direction. The product of the eigenvalues of Σ\Sigma is proportional to the squared volume of the corresponding uncertainty ellipsoid.

We use a complementary geometric interpretation of this Gaussian entropy formula, based on the LDL⊤LDL^\top decomposition of Σ\Sigma rather than its eigendecomposition. Square-integrable random variables defined on the same probability space can be considered as elements of a Hilbert space after centering, with the inner product defined as covariance. Specifically, let Yi=Xi−E[Xi]Y_i=X_i-\mathbb{E}[X_i]. We view each YiY_i as a vector in this Hilbert space and Y1,Y2,⋯ ,YnY_1,Y_2,\cdots,Y_n as a collection of vectors. Then, what is the Gram matrix of these vectors? It is precisely the covariance matrix Σ\Sigma. Thus, the determinant of Σ\Sigma can be viewed as the squared volume of the parallelotope constructed by the centered random-variable vectors Y1,Y2,⋯ ,YnY_1,Y_2,\cdots,Y_n. Consequently, Gaussian entropy can be interpreted, up to an additive constant, as the log-volume of this parallelotope. Through sequential orthogonalization, this volume is also connected to the product of innovation norms, corresponding to the LDL⊤LDL^\top decomposition of the covariance matrix.

This classical geometric connection motivates our approach: rather than relying on the exact Gaussian entropy interpretation, we apply the intuition of sequential innovations to measure newly introduced directions in other vector representations, such as hidden vectors encoding observation sentences in LLM Transformers. This is an important motivation for our method. Note that this principle provides a geometric guideline rather than an exact information-theoretic identity for LLM representations. In practice, we use the "plus one" transformation in 12log⁡(1+∣et∣2)\frac{1}{2}\log(1+|e_t|^2) and the threshold κ\kappa for numerical stability and selective basis retention. These practical choices do not preserve the exact Gaussian entropy decomposition.

W1. AAA's residual score belongs to the same geometric family as elliptical bonuses; the contribution of AAA is to use history-relative scores for outcome-conditioned step-credit allocation within multi-turn trajectories.

We agree. In a broad sense, AAA can be viewed as belonging to the same family of geometric novelty measures. For linearly independent vectors, the Gram determinant gives the squared volume of the parallelotope constructed by these vectors, as discussed above. The reviewer's identity-regularized matrix CtC_t is closely related to this geometry, although its log-det increment is not identical to the Gram–Schmidt residual score. We therefore do not claim novelty in the underlying geometric identity or orthogonalization procedure. Our main distinction from existing work lies in how history-relative innovations are used for outcome-conditioned step-credit assignment in multi-turn agent training.

First, we motivate the use of innovation-based step credit through an information-theoretic perspective on task success, with outcome conditioning as the key mechanism of our method.

Our key mechanism to align new information with contribution to task success is outcome conditioning: We assign non-zero intrinsic step rewards computed from innovations only for successful trajectories, while failed trajectories receive no additional intrinsic reward and retain their original GRPO advantages (Section 4.3). Let us consider the agentic task problem from an information-theoretic perspective. In many agentic tasks, solving a task T\mathcal{T} involves acquiring task-relevant information through sequential observations o1,o2,⋯ ,oTo_1,o_2,\cdots,o_T, alongside taking actions based on the accumulated information. From this perspective, actions and observations form a sequential interaction process in which new observations may reduce uncertainty relevant to task completion.

AuthorsOpenReview ↗

Response to Reviewer 4Gmy (2/9)

Now suppose that we are informed that the task is successful at the end of an episode and want to quantify the contribution of each step tt to task success. Information theory provides a natural motivation for assessing individual steps: If T\mathcal{T} denotes the task-relevant uncertainty, the new information provided by oto_t regarding T\mathcal{T} can be expressed as the conditional mutual information I(T;ot∣o<t)I(\mathcal{T};o_t\mid o_{<t}). However, the difficulty in this formulation is that it is generally intractable to define and model a variable T\mathcal{T} capturing the complicated structure of agentic tasks and compute this mutual information, except in relatively simple settings such as search or question answering. Our key motivation for circumventing this difficulty is that successful trajectories provide an outcome signal indicating that the sequence of interactions was sufficient to complete the task. In such trajectories, newly acquired observations may contain information that contributed to successful decisions. This motivates using history-relative innovation as a practical proxy for newly introduced information and allocating credit to these innovations conditioned on terminal success. Importantly, success does not imply that all information contained in the observations is task-relevant, nor that the conditional entropy H(ot∣o<t)H(o_t\mid o_{<t}) equals the task-relevant information gain I(T;ot∣o<t)I(\mathcal{T};o_t\mid o_{<t}). Observations may contain incidental or redundant information even in successful trajectories. Moreover, AAA computes geometric residuals of contextual LLM representations rather than these information-theoretic quantities directly. We therefore use this perspective as motivation for outcome-conditioned innovation scoring, not as a claim that our score exactly estimates task-relevant mutual information or causal contribution. On the other hand, in failed trajectories, a large innovation may simply reflect novel exploration without indicating progress toward task completion. Hence, we do not assign additional positive intrinsic rewards based on innovation scores in these trajectories. Instead, we retain their original GRPO advantages. This outcome conditioning reduces the risk of rewarding novelty independently of task success, while acknowledging that individual high-innovation steps within successful trajectories are not necessarily causally essential.

Second, we go beyond innovation-based reward design by integrating outcome-conditioned step credit into group-relative policy optimization.

As mentioned above, we construct intrinsic step rewards from innovations within successful trajectories. Then, how should failed trajectories be handled, and how should these step rewards be incorporated into GRPO training? Consider GRPO with multiple trajectory rollouts, where some trajectories succeed and others fail. Rather than replacing the original group-relative learning signal, AAA normalizes the innovation-based step rewards within each non-degenerate successful trajectory so that their sum equals the unit terminal-success credit. These rewards are then converted into step-level group-relative advantages, allowing actions within successful trajectories to receive differentiated learning signals. For failed trajectories, AAA retains the original GRPO advantages, preserving the success–failure contrast used in GRPO training. This design combines outcome-conditioned, history-relative step credit with group-relative policy optimization, without introducing an additional positive exploration bonus for failed trajectories. The within-trajectory normalization also prevents the total allocated intrinsic credit of a successful trajectory from increasing merely by adding more steps. Thus, the contribution of AAA is not the geometric innovation measure alone, but its integration into a practical step-credit assignment mechanism for sparse-reward multi-turn agent training.

 

Relationship to E3B and RepExp:

E3B [1] constructs an episodic exploration bonus from inverse-dynamics state embeddings. RepExp [2] uses language-model response representations for inference-time selection and RL post-training. AAA shares their representation-novelty motivation but differs in the comparison unit and training role:

MethodRepresentationComparison referenceUse
E3B [1]Inverse-dynamics state embeddingsEarlier states in the episodeExploration bonus added to environment reward
RepExp [2]Mean-pooled response embeddings; random projectionSame-prompt responses: selected responses at inference; response group in RLResponse selection; additive response-level RL bonus
GRPO+AAAPost-action contextual boundary representations; random projectionRetained directions from earlier interactions in the trajectoryNormalized step credit within successful trajectories; group-relative advantages
AuthorsOpenReview ↗

Response to Reviewer 4Gmy (3/9)

RepExp also investigates token-level inference-time generation. In its RL implementation, the bonus is disabled for prompt groups in which all responses are incorrect (Appendix C of [2]). AAA conditions credit at the individual trajectory level: for nondegenerate successful trajectories its allocated rewards sum to one, and failed trajectories retain their original GRPO advantages.

 

Q1. Basis expansion and agreement with regularized log-det.

At κ=0.03\kappa=0.03, AAA admits a new basis direction on 72.48% of eligible interior steps, and its within-trajectory raw-score rankings closely match regularized log-det increments on the same 94 successful ALFWorld trajectories (mean Spearman 0.9796).

We evaluated one frozen Qwen2.5-1.5B-Instruct GRPO+AAA step-150 checkpoint (one training run) once on each of the 140 tasks of the ALFWorld Seen split (configured evaluation seed 123). Of 119 successful episodes, 94 satisfy the existing correlation criterion of at least three interior scores; the other 25 have one or two. Successes are the relevant cohort because default AAA allocates intrinsic step rewards within successful trajectories.

MetricAggregationValue
Basis expansion (%)Step-weighted72.48
Basis expansion (%)Task-equal85.62
Raw-score SpearmanTask-equal0.9796
Reward-share dTVd_{\mathrm{TV}}Task-equal0.0072

Across these 94 trajectories, the basis expands on 632 of 872 eligible interior steps.

The step-weighted expansion rate pools admissions and eligible steps; the task-equal rate averages each task's expansion fraction. Giving longer trajectories more weight yields the lower pooled rate in this cohort. The first empty-basis and terminal steps are excluded from the expansion denominator, but the first valid vector remains in subsequent history. Every interior step is scored before the basis-admission test, including non-admitted steps.

Both scoring rules use identical projected representations from the actual prompts containing the two most recent interactions. AAA uses its retained basis, whereas the regularized log-det reference includes all preceding valid vectors with identity regularization:

Ct−1=I+∑k<tvkvk⊤,stLD=12log⁡(1+vt⊤Ct−1−1vt).C_{t-1}=I+\sum_{k\lt t}v_kv_k^\top,\qquad s_t^{\mathrm{LD}}=\tfrac12\log(1+v_t^\top C_{t-1}^{-1}v_t).

Spearman is computed over interior raw scores. TV compares whole-trajectory normalized shares after the same first/terminal mean-fill; both methods consequently assign identical boundary shares. Both metrics are averaged equally across the 94 tasks.

These fixed-checkpoint results show close empirical score-ranking agreement and quantify the difference in normalized allocations. Since selective retention and the treatment of within-span contributions change together, this comparison isolates neither factor. We did not train a policy with log-det rewards. We have updated Appendix E.8 and Table 10 with this analysis.  

W3 / Q2. Policy-dependent credit, optimization objective, and relation to reward-weighted fine-tuning

We agree with the reviewer that because rtir_t^i is policy- and trajectory-dependent, Eq. (8) is not an unbiased advantage estimator for a stationary Markov step-reward function. Rather, AAA redistributes unit terminal-success credit across steps to form policy-adaptive weights within a well-defined iteration-conditioned PPO surrogate, and our controlled ablations show where step-specific residual assignment contributes beyond uniform success reinforcement or the offset δ\delta.

1. Return preservation versus the task objective (W3). For every non-degenerate successful trajectory (Ti>2T_i > 2 and Si>ϵnormS_i > \epsilon_{\text{norm}}), AAA normalizes the boundary-corrected scores so that total trajectory credit equals one, while failed trajectories (and degenerate cases) receive zero intrinsic reward. Consequently, for non-degenerate trajectories, the allocated step rewards preserve the binary environment return: ∑t=1Tirti(τi;θold)=Renv(τi)∈0,1,Jtask(θ)=Eτ∼πθ[Renv(τ)].\sum_{t=1}^{T_i} r_t^i(\tau^i; \theta_{\text{old}}) = R_{\text{env}}(\tau^i) \in \\{0,1\\}, \qquad J_{\text{task}}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\bigl[R_{\text{env}}(\tau)\bigr]. Thus, although individual step shares rtir_t^i adapt with the policy's representations and trajectory normalization SiS_i, AAA does not alter which non-degenerate trajectories are rewarded at the episode level. At the same time, we explicitly distinguish return preservation from policy-gradient equivalence: because Eq. (8) applies rollout-group standardization, nonnegativity clipping, and a positive offset δ\delta to construct step-level update weights, we do not claim that the resulting updates form an unbiased policy gradient of Jtask(θ)J_{\text{task}}(\theta). Instead, AAA provides differentiated step-level learning signals among actions that share the same terminal outcome.

Reviewer_Vrc2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AAA (Align as Act) for step-level credit assignment in multi-step agents. The core idea is to treat hidden-state changes within a trajectory as sequentially arriving new information: a subspace basis is expanded step by step via random projection and Gram-Schmidt orthogonalization, the residual norm of each step measures how many new directions that step brings, and this residual becomes a training-free intrinsic reward added directly on top of GRPO or GiGPO. The whole pipeline trains no reward model and is purely parameter-free linear algebra. On ALFWorld and WebShop, both 1.5B and 7B models improve consistently; for example, the 1.5B success rate rises from 72.8 to 84.7 on ALFWorld and from 56.8 to 71.7 on WebShop.

Critical strengths
  • The method is simple and plug-and-play. The entire scoring pipeline consists of two matrix operations, random projection and orthogonalization; it trains no reward model, does not modify the GRPO or GiGPO core, and can be stacked on any group-relative algorithm.
  • The analysis is relatively thorough. The authors proactively label each step as progress, redundant, or invalid with text rules to test whether high-scoring steps are truly progress; they admit that large residuals in failed rollouts may correspond to a big step in a wrong direction; and they honestly report negative results in which HGPO surpasses AAA. Removing the success gate (rewards assigned only in successful rollouts) drops performance by 14.9 points.
Critical weaknesses
  1. The logic of the motivation chain is unclear: credit assignment asks "which step contributed," while the method encourages "which step brings new information," and these are not the same thing. The authors argue that step-level credit assignment needs step-level signals, that diverse interactions are more broadly distributed in representation space, and therefore the new direction of each step relative to history is used as the reward. But bringing new information does not equal moving toward the goal: detours, opening wrong containers, and inspecting irrelevant content all produce new directions. The paper's own labels confirm this: progress steps account for only about 0.5%, and even among the top 5% highest-residual steps, about 94% are not progress steps. This amounts to admitting that "new information approximates progress" holds at best within successful trajectories, which themselves still contain plenty of detours.
  2. No comparison against the most direct alternative: directly rewarding progress or penalizing redundancy. The authors already built a text-rule tool that labels each step as progress or redundant (E.3, used for analysis); the most natural baseline is to directly reward progress steps and penalize redundant steps in RL. Related schemes include the learned progress estimator SPA-RL, the process reward AgentPRM, and the classical intrinsic motivation methods RND and ICM. Table 1 compares only PPO, RLOO, GRPO, and GiGPO, leaving unanswered what AAA adds over directly rewarding progress or penalizing redundancy.
  3. The improvement over the strongest baseline is limited and within error bars. Against the strongest baseline GiGPO, on 1.5B ALFWorld GiGPO+AAA scores 88.9±2.0 versus 86.7±1.7, only 2.2 points apart with clearly overlapping standard deviations; the gaps on WebShop and at 7B are likewise 2-4 points, the same magnitude as the standard deviations.
Questions for authors
  1. The paper aims at step-level credit assignment, but to my understanding the method actually rewards "steps that bring new information," which is not equivalent. How do the authors justify that using "new directions" as the reward achieves the goal of credit assignment? Can the authors split high-scoring steps by the E.3 labels into "necessary but non-milestone actions" and "genuine detours," to show what the signal actually encourages?
  2. The paper does not compare against the more direct alternative: the authors already have text rules that label each step as progress or redundant (E.3), and directly rewarding progress steps while penalizing redundant ones is the most natural baseline. Can the authors demonstrate that AAA's gains come from the subspace residual itself, rather than from any signal that "favors non-redundant steps"?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Vrc2 (1/5)

We sincerely appreciate the reviewer's thoughtful evaluation and constructive feedback. We provide our detailed responses below.

 

W1 / Q1. Outcome-conditioned credit and task relevance of innovation scores

Our key mechanism to relate new information to task success is outcome conditioning: We assign non-zero step rewards computed from innovations only for successful trajectories, while failed trajectories receive no additional intrinsic reward and retain their original GRPO advantages (Section 4.3). Let us consider the agentic task problem from an information-theoretic perspective.

From an information-theoretic perspective, one can view solving the task T\mathcal{T} as requiring task-relevant information. This necessary information is gathered by making sequential observations o1,o2,⋯o_1,o_2,\cdots. When the accumulated observations and actions provide sufficient information and state changes for task completion, the task can be solved. In this perspective, the actions are intermediate steps to acquire information-bearing observations, as well as to affect the environment. Now suppose that we are informed that the task is successful at the end of an episode and want to quantify the contribution of each step tt to task success. Information theory provides a natural way to assess the information acquired at individual steps: The new information, so-called information gain, provided by oto_t regarding the task T\mathcal{T}, i.e., I(T;ot∣o<t)I(\mathcal{T};o_t\mid o_{<t}), can serve as a measure of task-relevant information at step tt. However, the difficulty in this formulation is that it is intractable to set a variable T\mathcal{T} capturing the complicated agentic task structure for computation of the mutual information, except for simple tasks such as search or question answering.

Our key insight to circumvent this difficulty and extend to general sparse agentic tasks is as follows: When the task is successful, the accumulated observations and actions have been sufficient for task completion, although they may also contain information unrelated to the task. This motivates using the conditional observation entropy H(ot∣o<t)H(o_t\mid o_{<t}) as a heuristic for newly acquired information, rather than assuming that it equals the task-relevant information gain I(T;ot∣o<t)I(\mathcal{T};o_t\mid o_{<t}). Still, this motivates considering the log innovation norm as a practical proxy for history-relative novelty in successful trajectories, although it is not perfect. The innovation norm is computed from LLM representations and does not directly estimate these information-theoretic quantities. On the other hand, in failed trajectories, success-based justification for assigning positive credit to innovations is unavailable. So, a big innovation step may not be a big step towards success at all. Hence, we do not assign additional positive intrinsic rewards to failed trajectories, but retain their original GRPO advantages. We also explore an extension of this intrinsic reward design, exploiting successful trajectories and non-negative least squares projection, in Appendix F.3.

Indeed, removing success gating lowers WebShop success from 71.1% to 56.2% (Table 16). Success gating prevents unsuccessful exploration from receiving intrinsic credit, but does not guarantee that every rewarded step within a successful trajectory is task-relevant.

Interpreting the 94% non-progress share. The progress labels in Appendix E.3 identify only explicit goal-milestone completions, comprising approximately 0.55% of all analyzed steps. Even an oracle that ranks every milestone within the top 5% would therefore attain at most approximately 11% precision (0.55/50.55/5), leaving at least 89% labeled non-progress. AAA achieves 10.62±1.78×10.62\pm1.78\times enrichment (approximately 5.9% precision, more than half this oracle ceiling), whereas NoQ retrieves no milestones in the top 5% across the three evaluation seeds. The coarse labeling also places necessary navigation, inspection, and other prerequisite actions in the non-progress category. However, the direct decomposition below shows that these actions account for only a small fraction of the highest-scoring steps.

AuthorsOpenReview ↗

Response to Reviewer Vrc2 (2/5)

Direct decomposition of the Figure 12b top-5% bin. Using the same trajectories, scores, and evaluation seeds (747 steps), we find 44 E.3 milestones (5.9%). Among the remaining 703 steps, 6 (0.9%) are additional milestones, 11.7% are task-relevant navigation or exploration (N+U), 43.7% are task-irrelevant interactions (D; mostly picking up non-target objects), and 43.8% are first-time invalid actions (R). Thus, we do not claim that the 94% non-milestone steps are predominantly task-relevant. Notably, repeated actions account for approximately 58% of all steps but none of the top-5% bin, while 97% of the bottom-half steps are redundant or invalid. In contrast, 97% of NoQ's top-5% steps are redundant or invalid. However, high-scoring steps are concentrated early (67% at t≤3t\leq3), and a position-only ranking achieves similar milestone enrichment (10.24±0.83×10.24\pm0.83\times); within-position enrichment is only 1.19×1.19\times. Figure 12b therefore demonstrates that innovation scores distinguish first-time interactions from repetition, but do not reliably identify goal progress independently of step position. Moreover, 97% of trajectories in this pool are unsuccessful (13 of 402 succeed) and receive no intrinsic innovation credit under base AAA. This raw-score analysis therefore differs from the distribution of steps actually rewarded during training.

Finer-grained analysis across training. We separately annotated five categories—milestone progress (M), necessary intermediate actions (N), task-relevant exploration (U), irrelevant detours (D), and redundant/invalid actions (R)—independently of AAA scores. The prefix-based annotator uses training-split object-location priors without evaluation ground-truth states. We analyzed 17,817 successful rollouts (195K steps) across 150 training steps from two related 1.5B GRPO+AAA runs (with the same successful-trajectory scoring rule as the base method, and small failure-credit weights λF∈0.005,0.01\lambda_F\in\\{0.005,0.01\\}).

Training stepsMedian successful trajectory lengthTask-relevant share in top 10% (overall base rate)Position-/length-controlled enrichment [95% CI]
1–102456% (27%)1.72×1.72\times [1.57, 1.87]
51–751173% (37%)1.26×1.26\times [1.22, 1.31]
126–150881% (58%)1.03×1.03\times [1.00, 1.05]

Task relevance = M + N + U. The top 10% are ranked by raw AAA innovation score. Confidence intervals are bootstrapped from pooled rollout data across two related runs, rather than independent training seeds.

In training steps 1–10, task-relevant actions account for 56% of the highest-scoring decile versus 27% overall and are 3.23×3.23\times as prevalent as in the bottom half of the score distribution. This enrichment remains 1.72×1.72\times after controlling for position and trajectory length; controlled milestone enrichment is 2.18×2.18\times. During training, the median length of successful trajectories falls from 24 to 8 steps, while task-relevant prevalence rises from 27% to 58%. The decline in controlled enrichment to 1.03×1.03\times is consistent with fewer repetitive or irrelevant steps remaining to distinguish as the trajectory distribution changes. Indeed, successful rollouts with at least 25 scored steps fall from 46% to 8% of successful rollouts (training steps 1–25 versus 101–150), while the within-rollout preference in the remaining long trajectories persists, as shown below. At final checkpoints, normalized credit shares by category are also close to category step shares (ratios 0.92–1.09 across four model/algorithm configurations), indicating limited category-level selectivity in the overall late-training distribution.

AuthorsOpenReview ↗

Response to Reviewer Vrc2 (3/5)

Effect on the applied advantage. We further examined the actual advantage assigned during training, comparing AAA's applied advantage with the original GRPO advantage on the same 17,817 successful rollouts. Under GRPO, task-relevant steps receive only 0.63×0.63\times [0.61, 0.65] the advantage of detour or redundant steps, decreasing from 1.08×1.08\times in training steps 1–25 to 0.48×0.48\times in steps 101–150. This reflects the group-relative outcome advantage: it vanishes when all rollouts in a task group succeed, leaving positive advantages concentrated on harder tasks with longer, noisier successful trajectories. Under AAA, the corresponding task-relevant-to-irrelevant advantage ratio increases to 1.16×1.16\times [1.12, 1.19] (1.401.40, 1.421.42, 1.471.47, and 1.09×1.09\times across training steps 1–25, 26–50, 51–100, and 101–150). Most of this shift comes from credit allocation across successful trajectories: within a task group, the shortest successful rollout receives 4.0×4.0\times [3.9, 4.2] the per-step advantage of the longest one, whereas GRPO assigns them the same outcome advantage. This effect follows from per-trajectory normalization rather than the innovation score itself. The innovation score additionally provides within-trajectory differentiation: among successful rollouts with at least 25 scored steps, task-relevant actions receive 1.21–1.24×1.21\text{--}1.24\times the advantage of detour or redundant actions, with this preference remaining relatively stable throughout training.

The analysis also identifies a limitation: early in training, irrelevant detours—often first visits to unrelated locations—are 1.4–2.3×1.4\text{--}2.3\times as prevalent among the highest-scoring step of each trajectory as under random selection. Thus, innovation scores do not identify task progress perfectly. Irrelevant steps within successful trajectories may still receive intrinsic credit, and the five-category annotations are rule-based rather than manually validated. In summary, the results support an interpretation: AAA provides an outcome-conditioned, novelty-based credit signal that suppresses repetition and is enriched for task-relevant actions in early, noisy successful trajectories, although it is imperfect and may not directly estimate goal progress. Its practical value relative to alternative step-level signals is evaluated in W2. We have revised Appendix E.3 accordingly.

Reviewer_8XWLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Align as Act (AAA), which uses sequential subspace residuals of policy hidden representations to allocate step-level credit within successful trajectories. It integrates this signal into GRPO and GiGPO without a separate evaluator. Experiments report improvements on ALFWorld and WebShop, with additional evaluations using a Llama backbone and search-augmented QA.

Critical strengths
  • AAA provides a practical way to obtain step-level learning signals from existing policy representations without additional supervision or a learned evaluator.

  • The reported improvements over GRPO are substantial and extend across model sizes and backbones. The representation analyses and component ablations provide useful supporting evidence.

Critical weaknesses
  1. The contribution of the residual scores is not sufficiently isolated from the accompanying normalization and advantage modifications. Table 3 uses only one training seed, and the uniform-reward control has substantial evaluation variance. Consequently, the results do not convincingly establish that residual-based allocation is responsible for the gains.

  2. The experimental results need reconciliation. Table 7 reports 74.5% ALFWorld success for “Original” GRPO+AAA, compared with 84.7% for the 1.5B model in Table 1. The separate hardware setup does not adequately explain this difference, and the precise configurations are unclear. This affects confidence in the reproducibility of the improvements.

Questions for authors
  1. What explains the difference between the GRPO+AAA results in Tables 1 and 7? Please specify the backbone, training seeds, checkpoint selection, and evaluation settings.

  2. Does AAA consistently outperform uniform rewards and within-trajectory shuffled scores across multiple training seeds, with the remaining advantage construction held fixed?

  3. Which Table 1 baselines were rerun under the same implementation and evaluation protocol, and which were taken from prior work?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8XWL (1/2)

We appreciate the reviewer's thorough review and helpful suggestions. We address the reviewer's comments in detail below.

W1 / Q2. Isolating the contribution of residual-based step-credit allocation

With the rest of the advantage construction held fixed, outcome-conditioned redistribution accounts for most of the gain on seen ALFWorld, whereas residual-based assignment yields higher mean success than Shuffled AAA, and higher than Uniform, on unseen ALFWorld and WebShop.

We compare four variants that differ only in how credit is allocated across the steps of a successful trajectory, while retaining the same trajectory-wise normalization, group standardization, nonnegativity clipping, success offset δ=0.1\delta=0.1, and original GRPO advantages for failed trajectories: GRPO+AAA, Uniform (1/Ti1/T_i per step), Shuffled AAA (interior residual scores permuted, preserving their distribution and total credit), and Position-only (a predetermined step-position rule).

Each variant is trained with three training seeds (0, 1, 2) for 150 training steps, and we evaluate the final checkpoint of each run. These checkpoints were trained independently of those reported in Table 1. ALFWorld results cover the full Seen (140) and Unseen (134) splits, each evaluated once with environment seed 123; WebShop results use 128 sampled validation episodes per evaluation seed (123, 456, 789), averaged within each training run.

Success rate (%), Qwen2.5-1.5B-Instruct, mean ± sample SD across three training runs

MethodALFWorld Seen (140)ALFWorld Unseen (134)WebShop (128)
GRPO+AAA86.4 ± 5.086.6 ± 2.073.2 ± 0.9
Uniform85.7 ± 2.684.3 ± 0.770.7 ± 5.2
Shuffled AAA84.8 ± 2.580.1 ± 5.464.9 ± 2.8
Position-only45.0 ± 3.832.3 ± 6.449.9 ± 3.0

On seen ALFWorld, GRPO+AAA, Uniform, and Shuffled AAA are within 1.6 points. On unseen ALFWorld and WebShop, GRPO+AAA exceeds Uniform by 2.3 and 2.5 points and Shuffled AAA by 6.5 and 8.3 points. Since Shuffled AAA keeps the residual-score distribution and changes only which step receives each score, these gaps point to step-specific assignment.

Across training seeds, the AAA–Shuffled AAA difference on unseen ALFWorld is positive in all three runs (+13.4, +5.2, and +0.7 points). Against Uniform, mean success is higher on unseen ALFWorld and WebShop, although the WebShop gap is within Uniform's seed variability (±5.2). Position-only performs substantially worse in all settings, although it does not isolate AAA's own position-dependent pattern. Per-run results are provided in Section 7.1 (Table 4(a)) and Appendix E.10.

Taken together, these controls indicate that residual-based allocation contributes beyond the shared normalization and advantage construction, most clearly relative to Shuffled AAA on unseen ALFWorld and WebShop.

 

AuthorsOpenReview ↗

Response to Reviewer 8XWL (2/2)

W2 / Q1. Discrepancy between Table 1 and Table 7

We identified that the discrepancy between Table 1 and Table 7 (Table 9 in the revised manuscript) was associated with configuration mismatches in the feature-mean centering ablation runs: λfail=0.5\lambda_{\mathrm{fail}}=0.5 for GRPO+AAA and λd=0.3\lambda_d=0.3 for GiGPO+AAA, instead of their respective main-experiment settings (λfail=0\lambda_{\mathrm{fail}}=0 and λd=1\lambda_d=1). We therefore reran the zero-mean variants under the main-experiment configurations and evaluation protocol.

Using Qwen2.5-1.5B-Instruct, each training run is evaluated at its final checkpoint, and each checkpoint is evaluated with environment seeds 123, 456, and 789; we average the three evaluations per checkpoint and report the mean and sample SD across training runs.

ALFWorld, Qwen2.5-1.5B-Instruct — Success rate (%)

MethodOriginalZero-mean
GRPO+AAA84.7 ± 0.783.5 ± 2.4
GiGPO+AAA88.9 ± 2.085.4 ± 2.6

Mean ± sample SD across training runs (Original: 3; Zero-mean: 2).

Under matched configurations, centering yields lower observed means by 1.2 and 3.5 points, respectively; given the small run counts, we do not draw strong conclusions about its effect. The corrected comparison replaces the original Table 7 (now Table 9).

 

Q3. Implementation and provenance of Table 1 results

All non-AAA entries in Table 1 are taken from Table 1 of Feng et al. (2025) [1], whereas GRPO+AAA and GiGPO+AAA are results from our own training runs. We build AAA on the released verl-agent codebase [2], retaining its core training framework and environment interfaces while adding post-action hidden-state extraction, sequential residual scoring, and step-level credit construction. Following the verl-agent defaults, GiGPO+AAA uses mean–std normalization on ALFWorld and mean normalization on WebShop (Appendix D).

As an implementation check, we also trained GRPO and GiGPO (w/ std) on ALFWorld with Qwen2.5-1.5B-Instruct, using one training seed per method. These runs achieved success rates of 74.74% and 89.06%, respectively, compared with the published 72.8% and 86.7% [1]. The results are in a similar performance range, and these verification runs were not used to populate Table 1.

We have clarified the baseline provenance in Section 6.1 and provided additional implementation and evaluation details in Appendix D.

References

[1] Feng et al. (2025). Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978

[2] verl-agent: Official GiGPO codebase. GitHub

Thank you for the additional experiments and clarifications. The three-seed allocation controls substantially address my concern about isolating the residual scores. The improvements over shuffled assignments on unseen ALFWorld and WebShop provide useful evidence that the assignment of scores to individual steps matters. The gains over uniform allocation remain modest, so I appreciate the more qualified claims.

The identified configuration mismatches also provide a concrete explanation for the discrepancy between Tables 1 and 7. Please ensure that the revised manuscript includes the corrected results and explicitly documents the differing settings.

The baseline provenance is now clear. The single-seed reproduction checks are helpful, although they leave uncertainty around the smaller improvements over GiGPO. Overall, the response substantially addresses my original concerns, with the strength of the gains over uniform allocation and GiGPO remaining the main limitations.

Comments on the paper

3 comments

Summary of revisions

We thank all reviewers for their constructive feedback. We have revised the manuscript to clarify AAA’s core contribution and strengthen its empirical evaluation through controlled ablations, additional baseline comparisons, and analyses of representation-based credit assignment. We also clarify the experimental protocols, baseline provenance, and connections to existing methods. Substantive revisions are highlighted in blue, while editorial changes to terminology, table headers, and numerical precision are left unmarked.

Importantly, we will remain actively engaged in the discussion and continue updating our responses and manuscript throughout the rebuttal period. We welcome any further questions or concerns and appreciate the reviewers’ continued interest and feedback.

The revisions addressing each reviewer’s concerns are summarized below.

Reviewer 8XWL

  • W1/Q2 (isolating residual-based allocation): Section 7.1 and Table 4(a); per-run results, including WebShop, in Table 14 (Appendix E.10).
  • W2/Q1 (Table 1 vs. Table 7): corrected comparison and run settings in Appendix E.5 and Table 9.
  • Q3 (baseline provenance): Section 6.1 and Appendix D; per-run results for the AAA entries of Table 1 in Table 17.

Reviewer Vrc2

  • W1/Q1 (new information vs. credit): revised framing of the method in Section 4; base-rate interpretation, composition of the top-5% bin, and a position-only control in Appendix E.3.
  • W2/Q2 (direct progress rewards and intrinsic baselines): Section 6.2 and Table 3; reward shaping in Table 15 (Appendix E.10).
  • W3 (gains over GiGPO): Section 6.1.

Reviewer 4Gmy

  • W1/Q1 (relation to elliptical scores): Appendix C.6; basis admission and agreement with log-det scoring in Appendix E.8 and Table 10.
  • W2 (information-theoretic reading): Sections 2 and 4.2, and Appendix C.4.
  • W3/Q2 (objective): Appendix C.5; offset sensitivity in Appendix E.4.
  • W4/Q3 (length and position): Section 7.1 and Table 4(a); position-only control in Appendix E.3.
  • Q5 (standard deviations): Appendix D and Table 17.
  • W5/Q6 (readout, hyperparameters, and rolling history): Appendix C.2, Appendix E.9 (Tables 11–12), and Appendix E.11.
  • W6/Q7 (GiGPO+AAA design): Appendix C.5.
  • W8 (NoQ, RAWΔ, and RND training ablations): Table 3 and Table 4(b).
  • W9/Q4 (Table 7 discrepancy): Appendix E.5 and Table 9, now referenced from Section 7.1; the HGPO comparison is now referenced from Section 6.1.
  • W10 (HGPO and recent methods): Appendix A and Appendix F.1.

Common Response: Additional Results and the Scope of AAA's Contribution

We thank all reviewers again for their constructive feedback. SSince our initial responses, we have completed additional experiments: controls for within-trajectory credit allocation, further GiGPO comparisons and step-level diagnostics, longer-horizon ScienceWorld tasks, and AAA combined with HGPO. These results let us state AAA's contribution more precisely.

AAA constructs step-specific credit within successful trajectories, without requiring a critic, a separately trained reward model, or state matching across rollouts.

In critic-free, outcome-supervised agent RL, every step of a trajectory shares one terminal outcome, and step-level credit assignment must turn that outcome into different update weights for those steps. Anchor-based methods do this through comparisons across matched steps: GiGPO compares returns among steps that share the current observation, and HGPO among steps that also share recent history. These anchor-group comparisons depend on the availability of matching steps.

AAA works within the trajectory instead. It scores each step's post-action policy representation by its Gram–Schmidt residual against directions retained from the trajectory's earlier steps; we do not claim novelty for this geometric quantity itself. Normalized within each successful trajectory, these scores set how much of the trajectory's outcome credit each step receives, and they can be incorporated into existing group-relative estimators through per-step advantage construction.

Our controls separate two effects of this construction: allocating outcome credit within successful trajectories, and assigning each residual score to its corresponding step. Uniform allocation alone already captures much of the gain on seen ALFWorld, but shuffling the residual-score assignments reduces performance on unseen ALFWorld and WebShop, showing that step-specific assignment matters. On longer-horizon ScienceWorld tasks, AAA also improves substantially over GRPO, and the HGPO experiments show how AAA's credit can be incorporated into a return-based anchor estimator.

The new experiments and analyses add the following:

Reviewer 8XWL

  • W1/Q2 and follow-up. Uniform and Shuffled AAA controls, complementary step-level information, and five-seed GiGPO comparisons

Reviewer Vrc2

  • W3. GiGPO five-training-seed results, task-relevance diagnostics, and matched WebShop evaluations

Reviewer 4Gmy

  • W7/Q8. AAA on longer-horizon tasks in ScienceWorld
  • W10. Why HGPO is strong, and what AAA adds to return-based anchor estimators

The discussion period closes on November 18 AoE, and we would welcome any remaining questions or concerns.

AuthorsOpenReview ↗

Summary of Discussion Status and Additional Results

We thank all reviewers for their initial feedback and Reviewer 8XWL for their constructive follow-up. As the discussion period approaches its end, we would like to briefly summarize the current status of our submission and the additional work completed in response to the reviews.

Reviewer responses and additional experiments:

  • Reviewer 8XWL: We appreciate the reviewer's follow-up acknowledging that several concerns were addressed by our additional analyses. In response to their remaining questions, we provided further clarifications, five-training-seed GiGPO comparisons, and step-level diagnostics.
  • Reviewer Vrc2: We provided detailed responses to the original concerns, followed by additional GiGPO comparisons, step-level analyses, and matched WebShop evaluations.
  • Reviewer 4Gmy: We addressed the reviewer's questions through manuscript revisions and extensive additional experiments, including longer-horizon ScienceWorld evaluations and HGPO comparisons.

Summary of additional evidence:

Beyond the initial rebuttal, we have conducted controlled credit-allocation ablations, multi-seed GiGPO comparisons, step-level diagnostics, longer-horizon evaluations, and experiments combining AAA with HGPO. Our main contributions and reviewer-specific results are summarized in our Common Response: Additional Results and the Scope of AAA's Contribution.

As the discussion period approaches its end, we would sincerely appreciate any further feedback from the reviewers and would be happy to address any remaining questions.

We hope that these additional results and revisions will be helpful in reassessing the concerns raised in the initial reviews, and that they will be carefully considered in the final assessment.

We thank the AC and all reviewers for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.