- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Align as Act (AAA), a step-level credit assignment method for LLM agents. For each rollout, AAA reads the policy's final-layer hidden state after each action's feedback. It projects that state with a fixed Gaussian random matrix and sequentially orthogonalizes it (Gram–Schmidt) against a basis of earlier residuals from the same trajectory. The residual component is mapped to a score , motivated by the innovations decomposition of Gaussian entropy. Scores are boundary-corrected and normalized within each successful trajectory. Failed rollouts receive default GRPO reward.
Critical strengths
- The method is simple, critic-free, and needs no learned reward model or step-level annotation. It is easy to add to an existing verl-agent pipeline.
- Gains over GRPO are consistent across two environments, two Qwens, and a Llama backbone.
- The diagnostics are informative: advantage dispersion in all-success groups, the advantage-composition analysis, which shows that the innovation term carries most of the advantage mass at .
- The authors report unfavourable or mixed results openly (HGPO) and document clearly algorithms and hyperparameters.
Critical weaknesses
W1. Positioning relative to elliptical and information-gain novelty scores.
Consider the regularized Gram matrix and its log-det increment:
This is the per-step information gain of GP-UCB with a linear kernel (Srinivas et al., 2010) and the log form of the episodic elliptical bonus used by E3B (Henaff et al., NeurIPS 2022). RepExp (Tuyls et al., ICLR 2026) uses elliptical bonuses on the LLM's own hidden states as an RL reward bonus, computed across samples rather than across steps. AAA's score can be read as a hard-thresholded Gram–Schmidt analogue of this increment.
W2. The "new information" interpretation.
The geometric reading (log-volume) holds for any vectors, and the paper gives it. The information-theoretic reading, however, relies on random variables with a covariance inner product and on Gaussianity, whereas are deterministic, uncentered vectors compared with the Euclidean inner product. The threshold also breaks the exact product decomposition .
W3. The objective
Eq. (8) is not the advantage of any single fixed reward. The reward is also non-stationary (it depends on ) and trajectory-dependent (through ), and the results are sensitive to .
W4. The contribution of the innovation scores is not isolated from length and position effects.
Sum-to-one normalization gives . Under "Uniform rewards" (), the step advantage depends only on trajectory length, so that ablation is essentially a test of a short-trajectory preference. It already reaches 67.7% WebShop success, against 71.1% for full AAA and 56.8% for GRPO in Table 1.
Position is a second confound. Residuals may shrink as the basis grows. Neither the training runs nor the enrichment analysis include a position-only baseline .
W5. and the AAA hyperparameters are under-ablated.
App. B.3 analyses readout position and layer only geometrically, on the untrained model. There is no downstream RL ablation of position, layer, , , the seed of . Value also encodes a truncated history window, so the residual may reflect old history dropping out of the prompt rather than new information in .
W6. GiGPO+AAA.
The paper notes that it does not reproduce GiGPO's estimator. Its final advantage is .
Still uses the immediate rather than returns from the anchor state, so GiGPO's comparison semantics is lost. The name overstates the relation to GiGPO. A variant with GiGPO's original advantage plus AAA, discounted AAA returns inside , and a sweep would clarify this.
W7. Short horizons only.
ALFWorld is limited at 50 steps and WebShop at 15. Jointly solved episodes average about 8–12 steps (Fig. 5). AAA may have specific long-horizon risks. At least one longer setting would be valuable, e.g. ScienceWorld, AppWorld, SWE environments, or MiniHack/Crafter.
W8. The score ablations are diagnostic only.
NoQ () and RAW () appear only in the enrichment analysis (Fig. 4), not as training rewards.
W9. statistical support.
- Table 7 reports GRPO+AAA on ALFWorld at 74.5 ± 3.6, versus 84.7 ± 0.7 in Table 1; GiGPO+AAA is at 86.5 versus 88.9. The model size and number of seeds for Table 7 are not stated. Please explain it and rerun GRPO under identical conditions.
- Most ablations use a single training seed: Tables 3 and 6, Figs. 6/15, E.7, and F.2.
- The two least favourable results (Table 7 and HGPO in F.1) are not referenced from the main text.
W10. Comparisons.
HGPO outperforms both AAA variants (Table 8), but only at 1.5B; HGPO+AAA is anticipated but not run, and there is no analysis of why HGPO is stronger. A discussion of other step-level credit methods would help:
- GraphGPO (ICML 2026);
- HiPER (ICML 2026), which reports higher ALFWorld success but changes the action structure;
- iStar (ICLR 2026);
- arXiv works: G2PO (already cited), HCAPO, and the concurrent SRPO (arXiv 2608.23493).
Questions for authors
- What fraction of steps extend the basis under , and how strongly does correlate with the log-det increment ?
- What objective does GRPO+AAA optimize? How does it relate to reward-weighted fine-tuning on successful trajectories?
- How do Uniform rewards and a position-only score compare to full AAA?
- What model size and how many seeds were used in Table 7, and why does ALFWorld drop by about 10 pp there?
- How is std computed in Table 1?
- How sensitive are results to readout position, layer, pooling, , , and the seed of ?
- Why are immediate rewards rather than returns used in , and why is not clipped for successful steps?
- How does AAA behave on longer horizons?
14 comments
Response to Reviewer 4Gmy (2/9)
Now suppose that we are informed that the task is successful at the end of an episode and want to quantify the contribution of each step to task success. Information theory provides a natural motivation for assessing individual steps: If denotes the task-relevant uncertainty, the new information provided by regarding can be expressed as the conditional mutual information . However, the difficulty in this formulation is that it is generally intractable to define and model a variable capturing the complicated structure of agentic tasks and compute this mutual information, except in relatively simple settings such as search or question answering. Our key motivation for circumventing this difficulty is that successful trajectories provide an outcome signal indicating that the sequence of interactions was sufficient to complete the task. In such trajectories, newly acquired observations may contain information that contributed to successful decisions. This motivates using history-relative innovation as a practical proxy for newly introduced information and allocating credit to these innovations conditioned on terminal success. Importantly, success does not imply that all information contained in the observations is task-relevant, nor that the conditional entropy equals the task-relevant information gain . Observations may contain incidental or redundant information even in successful trajectories. Moreover, AAA computes geometric residuals of contextual LLM representations rather than these information-theoretic quantities directly. We therefore use this perspective as motivation for outcome-conditioned innovation scoring, not as a claim that our score exactly estimates task-relevant mutual information or causal contribution. On the other hand, in failed trajectories, a large innovation may simply reflect novel exploration without indicating progress toward task completion. Hence, we do not assign additional positive intrinsic rewards based on innovation scores in these trajectories. Instead, we retain their original GRPO advantages. This outcome conditioning reduces the risk of rewarding novelty independently of task success, while acknowledging that individual high-innovation steps within successful trajectories are not necessarily causally essential.
Second, we go beyond innovation-based reward design by integrating outcome-conditioned step credit into group-relative policy optimization.
As mentioned above, we construct intrinsic step rewards from innovations within successful trajectories. Then, how should failed trajectories be handled, and how should these step rewards be incorporated into GRPO training? Consider GRPO with multiple trajectory rollouts, where some trajectories succeed and others fail. Rather than replacing the original group-relative learning signal, AAA normalizes the innovation-based step rewards within each non-degenerate successful trajectory so that their sum equals the unit terminal-success credit. These rewards are then converted into step-level group-relative advantages, allowing actions within successful trajectories to receive differentiated learning signals. For failed trajectories, AAA retains the original GRPO advantages, preserving the success–failure contrast used in GRPO training. This design combines outcome-conditioned, history-relative step credit with group-relative policy optimization, without introducing an additional positive exploration bonus for failed trajectories. The within-trajectory normalization also prevents the total allocated intrinsic credit of a successful trajectory from increasing merely by adding more steps. Thus, the contribution of AAA is not the geometric innovation measure alone, but its integration into a practical step-credit assignment mechanism for sparse-reward multi-turn agent training.
Relationship to E3B and RepExp:
E3B [1] constructs an episodic exploration bonus from inverse-dynamics state embeddings. RepExp [2] uses language-model response representations for inference-time selection and RL post-training. AAA shares their representation-novelty motivation but differs in the comparison unit and training role:
| Method | Representation | Comparison reference | Use |
|---|---|---|---|
| E3B [1] | Inverse-dynamics state embeddings | Earlier states in the episode | Exploration bonus added to environment reward |
| RepExp [2] | Mean-pooled response embeddings; random projection | Same-prompt responses: selected responses at inference; response group in RL | Response selection; additive response-level RL bonus |
| GRPO+AAA | Post-action contextual boundary representations; random projection | Retained directions from earlier interactions in the trajectory | Normalized step credit within successful trajectories; group-relative advantages |
Response to Reviewer 4Gmy (3/9)
RepExp also investigates token-level inference-time generation. In its RL implementation, the bonus is disabled for prompt groups in which all responses are incorrect (Appendix C of [2]). AAA conditions credit at the individual trajectory level: for nondegenerate successful trajectories its allocated rewards sum to one, and failed trajectories retain their original GRPO advantages.
Q1. Basis expansion and agreement with regularized log-det.
At , AAA admits a new basis direction on 72.48% of eligible interior steps, and its within-trajectory raw-score rankings closely match regularized log-det increments on the same 94 successful ALFWorld trajectories (mean Spearman 0.9796).
We evaluated one frozen Qwen2.5-1.5B-Instruct GRPO+AAA step-150 checkpoint (one training run) once on each of the 140 tasks of the ALFWorld Seen split (configured evaluation seed 123). Of 119 successful episodes, 94 satisfy the existing correlation criterion of at least three interior scores; the other 25 have one or two. Successes are the relevant cohort because default AAA allocates intrinsic step rewards within successful trajectories.
| Metric | Aggregation | Value |
|---|---|---|
| Basis expansion (%) | Step-weighted | 72.48 |
| Basis expansion (%) | Task-equal | 85.62 |
| Raw-score Spearman | Task-equal | 0.9796 |
| Reward-share | Task-equal | 0.0072 |
Across these 94 trajectories, the basis expands on 632 of 872 eligible interior steps.
The step-weighted expansion rate pools admissions and eligible steps; the task-equal rate averages each task's expansion fraction. Giving longer trajectories more weight yields the lower pooled rate in this cohort. The first empty-basis and terminal steps are excluded from the expansion denominator, but the first valid vector remains in subsequent history. Every interior step is scored before the basis-admission test, including non-admitted steps.
Both scoring rules use identical projected representations from the actual prompts containing the two most recent interactions. AAA uses its retained basis, whereas the regularized log-det reference includes all preceding valid vectors with identity regularization:
Spearman is computed over interior raw scores. TV compares whole-trajectory normalized shares after the same first/terminal mean-fill; both methods consequently assign identical boundary shares. Both metrics are averaged equally across the 94 tasks.
These fixed-checkpoint results show close empirical score-ranking agreement and quantify the difference in normalized allocations. Since selective retention and the treatment of within-span contributions change together, this comparison isolates neither factor. We did not train a policy with log-det rewards. We have updated Appendix E.8 and Table 10 with this analysis.
W3 / Q2. Policy-dependent credit, optimization objective, and relation to reward-weighted fine-tuning
We agree with the reviewer that because is policy- and trajectory-dependent, Eq. (8) is not an unbiased advantage estimator for a stationary Markov step-reward function. Rather, AAA redistributes unit terminal-success credit across steps to form policy-adaptive weights within a well-defined iteration-conditioned PPO surrogate, and our controlled ablations show where step-specific residual assignment contributes beyond uniform success reinforcement or the offset .
1. Return preservation versus the task objective (W3). For every non-degenerate successful trajectory ( and ), AAA normalizes the boundary-corrected scores so that total trajectory credit equals one, while failed trajectories (and degenerate cases) receive zero intrinsic reward. Consequently, for non-degenerate trajectories, the allocated step rewards preserve the binary environment return: Thus, although individual step shares adapt with the policy's representations and trajectory normalization , AAA does not alter which non-degenerate trajectories are rewarded at the episode level. At the same time, we explicitly distinguish return preservation from policy-gradient equivalence: because Eq. (8) applies rollout-group standardization, nonnegativity clipping, and a positive offset to construct step-level update weights, we do not claim that the resulting updates form an unbiased policy gradient of . Instead, AAA provides differentiated step-level learning signals among actions that share the same terminal outcome.
Response to Reviewer 4Gmy (1/9)
We thank the reviewer for their insightful comments and careful assessment. Our point-by-point responses are provided below.
W1/W2, Q1. Relationship to elliptical bonuses, theoretical motivation, and empirical agreement
W2. We address W2 first. As mentioned by the reviewer, the log-det formula arises in Gaussian entropy, where jointly Gaussian observation random variables X1,X2,⋯,Xn have covariance matrix Σ, i.e., (X1,X2,⋯,Xn)∼N(μ,Σ). Note that the entropy does not depend on the mean μ but only on Σ. For positive-definite Σ, the entropy is given by 21log((2πe)n∣Σ∣). By the maximum entropy principle, Gaussian variables achieve the maximum differential entropy under a fixed covariance constraint.
The classical geometric interpretation of this entropy formula is based on the volume of the uncertainty ellipsoid. This ellipsoid volume can be obtained by eigendecomposition of Σ=UΛU⊤, where each column vector in U represents an ellipsoid direction. The product of the eigenvalues of Σ is proportional to the squared volume of the corresponding uncertainty ellipsoid.
We use a complementary geometric interpretation of this Gaussian entropy formula, based on the LDL⊤ decomposition of Σ rather than its eigendecomposition. Square-integrable random variables defined on the same probability space can be considered as elements of a Hilbert space after centering, with the inner product defined as covariance. Specifically, let Yi=Xi−E[Xi]. We view each Yi as a vector in this Hilbert space and Y1,Y2,⋯,Yn as a collection of vectors. Then, what is the Gram matrix of these vectors? It is precisely the covariance matrix Σ. Thus, the determinant of Σ can be viewed as the squared volume of the parallelotope constructed by the centered random-variable vectors Y1,Y2,⋯,Yn. Consequently, Gaussian entropy can be interpreted, up to an additive constant, as the log-volume of this parallelotope. Through sequential orthogonalization, this volume is also connected to the product of innovation norms, corresponding to the LDL⊤ decomposition of the covariance matrix.
This classical geometric connection motivates our approach: rather than relying on the exact Gaussian entropy interpretation, we apply the intuition of sequential innovations to measure newly introduced directions in other vector representations, such as hidden vectors encoding observation sentences in LLM Transformers. This is an important motivation for our method. Note that this principle provides a geometric guideline rather than an exact information-theoretic identity for LLM representations. In practice, we use the "plus one" transformation in 21log(1+∣et∣2) and the threshold κ for numerical stability and selective basis retention. These practical choices do not preserve the exact Gaussian entropy decomposition.
W1. AAA's residual score belongs to the same geometric family as elliptical bonuses; the contribution of AAA is to use history-relative scores for outcome-conditioned step-credit allocation within multi-turn trajectories.
We agree. In a broad sense, AAA can be viewed as belonging to the same family of geometric novelty measures. For linearly independent vectors, the Gram determinant gives the squared volume of the parallelotope constructed by these vectors, as discussed above. The reviewer's identity-regularized matrix Ct is closely related to this geometry, although its log-det increment is not identical to the Gram–Schmidt residual score. We therefore do not claim novelty in the underlying geometric identity or orthogonalization procedure. Our main distinction from existing work lies in how history-relative innovations are used for outcome-conditioned step-credit assignment in multi-turn agent training.
First, we motivate the use of innovation-based step credit through an information-theoretic perspective on task success, with outcome conditioning as the key mechanism of our method.
Our key mechanism to align new information with contribution to task success is outcome conditioning: We assign non-zero intrinsic step rewards computed from innovations only for successful trajectories, while failed trajectories receive no additional intrinsic reward and retain their original GRPO advantages (Section 4.3). Let us consider the agentic task problem from an information-theoretic perspective. In many agentic tasks, solving a task T involves acquiring task-relevant information through sequential observations o1,o2,⋯,oT, alongside taking actions based on the accumulated information. From this perspective, actions and observations form a sequential interaction process in which new observations may reduce uncertainty relevant to task completion.