- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Utility-Augmented Transformer (UAT) for sequential decision-making in settings where action–reward feedback provides information about hidden task states. UAT encodes previous actions and rewards to modulate attention queries, keys, and values through additive and multiplicative pathways, while an exponentially smoothed utility state aggregates feedback within each context window. The theoretical analysis establishes containment of a restricted linear input-conditioning class and characterizes higher-order interactions under stylized assumptions, alongside a conditional belief-state interpretation. Experiments on controlled DarkRoom and semi-synthetic sepsis tasks report substantial improvements, while D4RL and robomimic evaluations demonstrate competitive performance. Additional ablations and feedback-corruption studies investigate component contributions and the conditions under which feedback modulation is beneficial.
Critical strengths
-
The feedback-conditioned attention design is reasonably motivated for tasks with hidden state changes. It provides a concrete mechanism for incorporating past action outcomes into history retrieval.
-
The results on DarkRoom and semi-synthetic sepsis suggest that the approach can be useful in controlled feedback-informative settings, providing some empirical support for the proposed design.
Critical weaknesses
-
The experiments do not sufficiently isolate the benefit of attention modulation. UAT and the main DT baseline differ in feedback representation, token layout, and explicit feedback aggregation. Although an input-only control is included, it covers limited settings and nearly matches UAT on D4RL-medium. Comparisons with matched nonlinear conditioning and separate additive/multiplicative ablations are needed to establish the contribution of the proposed mechanism.
-
The theoretical separation has limited relevance to the evaluated architecture. The main expressiveness result concerns attention logits under restricted single-layer, linear assumptions, whereas the deployed models use multiple layers and nonlinear components. The mechanism probe also removes these components to match the assumptions. Consequently, the analysis does not establish that the identified interaction explains the observed policy improvements, weakening support for a central claim.
Questions for authors
-
With identical feedback inputs, token layout, EMA aggregation, and comparable parameter counts, does UAT consistently outperform input-level conditioning on the controlled tasks? The input-only ablation covers sepsis but not DarkRoom; this comparison would help determine whether attention modulation explains the reported gains.
-
What evidence links the identified third-order interaction to improvements in the deployed multilayer model? In particular, do additive-only and multiplicative-only variants show that the multiplicative pathway contributes beyond the additive feedback pathway?
5 comments
Response to Reviewer TdYK (Part 2/3)
Q2. Evidence in the deployed multilayer model
We added additive-only and multiplicative-only variants trained from scratch in the deployed multilayer models, retaining the utility encoder and EMA and matching trainable parameter counts (Appendix D.5, pp. 29–30; Table 11, p. 33):
| Model | Sepsis accuracy | Sepsis return | D4RL (M) return |
|---|---|---|---|
| Additive-only | 0.211 ± 0.010† | −2.040 ± 0.185† | 56.057 ± 1.392 |
| Multiplicative-only | 0.314 ± 0.035 | 1.858 ± 1.048 | 56.126 ± 1.602 |
| UAT | 0.401 ± 0.031 | 2.027 ± 0.599 | 56.975 ± 1.899 |
Results are mean ± sample std over five seeds. D4RL additive-only initializes its nine additive projection tensors with .
† The sepsis additive-only projections remain zero throughout training: removing the multiplicative matrices while zero-initializing the additive maps and gate readouts makes gradients into the utility pathway vanish. The gates are zero, and their contribution is multiplied by the zero additive maps. The trained pathway comparison therefore comes from the D4RL runs (p. 30, lines 1586–1589).
On D4RL, the combined design has the highest mean return; removing either pathway lowers the mean, while the two single-path variants perform similarly. On sepsis, multiplicative-only retains much of the full model's return benefit. These results support the combined parameterization.
Figure 7 (p. 33, lines 1759–1761) further visualizes feedback-dependent attention changes and shows that disabling Q/K or V modulation lowers reference-action probabilities in the trained multilayer model.
The theoretical analysis and cubic probe characterize interaction capacity under their stated assumptions. The new pathway and conditioning controls test practical performance in the deployed architecture. They provide decision-level evidence for modulation in the deployed multilayer models.
Response to Reviewer TdYK: Responses to Weaknesses (Part 3/3)
The additional experiments and analyses presented in Q1–Q2 address the concerns raised in your critical weaknesses. We summarize how this evidence responds to each concern below.
W1. Isolating the contribution of attention modulation
The strengthened input-only comparisons (Table 4) examine conditioning location with the same nonlinear feedback representation and aggregation. UAT achieves higher mean accuracy and return on both controlled benchmarks, including the newly added DarkRoom comparison. This supports a practical benefit from attention-level conditioning beyond supplying and aggregating feedback.
The FiLM and AdaLN comparisons (Table 11) further examine alternative modulation designs and show higher mean performance for UAT across the reported metrics. These controls strengthen the evidence for UAT's particular conditioning design. The near-equivalence to input-only on D4RL also identifies a setting where input-level conditioning already performs competitively, clarifying the task-dependent value of additional modulation.
W2. Connecting the theoretical interaction to deployed-model performance
The pathway controls (Table 11) and trained-policy interventions (Figure 7) are both measured in the deployed multilayer models. On D4RL, full UAT achieves the highest mean return, and removing either pathway lowers it; on sepsis, the multiplicative pathway retains much of the benefit on its own. Disabling Q/K or V modulation in the trained model lowers reference-action probabilities. Together with the fixed-width theory in Section 5.5, these results provide decision-level support for the modulation design analyzed by the theory.
We have additionally tightened the theory's statement–proof consistency: Assumption 2 uses centering, mutual independence, and finite second moments; Theorem 3(b) explicitly states the width condition ; and Theorem 4 states the risk gap on the same high-probability event (p. 6, lines 322–323; p. 7, lines 330–333; p. 16, lines 848–862). Appendix A.6 (pp. 18–19) gives an explicit width-3 FiLM construction attaining the lower bound in Theorem 5.
Response to Reviewer TdYK (Part 1/3)
Thank you for the constructive review and for recognizing the motivation and controlled-task results. We have added matched conditioning comparisons and pathway controls addressing your two questions. References are to the revised manuscript.
Q1. Input-level versus attention-level conditioning
The new EMA-based input-only comparison shows higher mean accuracy and return for UAT on both controlled benchmarks. The input-only control uses shifted action–reward feedback, the nonlinear utility encoder, and EMA, followed by a linear projection added to observation embeddings. It incorporates feedback into existing observation embeddings without adding a separate feedback-token sequence.
UAT and this control share the data and feedback inputs within each comparison. On sepsis, the utility encoders are initialized identically within each seed, and trainable parameter counts are closely matched: 211,699 for input-only versus 211,638 for UAT.
Under equal training time, the results are (Table 4, p. 10, lines 517–525; mean ± sample std over five seeds):
Budgets are 28 minutes for DarkRoom and 130 minutes for sepsis, excluding initialization and evaluation. Models share hardware and precision within each task–seed, and we evaluate the final checkpoints at budget completion.
Table 4 provides the matched EMA-based comparison; the earlier sepsis input-only row in Table 11 does not use the EMA output and differs in data, environment, evaluation, and budget. Appendix D.6 (pp. 30–31) documents both protocols.
We also added FiLM and AdaLN controls retaining the feedback encoder and EMA, with matched data, training budgets, and parameter counts (Appendix D.5, p. 30, lines 1566–1576; Table 11, p. 33). On sepsis, FiLM modulates attention outputs before residual addition, while AdaLN replaces pre-attention, pre-FFN, and final normalization. On D4RL, both act on attention outputs:
These comparisons support the utility of UAT's projection-level conditioning beyond providing a nonlinear feedback encoder and EMA. Section 5.5 (pp. 7–8, lines 361–377 and 404–408) additionally formalizes a fixed-width advantage over single-branch FiLM through independent base and utility projections. In the specified bilinear-logit setting with invertible M0, Theorem 5 gives an if-and-only-if diagonalizability condition at matched width, with an explicit rotation witness. The near-equivalence between input-only and UAT on D4RL (M), 56.905 versus 56.975, also indicates that the benefit depends on the task.