acceptodds
Under review as a conference paper at ICLR 2027

Utility-Augmented Transformers: Feedback-Conditioned Retrieval for Sequential Decision Making

Abstract

Offline reinforcement learning has advanced sequential decision-making. Within this framework, conditional sequence models such as Decision Transformer achieve competitive performance by predicting actions from trajectory histories and target returns-to-go. These models typically incorporate actions and rewards as input features without directly conditioning attention projections on feedback, which may limit their effectiveness on feedback-informative tasks, where identical observations can require different actions depending on what previous actions achieved. This paper examines when input-level feedback conditioning is sufficient and when it is not. We introduce the Utility-Augmented Transformer (UAT), in which shifted action-reward feedback modulates queries, keys, and values, and an exponentially smoothed utility state supplies persistent conditioning, with zero initialization reducing UAT exactly to its vanilla Transformer backbone. The additive pathway provably contains the linear input-conditioning subclass, while the multiplicative pathway realizes a third-order query-key-feedback logit that input-level injection cannot represent, yielding a persistent approximation floor for the latter under bounded stylized designs. The utility state also admits a compressed belief-state reading under explicit sufficiency conditions. Extensive experiments show substantial gains of UAT on controlled tasks with informative, timely feedback and competitive performance across standard offline RL domains, under comparable parameter budgets and without increasing asymptotic attention complexity.

Primary area:
learning on time series and dynamical systems
Keywords:
Sequential decision-making, Offline reinforcement learning, Transformers, Non-stationary environments

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TdYKOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Utility-Augmented Transformer (UAT) for sequential decision-making in settings where action–reward feedback provides information about hidden task states. UAT encodes previous actions and rewards to modulate attention queries, keys, and values through additive and multiplicative pathways, while an exponentially smoothed utility state aggregates feedback within each context window. The theoretical analysis establishes containment of a restricted linear input-conditioning class and characterizes higher-order interactions under stylized assumptions, alongside a conditional belief-state interpretation. Experiments on controlled DarkRoom and semi-synthetic sepsis tasks report substantial improvements, while D4RL and robomimic evaluations demonstrate competitive performance. Additional ablations and feedback-corruption studies investigate component contributions and the conditions under which feedback modulation is beneficial.

Critical strengths
  1. The feedback-conditioned attention design is reasonably motivated for tasks with hidden state changes. It provides a concrete mechanism for incorporating past action outcomes into history retrieval.

  2. The results on DarkRoom and semi-synthetic sepsis suggest that the approach can be useful in controlled feedback-informative settings, providing some empirical support for the proposed design.

Critical weaknesses
  1. The experiments do not sufficiently isolate the benefit of attention modulation. UAT and the main DT baseline differ in feedback representation, token layout, and explicit feedback aggregation. Although an input-only control is included, it covers limited settings and nearly matches UAT on D4RL-medium. Comparisons with matched nonlinear conditioning and separate additive/multiplicative ablations are needed to establish the contribution of the proposed mechanism.

  2. The theoretical separation has limited relevance to the evaluated architecture. The main expressiveness result concerns attention logits under restricted single-layer, linear assumptions, whereas the deployed models use multiple layers and nonlinear components. The mechanism probe also removes these components to match the assumptions. Consequently, the analysis does not establish that the identified interaction explains the observed policy improvements, weakening support for a central claim.

Questions for authors
  1. With identical feedback inputs, token layout, EMA aggregation, and comparable parameter counts, does UAT consistently outperform input-level conditioning on the controlled tasks? The input-only ablation covers sepsis but not DarkRoom; this comparison would help determine whether attention modulation explains the reported gains.

  2. What evidence links the identified third-order interaction to improvements in the deployed multilayer model? In particular, do additive-only and multiplicative-only variants show that the multiplicative pathway contributes beyond the additive feedback pathway?

5 comments
AuthorsOpenReview ↗

Response to Reviewer TdYK (Part 1/3)

Thank you for the constructive review and for recognizing the motivation and controlled-task results. We have added matched conditioning comparisons and pathway controls addressing your two questions. References are to the revised manuscript.

Q1. Input-level versus attention-level conditioning

The new EMA-based input-only comparison shows higher mean accuracy and return for UAT on both controlled benchmarks. The input-only control uses shifted action–reward feedback, the nonlinear utility encoder, and EMA, followed by a linear projection added to observation embeddings. It incorporates feedback into existing observation embeddings without adding a separate feedback-token sequence.

UAT and this control share the data and feedback inputs within each comparison. On sepsis, the utility encoders are initialized identically within each seed, and trainable parameter counts are closely matched: 211,699 for input-only versus 211,638 for UAT.

Under equal training time, the results are (Table 4, p. 10, lines 517–525; mean ± sample std over five seeds):

ModelDarkRoom accuracyDarkRoom returnSepsis accuracySepsis return
Input-only0.518 ± 0.01558.074 ± 3.1090.333 ± 0.0250.702 ± 0.479
UAT0.540 ± 0.01762.148 ± 3.6120.431 ± 0.0242.677 ± 0.359

Budgets are 28 minutes for DarkRoom and 130 minutes for sepsis, excluding initialization and evaluation. Models share hardware and precision within each task–seed, and we evaluate the final checkpoints at budget completion.

Table 4 provides the matched EMA-based comparison; the earlier sepsis input-only row in Table 11 does not use the EMA output and differs in data, environment, evaluation, and budget. Appendix D.6 (pp. 30–31) documents both protocols.

We also added FiLM and AdaLN controls retaining the feedback encoder and EMA, with matched data, training budgets, and parameter counts (Appendix D.5, p. 30, lines 1566–1576; Table 11, p. 33). On sepsis, FiLM modulates attention outputs before residual addition, while AdaLN replaces pre-attention, pre-FFN, and final normalization. On D4RL, both act on attention outputs:

ModelSepsis accuracySepsis returnD4RL (M) return
FiLM0.357 ± 0.0181.814 ± 0.15554.696 ± 1.462
AdaLN0.313 ± 0.0221.604 ± 0.27953.553 ± 2.642
UAT0.401 ± 0.0312.027 ± 0.59956.975 ± 1.899

These comparisons support the utility of UAT's projection-level conditioning beyond providing a nonlinear feedback encoder and EMA. Section 5.5 (pp. 7–8, lines 361–377 and 404–408) additionally formalizes a fixed-width advantage over single-branch FiLM through independent base and utility projections. In the specified bilinear-logit setting with invertible M0M_0, Theorem 5 gives an if-and-only-if diagonalizability condition at matched width, with an explicit rotation witness. The near-equivalence between input-only and UAT on D4RL (M), 56.905 versus 56.975, also indicates that the benefit depends on the task.

AuthorsOpenReview ↗

Response to Reviewer TdYK (Part 2/3)

Q2. Evidence in the deployed multilayer model

We added additive-only and multiplicative-only variants trained from scratch in the deployed multilayer models, retaining the utility encoder and EMA and matching trainable parameter counts (Appendix D.5, pp. 29–30; Table 11, p. 33):

ModelSepsis accuracySepsis returnD4RL (M) return
Additive-only0.211 ± 0.010†−2.040 ± 0.185†56.057 ± 1.392
Multiplicative-only0.314 ± 0.0351.858 ± 1.04856.126 ± 1.602
UAT0.401 ± 0.0312.027 ± 0.59956.975 ± 1.899

Results are mean ± sample std over five seeds. D4RL additive-only initializes its nine additive projection tensors with N(0,0.12)\mathcal{N}(0,0.1^2).

† The sepsis additive-only projections remain zero throughout training: removing the multiplicative matrices while zero-initializing the additive maps and gate readouts makes gradients into the utility pathway vanish. The gates are zero, and their contribution is multiplied by the zero additive maps. The trained pathway comparison therefore comes from the D4RL runs (p. 30, lines 1586–1589).

On D4RL, the combined design has the highest mean return; removing either pathway lowers the mean, while the two single-path variants perform similarly. On sepsis, multiplicative-only retains much of the full model's return benefit. These results support the combined parameterization.

Figure 7 (p. 33, lines 1759–1761) further visualizes feedback-dependent attention changes and shows that disabling Q/K or V modulation lowers reference-action probabilities in the trained multilayer model.

The theoretical analysis and cubic probe characterize interaction capacity under their stated assumptions. The new pathway and conditioning controls test practical performance in the deployed architecture. They provide decision-level evidence for modulation in the deployed multilayer models.

AuthorsOpenReview ↗

Response to Reviewer TdYK: Responses to Weaknesses (Part 3/3)

The additional experiments and analyses presented in Q1–Q2 address the concerns raised in your critical weaknesses. We summarize how this evidence responds to each concern below.

W1. Isolating the contribution of attention modulation

The strengthened input-only comparisons (Table 4) examine conditioning location with the same nonlinear feedback representation and aggregation. UAT achieves higher mean accuracy and return on both controlled benchmarks, including the newly added DarkRoom comparison. This supports a practical benefit from attention-level conditioning beyond supplying and aggregating feedback.

The FiLM and AdaLN comparisons (Table 11) further examine alternative modulation designs and show higher mean performance for UAT across the reported metrics. These controls strengthen the evidence for UAT's particular conditioning design. The near-equivalence to input-only on D4RL also identifies a setting where input-level conditioning already performs competitively, clarifying the task-dependent value of additional modulation.

W2. Connecting the theoretical interaction to deployed-model performance

The pathway controls (Table 11) and trained-policy interventions (Figure 7) are both measured in the deployed multilayer models. On D4RL, full UAT achieves the highest mean return, and removing either pathway lowers it; on sepsis, the multiplicative pathway retains much of the benefit on its own. Disabling Q/K or V modulation in the trained model lowers reference-action probabilities. Together with the fixed-width theory in Section 5.5, these results provide decision-level support for the modulation design analyzed by the theory.

We have additionally tightened the theory's statement–proof consistency: Assumption 2 uses centering, mutual independence, and finite second moments; Theorem 3(b) explicitly states the width condition K≤dxK\leq d_x; and Theorem 4 states the risk gap on the same high-probability event (p. 6, lines 322–323; p. 7, lines 330–333; p. 16, lines 848–862). Appendix A.6 (pp. 18–19) gives an explicit width-3 FiLM construction attaining the lower bound in Theorem 5.

Reviewer_deQJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Utility-Augmented Transformer (UAT), which uses shifted action-reward feedback to modulate attention queries, keys, and values. Token-level utility features condition keys and values, while an exponentially smoothed, window-local utility state conditions queries. The model is trained through supervised action prediction on offline trajectories.

The theoretical analysis establishes a separation between observation-only and feedback-aware models, a restricted containment result for linear input conditioning, and a third-order attention-logit interaction unavailable to fixed linear input embeddings. Experiments evaluate controlled nonstationary DarkRoom and semi-synthetic sepsis tasks, alongside D4RL locomotion and robomimic.

The architecture is interesting, and the controlled-task results are encouraging. However, the theoretical separation concerns a restricted attention-logit class, several baselines differ substantially from their established implementations, and some mechanistic conclusions are not isolated by the experiments.

Critical strengths
  1. Clear architectural idea: Conditioning retrieval projections on observed action outcomes is a reasonable inductive bias for partially observed, nonstationary tasks. The causal shift of feedback is explicit, and the construction recovers the vanilla backbone when modulation vanishes.

  2. Substantial controlled-task improvements: UAT achieves strong results on the constructed DarkRoom and sepsis environments. These tasks directly test settings where feedback can provide information about hidden changes.

  3. Useful experimental controls: The author(s) report five model seeds, largely parameter-matched Transformer comparisons, final-checkpoint evaluation, and ablations retrained from scratch. The input-only feedback control retains the utility encoder and EMA, making it more informative than comparison with standard DT alone.

  4. Detailed implementation and scope disclosures: The appendices provide detailed baseline adaptations, reward construction, window-local state resets, parameter accounting, and limitations of the mechanism probes. The author(s) also report the substantial inference-throughput cost rather than relying solely on asymptotic complexity.

Critical weaknesses
  1. The theoretical separation is narrower than the architectural motivation suggests: Theorem 3(a) establishes an approximation floor for a scalar pre-softmax logit constructed from fixed linear embeddings. It does not establish a corresponding floor for attention outputs, action predictions, or policy returns. Nonlinear attention normalization, feed-forward layers, and depth can change the representational comparison.

This distinction is especially relevant because the deployed models have multiple layers and nonlinear gates. The depth-one mechanism probe disables LayerNorm, feed-forward computation, and residual mixing, and replaces the deployed nonlinear utility pathway with an affine one. Its cubic response is therefore an expected consequence of the restricted construction. The same response in activated random UAT further confirms that this is an architectural property, rather than evidence that training has discovered a useful decision mechanism.

I should mention that the author(s) appropriately acknowledge several of these limits. Nevertheless, the claimed explanation of the empirical gains would be stronger with tests showing that the relevant interactions improve prediction or control in the deployed multilayer model, rather than only demonstrating their existence.

  1. The main DT comparison does not isolate the location of feedback conditioning: In the main experiments, DT receives return-to-go, observation, and action tokens, whereas UAT receives shifted action--reward feedback through a nonlinear encoder and EMA and has no return-to-go token. The architectures therefore differ in feedback representation, temporal processing, tokenization, and conditioning location.

The input-only feedback ablation is a valuable response to this concern because it retains the encoder and EMA. However, it is reported only for sepsis and medium-level locomotion. On D4RL medium, input-only feedback achieves 56.905 compared with UAT's 56.975, providing little evidence of a practically meaningful retrieval-modulation advantage in that setting.

The matched input-only control should be included on DarkRoom and robomimic, and compared with a tuned nonlinear input-conditioning baseline. This is central to the paper's question of when input-level conditioning is insufficient.

  1. The evidence for broad practical advantage is mixed relative to the computational cost: On D4RL, the aggregate gains over DT are small, and UAT underperforms DT on Walker2d overall, including a decline from 61.6 to 51.3 on medium-replay. The Robomimic Can results are more encouraging, although uncertainty is substantial. These findings support competitive performance, rather than a consistent cross-domain improvement.

Meanwhile, inference throughput falls by approximately 41–51% relative to the Transformer controls. Unchanged asymptotic complexity does not imply negligible deployment cost. Performance- versus-wall-clock comparisons would be more informative than parameter matching alone.

The standard-benchmark claims also require more qualification. D4RL evaluation uses newer simulator environments rather than the original dataset environment versions, and robomimic rewards are relabelled using increments and a first-success bonus. These choices are shared within the reported comparisons, but limit direct comparability with standard benchmark results.

Questions for authors
  1. Does UAT outperform parameter- and training-matched FiLM, adaptive-normalization, or nonlinear input-conditioning models given the same utility features?

  2. Can the input-only ablation be extended to DarkRoom and robomimic, and can additive-only and multiplicative-only variants isolate the mechanism emphasized by the theory?

  3. How do the conclusions change with canonical baseline implementations and matched wall-clock budgets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer deQJ (Part 1/3)

Thank you for the thoughtful review and for recognizing the architectural design, controlled-task results, and implementation disclosures. We have added analyses and experiments addressing your questions. References are to the revised manuscript; Table 11 was Table 9 in the original submission.

Q1. Matched conditioning alternatives

UAT achieves higher mean performance than the added FiLM and AdaLN controls on sepsis and D4RL (M). These controls retain the feedback encoder and EMA, use the same data and training budget, and match trainable parameter counts through feed-forward width adjustment. On sepsis, FiLM modulates attention outputs before residual addition, while AdaLN replaces pre-attention, pre-FFN, and final normalization. On D4RL, both act on attention outputs (Appendix D.5, p. 30, lines 1566–1576).

Five-seed results (mean ± sample std; Table 11, p. 33):

ModelSepsis accuracySepsis returnD4RL (M) return
FiLM0.357 ± 0.0181.814 ± 0.15554.696 ± 1.462
AdaLN0.313 ± 0.0221.604 ± 0.27953.553 ± 2.642
UAT0.401 ± 0.0312.027 ± 0.59956.975 ± 1.899

We also evaluate nonlinear input conditioning through the same utility encoder and EMA, followed by a linear projection added to the observation embedding (Table 4). On sepsis, the input-only control and UAT have 211,699 and 211,638 parameters, with their utility encoders initialized identically within each seed. Under equal training time, UAT improves accuracy from 0.333 to 0.431 and return from 0.702 to 2.677. As clarified in Appendix D.6 (pp. 30–31), Table 4 provides the matched EMA-based input-only comparison; the earlier sepsis input-only result in Table 11 does not use the EMA output and differs in data, environment, evaluation, and budget.

We added a theoretical comparison in Section 5.5 (pp. 7–8, lines 361–377 and 404–408): independent base and utility projections give UAT a fixed-width expressivity advantage over single-branch FiLM in the specified bilinear-logit setting. For invertible M0M_0, Theorem 5 gives an if-and-only-if diagonalizability condition on M1M0−1M_1M_0^{-1} at matched inner width, with a planar-rotation witness (FiLM needs width at least 3, UAT's affine-gate core needs 2) and an explicit attaining construction. UAT is also a structured special case of a hypernetwork, using compact gates instead of explicitly generating dense weights.

AuthorsOpenReview ↗

Response to Reviewer deQJ (Part 2/3)

Q2. Extended input-only comparisons and mechanism controls

Input-only comparisons now cover DarkRoom and robomimic. Under equal wall-clock training on DarkRoom, UAT achieves accuracy 0.540 ± 0.017 and return 62.148 ± 3.612, versus 0.518 ± 0.015 and 58.074 ± 3.109 for input-only (Table 4, p. 10).

On robomimic, UAT has higher mean success in all four new settings (Table 9, p. 31; mean ± sample std over three seeds):

Task / feedbackInput-onlyUAT
Lift / sparse89.33 ± 3.0693.67 ± 2.08
Lift / terminal delay89.00 ± 4.5894.00 ± 3.61
Can / sparse45.00 ± 4.3648.67 ± 6.43
Can / terminal delay12.67 ± 6.1124.67 ± 0.58

Terminal delay accumulates the relabeled dense rewards until episode termination in both training and evaluation. This evaluates performance when reward feedback is unavailable before termination.

For the pathway controls, Table 11 (p. 33) reports the following results (mean ± sample std over five seeds):

ModelSepsis accuracySepsis returnD4RL (M) return
Additive-only0.211 ± 0.010†−2.040 ± 0.185†56.057 ± 1.392
Multiplicative-only0.314 ± 0.0351.858 ± 1.04856.126 ± 1.602
UAT0.401 ± 0.0312.027 ± 0.59956.975 ± 1.899

The variants retain the feedback encoder and EMA and are retrained with matched parameter counts. D4RL additive-only initializes its nine additive projection tensors with N(0,0.12)\mathcal{N}(0,0.1^2).

† The sepsis additive-only projections remain zero throughout training: with the multiplicative matrices removed and zero-initialized additive maps and gate readouts, the utility pathway receives zero gradient. Here gj=0g_j=0 makes ∂L/∂U∝gj\partial\mathcal{L}/\partial U\propto g_j vanish, and the gradient into WsW_s vanishes because its contribution is multiplied by the zero additive maps. The trained additive-versus-multiplicative comparison therefore comes from D4RL (Appendix D.5, p. 30, lines 1586–1589).

Multiplicative-only retains much of the sepsis return benefit, while full UAT achieves the highest mean across the reported metrics. On D4RL, the combined design has the highest mean return; removing either pathway lowers the mean, while the two single-path variants perform similarly.

These controls evaluate the deployed multilayer models. Figure 7 (p. 33, lines 1759–1761) additionally visualizes feedback-dependent attention changes and shows that disabling Q/K or V modulation lowers reference-action probabilities. The theory and cubic probe characterize a restricted logit class, including an architectural property present at random initialization. The new controls provide decision-level evidence for the modulation design.

Q3. Baseline implementations and equal-wall-clock comparison

We use the original authors' code as the basis for the baselines. Task-specific adaptations to the common offline setting are documented in Appendix C.2, and hyperparameter configurations are provided in Appendix C.3. To assess the performance–cost trade-off directly, we added equal-wall-clock comparisons with DT and the EMA-based input-only control (Section 6.4, p. 10, lines 500–504; Table 4, lines 517–525). Results are mean ± sample std over five seeds:

ModelDarkRoom accuracyDarkRoom returnSepsis accuracySepsis returnD4RL (M) return
Input-only0.518 ± 0.01558.074 ± 3.1090.333 ± 0.0250.702 ± 0.47954.113 ± 3.214
DT0.356 ± 0.0077.366 ± 0.7210.233 ± 0.022−1.895 ± 0.27255.509 ± 1.694
UAT0.540 ± 0.01762.148 ± 3.6120.431 ± 0.0242.677 ± 0.35955.052 ± 3.857

Training budgets are 28 minutes for DarkRoom, 130 minutes for sepsis, and 60 minutes per D4RL task, excluding initialization and evaluation. Within each task–seed, all models use the same hardware and precision, and we evaluate their final checkpoints at budget completion.

UAT achieves higher mean accuracy and return than DT and input-only on both controlled benchmarks, showing that its gains can offset the additional training cost. On D4RL, it remains close to DT with a slightly lower mean, reflecting a task-dependent performance–cost trade-off consistent with the reported training-throughput overhead.

AuthorsOpenReview ↗

Response to Reviewer deQJ (Part 3/3): Responses to Weaknesses

The additional experiments and analyses presented in Q1–Q3 address the concerns raised in your critical weaknesses. We summarize how this evidence responds to each concern below.

W1. Connecting the theoretical analysis to deployed models

The theory and cubic probe characterize an architectural interaction within the stated restricted logit class. The new experiments in Q2 examine the practical role of modulation in the deployed multilayer models.

The D4RL pathway ablations (Table 11) show that the combined additive–multiplicative design achieves the highest mean closed-loop return. The trained-policy interventions (Figure 7) further show that disabling Q/K or V modulation lowers reference-action probabilities. These results provide evidence at the prediction and control levels, complementing the structural insight from the theoretical analysis. We have also corrected Theorem 4's risk-gap statement to hold on the same high-probability event, since the input-level lower bound holds for every training sample (Appendix A.4, p. 16, lines 848–862). Assumption 2 now requires centering, mutual independence, and finite second moments, matching the conditions used in its proof (Appendix A.5, p. 17, lines 912–915).

W2. Isolating the contribution of conditioning location

The strengthened nonlinear input-only comparison (Table 4) shows higher mean accuracy and return for UAT on both controlled benchmarks. The expanded robomimic comparisons (Table 9) also show higher mean success than input-only across the tested sparse and terminal-delayed settings. These additions extend the comparison to the tasks requested in your review.

The FiLM and AdaLN controls (Table 11) further support the value of UAT's modulation design when alternative conditioning methods receive the same utility features. Together, these comparisons provide evidence for benefits beyond feedback encoding and aggregation. The near-equivalence to input-only on D4RL also identifies a setting where input-level conditioning already performs competitively and additional modulation brings little average benefit.

W3. Cost and benchmark protocols

The equal-wall-clock results (Table 4) show that the controlled-task gains persist after accounting for training time. On D4RL, UAT remains close to DT under equal time, with slightly lower mean return. The inference-throughput measurements in Table 3 quantify deployment cost separately from training cost.

Our D4RL evaluation uses newer simulator versions, and robomimic rewards are relabeled with increments and a first-success bonus. These choices are shared within the corresponding comparisons and documented in Appendix C.1.2 (p. 23, lines 1194–1199 and 1212–1217); they should be taken into account when comparing our results with published benchmark numbers.

Thank you for answering all my questions. I do not have any remaining concerns. However, as the work is not that novel, I tend to keep my rating 3.

Reviewer_biA7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Utility-Augmented Transformers for offline sequential decision making. UAT encodes previous actions and rewards and uses these features to modulate attention queries, keys, and values. The theory identifies a third-order interaction advantage in restricted single-layer models. Experiments show substantial gains on controlled hidden-regime tasks, but smaller and less consistent gains on standard benchmarks.

Critical strengths
  1. UAT offers a concrete design for feedback-conditioned attention. Local feedback modulates keys and values, while aggregated feedback modulates queries. The single-layer theory clarifies the resulting interaction structure.

  2. The controlled experiments support the motivation. The main results for controlled tasks show gains on hidden-regime tasks. Feedback-corruption tests and the matched input-only control provide useful supporting evidence.

Critical weaknesses
  1. The incremental contribution over existing conditioning methods remains unclear. UAT builds on feature modulation [1] and conditional weight generation [2]. Table 9 provides a useful input-only control, but lacks matched alternative modulation designs. The restricted single-layer theory and Figure 2 do not establish that Q/K/V modulation explains the practical gains.

  2. The experiments do not establish when attention-based retrieval is necessary or broadly beneficial. The controlled tasks may mainly reward recent-feedback integration. Matched short-window MLP and GRU controls are missing. The largest gains occur on DarkRoom and the custom synthetic sepsis task. Improvements on standard offline benchmarks are smaller or inconsistent: D4RL gains are marginal, and robomimic gains vary by task. Input-only feedback nearly matches UAT on D4RL in Table 9, despite 41–51% lower inference throughput in Table 3. Although the authors acknowledge limited generality, the advantage over simpler feedback models and its applicability beyond the controlled tasks remain unresolved.

References

[1] Perez et al. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2018.

[2] Ha et al. HyperNetworks. ICLR 2017.

Questions for authors
  1. What does Q/K/V modulation contribute beyond existing conditioning methods [1,2]? Please extend Table 9 with matched feature-modulation and adaptive-normalization controls using the same feedback encoder and EMA.

  2. Does the third-order interaction in Figure 2 explain the performance gains? Please add additive-only and multiplicative-only controls to Table 9 and report closed-loop returns.

  3. Do DarkRoom and the synthetic sepsis task require long-history retrieval, or mainly recent feedback? Please compare matched short-window MLP and GRU policies, and vary history length and reward delay.

  4. Why are the largest gains concentrated in the two controlled environments, while D4RL gains are marginal and robomimic gains vary by task? Please explain the near-equivalence to input-only feedback in Table 9 and test whether observability or regime switching determines the advantage.

  5. Beyond the acknowledged limitations, what evidence supports applicability to standard offline tasks with sparse or delayed rewards? A matched evaluation in such a setting would help establish the method’s scope.

  6. Given the throughput cost in Table 3, do the gains remain under equal wall-clock training budgets? Please compare UAT with DT and input-only feedback under this constraint.

I would be happy to vote for weak acceptance if the authors can address my concerns clearly.

5 comments
AuthorsOpenReview ↗

Response to Reviewer biA7 (Part 1/4)

Thank you for the constructive review. The revised manuscript incorporates our new analyses, experimental results, and protocol clarifications. We address your six questions below. References are to the revised manuscript; Table 11 was Table 9 in the original submission.

Q1. Contribution beyond FiLM and hypernetworks

We have added theoretical analysis showing a fixed-width expressivity advantage over single-branch FiLM in the specified bilinear-logit setting, and a compact alternative to explicit dense hypernetwork weight generation (Section 5.5, pp. 7–8, lines 361–377 and 404–408; proof in Appendix A.6, pp. 18–19, lines 938–996).

The key is independent base and utility projections: single-branch FiLM shares matrix factors across its base and feedback-dependent terms, whereas UAT can choose them independently. For invertible M0M_0, Theorem 5 states an if-and-only-if condition at matched width: a FiLM-type logit x⊤LD(s)Ryx^\top L D(s) R y at inner width k=dk=d realizes x⊤(M0+sM1)yx^\top(M_0+sM_1)y exactly when M1M0−1M_1M_0^{-1} is diagonalizable over R\mathbb{R}. The proof sets D0=D(0)D_0=D(0) and D1=D(1)−D(0)D_1=D(1)-D(0) to obtain M0=LD0RM_0=LD_0R and M1=LD1RM_1=LD_1R. Since L,D0,RL,D_0,R are invertible, M1M0−1=L(D1D0−1)L−1M_1M_0^{-1}=L(D_1D_0^{-1})L^{-1}; conversely, if M1M0−1=SΛS−1M_1M_0^{-1}=S\Lambda S^{-1} with real diagonal Λ\Lambda, choose L=SL=S, R=S−1M0R=S^{-1}M_0, and D(s)=I+sΛD(s)=I+s\Lambda. A planar-rotation target (M0=I2M_0=I_2, M1M_1 a 90∘90^\circ rotation with eigenvalues ±i\pm i) therefore needs FiLM inner width k≥3k\geq 3: rank excludes k<2k<2, and the diagonalizability condition excludes k=2k=2. UAT's affine-gate core needs only head width dm=2d_m=2, and the attaining k=3k=3 FiLM factorization is exhibited explicitly in Appendix A.6. An unrestricted hypernetwork can reproduce UAT; UAT uses dmd_m condition-dependent gates instead of explicitly generating dmdd_m d weight entries per projection.

We also added the requested FiLM and AdaLN controls, retaining the feedback encoder and EMA and matching trainable parameter counts through feed-forward width adjustment. On sepsis, FiLM modulates attention outputs before residual addition, while AdaLN replaces pre-attention, pre-FFN, and final normalization. On D4RL, both act on attention outputs (Appendix D.5, p. 30, lines 1566–1576). Table 11 (p. 33) reports the following five-seed results (mean ± sample std):

ModelSepsis accuracySepsis returnD4RL (M) return
FiLM0.357 ± 0.0181.814 ± 0.15554.696 ± 1.462
AdaLN0.313 ± 0.0221.604 ± 0.27953.553 ± 2.642
UAT0.401 ± 0.0312.027 ± 0.59956.975 ± 1.899

UAT has the highest mean on all three metrics, supporting benefits beyond providing a feedback encoder and EMA.

We additionally strengthened the input-only comparison in Table 4: shifted action–reward → encoder → EMA → linear projection → addition to the observation embedding. On sepsis, its encoder starts from the same-seed Full UAT encoder initialization, with 211,699 versus 211,638 trainable parameters. UAT still improves accuracy from 0.333 to 0.431 and return from 0.702 to 2.677. The earlier sepsis input-only row in Table 11 projects utility without using the EMA output and differs in data, environment, evaluation, and budget. The matched EMA-based comparison is within Table 4; Appendix D.6 (pp. 30–31) documents both protocols.

AuthorsOpenReview ↗

Response to Reviewer biA7 (Part 2/4)

Q2. Interaction structure and closed-loop pathway controls

Figure 2 identifies the interaction structure available to the architecture. We also add pathway experiments to examine how the additive and multiplicative components contribute in the deployed multilayer models (Appendix D.5, pp. 29–30; Table 11, p. 33):

ModelSepsis accuracySepsis returnD4RL (M) return
Additive-only0.211 ± 0.010†−2.040 ± 0.185†56.057 ± 1.392
Multiplicative-only0.314 ± 0.0351.858 ± 1.04856.126 ± 1.602
UAT0.401 ± 0.0312.027 ± 0.59956.975 ± 1.899

The variants retain the feedback encoder and EMA and are retrained with matched parameter counts. Results are mean ± sample std over five seeds. D4RL additive-only initializes its nine additive projection tensors with N(0,0.12)\mathcal{N}(0,0.1^2).

† On sepsis, the additive-only projections remain zero throughout training. With the multiplicative matrices removed and both additive maps and gate readouts zero-initialized, the gates satisfy gj=tanh⁡(Wsuj)=0g_j=\tanh(W_su_j)=0. Hence ∂L/∂UK,V∝gj=0\partial\mathcal{L}/\partial U_{K,V}\propto g_j=0, and the gradient into WsW_s vanishes because its contribution is multiplied by the zero additive maps; the query pathway has the same zero-product structure. This initialization leaves the utility pathway inactive (p. 30, lines 1586–1589).

The trained additive-versus-multiplicative comparison therefore comes from D4RL. There, the combined design achieves the highest mean return and each single-pathway variant lowers it. On sepsis, multiplicative-only retains much of the full model's return benefit (1.858 versus 2.027), while full UAT achieves higher accuracy (0.401 versus 0.314), supporting the combined parameterization.

The learned-policy interventions in Figure 7 (p. 33, lines 1759–1761) further visualize feedback-dependent attention changes and show that disabling Q/K or V modulation lowers reference-action probabilities.

Q3. Recent feedback versus long-history retrieval

We added parameter-matched MLP and GRU comparisons and examined history length and feedback delay (Appendix C.1.3, p. 23, lines 1222–1241; Figure 5, p. 24). The results clarify that both controlled benchmarks generally favor shorter histories and timely feedback; long-history retrieval is not necessary for strong performance.

On DarkRoom, MLP and GRU are competitive with short windows, but their returns decline as the history grows, whereas UAT maintains strong performance across the tested history lengths. On synthetic sepsis, shorter histories also perform best, and UAT achieves higher returns throughout the history-length sweep. Delaying feedback reduces UAT's returns on both tasks; substantial delay eventually reverses its advantage on sepsis, consistent with the reduced usefulness of stale outcomes under changing treatment regimes.

These findings clarify the benchmarks' temporal characteristics: recent, informative feedback is particularly valuable. UAT performs well in this setting while offering greater robustness to context length, especially on DarkRoom. Its observed benefits therefore extend to tasks where short histories already suffice. These comparisons match parameter and training budgets. In this study, MLP/GRU use observation–action–reward histories without RTG, while UAT retains RTG. In the delay sweep, MLP/GRU delay rewards only, whereas UAT delays utility actions and rewards.

AuthorsOpenReview ↗

Response to Reviewer biA7 (Part 3/4)

Q4. Why gains vary across tasks; observability and switching

Our interpretation is that feedback conditioning is most useful when it helps distinguish decision-relevant conditions that observations alone do not resolve. On D4RL (M), input-only feedback and full UAT obtain 56.905 ± 2.413 and 56.975 ± 1.899 (Table 11). This indicates that additional projection modulation brings little average benefit in that setting. We interpret the near-equivalence as evidence that the value of this inductive bias depends on the task.

To examine observability and switching, we added two interventions: fix the goal/regime while keeping it hidden, or reveal it while retaining switching. Within each condition, UAT and MLP share data, seeds, and training/evaluation budgets, with K=40K=40 and 50 final-checkpoint episodes per seed (Appendix E.6, p. 34, lines 1816–1835; Table 12, p. 35):

ConditionDarkRoom: UAT − MLP returnSepsis: UAT − MLP return
Hidden, switching+31.512+2.287
Hidden, fixed+48.692+1.416
Revealed, switching+40.414+2.350

UAT's advantage over this MLP baseline persists both without switches and with goal/regime information revealed. Its benefit therefore extends beyond the original hidden, switching setting. Together with D4RL and robomimic, these results support advantages whose magnitude depends on the task and feedback conditions. The hidden-switching reference reuses the clean K=40K=40 runs from the history-length study, following the configurations in Appendix C.1.3.

Q5. Sparse and terminal-delayed feedback on standard tasks

We added a matched comparison on robomimic Lift and Can under sparse rewards and terminal-delayed feedback (Section 6.2, p. 9, lines 466–470; Appendix D.7 and Table 9, p. 31). Success rates (%) are mean ± sample std over three seeds:

Task / feedbackInput-onlyDTUAT
Lift / sparse89.33 ± 3.0692.67 ± 2.3193.67 ± 2.08
Lift / terminal delay89.00 ± 4.5891.33 ± 2.5294.00 ± 3.61
Can / sparse45.00 ± 4.3649.00 ± 5.2048.67 ± 6.43
Can / terminal delay12.67 ± 6.1121.33 ± 8.0224.67 ± 0.58

UAT has a higher mean than DT in three settings and is nearly tied in Can/sparse; it has a higher mean than input-only in all four.

All models use matched task settings. Terminal delay accumulates the relabeled dense rewards until episode termination in both training and evaluation, with DT's training RTG recomputed accordingly. These results support UAT's competitive performance under sparse and terminal-delayed feedback on standard robotic tasks.

Q6. Equal-wall-clock training budgets

Yes, UAT's gains on both controlled benchmarks remain under equal training time. We added comparisons against DT and the EMA-based input-only control described in Q1 (Section 6.4, p. 10, lines 500–504; Table 4, lines 517–525). Results are mean ± sample std over five seeds:

ModelDarkRoom accuracyDarkRoom returnSepsis accuracySepsis returnD4RL (M) return
Input-only0.518 ± 0.01558.074 ± 3.1090.333 ± 0.0250.702 ± 0.47954.113 ± 3.214
DT0.356 ± 0.0077.366 ± 0.7210.233 ± 0.022−1.895 ± 0.27255.509 ± 1.694
UAT0.540 ± 0.01762.148 ± 3.6120.431 ± 0.0242.677 ± 0.35955.052 ± 3.857

Training budgets are 28 minutes for DarkRoom, 130 minutes for sepsis, and 60 minutes per D4RL task, excluding initialization and evaluation. Within each task–seed, all models use the same hardware and precision, and we evaluate their final checkpoints at budget completion.

UAT achieves higher mean accuracy and return than DT and input-only on both controlled benchmarks, showing that its gains can offset the additional training cost. On D4RL, it remains close to DT with a slightly lower mean, reflecting a task-dependent performance–cost trade-off consistent with the reported training-throughput overhead.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Global Response to Reviewers (1/2)

Thank you to all reviewers for the constructive feedback. We are grateful that the reviewers found the architectural idea interesting (Reviewer 2), the controlled-task evidence encouraging (Reviewer 2) with useful experimental controls (Reviewer 2), and the problem understanding substantially deeper (Reviewer 3). Reviewer 1 explicitly offered to vote for acceptance if the concerns are addressed. We have seriously looked into each issue and addressed every concern below.

The reviews raise four central issues: isolating the contribution of attention-level conditioning, connecting the theory to deployed policies, establishing when feedback modulation is useful, and assessing its computational cost and benchmark scope. We summarize the new evidence below. References are to the revised manuscript; additions in the manuscript are highlighted in blue. Protocol details and implementation clarifications appear in our individual responses.

1. Contribution beyond input conditioning and existing modulation methods.

We added FiLM and AdaLN controls using the same feedback encoder and EMA, with matched training budgets and parameter counts (Table 11). UAT achieves the highest mean sepsis accuracy, sepsis return, and D4RL (M) return among these controls.

The strengthened EMA-based input-only comparison now covers both controlled benchmarks (Table 4). Under equal training time, UAT improves mean return from 58.074 to 62.148 on DarkRoom and from 0.702 to 2.677 on sepsis, with higher mean accuracy on both. The new robomimic comparisons also include input-only feedback (Table 9).

Section 5.5 adds a theoretical comparison: independent base and utility projections give UAT a fixed-width expressivity advantage over single-branch FiLM in the specified bilinear-logit setting. Theorem 5 makes this precise in both directions: for invertible M0M_0, at matched inner width k=dk=d, a FiLM-type logit x⊤LD(s)Ryx^\top L D(s) R y can realize the target pair (M0,M1)(M_0,M_1) if and only if M1M0−1M_1M_0^{-1} is diagonalizable over R\mathbb{R}, because its base and feedback-dependent blocks must share the rank-one factors determined by LL and RR. UAT's affine-gate core realizes every such pair at head width dm=dd_m=d through independent base and utility projections; a planar-rotation target requires FiLM inner width k≥3k\geq 3 but only UAT width 2, and the k=3k=3 construction is given explicitly. UAT also provides a compact, structured form of conditional weight generation.

2. Evidence in the deployed multilayer models.

We added retrained additive-only and multiplicative-only variants (Table 11). On D4RL (M), full UAT has the highest mean return (56.975), compared with 56.057 for additive-only and 56.126 for multiplicative-only. The two single-path variants perform similarly, while the combined design gives the best mean. The sepsis additive-only run has an initialization deadlock: with the multiplicative matrices removed and both the additive maps and gate readouts zero-initialized, gradients into the utility pathway vanish. The trained pathway-level comparison therefore comes from the D4RL runs, whose additive projection tensors are initialized with N(0,0.12)\mathcal{N}(0,0.1^2); the sepsis multiplicative-only variant retains much of the full model's return benefit.

Interventions in the trained multilayer model further show that disabling Q/K or V modulation lowers reference-action probabilities; Figure 7 also visualizes feedback-dependent attention changes. These experiments connect the modulation design to prediction and closed-loop performance. The theory and cubic probe characterize interaction capacity under their stated assumptions.

AuthorsOpenReview ↗

Global Response to Reviewers (2/2)

3. Temporal requirements and applicability across tasks.

Parameter-matched MLP and GRU comparisons, together with history-length and delay sweeps (Figure 5; Appendix C.1.3), show that the controlled benchmarks favor recent, informative feedback. Short histories can already be effective, and UAT is particularly robust to increasing context length on DarkRoom. Delayed feedback reduces its benefit. Additional interventions show that UAT's advantage over the tested MLP persists when the goal/regime is fixed or revealed (Table 12).

On standard robotic tasks, we added sparse and terminal-delayed reward evaluations on robomimic Lift and Can (Table 9). UAT has a higher mean success rate than DT in three of four settings and is nearly tied in the fourth; it exceeds input-only in all four. Terminal-delayed rewards are released only at episode termination, testing performance without intermediate reward feedback.

The standard-task results show competitive, task-dependent performance. Input-only and UAT are nearly tied on D4RL (M), and UAT trails DT on Walker2d medium-replay (51.3 versus 61.6; Table 8).

4. Computational cost and comparison protocols.

The new equal-wall-clock experiments use 28 minutes on DarkRoom, 130 minutes on sepsis, and 60 minutes per D4RL task, with the same hardware and precision within each task–seed (Table 4). UAT retains higher mean accuracy and return than DT and input-only on both controlled benchmarks, showing that its gains can offset the training overhead. On D4RL (M), UAT is close to DT with a slightly lower mean (55.052 versus 55.509). Table 3 separately reports the 41–51% inference-throughput reduction.

Our baselines use the original authors' code with task-specific adaptations and hyperparameters documented in Appendices C.2–C.3. D4RL evaluation uses newer simulator versions, and robomimic uses reward increments plus a first-success bonus. These settings are shared within each corresponding comparison, and the scores should be interpreted under the reported protocols.

Summary

  1. Contribution. Matched controls support added value from attention-level conditioning, while the theory clarifies the role of independent base and utility projections.

  2. Mechanism. Multilayer pathway ablations and trained-policy interventions connect the modulation design to prediction and closed-loop performance.

  3. Scope. UAT shows strong controlled-task gains, context-length robustness on DarkRoom, and competitive performance under sparse and terminal-delayed rewards.

  4. Cost and protocols. Controlled-task gains persist under equal training time, while D4RL remains close to DT. Inference costs and benchmark adaptations are explicitly reported.

AuthorsOpenReview ↗

Global Response: Final Update for the Area Chair

Dear Area Chair,

As the discussion concludes, we provide a final summary of how the revision and subsequent experiments strengthen the paper, complementing our two earlier global responses.

A central question across the reviews was whether UAT’s attention-level conditioning offers benefits over matched input-level conditioning and alternative modulation methods. The added input-only, FiLM, and AdaLN comparisons provide evidence for this contribution under matched conditions. On both controlled benchmarks, UAT’s advantages over input-only conditioning persist under equal training time. The revised theory further identifies a fixed-width expressivity advantage over single-branch FiLM through independent base and utility projections in the specified bilinear-logit setting. The broader evaluations clarify how these benefits depend on the task, feedback availability, and computational budget.

The discussion has reflected this progress. Reviewer biA7 raised their rating from 2 to 3 after our explanations and experiments addressed most of their concerns. Reviewer deQJ reported no remaining concerns and retained a rating of 3, with a reservation about novelty. Reviewer TdYK acknowledged that the matched input-only comparisons helped address their first concern, while maintaining a rating of 2 and seeking stronger evidence connecting the theoretical interaction to performance in trained multilayer policies.

Our latest follow-up, posted after TdYK’s assessment, directly examines this connection. We selectively removed the corresponding interaction terms from the attention logits of trained three-layer UAT policies. Removing the key-side term reduced mean normalized return by 4.17 points on D4RL (M); jointly removing the query-side and key-side terms reduced mean return by 7.26 points on cyclical DarkRoom. Each intervention lowered return relative to the intact policy in all five training seeds. Our follow-up to Reviewer TdYK at the bottom of this page reports the intervention details, norm-matched controls, and paired confidence intervals.

These interventions add direct evidence from trained policies to the theoretical analysis and retrained comparisons. Together, the results support the paper’s central contribution: using action–reward feedback to condition retrieval from history, with a theoretical characterization of the parameterization and empirical evidence identifying where it is useful.

We believe the revised manuscript and subsequent experiments substantially address the main technical concerns raised during review. We respectfully ask the Area Chair to consider the complete discussion, including the reviewer updates and the additional posted evidence, in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.