acceptodds
Withdrawn from ICLR 2027

BLUR: Bootstrapped Latent Utility Reward Learning from Trajectory-Level Performance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In complex reinforcement learning tasks, handcrafted dense step-wise rewards can substantially improve exploration efficiency and policy performance. However, their construction requires extensive domain knowledge and scales poorly to new tasks. Existing automatic reward-learning methods reduce the need for manual intervention, but often struggle to model high-dimensional sequential behaviors and suffer from severe cold-start problems when successful trajectories are scarce. To address these limitations, we propose Bootstrapped Latent Utility Reward Learning (BLUR), which learns context-dependent dense rewards solely from trajectory-level performance signals. BLUR first constructs relative utility supervision by comparing trajectory performance against the recent policy level, guiding the policy from random failures toward progressively better failures and eventually successful behaviors. The temporary relative model is then discarded, and a new reward model with fixed absolute semantics is trained from both successful and unsuccessful trajectories. Finally, this model is frozen and used to optimize an independently initialized target policy. The reward model employs an LSTM to encode temporal behavioral context and VaDE to model latent behavioral structures, from which step-wise rewards are generated using trajectory-performance statistics.

Primary area:
reinforcement learning
Keywords:
Deep Reinforcement Learning, Reward Shaping, Clustering, Cold-start Warmup

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZeLnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses automatic reward shaping in reinforcement learning (RL). In contrast to handcrafted rewards, which induce designer bias, i.e., pre-specifying useful behavior while restricting potential informative behavior for training, the paper proposes BLUR to learn a dense reward function for easy learning. It compresses the experience trajectories into latent representations to cluster behavioral patterns. It then assigns performance values to each cluster to learn a reward model that can associate between behavioral patterns and the actual task performance. BLUE is evaluated on cooperative multi-agent tasks and compared with standard baselines, such as MAPPO, as well as a re-implementation of SASR, an adaptive reward shaping approach for single-agent RL tasks.

Critical strengths

The paper addresses an important and long-standing problem in reinforcement learning, where sparse rewards can lead to suboptimal learning due to lacking exploration. Handcrafting rewards is tedious and can yield unintended side effects, which needs to be addressed adequately.

Critical weaknesses
  1. Clarification of the setting:
  • Overall, the paper addresses a single-agent RL problem by focusing on a single scalar reward, without considering any agent interactions, conflicts, or credit assignments. However, it uses multi-agent environments to evaluate its approach with multi-agent benchmarks (SMAC) and baselines (MAPPO) rather than related single-agent work (SIL, RUDDER, etc.).
  • Although it briefly states that multi-agent tasks could be collapsed to single-agent problems, the reverse does not hold: Having several self-interested agents could lead to emergent dilemmas, which cannot be addressed trivially by using single-agent methods [1]. Thus, I suggest a clearer focus toward single-agent MDPs instead of Dec-POMDPs, which seems like an over-design to me.
  • Furthermore, SMAC is noted to be outdated [2].
  • Despite the paper's criticsm against handcrafted rewards, the approach itself requires domain-dependent design, e.g., to define the statistics necessary for P(τ)P(\tau) (Equation 1), which is necessary to conduct supervised reward learning (Equation 5). Thus, it does not seem to solve the actual issue of handcrafting.
  1. Evaluation
  • The paper only reports the final test performance of BLUR in a table. Figure 5 reports the training curves compared with handcrafted rewards. However, I do not see any substantial advantage of BLUE, i.e., evidence about how it overcomes the designer bias mentioned above.
  • Due to the actual single-agent RL focus, the multi-agent evaluation appears misleading. The underperformance of SASR might be confounded by multi-agent aspects (e.g., credit assignment, which could affect the single-agent reward adaptation). Conversely, other adaptive reward methods that the paper did not compare with could still outperform BLUE (either in single-agent or multi-agent settings). Due to the missing experimental comparison, I cannot confirm the significance of the results.

[1] Leibo et al., "Multi-agent Reinforcement Learning in Sequential Social Dilemmas", AAMAS-17

[2] Ellis et al., "SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning", NeurIPS Benchmarks 2023

Questions for authors
  1. What is meant by “consequence” regarding xtx_t (Equation 2)? How is it represented?
  2. Why does the auxiliary performance-prediction head require pooling of all hidden states (Equation 4)? hTh_T should already contain all preceding information, according to the general RNN design.
Reviewer_fY79OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BLUR, a method for learning dense reward shaping signals from coarse trajectory-level performance evaluations. First, the trajectory-level reward function P(τ) is designed based on human priors. Second, the trajectories are collected, and BLUR learns statistical associations between recurring behavioral patterns and trajectory-level reward. Specifically, LSTM encodes trajectories and VaDE clusters embeddings into different latent patterns. Third, BLUR converts these associations into step-wise dense shaping signals by a posterior-weighted combination of cluster utilities of each step. Experiments on SMAC and HighwayEnv validate the proposed method, which outperforms handcrafted dense-reward baselines and SASR.

Critical strengths
  1. The three-step pipeline is clean and easy to understand. The estimation of pattern utility from statistics is reasonable.

  2. The whole method and training pipeline are well designed. Although the method seems complex, it works well in the tested environments.

  3. The studied topic is a long-standing problem in MARL.

Critical weaknesses
  1. This method still needs a human-designed trajectory-level reward. Furthermore, the trajectory-level reward functions in this paper are also complex and require several coefficients.

  2. There is no mechanism for handling unseen behavioral modes. If the latent pattern does not exist in the collected trajectories but exists in the latter training stage, its utility may not be accurate.

  3. Each valid timestep contributes one posterior-weighted sample, so longer trajectories or repeated visits to the same pattern contribute proportionally more weight.

  4. BLUR requires a Relative Utility Bootstrap stage to collect trajectories and train the relative LSTM–VaDE model, and an Absolute Utility Learning stage that again collects data and trains a new absolute LSTM–VaDE model from scratch without updating the policy.

Questions for authors
  1. How sensitive is BLUR to the weights in P(τ)? For example, in SMAC, what happens if the win coefficient is changed from 100 to 10 or 1000?

  2. How was the number of latent patterns K=8 chosen? Is there a sensitivity analysis?

  3. What would happen if a latent pattern appears only in the training stage?

  4. Why does SASR perform poorly in SMAC? Would the authors compare with more baselines, such as the credit assignment methods from single-agent RL (such as RUDDER)?

Reviewer_8WipOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes a method for learning a dense reward function from trajectory-level supervision. The key idea is to learn history representations by regressing the trajectory-level signal on pooled representations, cluster them, aggregate trajectory-level returns over clusters, and then distribute it to each history according to soft clustering weights. This idea is applied over three stages: first, a reward model is fitted on “normalized” trajectory-level returns, then a second reward model is fit on an “unnormalized” variant, and the latter is used as a reward shaping term for policy learning.

The idea is applied to both single- and multi-agent RL across two benchmark. It approaches the performance of handcrafted reward shaping, and significantly outperform a single baseline. The authors additionally ablate the method’s components and the training scheme.

Critical strengths
  • The method is simple and presented in a clear and transparent way; the authors do not overclaim their contributions.
Critical weaknesses
  • The main issue is, in my opinion, the absence of two baselines. The first one is simply the optimization of the sparse reward alone, i.e. setting γabs=0\gamma_{abs}=0. The second would be decomposing episodic rewards, for instance by computing r(ht’)=P((ot,at)0t’−1)−P((ot,at)0t’−2)r(h_t’) = P((o_t, a_t)_0^{t’-1}) - P((o_t, a_t)_0^{t’-2}).
  • Some algorithmic component are environment dependent (line 161, 300). As there are only two domains, the generality of the method is not well supported. I would suggest the authors to either clarify why these differences are needed, or extend the evaluation to demonstrate some sort of generality in the implementation of BLUR.
  • The performance of SASR is almost zero across the evaluation, which raises questions about its implementation. Is this performance expected? How can it be explained?

Minor:

  • MARL’s introduction in section 3 is abrupt, it’s perhaps woth mentioning it in the introduction.
  • PabsP_{abs} is not introduced on line 160.
  • \citet is often used instead of \citep
Questions for authors
  • What is the reasoning for excluding a baseline decomposing episodic rewards?
  • Are the environment-specific implementation choices necessary?
  • The paragraphs on Relative Utility Bootstrap and Absolute Learning utility seem to be redundant. The latter seems to simply set bn=0b_n=0. What other differences are there? Can the presentation be simplified accordingly?
  • The authors state that “Because no significance tests are performed […] SASR’s zero-valued results alone do not establish general superiority over automatic reward-learning methods.” Does this mean that the comparison with SASR should not be considered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.