- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SARC, a state adaptive reward composition method for sparse reward RL. An LLM first generates several interpretable per-step latent reward functions, and SARC then learns state dependent weights over these rewards through bilevel optimization. The policy is trained with the composed reward, while the intention function is optimized using the original environment reward. Experiments show that SARC generally outperforms standard RL, intrinsic motivation, and LLM based reward shaping baselines. Intervention studies show that the learned weights influence policy behavior, and ablations support the proposed stabilization mechanisms.
Critical strengths
- The paper introduces an intuitive and interpretable reward composition framework. The state dependent weights provide a clear interface between task priorities and policy behavior, and intervention experiments give evidence that the policy actually responds to these weights.
- The empirical evaluation is broad. And the ablations and sensitivity studies provide useful evidence for the importance of the proposed components.
Critical weaknesses
- The core bilevel state-dependent reward weighting is closely related to [1]. The paper does not clearly separate its new algorithmic contribution from this prior work.
- The proposed Q-value approximation under dynamic intention weights is not fully validated. The theory bounds part of the bias, but the actual gap to the true dynamic return is not measured.
[1] Learning to utilize shaping rewards: A new approach of reward shaping.
Questions for authors
- What is the main algorithmic difference between SARC and [1] (above)? [1] already support state or action dependent shaping weights, multiple shaping rewards, and policy conditioning on the weights. Can the authors include a direct [1] style baseline under the same experimental setting and training budget?
- Is the improvement mainly due to the LLM generated reward basis, the state adaptive weights, or the stabilization mechanisms? The current ablations study these components separately, but it remains unclear how much each component contributes relative to the core bilevel reward weighting framework.
- The same task and prompt may lead to different valid reward functions. How does SARC handle this variability? In particular, how stable are the learned intention weights when the LLM, prompt, or sampling seed changes?
- If the LLM generates a misleading, redundant, or poorly scaled reward dimension, can the upper level optimization suppress its effect reliably? Could such a dimension still affect policy learning or the interpretation of the learned intention weights?
- Is the proposed decomposed Q value intended to be an exact Q function for the dynamic composed reward, or only a policy learning surrogate? The appendix bounds the freezing and discounting discrepancy, but does not quantify the remaining critic and bootstrapping error. What conditions are needed for this approximation to be reliable?
I would be happy to reconsider my score upward if the authors can clarify these points.
5 comments
Response to Reviewer bgmd (2/2)
We continue with the remaining questions.
Q3-4. How robust is SARC to variations and flaws in the LLM-generated reward functions?
What matters is not that two generators produce identical code, but that a reasonable set of reward functions remains usable and a misleading dimension can be suppressed. The encoder experiments test the first; the corrupted-reward experiments test the second.
LLM-based reward generation. LLMs are effective at identifying task-relevant quantities and writing them as reward functions that can score a sub-goal of the task; assigning a numerical weight to each sub-goal remains difficult. Following LaRe [2], we separate these responsibilities: the LLM writes the reward dimensions, while SARC learns their state-dependent importance. Each factor is independently computed, normalized, and range-checked before training, with LaRe's pre-verification to filter implementation errors.
Stability under reasonable reward functions. When the generated functions cover the core quantities of the task, the intention absorbs the remaining variation across generators. We regenerate the reward functions for the four classic tasks by varying the generator LLM, prompt, and sampling seed. Across four generator LLMs, two prompts, and three samples, performance fluctuates by at most about from the paper encoder (Appendix B.3, Table A4).
A misleading, redundant, or poorly scaled dimension. When a dimension is misleading or poorly scaled, the question is whether the upper-level optimization can suppress it. We introduce four types of imperfections: a misleading dimension, only the misleading dimension, a duplicated dimension, and a tenfold scale mismatch. An appended misleading dimension is suppressed once successes exist, because the upper level is scored by the environment reward alone (Figure A1); redundant copies are ignored. SARC cannot recover missing core information, and a severely mismatched scale cannot be suppressed either, because the simplex bounds the weights but not the magnitudes. ScaleUp bypasses the per-factor normalization and range check used before training, so this magnitude failure does not arise in the normal pipeline. An appended misleading dimension also does not change how the remaining intention weights should be read (Appendix F.6).
| Task | Variant | Clean | MisApp | MisOnly | Dup | ScaleUp |
|---|---|---|---|---|---|---|
| MountainCar | SARC | 1.0±0.0 | 0.8±0.4 | 0.0±0.0 | 1.0±0.0 | 0.0±0.0 |
| Uniform | 0.4±0.5 | 0.0±0.0 | 0.0±0.0 | - | - | |
| - | 0.15 | 0.67 | - | - | ||
| Pendulum | SARC | 20.5±0.4 | 19.3±1.2 | 13.1±6.7 | 19.5±0.2 | 16.2±2.7 |
| Uniform | 17.7±0.4 | 11.0±5.1 | 0.1±0.1 | - | - | |
| - | 0.10 | 0.31 | - | - | ||
| Acrobot | SARC | 31.1±1.2 | 28.3±0.3 | 0.0±0.0 | 30.1±1.3 | 11.0±13.4 |
| Uniform | 28.8±0.6 | 7.2±8.8 | 0.0±0.0 | - | - | |
| - | 0.00 | 0.14 | - | - |
Table A5: return and mean intention .
Non-uniqueness and interpretability of intentions. Different seeds need not converge to identical intention weights, particularly when correlated reward dimensions or alternative compositions support similarly effective behaviours. For example, both position progress and speed can contribute to successful hill climbing in MountainCar. Table A3 reports the visited-state mean intention of the paper encoder across training seeds. Thus, numerical agreement is not required for interpretability. Instead, we assess whether the learned priorities reflect meaningful task structure and produce corresponding behavioural changes under intervention (Figure 3, Appendix F).
Q5. Is the decomposed Q-value exact or a surrogate, and under what conditions can it be trusted?
Nature of the approximation. Our decomposed Q-value is a policy-learning surrogate rather than an exact Q function of the dynamically composed reward. SARC introduces state-dependent weights and different discount factors, making the linear decomposition approximate. Appendix A (Theorem 1 and Eq. A.8) characterizes the freeze of the intention at the query state and the discount mismatch.
Conditions for reliable action selection. The surrogate is reliable when the intention drifts little over the lower-level horizon, which Theorem 1 bounds and dual-gamma discounting enforces. Whether the remaining critic and bootstrapping error is small is task-dependent (Appendix A.2). None of these discrepancies enters the upper-level objective, which scores intentions with alone.
We thank the reviewer again for the careful feedback. If these responses address the concerns raised, we would appreciate your reconsideration of the score.
References
[1] Hu et al., NeurIPS 2020. [2] Qu et al., AAAI 2025.
Thank you for the detailed clarifications and additional comparisons, which address most of my concerns. Could you clarify whether dual gamma discounting guarantees small intention drift or simply reduces the influence of future intention changes on the approximation? Also, although the upper level objective uses only , could surrogate errors still affect upper level optimization indirectly through the learned policy and its data distribution?
Response to Reviewer bgmd (1/2)
Thank you for the detailed and constructive review. We greatly appreciate your recognition of the interpretable composition framework and the intervention experiments. Revisions are marked in blue in the updated PDF. We address each concern below.
Q1. How does SARC differ from BiPaRS, and how does BiPaRS perform in the same setting?
Since BiPaRS [1], bilevel reward optimization has adapted how shaping rewards are used according to the policy's task performance. SARC builds on this idea, but uses an LLM to construct multi-dimensional latent rewards with explicit semantics, and models the weights as a simplex, state-dependent intention over those dimensions rather than as unconstrained coefficients on shaping rewards. This makes reward priorities interpretable and the weights easier to explore. Differences in optimization and stabilization are listed in Table A11.
The performance of BiPaRS. We reproduce BiPaRS-PPO following the MuJoCo implementation of BiPaRS and build BiPaRS-TD3 on SARC's TD3 skeleton on the six tasks. Both use SARC's latent rewards and interaction budget, and we report the Score of Figure 4. SARC substantially outperforms both. Figure A11 (BiPaRS-TD3) shows the reason. The shaping weights grow by one to three orders of magnitude and can change sign without a constraint. A factor meant to shape behaviour can thus turn into a penalty, and the shaped return ends up on a very different scale from the sparse environment reward. BARFI and MORSE score 0.00 and 0.08 on the same rewards and budget (Table A18).
BiPaRS is Table A12.
Q2. Where does SARC's improvement come from?
We use two progressive ladders on the same six tasks, five seeds, and training budget. The first separates the LLM-generated reward functions from how they are composed (sparse reward, uniform weights, a learned state-independent intention, tuned static weights, SARC's frozen mean intention, and full SARC). The second starts from BiPaRS-TD3 and adds simplex weights with zero compensation, component-wise critics, and DG, CG, and CE (dual-gamma discounting, correlation guidance, and consistent exploration) in turn.
The results identify three contributions (Table 2). Uniform composition already improves over the sparse reward, so the LLM-generated reward functions are a useful prior. The learned state-independent intention keeps CG and CE and scores 0.61 against SARC's 0.74, so the intention needs to depend on the state. Adding simplex geometry, component-wise critics, and the stabilization mechanisms to BiPaRS-TD3 each removes a different obstacle: unconstrained weights drowning the sparse signal, moving weights interfering with value estimation, and unstable optimization on the harder tasks.
Table 2.
Q3–Q5 follow in the next comment.