acceptodds
Under review as a conference paper at ICLR 2027

SARC: Interpretable State-Adaptive Reward Composition for Sparse Reward Reinforcement Learning

Abstract

Sparse-reward reinforcement learning suffers from severe credit assignment challenges, as agents often receive no informative feedback within an episode. Reward decomposition and shaping methods alleviate this problem but often introduce redundant information or reward hacking. Recent latent-reward approaches provide structured, multi-dimensional signals, but their composition does not necessarily expose how individual factors influence policy behavior across states. We propose State-Adaptive Reward Composition (SARC), which composes the action reward as a linear combination of the environment reward and multi-dimensional latent rewards. The composition weights are produced by a state-dependent intention function trained through bilevel optimization: the upper level optimizes the intention function to maximize environment return, while the lower level optimizes the policy to maximize the composed return. The learned intention weights explicitly and interpretably reveal the agent's task priorities at each state. We further introduce stabilization techniques for this bilevel training. Across eight sparse-reward continuous-control tasks, SARC achieves strong performance against competitive baselines. Intervention experiments show that the learned weights causally steer behavior, while ablations characterize the contributions of the main components.

Primary area:
reinforcement learning
Keywords:
Sparse-Reward Reinforcement Learning, Credit Assignment, Reward Composition, Latent Rewards, Bilevel Optimization, Interpretable Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bgmdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SARC, a state adaptive reward composition method for sparse reward RL. An LLM first generates several interpretable per-step latent reward functions, and SARC then learns state dependent weights over these rewards through bilevel optimization. The policy is trained with the composed reward, while the intention function is optimized using the original environment reward. Experiments show that SARC generally outperforms standard RL, intrinsic motivation, and LLM based reward shaping baselines. Intervention studies show that the learned weights influence policy behavior, and ablations support the proposed stabilization mechanisms.

Critical strengths
  1. The paper introduces an intuitive and interpretable reward composition framework. The state dependent weights provide a clear interface between task priorities and policy behavior, and intervention experiments give evidence that the policy actually responds to these weights.
  2. The empirical evaluation is broad. And the ablations and sensitivity studies provide useful evidence for the importance of the proposed components.
Critical weaknesses
  1. The core bilevel state-dependent reward weighting is closely related to [1]. The paper does not clearly separate its new algorithmic contribution from this prior work.
  2. The proposed Q-value approximation under dynamic intention weights is not fully validated. The theory bounds part of the bias, but the actual gap to the true dynamic return is not measured.

[1] Learning to utilize shaping rewards: A new approach of reward shaping.

Questions for authors
  1. What is the main algorithmic difference between SARC and [1] (above)? [1] already support state or action dependent shaping weights, multiple shaping rewards, and policy conditioning on the weights. Can the authors include a direct [1] style baseline under the same experimental setting and training budget?
  2. Is the improvement mainly due to the LLM generated reward basis, the state adaptive weights, or the stabilization mechanisms? The current ablations study these components separately, but it remains unclear how much each component contributes relative to the core bilevel reward weighting framework.
  3. The same task and prompt may lead to different valid reward functions. How does SARC handle this variability? In particular, how stable are the learned intention weights when the LLM, prompt, or sampling seed changes?
  4. If the LLM generates a misleading, redundant, or poorly scaled reward dimension, can the upper level optimization suppress its effect reliably? Could such a dimension still affect policy learning or the interpretation of the learned intention weights?
  5. Is the proposed decomposed Q value intended to be an exact Q function for the dynamic composed reward, or only a policy learning surrogate? The appendix bounds the freezing and discounting discrepancy, but does not quantify the remaining critic and bootstrapping error. What conditions are needed for this approximation to be reliable?

I would be happy to reconsider my score upward if the authors can clarify these points.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bgmd (1/2)

Thank you for the detailed and constructive review. We greatly appreciate your recognition of the interpretable composition framework and the intervention experiments. Revisions are marked in blue in the updated PDF. We address each concern below.

Q1. How does SARC differ from BiPaRS, and how does BiPaRS perform in the same setting?

Since BiPaRS [1], bilevel reward optimization has adapted how shaping rewards are used according to the policy's task performance. SARC builds on this idea, but uses an LLM to construct multi-dimensional latent rewards with explicit semantics, and models the weights as a simplex, state-dependent intention over those dimensions rather than as unconstrained coefficients on shaping rewards. This makes reward priorities interpretable and the weights easier to explore. Differences in optimization and stabilization are listed in Table A11.

The performance of BiPaRS. We reproduce BiPaRS-PPO following the MuJoCo implementation of BiPaRS and build BiPaRS-TD3 on SARC's TD3 skeleton on the six tasks. Both use SARC's latent rewards and interaction budget, and we report the Score of Figure 4. SARC substantially outperforms both. Figure A11 (BiPaRS-TD3) shows the reason. The shaping weights grow by one to three orders of magnitude and can change sign without a constraint. A factor meant to shape behaviour can thus turn into a penalty, and the shaped return ends up on a very different scale from the sparse environment reward. BARFI and MORSE score 0.00 and 0.08 on the same rewards and budget (Table A18).

MethodMountainCarPendulumAcrobotHalfCheetahAntHumanoidScore
BiPaRS-PPO1.0±0.05.0±0.823.0±0.628.0±34.00.1±0.10.0±0.00.21
BiPaRS-TD30.0±0.00.0±0.015.8±9.9447.0±235.00.8±0.60.8±0.50.09
SARC1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74

BiPaRS is Table A12.

Q2. Where does SARC's improvement come from?

We use two progressive ladders on the same six tasks, five seeds, and training budget. The first separates the LLM-generated reward functions from how they are composed (sparse reward, uniform weights, a learned state-independent intention, tuned static weights, SARC's frozen mean intention, and full SARC). The second starts from BiPaRS-TD3 and adds simplex weights with zero compensation, component-wise critics, and DG, CG, and CE (dual-gamma discounting, correlation guidance, and consistent exploration) in turn.

The results identify three contributions (Table 2). Uniform composition already improves over the sparse reward, so the LLM-generated reward functions are a useful prior. The learned state-independent intention keeps CG and CE and scores 0.61 against SARC's 0.74, so the intention needs to depend on the state. Adding simplex geometry, component-wise critics, and the stabilization mechanisms to BiPaRS-TD3 each removes a different obstacle: unconstrained weights drowning the sparse signal, moving weights interfering with value estimation, and unstable optimization on the harder tasks.

VariantMountainCarPendulumAcrobotHalfCheetahAntHumanoidScore
How the reward functions are weightedNone (sparse reward only)0.5±0.410.0±3.00.0±0.0836.0±88.0537.0±61.0121.0±148.00.36
Fixed: uniform0.4±0.517.7±0.428.8±0.6981.0±1.0322.0±394.0920.0±45.00.52
Learned: state-independent0.8±0.418.1±0.923.0±11.5788.0±394.0709.0±178.0893.0±51.00.61
Fixed: tuned1.0±0.015.7±0.422.8±11.5985.0±0.0555.0±242.0814.0±322.00.63
Fixed: SARC's mean intention1.0±0.016.9±1.424.6±12.4983.0±1.0501.0±61.0941.0±52.00.65
State-adaptive (SARC)1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74
From BiPaRS to SARCBiPaRS-TD30.0±0.00.0±0.015.8±9.9447.0±235.00.8±0.60.8±0.50.09
+ simplex weights, zero compensation0.0±0.012.0±6.126.8±0.9929.0±57.0354.0±157.0124.0±67.00.30
+ component-wise critics0.8±0.419.3±0.328.4±0.9984.0±1.0720.0±241.0182.0±134.00.54
+ DG, CG, CE (SARC)1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74

Table 2.

Q3–Q5 follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer bgmd (2/2)

We continue with the remaining questions.

Q3-4. How robust is SARC to variations and flaws in the LLM-generated reward functions?

What matters is not that two generators produce identical code, but that a reasonable set of reward functions remains usable and a misleading dimension can be suppressed. The encoder experiments test the first; the corrupted-reward experiments test the second.

LLM-based reward generation. LLMs are effective at identifying task-relevant quantities and writing them as reward functions that can score a sub-goal of the task; assigning a numerical weight to each sub-goal remains difficult. Following LaRe [2], we separate these responsibilities: the LLM writes the reward dimensions, while SARC learns their state-dependent importance. Each factor is independently computed, normalized, and range-checked before training, with LaRe's pre-verification to filter implementation errors.

Stability under reasonable reward functions. When the generated functions cover the core quantities of the task, the intention absorbs the remaining variation across generators. We regenerate the reward functions for the four classic tasks by varying the generator LLM, prompt, and sampling seed. Across four generator LLMs, two prompts, and three samples, performance fluctuates by at most about 1212\\% from the paper encoder (Appendix B.3, Table A4).

A misleading, redundant, or poorly scaled dimension. When a dimension is misleading or poorly scaled, the question is whether the upper-level optimization can suppress it. We introduce four types of imperfections: a misleading dimension, only the misleading dimension, a duplicated dimension, and a tenfold scale mismatch. An appended misleading dimension is suppressed once successes exist, because the upper level is scored by the environment reward alone (Figure A1); redundant copies are ignored. SARC cannot recover missing core information, and a severely mismatched scale cannot be suppressed either, because the simplex bounds the weights but not the magnitudes. ScaleUp bypasses the per-factor normalization and range check used before training, so this magnitude failure does not arise in the normal pipeline. An appended misleading dimension also does not change how the remaining intention weights should be read (Appendix F.6).

TaskVariantCleanMisAppMisOnlyDupScaleUp
MountainCarSARC1.0±0.00.8±0.40.0±0.01.0±0.00.0±0.0
Uniform0.4±0.50.0±0.00.0±0.0--
ϕmis\phi_{\mathrm{mis}}-0.150.67--
PendulumSARC20.5±0.419.3±1.213.1±6.719.5±0.216.2±2.7
Uniform17.7±0.411.0±5.10.1±0.1--
ϕmis\phi_{\mathrm{mis}}-0.100.31--
AcrobotSARC31.1±1.228.3±0.30.0±0.030.1±1.311.0±13.4
Uniform28.8±0.67.2±8.80.0±0.0--
ϕmis\phi_{\mathrm{mis}}-0.000.14--

Table A5: return and mean intention ϕmis\phi_{\mathrm{mis}}.

Non-uniqueness and interpretability of intentions. Different seeds need not converge to identical intention weights, particularly when correlated reward dimensions or alternative compositions support similarly effective behaviours. For example, both position progress and speed can contribute to successful hill climbing in MountainCar. Table A3 reports the visited-state mean intention of the paper encoder across training seeds. Thus, numerical agreement is not required for interpretability. Instead, we assess whether the learned priorities reflect meaningful task structure and produce corresponding behavioural changes under intervention (Figure 3, Appendix F).

Q5. Is the decomposed Q-value exact or a surrogate, and under what conditions can it be trusted?

Nature of the approximation. Our decomposed Q-value is a policy-learning surrogate rather than an exact Q function of the dynamically composed reward. SARC introduces state-dependent weights and different discount factors, making the linear decomposition approximate. Appendix A (Theorem 1 and Eq. A.8) characterizes the freeze of the intention at the query state and the discount mismatch.

Conditions for reliable action selection. The surrogate is reliable when the intention drifts little over the lower-level horizon, which Theorem 1 bounds and dual-gamma discounting enforces. Whether the remaining critic and bootstrapping error is small is task-dependent (Appendix A.2). None of these discrepancies enters the upper-level objective, which scores intentions with QrealQ_{\mathrm{real}} alone.

We thank the reviewer again for the careful feedback. If these responses address the concerns raised, we would appreciate your reconsideration of the score.

References

[1] Hu et al., NeurIPS 2020. [2] Qu et al., AAAI 2025.

Thank you for the detailed clarifications and additional comparisons, which address most of my concerns. Could you clarify whether dual gamma discounting guarantees small intention drift or simply reduces the influence of future intention changes on the approximation? Also, although the upper level objective uses only QrealQ_{\mathrm{real}}, could surrogate errors still affect upper level optimization indirectly through the learned policy and its data distribution?

Reviewer_Ka7tOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SARC learns state-dependent weights over interpretable auxiliary rewards, such as progress, speed, and stability. These weights both guide policy training and serve as inputs to the policy. The method uses separate critics for each reward component and introduces three mechanisms to stabilize learning. Experiments on eight continuous-control tasks with binary rewards show strong performance across five seeds. Ablations and interventions examine the contribution of the components and whether changing the weights produces the expected behavioral changes.

Critical strengths

The paper is well written, and the figures make the idea easy to follow. I also liked the experiments that change the learned weights and check how the agent responds. These help show whether the weights actually control the behaviors their labels suggest.

SARC performs consistently well across the eight tasks, including both classic control and higher-dimensional locomotion. The results across multiple seeds and the component ablations support the practical value of the overall approach.

Critical weaknesses

The experiments do not adequately isolate the benefit of learning state-dependent reward weights. The closest adaptive reward methods discussed in related work (BiPaRS, BARFI, and MORSE) are also absent from the comparisons. There is also no comparison with separately trained, tuned fixed-weight mixtures using the same reward components. Freezing the weights of an already-trained SARC policy tests its dependence on that input, but does not establish an advantage over training with fixed weights.

I am unsure whether the learned reward weights could favor behavior that is suboptimal for the original task. The zero-reward component lets SARC ignore the auxiliary rewards, but it is unclear when training will learn to do so. The experiments show good returns, but do not directly test what could happen for instance when an auxiliary reward conflicts with the task objective.

Questions for authors

Does the zero-reward component provide any guarantee that SARC can recover the original optimal policy, or does it only make this possible in principle? What can be said about whether the practical updates learn to ignore misleading auxiliary rewards?

Do you have results for separately trained, tuned fixed-weight mixtures or the closest adaptive reward methods, using the same reward components? Freezing the weights after training seems to answer a different question.

How often does each task give a nonzero reward under random exploration and early in training? Since most tasks reward every successful timestep, I would like to understand how sparse the feedback actually is.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ka7t (1/2)

Thank you for the detailed and constructive review. We greatly appreciate your recognition of the intervention experiments and SARC's performance. Revisions are marked in blue in the updated PDF. We address each concern below.

Q1. Does the zero-reward component recover the original optimum, and can training ignore misleading auxiliary rewards?

Does the zero-reward component recover the original optimum? The zero-reward component puts the original MDP inside the composed family. All mass on that coordinate recovers the environment return (Proposition 2). In the ideal bilevel problem the outer objective is that return, so a global bilevel optimum cannot be worse than the original task, but only in that ideal realization.

Can training ignore a misleading auxiliary reward? The inner loop follows the composed reward. The outer loop sees only the sparse environment return, so a misleading auxiliary is down-weighted only after the first successes. We corrupt the reward functions on MountainCar, Pendulum, and Acrobot (Table A5). MisApp appends the opposite of success. MisOnly keeps only that dimension. Dup appends a copy of a core dimension and its square. ScaleUp rescales one dimension tenfold. On MisApp and MisOnly we also train uniform weights, which cannot change ϕ\phi. If SARC stays above this control while the control collapses, the suppression came from the outer loop. ScaleUp is a failure of magnitudes, and the simplex cannot bound it. It bypasses the per-factor normalization and range check used before training, so this magnitude failure does not arise in the normal pipeline. When the core signals remain, the misleading weight falls once successes exist (Figure A1), and SARC stays above the uniform control. On Pendulum and Acrobot it also keeps most of the clean return. The protection ends at MisOnly and ScaleUp (Appendix B.4).

TaskCleanMisAppMisOnlyDupScaleUp
MountainCarSARC1.0±0.00.8±0.40.0±0.01.0±0.00.0±0.0
Uniform0.4±0.50.0±0.00.0±0.0--
ϕmis\phi_{\mathrm{mis}}-0.150.67--
PendulumSARC20.5±0.419.3±1.213.1±6.719.5±0.216.2±2.7
Uniform17.7±0.411.0±5.10.1±0.1--
ϕmis\phi_{\mathrm{mis}}-0.100.31--
AcrobotSARC31.1±1.228.3±0.30.0±0.030.1±1.311.0±13.4
Uniform28.8±0.67.2±8.80.0±0.0--
ϕmis\phi_{\mathrm{mis}}-0.000.14--

Table A5: return and mean intention ϕmis\phi_{\mathrm{mis}}.

Q2. Do state-dependent weights outperform separately trained fixed weights and the closest adaptive reward methods?

Separately trained fixed weights. Table 2 (top) uses the same latent rewards, backbone, and budget, and reports the Score of Figure 4. Learned: state-independent is a state-independent intention trained from scratch. Fixed: tuned uses ten static weight vectors proposed by an LLM. We screen one vector per task on three held-out seeds and train from scratch with ϕ\phi frozen. Fixed: SARC's mean intention freezes the mean intention of a trained SARC run, which is the freeze-after-training control. All three trail State-adaptive (SARC), so state-adaptivity is better.

VariantMountainCarPendulumAcrobotHalfCheetahAntHumanoidScore
None (sparse reward only)0.5±0.410.0±3.00.0±0.0836.0±88.0537.0±61.0121.0±148.00.36
Fixed: uniform0.4±0.517.7±0.428.8±0.6981.0±1.0322.0±394.0920.0±45.00.52
Learned: state-independent0.8±0.418.1±0.923.0±11.5788.0±394.0709.0±178.0893.0±51.00.61
Fixed: tuned1.0±0.015.7±0.422.8±11.5985.0±0.0555.0±242.0814.0±322.00.63
Fixed: SARC's mean intention1.0±0.016.9±1.424.6±12.4983.0±1.0501.0±61.0941.0±52.00.65
State-adaptive (SARC)1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74

Table 2, top.

The comparison with BiPaRS, BARFI, and MORSE, and Q3, follow in the next comment.

AuthorsOpenReview ↗

Response to Reviewer Ka7t (2/2)

We continue with the remaining questions.

Closest adaptive reward methods. On the same latent rewards and budget we report BiPaRS-PPO [1], BiPaRS-TD3, BARFI [2], and MORSE [3] (Tables A12 and A18). BiPaRS lets shaping weights grow without a bound and change sign, so a helpful term can become a penalty (Figure A11). BARFI's regularizer shuts off the auxiliary rewards before any success arrives, leaving only the sparse reward. MORSE mixes heuristic rewards without a simplex bound, so a latent term can drown the sparse reward. None approaches SARC (Appendix G.3).

MethodMountainCarPendulumAcrobotHalfCheetahAntHumanoidScore
BiPaRS-PPO1.0±0.05.0±0.823.0±0.628.0±34.00.1±0.10.0±0.00.21
BiPaRS-TD30.0±0.00.0±0.015.8±9.9447.0±235.00.8±0.60.8±0.50.09
BARFI0.0±0.00.1±0.00.0±0.00.0±0.00.0±0.00.1±0.00.00
MORSE0.3±0.40.1±0.026.1±0.80.0±0.01.0±0.60.5±0.20.08
SARC1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74

BiPaRS is Table A12. BARFI and MORSE are Table A18.

Q3. How sparse is the reward under random exploration and early in training?

Sparsity for learning is whether undirected behaviour ever produces the signal, not whether the reward is granted per step. A random policy never obtains it on Acrobot, HalfCheetah, and Humanoid. Ant is the instructive case: more than a third of random episodes contain a rewarded step, yet those steps are isolated, about one in a thousand, and do not teach a gait. Early in training the first reward arrives after thousands to tens of thousands of steps, and becomes frequent only once the behaviour is already largely correct.

TaskRandom episodesRandom stepsFirst 10% episodesFirst reward (step)
Simple14/3000.23%0.22 [0.05, 0.34]75–3.1k
MountainCar1/300<0.001%0.24 [0.05, 0.58]9.8k–13k
Pendulum15/3000.37%0.06 [0.03, 0.08]350–5.2k
Acrobot0/30000.23 [0.00, 0.47]19k–31k
Hopper9/3000.03%0.57 [0.45, 0.64]1k–13k
HalfCheetah0/30000.00 [0.00, 0.01]≥100k
Ant111/3000.10%0.41 [0.36, 0.45]1k–2k
Humanoid0/30000.20 [0.00, 0.37]4.2k–6.3k

Table A9.

We thank the reviewer again for the careful feedback. If these responses address the concerns raised, we would appreciate your reconsideration of the score.

References

[1] Hu et al., NeurIPS 2020. [2] Gupta et al., NeurIPS 2023. [3] Xie et al., arXiv:2512.15120, 2025.

Based on the additional results described in the response, my main concerns appear substantially addressed, and I am inclined to raise my recommendation to weak acceptance. My remaining reservation is whether the numerical comparisons show consistent, meaningful gains relative to seed variability, particularly against the tuned fixed-weight alternatives.

Reviewer_tJnjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SARC learns state-dependent weights over named auxiliary rewards and uses those weights both to combine value estimates and to condition the policy. The weights are trained using environment returns and a correlation signal. The method also uses separate discounts and episode-level exploration in the weight space. Experiments on eight continuous-control tasks test performance, component ablations, and how changing the weights affects trained policies.

Critical strengths
  1. The weight interface can be tested directly. The interventions change the policy's intention input while holding network parameters and initial states fixed. They test whether behavior changes and whether the changes follow the reward labels. This goes beyond displaying weights as an explanation, and the appendix acknowledges dimensions that cannot be steered independently (pages 27–30).

  2. The full system performs well across the reported tasks. Table 1 gives mean returns of 910.8 on Ant and 953.3 on Humanoid, with results over five training seeds (page 7). Reward definitions, baseline settings, and extensive ablations make the empirical results easier to assess.

Critical weaknesses
  1. The benefit of state-adaptive composition is not sufficiently isolated. The paper cites related adaptive shaping methods but does not compare against them. Fixing a trained policy's weights to a single reward tests its response to that intervention, not whether separately trained fixed-weight policies perform worse; the appendix correctly notes this limitation. The LaRe comparison uses the same reward inputs but also changes exploration, policy conditioning, and value learning. A matched comparison is needed to identify what the proposed composition adds (pages 3, 25, and 30).

  2. The stated bilevel objective and practical algorithm are not fully aligned. The upper update uses the current actor, without showing how it accounts for the actor's response to new reward weights. The actor also optimizes an approximate score with two discounts. In addition, Algorithm 1 delays both actor and intention updates, while Appendix D.3 says only intention updates are delayed. These differences make it unclear which algorithm produced the results (pages 4–6 and 23–24).

Questions for authors
  1. Can you compare against a learned fixed-weight mixture or a related adaptive shaping method using the same rewards, backbone, and interaction budget?

  2. How do the practical updates approximate the stated bilevel objective? Which actor/intention update schedule and target-smoothing procedure were actually used in the reported experiments?

2 comments
AuthorsOpenReview ↗

Response to Reviewer tJnj (1/2)

Thank you for the detailed and constructive review. We greatly appreciate your recognition of the intervention experiments and the empirical evaluation. Revisions are marked in blue in the updated PDF. We address each concern below.

Q1. How does SARC compare with a learned state-independent intention and with related adaptive shaping, under the same setting?

We keep SARC's latent rewards, backbone, and interaction budget, and report the Score of Figure 4 (Table 2).

State-independent intention and fixed weights. Learned: state-independent is a state-independent intention trained from scratch. Fixed: tuned uses ten static weight vectors proposed by an LLM. We screen one vector per task on three held-out seeds and train from scratch with ϕ\phi frozen. Fixed: SARC's mean intention freezes the mean intention of a trained SARC run, which is the freeze-after-training control. All three trail State-adaptive (SARC), so state-adaptivity is better.

VariantMountainCarPendulumAcrobotHalfCheetahAntHumanoidScore
None (sparse reward only)0.5±0.410.0±3.00.0±0.0836.0±88.0537.0±61.0121.0±148.00.36
Fixed: uniform0.4±0.517.7±0.428.8±0.6981.0±1.0322.0±394.0920.0±45.00.52
Learned: state-independent0.8±0.418.1±0.923.0±11.5788.0±394.0709.0±178.0893.0±51.00.61
Fixed: tuned1.0±0.015.7±0.422.8±11.5985.0±0.0555.0±242.0814.0±322.00.63
Fixed: SARC's mean intention1.0±0.016.9±1.424.6±12.4983.0±1.0501.0±61.0941.0±52.00.65
State-adaptive (SARC)1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74

Table 2, top.

Related adaptive shaping: what the composition adds. We start from BiPaRS-TD3 on that same skeleton and add simplex weights with zero compensation, component-wise critics, and DG, CG, and CE (dual-gamma discounting, correlation guidance, and consistent exploration) in turn (Table 2, bottom). Holding this baseline fixed is the matched comparison that isolates what the composition adds. Each step removes a different obstacle: unconstrained weights drowning the sparse signal, moving weights interfering with value estimation, and unstable optimization on the harder tasks. The unconstrained weights of BiPaRS-TD3 grow by orders of magnitude and change sign (Figure A11). BiPaRS-PPO (0.21, Table A12), BARFI (0.00), and MORSE (0.08, Table A18) use the same rewards and budget. None approaches SARC (0.74).

VariantMountainCarPendulumAcrobotHalfCheetahAntHumanoidScore
BiPaRS-TD30.0±0.00.0±0.015.8±9.9447.0±235.00.8±0.60.8±0.50.09
+ simplex weights, zero compensation0.0±0.012.0±6.126.8±0.9929.0±57.0354.0±157.0124.0±67.00.30
+ component-wise critics0.8±0.419.3±0.328.4±0.9984.0±1.0720.0±241.0182.0±134.00.54
+ DG, CG, CE (SARC)1.0±0.020.5±0.431.1±1.2983.4±0.4910.8±51.3953.3±6.10.74

Table 2, bottom.

Table 1's LaRe comparison [1] is our TD3 adaptation. It uses the same latent rewards but also changes exploration, policy conditioning, and value learning. We therefore put the LaRe decoder on SARC's backbone (LaRe-M). It shares SARC's critics, budget, and latent rewards. The intention is replaced by the decoder and the actor receives a zero intention. LaRe-M has no intention weights to explore. We compare it with SARC w/o CE (Table A16), which uses per-step rather than episode-consistent intention noise (Appendix G.1). LaRe-M scores 0.37 against 0.65 for SARC w/o CE. Before the first successes the decoder's return-decomposition target is zero. The intention composes named quantities from the first step.

Q2 follows in the next comment.

References

[1] Qu et al., AAAI 2025.

AuthorsOpenReview ↗

Response to Reviewer tJnj (2/2)

We continue with the remaining question.

Q2. How do the practical updates approximate the bilevel objective, and which schedule was used?

The actor's response to new reward weights. The upper update uses the current actor. This is the standard first-order device of BiPaRS [1]. The actor is conditioned on the weights, so differentiating QrealQ_{\mathrm{real}} through that input accounts for that response without unrolling the inner optimization. The actor is trained on stored behaviour intentions, as a goal-conditioned policy is trained on stored goals, and therefore represents the response to a family of intentions. The same conditioning can be checked at run time. Overwriting only the intention input of a frozen actor already changes behaviour on all eight tasks (Figure 3c and 3d).

Approximate score with two discounts. The actor optimizes an approximate score with two discounts, a surrogate rather than an exact value of the dynamically composed reward. Under fixed weights and a common discount, linear decomposition follows from successor features [2]. State-dependent weights and dual-gamma discounting make it approximate (Appendix A, Theorem 1 and Eq. A.8). None of this discrepancy enters the upper-level objective, which scores intentions with QrealQ_{\mathrm{real}} alone.

Training clarification. Reported runs update the critics and the actor every step and delay only the intention (d=2d=2). Algorithm 1 is updated to match. Target smoothing adds N(0,σtarg2)\mathcal{N}(0,\sigma_{\mathrm{targ}}^{2}) noise, clipped, to the target intention logits of the environment-reward critic. Both settings are in Table A10. The latent critics use the unperturbed target intention.

We thank the reviewer again for the careful feedback. If these responses address the concerns raised, we would appreciate your reconsideration of the score.

References

[1] Hu et al., NeurIPS 2020. [2] Barreto et al., NeurIPS 2017.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary Comment

Dear Area Chair,

We thank you and the reviewers for the constructive feedback. This work addresses a core challenge in sparse-reward reinforcement learning: how to guide policy learning with auxiliary rewards without costly manual tuning or reward hacking, while keeping decisions interpretable and steerable. Large language models excel at writing semantic reward components such as progress, speed, and stability, yet they struggle to set numerical trade-offs among them. We propose State-Adaptive Reward Composition (SARC), which pairs those LLM-generated bases with a bilevel, state-dependent intention:

  • Bilevel state-adaptive framework. The upper level scores a simplex intention ϕψ(s)\phi_\psi(s) by environment return. The lower level trains the policy with component-wise critics, dual-gamma discounting, correlation guidance, and consistent exploration.
  • Causal interface. Because ϕψ(s)\phi_\psi(s) both weights the latent critics and conditions the actor, it exposes phase-dependent priorities and can be overwritten after training. Changing only that input steers behavior toward the corresponding semantic quantity on seven of eight tasks.
  • Empirical results. SARC attains the highest or tied-highest mean sparse return on seven of eight tasks, and ranks second on the remaining one.

Reviewers recognized an "intuitive and interpretable reward composition framework" (bgmd) whose "weight interface can be tested directly... beyond displaying weights as an explanation" (tJnj), and that the paper is "well written" (Ka7t) and "performs consistently well across the eight tasks" (Ka7t, tJnj, bgmd). Their shared request was to isolate state-adaptive weighting from the LLM reward basis and from prior adaptive shaping, together with questions on imperfect LLM rewards and the decomposed-value surrogate. We added matched controls and the analyses below.

1. Major Revisions

  • State-adaptive vs. fixed weights. The same latent rewards, backbone, and budget test whether the weights must depend on the state. Uniform composition already improves over sparse reward, but even Fixed: tuned remains below SARC. Task-seed-stratified IQM is 0.84 [0.83, 0.86] for SARC against 0.69 [0.57, 0.77] for Fixed: tuned; the intervals do not overlap, so the gap is not a seed artifact.
  • Matched adaptive shaping and LaRe. The same rewards and budget isolate what the composition adds. BiPaRS-PPO [1], BiPaRS-TD3, BARFI [2], and MORSE [3] score at most 0.21: unconstrained BiPaRS weights grow and change sign, and BARFI shuts auxiliaries off before the first success. Starting from BiPaRS-TD3 (0.09), simplex weights with zero compensation bound that scale (0.30), component-wise critics stabilize value estimates as weights move (0.54), and dual-gamma discounting, correlation guidance, and consistent exploration recover SARC at 0.74. A matched LaRe decoder [4] (LaRe-M) scores 0.37 against 0.65 for SARC without consistent exploration. The gain is the composition, not the LLM basis or bilevel shaping alone.
  • Imperfect LLM rewards. Across generator LLMs, prompts, and samples, return fluctuates by at most about 1212\\%. When a misleading dimension is appended, uniform weights collapse, while SARC stays near the clean return once successes appear and the upper level down-weights that dimension.

2. Bilevel Objective and Updates

Reviewers tJnj and bgmd asked how the practical updates relate to the stated objective. The upper step uses the current actor, the first-order device of BiPaRS [1]. Because the actor is conditioned on stored intentions, differentiating QrealQ_{\mathrm{real}} through that input accounts for the response without unrolling. QcompQ_{\mathrm{comp}} is a surrogate, not an exact value of the dynamic reward; Theorem 1 bounds the freeze-and-discount gap, and dual-gamma discounting tightens that bound rather than guaranteeing small intention drift. Surrogate error can still affect the data distribution, but the upper-level criterion remains environment return.

3. Discussion

  • Reviewer Ka7t wrote that their main concerns were substantially addressed and raised the rating to Weak Acceptance. We then answered their remaining reservation on seed variability with the IQM above.
  • Reviewer bgmd wrote that the additional comparisons address most of their concerns; we then answered the follow-up on dual-gamma discounting and indirect surrogate error.
  • Reviewer tJnj did not respond during the discussion period, but we have provided answers on matched comparisons and the bilevel updates.

We thank the Area Chair for the time and consideration, and the reviewers for the constructive discussion.

Best regards,
The Authors

References

[1] Hu et al., Learning to Utilize Shaping Rewards (BiPaRS), NeurIPS 2020.

[2] Gupta et al., BARFI, NeurIPS 2023.

[3] Xie et al., MORSE, arXiv:2512.15120, 2025.

[4] Qu et al., Latent Reward (LaRe), AAAI 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.