- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies for RL agent training, how to acquire fundamental, universally applicable skills by observing demonstrations that stem from different goals—or even involve conflicting behaviors?
For instance, in a single game, some players might focus on gathering wood, while others prioritize collecting stone. If one were to simply imitate all these actions indiscriminately, the resulting strategy might fail to achieve any of the intended goals effectively. GRID aims to first learn general-purpose capabilities, which can then be adapted to suit specific objectives.
The hypothesis it follows is: While individuals may pursue different objectives, they often share common fundamental capabilities. Useful, shared reward structures do, in fact, exist across different tasks.
I feel that is a reaosnable hypothesis, not very strong.
Critical strengths
S1: empricial results are strong - GRID-L2 Hopper: 2997.6 (best baseline: 2099.1); GRID-L1 Highway: 13.41 (aggregated AIRL: 12.62).
S2: empirical results are tracable - the experimental setup and relevant baselines have been made public, including those with poorer performance (reviewer highly appreciate that)
Critical weaknesses
W1: Evidence are missing for these labeled "shared rewards" learned truly represent shared behaviors. Plz add
- discussion if the shared rewards learned are truly common objectives, or are they merely an illusion created by numerical variations in the reward models?
- normalizing numerical ranges does not equate to aligning meanings. Scaling values to the interval merely ensures that the ranges are identical; it provides no guarantee that the rewards associated with a given state-action pair represent comparable underlying objectives. How to justify it is?
- the fact that there are few non-zero entries overall can not guarantee that a common component can be identified for every specific state. Under what condition it exists udner waht it not?
- verify that the decomposition is indeed correct - by comparing the rewards for various tasks under identical states and actions, using an environment with a known shared reward component, one can test the complete pipeline—"Demonstrations → AIRL → GRID"—to see whether the known shared component can be successfully recovered. Simply decomposing a manually constructed reward function only allows for partial verification.
W2: performance lifting is hard to be attributed solely to "decomposing rewards into shared and individual components" because 3 other factors bundled:
- reward scaling: standardizing the reward range may lead to more stable training.
- k-NN support penalty: encourages the agent to remain within regions covered by demonstrations, which could improve performance independently of reward decomposition.
- larger network architecture: The 4×1024 configuration (four layers with 1,024 neurons each) offers significantly greater capacity than the 1×32 configuration (a single layer with 32 neurons), which could also influence the results. As a result, plz provide control exps:
- Removing the support penalty: to determine how much improvement is contributed from the reward decomposition itself.
- Applying the same scaling and support penalty to the baseline: to assess the remaining performance gap when both methods utilize the same training setup.
- Training with only the support signal: ablating reward decomposition and focusing solely on encouraging proximity to the demonstration region, to see what level of performance can be achieved.
W3: unclear to me if the information bottleneck theory really support the loss function employed in the paper? I mean the loss function may be practical, but the theoretical justification for why it can isolatea "shared" reward remains uncear to me. Plz
- add more explantion and justifciaon to that point
One could either explicitly incorporate a noise model and re-derive the KL divergence, or reframe the approach as a "reward regression decomposition with regularization." Thus it needs a comparative study to clarify what additional value the proposed method actually provides.
- evaluating the full method against baselines such as removing identity terms, simple averaging, or taking the simple median
W4: clarify compared to MRSD with shared reward, task-speifi reward, L2 regularziation, what are the novel things from the paper here? If it is combo, then more comprehesnvie abaltion study is needed to asees the contribution of indvidual modificaiton/ combo decisions.
W5: The claim of faster adaptation lacks sufficient evidence. To demonstrate faster learning, it must compare performance trajectories as a function of training steps—ideally against a baseline trained from scratch. A high final score does not necessarily equate to a fast learning rate.
Questions for authors
All qs to followup and action itesm listed for each weakness point above, plz read and follow up them
2 comments
Supplementary Response
(2/2) Remaining questions, setup, and details.
W3: Theoretical justification of the loss.
Why isolates shared rewards. depends only on , making it task-independent. Reconstruction requires , while compression penalizes the magnitude of . Under exact reconstruction, . Thus, minimizing L1 compression gives
while squared L2 compression gives
These characterize the inductive bias: L1 favors the majority-supported reward, while L2 captures the mean structure. If , exact recovery occurs when task-specific deviations have zero median (L1) or zero mean (L2) at each . This analysis describes an idealized setting (equal task weighting, sufficient expressivity, exact reconstruction, no identity term) and explains each variant's inductive bias; the full model builds on it with the identity loss and soft reconstruction. In Highway-Env and MuJoCo, each task reward is evaluated only on its own , so the rewards are not compared at the same . Thus, the pointwise result does not apply directly. This agrees with our results: with kNN support, GRID outperforms pointwise aggregation. The identity loss also keeps task-discriminative.
Noise model. Following the reviewer's suggestion, consider an auxiliary stochastic encoder , where and is fixed. Then , where the first term is the negative entropy and does not depend on . For a Gaussian prior , and , so
which yields squared L2 compression exactly. For a Laplace prior ,
By Jensen's and the triangle inequalities,
Thus, the -dependent KL term approaches uniformly as . GRID directly regularizes deterministic specific-reward outputs; the stochastic formulation motivates these penalties rather than describing sampling during training. GRID can thus be viewed as regularized reward regression, with the IB view motivating compression (R2), identity (R1), and reconstruction (R3). We will clarify this derivation in the appendix.
Comparative study. With kNN support (Comment 1), GRID-L2 outperforms the mean (0.71 vs. 0.64 generalist; 0.80 vs. 0.55 fine-tune), while GRID-L1 outperforms the median (0.93 vs. 0.69; 0.58 vs. 0.50). We additionally ablate the identity and compression losses (basis functions, MLP [32,32], 5 seeds; MSE vs. ground truth):
| Variant | Total ↓ | General ↓ | Specific ↓ |
|---|---|---|---|
| Full (GRID-L1) | 0.194 ± 0.002 | 0.0013 ± 0.0002 | 0.193 ± 0.002 |
| No compression | 0.008 ± 0.008 | 1.20 ± 0.70 | 1.20 ± 0.68 |
| No identity | 0.576 ± 0.271 | 0.0011 ± 0.0002 | 0.576 ± 0.270 |
| Neither | 0.363 ± 0.258 | 1.22 ± 0.69 | 1.59 ± 0.63 |
Without compression, shared-reward recovery fails (general MSE 1.20). Removing identity increases specific-reward MSE approximately threefold (0.193 → 0.576), while leaving general-reward MSE nearly unchanged.
W4: Novelty compared with MSRD. (i) Problem: MSRD assumes demonstrations of the same task with different strategies; GRID handles different tasks. (ii) Two-stage learning: GRID first learns individual rewards and then disentangles them, whereas MSRD jointly learns and distills rewards adversarially. (iii) Identity and embeddings: GRID uses a single specific-reward network conditioned on learned task embeddings with an identity loss; MSRD uses separate strategy networks, requiring approximately times as many parameters (6.7× on pendulum). (iv) Generalist pipeline: GRID incorporates kNN support and KL-control fine-tuning. Ablations: identity (above), L1 vs. L2, support penalty, and matched-capacity GRID vs. MSRD (Comment 1).
Setup. All methods share RL settings, steps, seeds, evaluation, and KL weight (Highway-Env: 0.03; MuJoCo: 0.01). Score: mean return over the final 10% of training; 5 seeds per task. Normalization: To aggregate tasks with different reward scales, each task/column is normalized as . Intervals: 95% stratified bootstrap CIs (Agarwal et al., NeurIPS 2021), using 10,000 resamples of 5 seeds within each task. All ± values denote standard errors.
Main Response
(1/2) Main results. We thank the reviewer for the detailed feedback and for valuing our transparency. This comment answers W1, W2, and W5. Comment 2/2 answers W3 and W4 and gives the setup.
W1: Do the shared rewards represent truly shared behaviors?
Bullet point 1: Shared behavior or numerical artifact? An artifact of numerical variation would not produce a competent generalist policy. In Highway-Env and MuJoCo, the shared component is known by construction: all agents share the environment reward (high speed and no crashes in Highway-Env; the locomotion reward in MuJoCo), and only the task-specific part differs (lane preference; DIAYN skill). We train the generalist only on R_g and evaluate it on this ground-truth shared reward. GRID's generalist achieves the highest shared return (normalized score 0.93 for GRID-L1 with kNN; W2 table), and it drives around slower vehicles at high speed (Fig. 6). This is the behavior that all demonstrators share.
Bullet point 2: Normalization does not align meanings. We agree. The [−1, 1] scaling only removes differences in reward scale and offset across tasks (it is invariant to aR + b, a > 0); it is not meant to align meanings. GRID does not assume that the task rewards have the same meaning. Because the compression loss penalizes R_s, R_g learns to explain what all tasks agree on, and any part that does not agree is left to the task-specific R_s. Whether R_g captures a meaningful shared objective is then tested by the generalist's ground-truth return, as in Bullet point 1.
Bullet point 3: When does a shared component exist? Did the reviewer mean that a sparse R_s does not guarantee that a shared component exists at every state, and ask under which conditions it exists? If so, GRID assumes that the demonstrated tasks share some structure (Sec. 4.1); it does not guarantee a shared component at every state. Where the task rewards agree, R_g captures the common value. Where they disagree, the difference goes to R_s, which is non-zero only there. At states that no demonstration covers, R_g is not defined well, which is why GRID uses the kNN support penalty (Sec. 4.3).
Bullet point 4: Full-pipeline verification. Highway-Env and MuJoCo are exactly this test: the shared reward is known, and the full pipeline "Demonstrations → AIRL → GRID → generalist" recovers a reward whose policy achieves the known shared objective, better than all baselines (W2). Craftax and the basis functions complement this with directly interpretable rewards.
W2: Isolating the effect of decomposition. We ran all three requested controls. All methods use the same [−1, 1] scaling and the same reward-network size (MLP [32, 32]), which removes the capacity difference. Entries are normalized scores over all tasks with 95% stratified bootstrap CIs (Comment 2); compare values only within a column.
W5: Faster adaptation. We measured the environment steps needed to reach fine-tuning rewards of 1.0, 1.5, and 2.0 in Highway-Env (5 seeds), against a policy trained from scratch on the full task reward. GRID receives only the fine-tuning reward and the KL term.
GRID reaches 1.5 and 2.0 in about 45% and 62% of the steps needed from scratch. We will add fine-tuning curves to the appendix.