acceptodds
Under review as a conference paper at ICLR 2027

Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents

Abstract

Humans often acquire new skills by observing others, since observed behaviors implicitly reveal how to act reasonably in an environment. However, observations drawn from a heterogeneous population introduce conflicting behavioral signals, making it difficult to determine which behaviors are worth imitating. We address this challenge with General Reward Inference and Disentanglement (GRID), a social learning method that extracts universally useful behaviors from a heterogeneous population of demonstrators pursuing different goals. GRID decomposes per-agent reward functions into a general reward, capturing behaviors shared across all agents, and specific rewards, capturing individual preferences and objectives, through an information bottleneck. Training exclusively on the general reward provides a new paradigm of generalist pretraining. It yields a generalist agent that internalizes universal environmental competencies, such as safety and basic task proficiency, without the mode-averaging bias that afflicts standard learning from demonstration techniques. This generalist serves as a strong prior for fine-tuning to downstream tasks, including preferences unseen during training. Experiments across a synthetic basis function decomposition, multi-agent Craftax, continuous control tasks (MuJoCo Gym) and an autonomous driving simulator (Highway-Env) confirm that GRID successfully disentangles reward structure in a semantically meaningful way, outperforms standard learning from demonstration baselines, and enables more efficient and stable specialization.

Primary area:
reinforcement learning
Keywords:
Learning from Demonstration, Imitation Learning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uzoMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies for RL agent training, how to acquire fundamental, universally applicable skills by observing demonstrations that stem from different goals—or even involve conflicting behaviors?

For instance, in a single game, some players might focus on gathering wood, while others prioritize collecting stone. If one were to simply imitate all these actions indiscriminately, the resulting strategy might fail to achieve any of the intended goals effectively. GRID aims to first learn general-purpose capabilities, which can then be adapted to suit specific objectives.

The hypothesis it follows is: While individuals may pursue different objectives, they often share common fundamental capabilities. Useful, shared reward structures do, in fact, exist across different tasks.

I feel that is a reaosnable hypothesis, not very strong.

Critical strengths

S1: empricial results are strong - GRID-L2 Hopper: 2997.6 (best baseline: 2099.1); GRID-L1 Highway: 13.41 (aggregated AIRL: 12.62).

S2: empirical results are tracable - the experimental setup and relevant baselines have been made public, including those with poorer performance (reviewer highly appreciate that)

Critical weaknesses

W1: Evidence are missing for these labeled "shared rewards" learned ​​truly represent shared behaviors. Plz add

  • discussion if the shared rewards learned are truly common objectives, or are they merely an illusion created by numerical variations in the reward models?
  • normalizing numerical ranges does not equate to aligning meanings. Scaling values ​​to the interval [−1,1][-1, 1] merely ensures that the ranges are identical; it provides no guarantee that the rewards associated with a given state-action pair represent comparable underlying objectives. How to justify it is?
  • the fact that there are few non-zero entries overall can not guarantee that a common component can be identified for every specific state. Under what condition it exists udner waht it not?
  • verify that the decomposition is indeed correct - by comparing the rewards for various tasks under identical states and actions, using an environment with a known shared reward component, one can test the complete pipeline—"Demonstrations → AIRL → GRID"—to see whether the known shared component can be successfully recovered. Simply decomposing a manually constructed reward function only allows for partial verification.

W2: performance lifting is hard to be attributed solely to "decomposing rewards into shared and individual components" because 3 other factors bundled:

  • reward scaling: standardizing the reward range may lead to more stable training.
  • k-NN support penalty: encourages the agent to remain within regions covered by demonstrations, which could improve performance independently of reward decomposition.
  • larger network architecture: The 4×1024 configuration (four layers with 1,024 neurons each) offers significantly greater capacity than the 1×32 configuration (a single layer with 32 neurons), which could also influence the results. As a result, plz provide control exps:
  • Removing the support penalty: to determine how much improvement is contributed from the reward decomposition itself.
  • Applying the same scaling and support penalty to the baseline: to assess the remaining performance gap when both methods utilize the same training setup.
  • Training with only the support signal: ablating reward decomposition and focusing solely on encouraging proximity to the demonstration region, to see what level of performance can be achieved.

W3: unclear to me if the information bottleneck theory really support the loss function employed in the paper? I mean the loss function may be practical, but the theoretical justification for why it can isolatea "shared" reward remains uncear to me. Plz

  • add more explantion and justifciaon to that point

One could either explicitly incorporate a noise model and re-derive the KL divergence, or reframe the approach as a "reward regression decomposition with regularization." Thus it needs a comparative study to clarify what additional value the proposed method actually provides.

  • evaluating the full method against baselines such as removing identity terms, simple averaging, or taking the simple median

W4: clarify compared to MRSD with shared reward, task-speifi reward, L2 regularziation, what are the novel things from the paper here? If it is combo, then more comprehesnvie abaltion study is needed to asees the contribution of indvidual modificaiton/ combo decisions.

W5: The claim of faster adaptation lacks sufficient evidence. To demonstrate faster learning, it must compare performance trajectories as a function of training steps—ideally against a baseline trained from scratch. A high final score does not necessarily equate to a fast learning rate.

Questions for authors

All qs to followup and action itesm listed for each weakness point above, plz read and follow up them

2 comments
AuthorsOpenReview ↗

Main Response

(1/2) Main results. We thank the reviewer for the detailed feedback and for valuing our transparency. This comment answers W1, W2, and W5. Comment 2/2 answers W3 and W4 and gives the setup.

W1: Do the shared rewards represent truly shared behaviors?

Bullet point 1: Shared behavior or numerical artifact? An artifact of numerical variation would not produce a competent generalist policy. In Highway-Env and MuJoCo, the shared component is known by construction: all agents share the environment reward (high speed and no crashes in Highway-Env; the locomotion reward in MuJoCo), and only the task-specific part differs (lane preference; DIAYN skill). We train the generalist only on R_g and evaluate it on this ground-truth shared reward. GRID's generalist achieves the highest shared return (normalized score 0.93 for GRID-L1 with kNN; W2 table), and it drives around slower vehicles at high speed (Fig. 6). This is the behavior that all demonstrators share.

Bullet point 2: Normalization does not align meanings. We agree. The [−1, 1] scaling only removes differences in reward scale and offset across tasks (it is invariant to aR + b, a > 0); it is not meant to align meanings. GRID does not assume that the task rewards have the same meaning. Because the compression loss penalizes R_s, R_g learns to explain what all tasks agree on, and any part that does not agree is left to the task-specific R_s. Whether R_g captures a meaningful shared objective is then tested by the generalist's ground-truth return, as in Bullet point 1.

Bullet point 3: When does a shared component exist? Did the reviewer mean that a sparse R_s does not guarantee that a shared component exists at every state, and ask under which conditions it exists? If so, GRID assumes that the demonstrated tasks share some structure (Sec. 4.1); it does not guarantee a shared component at every state. Where the task rewards agree, R_g captures the common value. Where they disagree, the difference goes to R_s, which is non-zero only there. At states that no demonstration covers, R_g is not defined well, which is why GRID uses the kNN support penalty (Sec. 4.3).

Bullet point 4: Full-pipeline verification. Highway-Env and MuJoCo are exactly this test: the shared reward is known, and the full pipeline "Demonstrations → AIRL → GRID → generalist" recovers a reward whose policy achieves the known shared objective, better than all baselines (W2). Craftax and the basis functions complement this with directly interpretable rewards.

W2: Isolating the effect of decomposition. We ran all three requested controls. All methods use the same [−1, 1] scaling and the same reward-network size (MLP [32, 32]), which removes the capacity difference. Entries are normalized scores over all tasks with 95% stratified bootstrap CIs (Comment 2); compare values only within a column.

MethodGeneralist, kNNGeneralist, no kNNFine-tune, kNNFine-tune, no kNN
AIRL (pooled)0.62 [0.39, 0.83]0.82 [0.65, 0.98]0.46 [0.14, 0.74]0.50 [0.13, 0.81]
MSRD0.61 [0.51, 0.68]0.77 [0.55, 0.98]0.51 [0.22, 0.76]0.32 [−0.00, 0.62]
FLAIR0.38 [0.20, 0.56]0.81 [0.60, 1.02]0.65 [0.49, 0.85]0.72 [0.51, 0.94]
REPRESENT0.38 [−0.31, 0.85]0.67 [0.64, 0.70]0.42 [−0.02, 0.83]0.76 [0.64, 0.91]
Pointwise mean0.64 [0.44, 0.83]0.84 [0.66, 1.00]0.55 [0.34, 0.73]0.68 [0.45, 0.89]
Pointwise median0.69 [0.51, 0.88]0.55 [0.37, 0.75]0.50 [0.18, 0.76]0.64 [0.26, 0.93]
GRID-L10.93 [0.66, 1.13]0.02 [−0.24, 0.23]0.58 [0.41, 0.74]0.43 [0.21, 0.67]
GRID-L20.71 [0.46, 0.96]0.09 [−0.12, 0.30]0.80 [0.61, 0.95]0.36 [0.05, 0.66]
  • Same scaling and penalty for all methods ("kNN" columns): GRID outperforms all baselines. GRID-L2 has a higher mean than every baseline in both settings.
  • Without the penalty ("no kNN" columns): GRID is weaker, because R_g is trained only on expert state-action pairs. kNN support is an essential design component of GRID.
  • Support signal only: a policy trained on the kNN support reward alone reaches 4.57 [3.10, 6.04] in Highway-Env, versus 13.35 [12.74, 13.96] for GRID-L1 with kNN. Thus, the support penalty alone is not sufficient; the decomposed reward R_g provides the gain.

W5: Faster adaptation. We measured the environment steps needed to reach fine-tuning rewards of 1.0, 1.5, and 2.0 in Highway-Env (5 seeds), against a policy trained from scratch on the full task reward. GRID receives only the fine-tuning reward and the KL term.

Method1.01.52.0
GRID-L1 (β = 0.03)3.0k ± 0.3k5.1k ± 1.0k10.4k ± 3.1k
From scratch, full task reward3.1k ± 0.1k11.4k ± 3.3k16.7k ± 4.3k

GRID reaches 1.5 and 2.0 in about 45% and 62% of the steps needed from scratch. We will add fine-tuning curves to the appendix.

AuthorsOpenReview ↗

Supplementary Response

(2/2) Remaining questions, setup, and details.

W3: Theoretical justification of the loss.

Why RgR_g isolates shared rewards. RgR_g depends only on (s,a)(s,a), making it task-independent. Reconstruction requires Rg+Rs(p)≈R(p)R_g+R_s^{(p)}\approx R^{(p)}, while compression penalizes the magnitude of Rs(p)R_s^{(p)}. Under exact reconstruction, Rs(p)=R(p)−RgR_s^{(p)}=R^{(p)}-R_g. Thus, minimizing L1 compression gives

Rg∗(s,a)=Median⁡p{R(p)(s,a)},R_g^{\ast}(s,a)=\operatorname{Median}_p\lbrace R^{(p)}(s,a)\rbrace ,

while squared L2 compression gives

Rg∗(s,a)=1N∑pR(p)(s,a).R_g^{\ast}(s,a)=\frac1N\sum_p R^{(p)}(s,a).

These characterize the inductive bias: L1 favors the majority-supported reward, while L2 captures the mean structure. If R(p)=Rshared+Δ(p)R^{(p)}=R_{\mathrm{shared}}+\Delta^{(p)}, exact recovery occurs when task-specific deviations have zero median (L1) or zero mean (L2) at each (s,a)(s,a). This analysis describes an idealized setting (equal task weighting, sufficient expressivity, exact reconstruction, no identity term) and explains each variant's inductive bias; the full model builds on it with the identity loss and soft reconstruction. In Highway-Env and MuJoCo, each task reward is evaluated only on its own DpD_p, so the rewards are not compared at the same (s,a)(s,a). Thus, the pointwise result does not apply directly. This agrees with our results: with kNN support, GRID outperforms pointwise aggregation. The identity loss also keeps Rs(p)R_s^{(p)} task-discriminative.

Noise model. Following the reviewer's suggestion, consider an auxiliary stochastic encoder p(z∣x)=N(μs(x),σ2)p(z\mid x)=\mathcal N(\mu_s(x),\sigma^2), where μs(x)=Rs(p)(x)\mu_s(x)=R_s^{(p)}(x) and σ>0\sigma>0 is fixed. Then DKL(p∥r)=Ep[log⁡p(z∣x)]−Ep[log⁡r(z)]D_{\mathrm{KL}}(p\Vert r)=\mathbb E_p[\log p(z\mid x)]-\mathbb E_p[\log r(z)], where the first term is the negative entropy −12log⁡(2πeσ2)-\tfrac12\log(2\pi e\sigma^2) and does not depend on μs\mu_s. For a Gaussian prior r(z)=N(0,s2)r(z)=\mathcal N(0,s^2), log⁡r(z)=−12log⁡(2πs2)−z2/(2s2)\log r(z)=-\tfrac12\log(2\pi s^2)-z^2/(2s^2) and Ep[z2]=μs2+σ2\mathbb E_p[z^2]=\mu_s^2+\sigma^2, so

DKL(p∥r)=μs(x)22s2+const,D_{\mathrm{KL}}(p\Vert r)=\frac{\mu_s(x)^2}{2s^2}+\mathrm{const},

which yields squared L2 compression exactly. For a Laplace prior r(z)∝e−∣z∣/br(z)\propto e^{-|z|/b},

DKL(p∥r)=Ep[∣z∣]b+const.D_{\mathrm{KL}}(p\Vert r)=\frac{\mathbb E_p[|z|]}b+\mathrm{const}.

By Jensen's and the triangle inequalities,

∣μs∣≤Ep[∣z∣]≤∣μs∣+σ2/π.|\mu_s|\leq\mathbb E_p[|z|]\leq|\mu_s|+\sigma\sqrt{2/\pi}.

Thus, the μs\mu_s-dependent KL term approaches ∣μs∣/b|\mu_s|/b uniformly as σ→0\sigma\to0. GRID directly regularizes deterministic specific-reward outputs; the stochastic formulation motivates these penalties rather than describing sampling during training. GRID can thus be viewed as regularized reward regression, with the IB view motivating compression (R2), identity (R1), and reconstruction (R3). We will clarify this derivation in the appendix.

Comparative study. With kNN support (Comment 1), GRID-L2 outperforms the mean (0.71 vs. 0.64 generalist; 0.80 vs. 0.55 fine-tune), while GRID-L1 outperforms the median (0.93 vs. 0.69; 0.58 vs. 0.50). We additionally ablate the identity and compression losses (basis functions, MLP [32,32], 5 seeds; MSE vs. ground truth):

VariantTotal ↓General ↓Specific ↓
Full (GRID-L1)0.194 ± 0.0020.0013 ± 0.00020.193 ± 0.002
No compression0.008 ± 0.0081.20 ± 0.701.20 ± 0.68
No identity0.576 ± 0.2710.0011 ± 0.00020.576 ± 0.270
Neither0.363 ± 0.2581.22 ± 0.691.59 ± 0.63

Without compression, shared-reward recovery fails (general MSE 1.20). Removing identity increases specific-reward MSE approximately threefold (0.193 → 0.576), while leaving general-reward MSE nearly unchanged.

W4: Novelty compared with MSRD. (i) Problem: MSRD assumes demonstrations of the same task with different strategies; GRID handles different tasks. (ii) Two-stage learning: GRID first learns individual rewards and then disentangles them, whereas MSRD jointly learns and distills rewards adversarially. (iii) Identity and embeddings: GRID uses a single specific-reward network conditioned on learned task embeddings with an identity loss; MSRD uses separate strategy networks, requiring approximately (K+1)/3(K+1)/3 times as many parameters (6.7× on pendulum). (iv) Generalist pipeline: GRID incorporates kNN support and KL-control fine-tuning. Ablations: identity (above), L1 vs. L2, support penalty, and matched-capacity GRID vs. MSRD (Comment 1).


Setup. All methods share RL settings, steps, seeds, evaluation, and KL weight β\beta (Highway-Env: 0.03; MuJoCo: 0.01). Score: mean return over the final 10% of training; 5 seeds per task. Normalization: To aggregate tasks with different reward scales, each task/column is normalized as (x−worst method mean)/(best method mean−worst method mean)(x-\text{worst method mean})/(\text{best method mean}-\text{worst method mean}). Intervals: 95% stratified bootstrap CIs (Agarwal et al., NeurIPS 2021), using 10,000 resamples of 5 seeds within each task. All ± values denote standard errors.

Reviewer_xjxjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GRID, which decomposes task-specific rewards into a shared reward and task-specific components, then uses the shared reward to train a generalist policy before KL-controlled fine-tuning. The results show improvements in some generalist settings, but the benefits of decomposition and fine-tuning are not consistently demonstrated.

Critical strengths
  1. The problem is relevant, and the overall pipeline is practical. Separating shared behavior from task-specific preferences is a reasonable way to leverage heterogeneous demonstrations. GRID also decouples reward learning from decomposition, avoiding the joint adversarial training used by MSRD, FLAIR, and REPRESENT.
  2. The evaluation includes relevant decomposition baselines and environments where learned rewards can be inspected. The Craftax reward maps and Highway lane statistics are useful for assessing what the shared reward captures beyond policy returns alone.
Critical weaknesses
  1. For a sufficiently flexible task-specific component, the L2 objective appears to make the shared reward a pointwise mean of the task rewards, while the L1 variant behaves more like a robust location estimate, approaching the median in some regimes. The paper does not establish that the identity loss meaningfully changes this behavior, nor does it compare GRID against simple mean and median baselines.
  2. GRID uses a rescaled shared reward combined with a kNN-based out-of-distribution penalty, while the baselines use different reward scaling and generally lack the same penalty.
  3. The missing β values further complicate the interpretation. The paper also lacks a from-scratch RL baseline and direct measurements of sample efficiency. Therefore, the claims about more efficient and stable specialization are not yet well supported.
Questions for authors
  1. How does GRID compare with pointwise mean and median reward aggregation? Please evaluate these baselines using the same generalist training and fine-tuning pipeline,.
  2. Which state-action pairs are used to evaluate each task reward during decomposition training: task-specific demonstrations or the pooled dataset? How are rewards normalized across tasks?
  3. Can the baselines be evaluated with the same reward scaling and kNN penalty as GRID? Conversely, how does GRID perform without the penalty?
  4. Please report the fine-tuning β values, the number of Highway-Env seeds, and whether the reported ± values are standard deviations or standard errors.
  5. In Craftax, are the task rewards taken from the environment or learned through IRL?
2 comments
AuthorsOpenReview ↗

Main Response

(1/2) Main results. We thank the reviewer for the careful analysis. This comment answers W1/Q1, W2/Q3, and W3. Comment 2/2 answers Q2, Q4, and Q5 and gives the setup.

W1/Q1: Comparison with pointwise mean and median. We agree with the reviewer's analysis in the limit: with exact reconstruction and no identity loss, the L2 and L1 compression terms are minimized by the pointwise mean and median of the task rewards. GRID differs from this limit in two ways. First, reconstruction is a soft loss that is optimized jointly with the identity loss, which requires R_s to stay informative about the task. Second, in the environments with learned rewards (Highway-Env and MuJoCo), each task reward is evaluated only on that task's own demonstrations D_p, not on a shared set of states. In states that only one task visits, R_g fits that task's reward, not an average over tasks. R_g is a single network of (s, a) that generalizes across these states, so it is not a pointwise mean or median. To test whether this matters, we added the pointwise mean and median of the learned rewards as baselines. They use the same generalist training and fine-tuning pipeline as all other methods:

MethodGeneralist, kNNGeneralist, no kNNFine-tune, kNNFine-tune, no kNN
AIRL (pooled)0.62 [0.39, 0.83]0.82 [0.65, 0.98]0.46 [0.14, 0.74]0.50 [0.13, 0.81]
MSRD0.61 [0.51, 0.68]0.77 [0.55, 0.98]0.51 [0.22, 0.76]0.32 [−0.00, 0.62]
FLAIR0.38 [0.20, 0.56]0.81 [0.60, 1.02]0.65 [0.49, 0.85]0.72 [0.51, 0.94]
REPRESENT0.38 [−0.31, 0.85]0.67 [0.64, 0.70]0.42 [−0.02, 0.83]0.76 [0.64, 0.91]
Pointwise mean0.64 [0.44, 0.83]0.84 [0.66, 1.00]0.55 [0.34, 0.73]0.68 [0.45, 0.89]
Pointwise median0.69 [0.51, 0.88]0.55 [0.37, 0.75]0.50 [0.18, 0.76]0.64 [0.26, 0.93]
GRID-L10.93 [0.66, 1.13]0.02 [−0.24, 0.23]0.58 [0.41, 0.74]0.43 [0.21, 0.67]
GRID-L20.71 [0.46, 0.96]0.09 [−0.12, 0.30]0.80 [0.61, 0.95]0.36 [0.05, 0.66]

Entries are normalized scores over all tasks with 95% stratified bootstrap CIs [1] (Comment 2). Each column is normalized separately, so values can be compared only within a column. With kNN support, GRID-L2 outperforms the pointwise mean (0.71 vs. 0.64 generalist; 0.80 vs. 0.55 fine-tune), and GRID-L1 outperforms the pointwise median (0.93 vs. 0.69; 0.58 vs. 0.50). Thus, GRID does not reduce to its mean or median limit in practice. In our ablation (Comment 2), removing the identity loss raises the specific-reward error from 0.193 to 0.576, so the identity loss changes the learned decomposition.

W2/Q3: Matched reward scaling and kNN penalty. All methods in the table use the same [−1, 1] reward scaling and the same reward-network size (MLP [32, 32]), and each method is evaluated both with and without the kNN penalty. With kNN support, GRID outperforms all baselines, and GRID-L2 has a higher mean than every baseline in both settings. Its fine-tuning CIs (width 0.33–0.34) are also narrower than those of MSRD, FLAIR, and REPRESENT (0.36–0.85), although these baselines have up to 6.7× more weights. Without the penalty, GRID is weaker, because R_g is trained only on expert state-action pairs and is not defined well outside them. kNN support is thus an essential design component of GRID (Sec. 4.3). However, kNN support alone is not sufficient: a policy trained on the kNN support reward only reaches 4.57 [3.10, 6.04] in Highway-Env, versus 13.35 [12.74, 13.96] for GRID-L1 with kNN.

W3: From-scratch baseline and sample efficiency. We measured the environment steps needed to reach fine-tuning rewards of 1.0, 1.5, and 2.0 in Highway-Env (5 seeds). We compare the GRID specialist with a policy trained from scratch on the full task reward. GRID receives only the fine-tuning reward and the KL term.

Method1.01.52.0
GRID-L1 (β = 0.03)3.0k ± 0.3k5.1k ± 1.0k10.4k ± 3.1k
From scratch, full task reward3.1k ± 0.1k11.4k ± 3.3k16.7k ± 4.3k

GRID reaches 1.5 and 2.0 in about 45% and 62% of the steps needed from scratch. The generalist prior already encodes shared driving skills, so fine-tuning only learns the lane preference. We will add fine-tuning curves to the appendix. The β values are in Comment 2 (Q4).

AuthorsOpenReview ↗

Supplementary Response

(2/2) Remaining questions, setup, and details.

Q2: State-action pairs and normalization. The pairs depend on the domain. In the basis function decomposition, every task is evaluated on the same 2,000 points in [−4π,4π][-4\pi, 4\pi], where ground-truth values are available. In Craftax, we use the pooled dataset. In Highway-Env and MuJoCo, each task reward R(p)R^{(p)} is evaluated only on that task's own demonstrations DpD_p. Before GRID, we min-max scale each task's learned reward on its own demonstration pairs DpD_p:

R~(p)(s,a)=2R(p)(s,a)−min⁡DpR(p)max⁡DpR(p)−min⁡DpR(p)−1.\tilde R^{(p)}(s,a) = 2\frac{R^{(p)}(s,a) - \min_{D_p} R^{(p)}}{\max_{D_p} R^{(p)} - \min_{D_p} R^{(p)}} - 1.

If R(p)R^{(p)} is replaced by aR(p)+baR^{(p)} + b with a>0a > 0, R~(p)\tilde R^{(p)} does not change. Thus, GRID is insensitive to differences in reward scale and offset across tasks by construction. All baselines use the same [−1, 1] scaling.

Q4: β\beta values, seeds, and ± values. The fine-tuning KL weights are β=0.03\beta = 0.03 for Highway-Env (DQN) and β=0.01\beta = 0.01 for Pendulum and Hopper (SAC). Highway-Env uses 5 seeds. All ± values in the paper and in this response are standard errors. We will add all of this to the paper.

Q5: Craftax rewards. We use the ground-truth rewards to train GRID in Craftax and in the basis function decomposition only. All other environments use learned rewards (AIRL).


Pipeline. Each method learns a reward, trains a generalist on it, and fine-tunes it with the new-task reward plus the KL-control term (Sec. 4.3). All methods use the same RL settings, number of steps, seeds, evaluation, and β\beta.

Capacity. All reward networks are MLP [32, 32] with [−1, 1] reward scaling. MSRD, FLAIR, and REPRESENT use one network per strategy plus one task network (K+1K + 1 networks). GRID uses three networks shared by all strategies plus a 2-D embedding per strategy. Thus, the baselines have about (K+1)/3(K + 1)/3 times the parameters of GRID: 6.7× on pendulum (K=19K = 19), 4.7× on hopper (K=13K = 13), and 1.3× on highway (K=3K = 3).

Tasks. Generalist: Highway, Pendulum, Hopper. Fine-tuning: Highway lane-3 (DQN, β=0.03\beta = 0.03); Pendulum new task, Hopper new tasks 1 and 2 (SAC, β=0.01\beta = 0.01). For MuJoCo, the new-task reward is the environment reward plus a skill reward. The score of a run is the mean evaluation return over the last 10% of training. With 5 seeds per task, there are 15 generalist runs and 20 fine-tuning runs per method.

kNN columns. "kNN": every method's generalist uses the OOD kNN blend. "No kNN": no method uses it. Fine-tuning never uses the blend. The kNN-only policy uses the kNN support reward with no learned reward. Each column is normalized separately, so values can be compared only within a column, not between columns.

Normalization. Raw returns differ greatly in scale across tasks (e.g., about 13 in Highway-Env and up to about 3,000 in Hopper). To combine all tasks into one concise table, we normalize the returns so that they are comparable. For each task and kNN setting,

score=x−worst method meanbest method mean−worst method mean.\text{score} = \frac{x - \text{worst method mean}}{\text{best method mean} - \text{worst method mean}}.

Each entry is the mean of a method's 15 (generalist) or 20 (fine-tune) scores, so all tasks have equal weight. Each column has its own normalization points.

Interval. We report the 95% stratified bootstrap CI [1]. In each of 10,000 resamples, we draw 5 seeds with replacement within each task, pool the scores, and compute the mean. The interval is the 2.5th–97.5th percentile of these means.

Ablation (basis function decomposition, MLP [32, 32], 5 seeds, MSE vs. ground truth).

VariantTotal ↓General ↓Specific ↓
Full (GRID-L1)0.194 ± 0.0020.0013 ± 0.00020.193 ± 0.002
No compression0.008 ± 0.0081.20 ± 0.701.20 ± 0.68
No identity0.576 ± 0.2710.0011 ± 0.00020.576 ± 0.270
Neither0.363 ± 0.2581.22 ± 0.691.59 ± 0.63

Compression makes RgR_g explain the shared behavior; without it, the split fails. The identity loss makes RsR_s capture task-specific behavior.

References

[1] R. Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.

Reviewer_NQ7zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes General Reward Inference and Disentanglement (GRID), a framework for learning shared behaviors from heterogeneous agents pursuing different objectives. GRID first obtains task-specific reward functions, either directly from the environment or through inverse reinforcement learning, and decomposes them into a shared general reward and agent-specific rewards. The decomposition is learned using a reconstruction loss and an information bottleneck objective that encourages the specific rewards to preserve agent identity information while remaining compact. The learned general reward is then used to train a generalist policy, which is fine-tuned on downstream task-specific rewards using KL-control to preserve the generalist prior. Experiments on synthetic reward functions, Craftax, Highway-Env, and MuJoCo demonstrate meaningful reward decomposition and improvements in generalist learning and downstream adaptation.

Critical strengths
  1. The paper addresses an interesting and important problem of learning shared competencies from heterogeneous agents with different objectives. The proposed reward decomposition framework offers an intuitive approach to separating general environmental behaviors from individual preferences.

  2. The experiments cover several domains, including synthetic reward decomposition, discrete gridworlds, autonomous driving, and continuous control. The qualitative results provide interpretable examples of reward disentanglement, while the generalist learning experiments demonstrate promising performance improvements over existing baselines.

Critical weaknesses
  1. The method relies on assumptions about reward estimation and demonstration quality that are not sufficiently examined. In particular, reward functions independently recovered through AIRL may not be directly comparable across agents. The effects of demonstration quality and state-action coverage are also not evaluated.

  2. The effectiveness of the proposed information bottleneck objective is not sufficiently demonstrated. Since the identity embeddings are jointly optimized and the auxiliary network also receives state-action information, it is unclear whether the identity loss effectively encourages the specific rewards to capture agent-specific information. Ablations on the loss components and embedding design are also missing.

  3. The experimental comparisons may be affected by differences in reward processing and model capacity. GRID uses additional reward normalization and OOD reward blending that are not consistently applied to the baselines, as well as a substantially larger general reward network. These differences make it difficult to isolate the contribution of the proposed reward disentanglement.

  4. The experimental results do not fully support the claimed benefits for downstream adaptation. GRID's fine-tuning performance is mixed across tasks, and the reported best returns provide limited evidence for improved adaptation efficiency.

Questions for authors
  1. How sensitive is GRID to demonstration quality and state-action coverage? Section 4.1 assumes that demonstrations are optimal and representative of environmental norms. However, expert policies may cover only a limited part of the state-action space, while broader exploration may introduce suboptimal behaviors. In Craftax, demonstrations are collected from IPPO checkpoints at different training stages. Could the authors investigate this trade-off by varying demonstration quality and state-action coverage under matched data budgets?

  2. What prevents the jointly optimized agent identity embeddings from collapsing or becoming indistinguishable? Since the auxiliary network also receives state-action information, could it predict agent identity without relying on the specific reward? Ablations using fixed embeddings, removing the specific reward input to the auxiliary network, and removing individual loss components would help validate the proposed information bottleneck objective.

  3. In Figure 2, the input to the Embedding Network appears to be the reconstructed total reward, whereas Eq. (5) defines its input as the specific reward, together with state-action information. Could the authors clarify and correct this inconsistency?

  4. How are the reward functions independently recovered through AIRL aligned or normalized before being used by GRID in Highway-Env and MuJoCo? Given the ambiguity of IRL reward estimates, how sensitive is GRID to differences in reward scale or representation across agents?

  5. Appendix Table 7 shows that GRID uses reward normalization and OOD reward blending during generalist training, while the baselines use different reward processing strategies without OOD blending. These differences may affect policy performance independently of reward decomposition. Could the authors evaluate GRID without OOD blending and compare against baselines using matched reward normalization and OOD handling to better isolate the contribution of the proposed method?

  6. How sensitive is GRID to the capacity of its reward networks? The general reward network is substantially larger than those used by several baselines. Could the authors provide comparisons with more closely matched network capacities?

  7. Could the authors provide full fine-tuning learning curves to support the claim of more efficient downstream adaptation? The current results are mixed across tasks, and reporting the best achieved returns does not fully establish improvements in sample efficiency.

2 comments
AuthorsOpenReview ↗

Main Response

(1/2) Main results. We thank the reviewer for the constructive feedback. This comment answers Q5, Q6, Q2, and Q7. Comment 2/2 answers Q4, Q1, and Q3 and gives the setup.

Q5 and Q6: Matched reward processing and capacity. We reran all methods with the same reward-network size (MLP [32, 32]) and [−1, 1] reward scaling, with and without the OOD kNN blend. We also added the pointwise mean and median of the learned rewards as baselines. Entries are normalized scores over all tasks with 95% stratified bootstrap CIs [1].

MethodGeneralist, kNNGeneralist, no kNNFine-tune, kNNFine-tune, no kNN
AIRL (pooled)0.62 [0.39, 0.83]0.82 [0.65, 0.98]0.46 [0.14, 0.74]0.50 [0.13, 0.81]
MSRD0.61 [0.51, 0.68]0.77 [0.55, 0.98]0.51 [0.22, 0.76]0.32 [−0.00, 0.62]
FLAIR0.38 [0.20, 0.56]0.81 [0.60, 1.02]0.65 [0.49, 0.85]0.72 [0.51, 0.94]
REPRESENT0.38 [−0.31, 0.85]0.67 [0.64, 0.70]0.42 [−0.02, 0.83]0.76 [0.64, 0.91]
Pointwise mean0.64 [0.44, 0.83]0.84 [0.66, 1.00]0.55 [0.34, 0.73]0.68 [0.45, 0.89]
Pointwise median0.69 [0.51, 0.88]0.55 [0.37, 0.75]0.50 [0.18, 0.76]0.64 [0.26, 0.93]
GRID-L10.93 [0.66, 1.13]0.02 [−0.24, 0.23]0.58 [0.41, 0.74]0.43 [0.21, 0.67]
GRID-L20.71 [0.46, 0.96]0.09 [−0.12, 0.30]0.80 [0.61, 0.95]0.36 [0.05, 0.66]

Analysis. Overall, GRID with kNN support outperforms the baselines on both the generalist and fine-tuning tasks. GRID-L2 has a higher mean than every baseline in both settings, and GRID-L1 has the best generalist score (0.93). For fine-tuning, the baselines that jointly train adversarially and disentangle shared and specific components (MSRD, FLAIR, REPRESENT) have CI widths of 0.36–0.85, versus 0.33–0.34 for GRID, although they have up to 6.7× more weights. In these baselines, adversarial reward learning and disentanglement compete during training, so some seeds collapse. GRID separates them: AIRL first learns each reward, and GRID then disentangles it with a supervised objective, which is more consistent. As expected, GRID without kNN is weaker, because R_g is trained only on the expert state-action distribution and is not defined well outside it. kNN support is thus an essential design component of GRID (Sec. 4.3). Compare values only within a column.

Is the gain due to kNN alone? No. A policy trained on the kNN support reward only (Highway, 5 seeds) reaches 4.57 [3.10, 6.04], versus 13.35 [12.74, 13.96] for GRID-L1. kNN keeps the agent near demonstrated states; R_g tells it which actions are good there.

Q2: Ablation. Basis functions, MLP [32, 32], 5 seeds, MSE vs. ground truth.

VariantTotal ↓General ↓Specific ↓
Full (GRID-L1)0.194 ± 0.0020.0013 ± 0.00020.193 ± 0.002
No compression0.008 ± 0.0081.20 ± 0.701.20 ± 0.68
No identity0.576 ± 0.2710.0011 ± 0.00020.576 ± 0.270
Neither0.363 ± 0.2581.22 ± 0.691.59 ± 0.63
Fixed embeddings1.63 ± 0.930.22 ± 0.201.80 ± 0.92
q without R_s0.200 ± 0.0020.0011 ± 0.00020.199 ± 0.002
  • No compression: the total reward is accurate, but the split fails (general MSE 1.20). Compression forces R_g to explain the shared behavior (R2).
  • No identity loss: R_g stays accurate, but R_s degrades (0.576 vs. 0.193). The identity loss makes R_s capture agent-specific behavior (R1).
  • Neither / fixed embeddings: all three rewards degrade, so learnable embeddings are needed.
  • q without R_s: the rewards barely change, but q can no longer identify the tasks:
VariantEmbedding spread ↑Identification error ↓
Full2.96 ± 0.270.8%
q without R_s1.78 ± 0.1166.7%

With R_s as input, the embeddings are 1.7× more spread out (2.96 vs. 1.78), and the identification error is only 0.8%. Without R_s, the error is 66.7%, the rate of random guessing among three tasks. Thus, the embeddings do not collapse, and q relies on R_s to identify the task. In future work, we will use these embeddings to characterize demonstrators for personalization, as done in [2, 3].

Q7: Sample efficiency. We measured the environment steps needed to reach fine-tuning rewards of 1.0, 1.5, and 2.0 in Highway-Env (5 seeds). We compare the GRID specialist with a policy trained from scratch on the full task reward. GRID receives only the fine-tuning reward and the KL term.

Method1.01.52.0
GRID-L1 (β = 0.03)3.0k ± 0.3k5.1k ± 1.0k10.4k ± 3.1k
From scratch, full task reward3.1k ± 0.1k11.4k ± 3.3k16.7k ± 4.3k

GRID reaches 1.5 and 2.0 in about 45% and 62% of the steps needed from scratch. The generalist prior already encodes shared driving skills, so fine-tuning only learns the lane preference. We will add fine-tuning curves to the appendix.

AuthorsOpenReview ↗

Supplementary Repsonse

(2/2) Remaining questions, setup, and details.

Q4: Reward alignment. Before GRID, we min-max scale each agent's learned reward on its own demonstration pairs D_p:

R~(p)(s,a)=2R(p)(s,a)−min⁡DpR(p)max⁡DpR(p)−min⁡DpR(p)−1.\tilde R^{(p)}(s,a) = 2\frac{R^{(p)}(s,a) - \min_{D_p} R^{(p)}}{\max_{D_p} R^{(p)} - \min_{D_p} R^{(p)}} - 1.

If R(p)R^{(p)} is replaced by aR(p)+baR^{(p)} + b with a>0a > 0, R~(p)\tilde R^{(p)} does not change. Thus, GRID is insensitive to reward scale and offset across agents by construction.

Q1: Demonstration quality and coverage. Could the reviewer elaborate on the "trade-off by varying demonstration quality and state-action coverage under matched data budgets"? GRID assumes that tasks are demonstrated optimally, and it handles OOD states with kNN support, as described in Comment 1.

Q3: Figure 2. Thank you for noticing this inconsistency. We will correct Figure 2 in the paper. Please refer to the mathematical formulation in Eq. (5) instead of the diagram.


Pipeline. Each method learns a reward, trains a generalist on it, and fine-tunes it with the new-task reward plus the KL-control term (Sec. 4.3). All methods use the same RL settings, number of steps, seeds, evaluation, and KL weight β.

Capacity. All reward networks are MLP [32, 32] with [−1, 1] reward scaling. MSRD, FLAIR, and REPRESENT use one network per strategy plus one task network, so they have K + 1 networks. GRID uses three networks shared by all strategies plus a 2-D embedding per strategy. Thus, the baselines have about (K + 1)/3 times the parameters of GRID: 6.7× on pendulum (K = 19), 4.7× on hopper (K = 13), and 1.3× on highway (K = 3).

Tasks. Generalist: Highway, Pendulum, Hopper. Fine-tuning: Highway lane-3 (DQN, β = 0.03); Pendulum new task, Hopper new tasks 1 and 2 (SAC, β = 0.01). For MuJoCo, the new-task reward is the environment reward plus a skill reward. The score of a run is the mean evaluation return over the last 10% of training, not the best return. With 5 seeds per task, there are 15 generalist runs and 20 fine-tuning runs per method.

kNN columns. "kNN": every method's generalist uses the OOD kNN blend. "No kNN": no method uses it. Fine-tuning never uses the blend. The kNN-only policy uses the kNN support reward with no learned reward.

Normalization. Raw returns differ greatly in scale across tasks (e.g., about 13 in Highway-Env and up to about 3,000 in Hopper). To combine all tasks into one concise table, we normalize the returns so that they are comparable. For each task and kNN setting, score = (x − worst method mean) / (best method mean − worst method mean). A score of 0 is the worst method mean and 1 is the best method mean; single runs can fall outside [0, 1]. Each entry is the mean of a method's 15 (generalist) or 20 (fine-tune) scores, so all tasks have equal weight. Each column has its own normalization points.

Interval. We report the 95% stratified bootstrap confidence interval [1]. In each of 10,000 resamples, we draw 5 seeds with replacement within each task, pool the scores, and compute the mean. The interval is the 2.5th–97.5th percentile of these means. We use the stratified bootstrap instead of ± SE because some seeds collapse, so scores are not normal.

Ablation details. In the full model, the higher total MSE than without compression is the expected cost of the bottleneck trade-off (β in Eq. 1). Metrics: spread S=1N∑p=1N∥ω(p)−ωavg∥2S = \frac{1}{N}\sum_{p=1}^{N} \lVert \omega^{(p)} - \omega_{\text{avg}} \rVert^2, with N=3N = 3. For identification error, we assign each of the 6,000 data points to the embedding nearest to q's output and report the fraction assigned to the wrong task; chance is 2/3 for three tasks. q receives (s, a, R_s) in the full model and (s, a) only in the ablation.

References

[1] R. Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.

[2] M. L. Schrum et al. MIND MELD: Personalized meta-learning for robot-centric imitation learning. HRI, 2022.

[3] M. L. Schrum et al. Reciprocal MIND MELD: Improving learning from demonstration via personalized, reciprocal teaching. CoRL, 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks

Dear Area Chair and Reviewers,

We thank the Area Chair and all reviewers for their time evaluating our work. The reviewers highlighted the reward decomposition approach (NQ7z, xjxj), interpretable qualitative results (NQ7z, xjxj), and promising empirical results (NQ7z, uzoM). Below, we summarize GRID's core contributions and how we addressed the main concerns.

Core Contributions of GRID

GRID answers the question: what should an agent learn from a population of heterogeneous agents that pursue different goals? We posit that the most valuable signals are those shared across agents, as they reflect general competence and norms. GRID represents these shared signals as a general reward and the agent-specific signals as specific rewards. The sum of the general reward and an agent's specific reward gives that agent's total reward, which explains its behavior. The specific rewards are learned through an information bottleneck that compresses them while maximizing their mutual information with agent identity. Thus, each specific reward keeps only what is needed to describe its agent, and the general reward explains most of the population's behavior. Training on the general reward with our OOD kNN support, which keeps the agent near demonstrated states, yields a generalist agent that internalizes the competencies shared by the population. The generalist is a strong prior that learns new tasks quickly.

1. Baseline fairness (all reviewers)

All reviewers asked whether the comparison was fair, because only GRID used OOD kNN support and the reward networks had different sizes. We used the same reward-network size (MLP [32, 32]) and [−1, 1] reward scaling for all methods, and we reran every method with and without OOD kNN support. The table shows the normalized scores with kNN support, averaged over all tasks, with 95% stratified bootstrap CIs:

MethodGeneralistFine-tune
AIRL (pooled)0.62 [0.39, 0.83]0.46 [0.14, 0.74]
MSRD0.61 [0.51, 0.68]0.51 [0.22, 0.76]
FLAIR0.38 [0.20, 0.56]0.65 [0.49, 0.85]
REPRESENT0.38 [−0.31, 0.85]0.42 [−0.02, 0.83]
Pointwise mean0.64 [0.44, 0.83]0.55 [0.34, 0.73]
Pointwise median0.69 [0.51, 0.88]0.50 [0.18, 0.76]
GRID-L10.93 [0.66, 1.13]0.58 [0.41, 0.74]
GRID-L20.71 [0.46, 0.96]0.80 [0.61, 0.95]

With OOD kNN support, GRID outperforms all baselines, which we attribute to separating reward learning from decomposition. kNN support is an essential part of GRID, because the general reward is fit only on demonstrated state-action pairs (the results without kNN are in our responses). However, kNN support alone is not sufficient. For example, in Highway-Env, a policy trained on the kNN support reward alone reaches a return of 4.57, vs. 13.35 for GRID-L1 with kNN support.

2. Does GRID reduce to a pointwise mean or median? (xjxj, uzoM)

In the idealized case (exact reconstruction, no identity loss), the L2 and L1 compression terms are minimized by the pointwise mean and median. In practice, GRID differs: each task reward is evaluated only on its own demonstrations, and the identity loss keeps the specific rewards informative. Empirically, with the same pipeline, GRID-L2 outperforms the pointwise mean (0.80 vs. 0.55 for fine-tuning), and GRID-L1 outperforms the pointwise median (0.93 vs. 0.69 for the generalist).

3. Effectiveness of the information bottleneck (NQ7z, uzoM)

Ablations show that each loss is necessary. Without compression, the decomposition fails (general-reward MSE 0.0013 → 1.20). Without the identity loss, the specific-reward MSE triples (0.193 → 0.576). Without the specific reward as an input to the identity network, task identification drops to chance (error 0.8% → 66.7%).

4. Faster adaptation (all reviewers)

We report the fine-tuning reward at three points of training (GRID-L2, the variant with the best fine-tuning score, 5 seeds). GRID-L2 receives only the fine-tuning reward and the KL term. Training from scratch receives the full task reward, so it must also learn the shared skills that GRID's prior already provides.

TaskStepsGRID-L2From scratch
Highway (lane-3 bonus)5k / 10k / 20k1.82 / 2.12 / 2.481.07 / 1.65 / 1.48
Pendulum (skill reward)1k / 2k / 3k99 / 1531 / 2925230 / 614 / 1760
Hopper task 1 (skill reward)25k / 50k / 100k244 / 439 / 522418 / 371 / 518
Hopper task 2 (skill reward)25k / 50k / 100k554 / 1163 / 1840317 / 481 / 725

GRID-L2 adapts faster on three of the four tasks. On Hopper task 1, it leads training from scratch by the middle of training and matches it by the end.

Overall, the reviewers' feedback has made our evaluation more rigorous. We are grateful for their insights, and we ask the Area Chair and the reviewers to consider these results and clarifications in the final evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.