acceptodds
Under review as a conference paper at ICLR 2027

The Reward-Aware Laplacian Representation in Zero-Shot Reinforcement Learning

23 views

Abstract

Representations based on the reward-agnostic successor measure have been shown to be effective in a variety of applications, and, in particular, have led to many advances in zero-shot RL. In this paper, we consider the setting in which, across different tasks, there are shared reward structures that reflect inherent environmental properties, e.g., the negative rewards of wear and tear in robotics. We demonstrate that in this setting, standard zero-shot RL methods can fail to represent downstream reward functions well, as large approximation errors for the shared reward structures can dominate task-specific reward information during inference. We address this problem by learning a reward-aware representation that reduces the burden of representing shared reward structures during inference, enabling better reconstruction of task-specific rewards. We do so by proposing ReAL, a reward-aware generalization of the Laplacian representation. Empirically, we show that compared to reward-agnostic representations, the ReAL representation enables better downstream reward fitting and stronger zero-shot performance in zero-shot goal-reaching tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Representation Learning, Laplacian Representation, Zero-shot RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E1q9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors address the zero-shot RL (ZSRL) setting where all tasks have a shared permanent reward component. They claim that in this setting, learning a reward free latent representation and extracting a task-specific reward representation at inference is inefficient, as small approximation errors on the shared reward component can dominate the signal from the learned task-specific reward representation component. To mitigate this issue, they introduce the reward-aware Laplacian representation (ReAL), a modification of the standard Laplacian representation used in standard ZSRL research. They show experimentally that ReAL outperforms FB and the standard Laplacian method in a series of antmaze environment variants with a low reward region.

Critical strengths
  • the authors clearly identify a limitation of the FB and standard Laplacian frameworks, which has to be addressed
Critical weaknesses
  • The paragraph "Choice of f" seems to imply that this function has to be carefully chosen depending on the properties of such or such shared reward function component: here it is defined for shared reward functions corresponding to low negative rewards. If this is correct, this is a strong limitation of the method: for instance, what if the shared reward component is positive, but lower than that of the goal? Do we have to define another f function?

  • the experimental study is focused on a very narrow set of environments that all share the same permanent reward component structure. It is not clear at all whether the proposed issue and mitigation extend to other domains than maze navigation with a permanent negative reward area. For instance, locomotion tasks (e.g. cheetah penalizing back-leg use) are missing.

  • I believe that the paragraph "Relationships with other representations" should be more forward in the introduction as it contains important elements for the positioning of the work. In particular, if the authors fail to extend their experimental study beyond their narrow ZESP instances, another positioning with respect to these related methods might be possible.

  • the comparison with the baselines is not clean: The FB baseline is not classic FB. Training z’s are replaced by z+zpz + z_p (or z+b⋅zpz + b·z_p), renormalized, with zpz_p periodically re-fit, and a single USFA is shared across all h (Appendix A.2.3). Since FB trains the encoder and policy jointly, this changes what the representation learns. There are signs that this handicaps the baselines: at h=0h = 0 the test reward is the standard goal reward, yet FB drops from 98.8% (Table 6) to 74.0% (Table 8) on antmaze-mn, and from 66.4% to 13.6% on antmaze-ls. It also means the ReAL − FB gap at h=0h = 0 (+24.0 on mn, +40.4 on ln) cannot come from reward-awareness, since no penalty exists there.

  • the mechanism and generality of the claim are not validated. The Gibbs-phenomenon explanation is an analogy backed only by indirect evidence. The approximation error of the permanent reward is never measured, and no theorem shows ReAL reduces it. The permanent reward is also a sharp discontinuity, exactly where Gibbs errors arise, so it is unclear what happens with a smooth penalty (e.g. an inverted Gaussian).

Questions for authors
  • In the maze setting of Fig. 1, are the results the same if the agent gets a large positive reward for reaching the goal, a penalty in the low reward region and 0 elsewhere? And if the low reward region corresponds to a positive reward lower than that of the goal?

  • to address these other settings, do you have to change the ff function or other parameters?

  • Success rate and permanent reward: Success only requires reaching the goal, so crossing the low-reward region still counts, and the return formula is never stated. Many baseline returns are near −1000 (failed episodes), so the return gap mostly reflects "reached the goal or not." Could you state how the return is computed, and report a success rate that only counts episodes reaching the goal without crossing the low-reward region?

  • Could you add FB (and the Laplacian) with standard z-sampling, and with a separate USFA per h, to show the gains are not due to the modified pipeline?

Minor issues, typos:

  • Proposition 4.3 is trivial and could be presented as a remark.
  • line 13: RL -> reinforcement learning (RL)
  • Fig. 2 left, it is not clearly specified where are ground goal localizations (red areas ?) and which corresponds to which infered goal localization
7 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Summary. We thank the reviewer for the helpful feedback. We extended the experiments to humanoidmaze, visual antmaze and the DeepMind Control Suite (Appendix D), where ReAL largely outperforms the baselines, using the same f,cf, c, and λ\lambda as in the existing antmaze experiments with no further tuning. Because the permanent reward is normalized, our algorithm can be directly applied without changing ff even when the shared reward is positive. We stated how returns are computed and reported the number of successful episodes that avoid the low-reward region, where ReAL leads by a wide margin. We retrained FB and the Laplacian with standard z-sampling and with a separate USFA per h, and none of these variants closes the gap to ReAL. We also showed that ReAL’s advantage holds for a smooth inverted-Gaussian penalty.


Q1: The experimental study is focused on a very narrow set of environments.

A1: We further include experiment results in humanoidmaze, visual antmaze, and also Deepmind Control Suite in Appendix D.

For the extra domains in OGBench, apart from very challenging settings such as humanoidmaze-large and visual antmaze-ls, where all methods struggle to perform well, the ReAL representation largely outperforms the Laplacian and FB representations. For DMC, the ReAL representation achieves significantly higher returns compared to the Laplacian and FB representations when the permanent reward component is non-zero.

Our findings in these extra domains confirm our previous findings for antmaze, demonstrating that our results hold across a wide variety of settings.

Q2: It seems like the function ff has to be carefully chosen. What if the shared reward component is positive, but lower than that of the goal?

A2:

Summary: For our experiments, we have not performed extensive tuning of ff, and simply used the sigmoid function due to its robustness. Furthermore, we normalize the shared reward component to have zero mean and unit variance, so our existing procedure can be directly applied to the case where the shared reward is positive, but lower than that of the goal.

Details: In our paper, we simply use the sigmoid function as ff for our OGBench experiments without exploring other choices. We chose sigmoid as it is bounded and robust to outliers. Though we did not extensively tune the choice of ff, as shown in the paper, the ReAL representation significantly outperforms the baselines.

Furthermore, as detailed in Appendix A.1, we normalize the permanent reward component over the dataset to have zero mean and unit variance. This, again, is a design choice that enables the algorithm to be applied robustly to different permanent reward components, without being heavily affected by their respective offsets and scales. As a result, even if the shared reward component is positive but lower than that of the goal, we can simply apply the existing procedure and learn the reward-aware basis functions.

Finally, we directly used the same function ff with the same parameters (λ=1\lambda=1, c=0.5c=0.5) that we used for antmaze for running experiments in humanoidmaze, visual antmaze, and Deepmind Control Suite without any tuning. As shown in Appendix D of the revised paper, the ReAL representation still significantly outperforms the baselines.

In general, while, same as any hyperparameter, extensive tuning could possibly further improve performance, we do not expect the benefits offered by the ReAL representation to only manifest with extensive tuning of ff.

Q3: Could you state how the return is computed, and report a success rate that only counts episodes reaching the goal without crossing the low-reward region?

A3: The example we used in Section 3.2 demonstrates how the downstream reward function is computed (near line 149). For the general case, at every time step, the reward is simply 0 for reaching the goal (upon which the episode terminates), −h−1-h-1 for being in the low-reward region, and −1-1 otherwise. Therefore, the return that we measure already captures the reward for reaching both the goal and low-reward regions. An agent not reaching the goal will accumulate negative reward in all of the 1000 time steps and receive a return ≤−1000\leq -1000. If two agents both reach the goal using the same number of steps, the one that reaches the goal without entering the low-reward region receives a higher return.

We generate 200 episodes for each downstream task in each of mn, ms, ln, ls, and report the number of successful episodes without crossing the low-reward region for ReAL, Laplacian, and FB. The max is 4000 episodes (200 episodes for each of the 20 tasks). ReAL has significantly more such successes:

h01234
ReAL19972003177814941402
Laplacian1479695396321245
FB15051113898656525
AuthorsOpenReview ↗

Rebuttal (2/2)

Q4: Could you add FB (and the Laplacian) with standard z-sampling, and with a separate USFA per h, to show the gains are not due to the modified pipeline?

A4: None of the requested baselines close the significant performance gap between FB/Laplacian and the ReAL representation, confirming that the gain of the ReAL representation does not come from the training paradigm. The success rates for the requested baselines are below.

The success rates for FB with standard z-sampling:

h01234
mn98.085.664.452.440.4
ms81.633.225.624.420.4
ln82.426.815.66.85.6
ls77.62.81.62.82.0

The success rates for FB where each hh is trained separately:

h01234
mn97.694.874.434.438.4
ms76.427.219.610.815.2
ln76.46.81.61.20.4
ls72.01.60.40.00.4

The success rates for Laplacian with standard z-sampling:

h01234
mn98.476.459.644.433.2
ms97.624.012.814.09.6
ln90.429.216.014.89.2
ls47.24.02.43.22.0

The success rates for Laplacian where each hh is trained separately:

h01234
mn95.250.028.015.615.2
ms98.028.020.415.68.4
ln92.85.63.21.63.2
ls50.83.64.84.04.4

Finally, the success rates of ReAL from the paper, for easy comparison here:

h01234
mn98.097.698.497.297.2
ms96.897.675.265.246.0
ln87.676.864.449.639.2
ls34.431.224.415.210.0

Q5: The permanent reward is also a sharp discontinuity, exactly where Gibbs errors arise, so it is unclear what happens with a smooth penalty (e.g. an inverted Gaussian).

A5: We further run a version of the experiments where the penalty is an inverted Gaussian multiplied by hh. Note that for FB and Laplacian, we choose the training paradigm (mentioned in Q4) that gives the highest success rates.

Success rate of OGBench with inverted Gaussian penalty, averaged over all four antmaze settings:

h01234
Laplacian84.235.724.119.717.5
FB87.137.024.720.317.8
ReAL71.064.054.846.838.6

We can see that ReAL still significantly outperforms baselines when h>0h > 0. We further note that the lower number when h=0h = 0 is possibly an artifact of the shared USFA training for different hh. Note that Laplacian and FB numbers here are trained with standard z-sampling, and thus do not suffer from this issue.

Q6: The mechanism and generality of the claim are not validated. The approximation error of the permanent reward is never measured, and no theorem shows ReAL reduces it.

A6: Measuring the approximation error of the permanent reward is not sufficient on its own. This is because approximation errors for just the permanent reward component could be cancelled or amplified by the approximation errors for the task specific component. Therefore, looking at how well each component is approximated in isolation does not inform us of whether task specific information is retained in the full approximated reward.

This is the reason we report metrics on the full approximated reward, and we have provided different forms of evidence validating the mechanism. Figure 1, 6, 7, and 8 are qualitative evidence that directly shows visually that local approximation errors in the fitted reward function (red peak with the black marker) overwhelm the goal peak (green marker). Figures 2 and 4 are quantitative evidence measuring whether the peak of fitted reward function lies near the actual goal.


Conclusion. We believe these results address the reviewer’s concerns about generality, the fairness of the baseline comparison and the mechanism, and we kindly ask the reviewer to consider raising their score.

Reviewer_E1q9OpenReview ↗

Revised version?

The answers to my questions seem to all go in the right direction, but I would like to see the corresponding changes in the paper to make sure. Could the authors quickly share a revised version with the changes highlighted (e.g. in blue)? In particular, I want to make sure that the paper makes appropriate claims about the types of "common reward structures" that the algorithm satisfactorily deals with.

Reviewer_dbW1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a zero-shot RL method, where downstream task rewards consist of a shared permanent penalty observed during pre-training and a task-specific transient goal reward observed only as part of the combined reward at inference. Because smooth reward-agnostic representations such as the Laplacian and Forward-Backward (FB) representations suffer from Gibbs-phenomenon overshoot near sharp penalty boundaries, their linear reward fits obscure the goal location. To address this, the authors propose the Reward-Aware Laplacian (ReAL) representation, which uses a sigmoid function of the shared reward to modulate the magnitude of the Laplacian eigenvectors via a reward-aware augmented Lagrangian objective.

Critical strengths
  • Framing the failure of smooth spectral representations on discontinuous reward penalties through the Gibbs phenomenon is intuitive.
  • Augmenting the standard Laplacian objective with a bounded sigmoid reward penalty is simple yet effective.
Critical weaknesses
  • Since the permanent reward is available in the pre-training dataset, the paper omits two natural baselines: (1) training a scalar reward predictor to append as an extra basis function to the Laplacian or FB representation, and (2) regressing out the predicted permanent reward from the downstream rewards at inference before fitting the task vector.
  • Experiments are restricted to 2D antmaze navigation omitting other OGBench problems.
Questions for authors
  • How does ReAL compare with simple baselines that directly use the known reward?
  • Can ReAL be combined with or compared against one-step and flow-map policies such as FQL (Park et al., ICML 2025) and FMQ (Ziakas et al., NeurIPS 2026)?
2 comments

Summary. We thank the reviewer for the constructive feedback. In response, we have added three sets of experiments.

First, Appendix C compares ReAL with the baseline the reviewer proposed: a learned permanent-reward predictor appended as an extra basis function to the Laplacian and FB representations, with the downstream task vector fitted by least squares. Note that regressing out the predicted reward before fitting the task vector is a special case of this least-squares fit and performs similarly in practice. While this baseline slightly improves both Laplacian and FB, ReAL still significantly outperforms it.

Second, Appendix D now extends the evaluation beyond antmaze to humanoidmaze, visual antmaze and the DeepMind Control Suite (DMC). For extra OGBench domains, ReAL largely outperforms the reward-agnostic representations there, except in the hardest settings (humanoidmaze-large and visual antmaze-ls), where all methods struggle. For DMC, ReAL significantly outperforms the reward-agnostic representations when the permanent reward component is non-zero.

Third, Appendix E now combines ReAL with FQL, which also significantly outperforms the reward-agnostic baselines, showing that ReAL’s benefits are not tied to TD3+BC.


Q1: How does ReAL compare to a baseline that trains an extra scalar reward predictor, appending the predicted permanent rewards as an extra basis function.

A1: We added Appendix C, which compares ReAL with additional baselines, including the one raised by the reviewer. Note that baseline (2) is a special case of baseline (1), corresponding to a least-squares fit with a fixed coefficient for the predicted reward basis. The least-squares fit for (1) should fit the downstream reward at least as well as (2). In practice, we observe them to give similar performance when combined with the Laplacian representation, and thus we only report the numbers of (1) in Appendix C. From Appendix C, we can see that while appending the extra predicted rewards as a basis function slightly boosts performance of Laplacian and FB, they still significantly underperform ReAL.

Q2: Experiments are restricted to 2D antmaze navigation, omitting other OGBench problems.

A2: We further include experiment results in humanoidmaze, visual antmaze, and also Deepmind Control Suite in Appendix D.

For the extra domains in OGBench, apart from very challenging settings such as humanoidmaze-large and visual antmaze-ls, where all methods struggle to perform well, the ReAL representation largely outperforms the Laplacian and FB representations. For DMC, the ReAL representation achieves significantly higher returns compared to the Laplacian and FB representations when the permanent reward component is non-zero.

Our findings in these extra domains confirm our previous findings for antmaze, demonstrating that our results hold across a wide variety of settings.

Q3: Can ReAL be combined with or compared against one-step and flow-map policies such as FQL (Park et al., ICML 2025) and FMQ (Ziakas et al., NeurIPS 2026)?

A3: We include results of combining ReAL with FQL in Appendix E. While it performs slightly worse compared to ReAL with TD3+BC, it still significantly outperforms reward-agnostic baselines, demonstrating that the benefits of ReAL are not tied to TD3+BC. We also note that it is possible that the performance can be further improved with more extensive hyperparameter tuning, and FQL remains a reasonable alternative.


Conclusion. We believe these additions address each of the reviewer’s concerns. ReAL outperforms simple baselines that use the known permanent reward, its advantage holds across domains well beyond 2D antmaze, and it remains effective with a different policy-extraction method. We hope the reviewer agrees that these results strengthen the paper, and we kindly ask the reviewer to consider raising the score. We would be glad to answer any remaining questions.

As we are approaching the end of the discussion phase, we would like to check whether we have addressed all your concerns. We appreciate the work that goes into the review, and we understand that the reviewer might be busy with their own paper, but we would still like to make sure that the reviewer is satisfied with our answers. If the reviewer has any further concerns, we are happy to address them. We also uploaded another revision, which further incorporates the additional experiment results into the main text, with major changes highlighted in blue.

Reviewer_vparOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies zero-shot RL where downstream rewards share a permanent component observed during pre-training. It argues that reward-agnostic bases such as the Laplacian and FB produce Gibbs overshoots when fitting this component, obscuring the task-specific goal. ReAL adds a reward-dependent diagonal term to the Laplacian, Lr=I−P+cRL_r=I-P+cR, with R=diag(σ(−r/λ))R=\mathrm{diag}(\sigma(-r/\lambda)), and learns its eigenvectors by modifying ALLO. On four OGBench antmaze datasets with added low-reward regions, ReAL preserves goal localization more reliably than the baselines. For penalty magnitudes h=1h=1–44, its average success rate is 36–55 percentage points higher.

Critical strengths
  1. The problem is real and well illustrated. A large shared reward component can swamp the task-specific signal when rewards are fitted in a fixed basis. The Gibbs interpretation in Sec. 3.2 is useful, and Fig. 4 and Tables 7–8 show the failure clearly for both baselines.

  2. ReAL requires only one additional term in ALLO, and the empirical gains are large across all four datasets. The paper reports returns alongside success rates and checks its baseline implementations on standard OGBench (App. B.1).

Critical weaknesses

Some checks below use an idealised toy: exact eigenvectors of a random walk on a 0.5-unit grid over the OGBench mazes, uniform data, d=50d=50, and the low-reward cells shown in Fig. 5. I use the paper's reward function with a goal radius of 1 and its localization threshold of 3.5. Large-maze results average over the four distinct goals. Barrier strengths are relative to the 50th Laplacian eigenvalue, λ50\lambda_{50}. I can share the code through the AC.

Weakness 1. The claimed explanation for ReAL's gains is untested.

Sections 4.2 and 5.1 attribute the gains to a basis that better captures the permanent reward. Yet the operator suppresses the basis functions on the low-reward region WW. Strong suppression can keep the goal visible in the fitted reward while losing most of the penalty.

Because the permanent reward is binary, the diagonal potential takes only two values. We can write

\qquad \kappa=c[f(x_W)-f(x_{\bar W})],$$ Here $f(x)=\sigma(-x)$, $x_W,x_{\bar W}$ are the standardised rewards, and $C=cf(x_{\bar W})$. At the paper's $c=0.5$ and $\lambda=1$, $\kappa\approx0.27$ when $W$ accounts for 3–10% of the data. The exact eigenvectors depend on $f$ only through $\kappa$, so the experiments test only the barrier height $\kappa$, not the shape of $f$. Let $\mu_d$ be the $d$-th eigenvalue of $L_r-CI$, and let $\Pi$ project onto its first $d$ eigenvectors. Since $L\succeq0$, every $u$ in this span satisfies $$\kappa\\,\mathbb E_\rho[\mathbf1_Wu^2] \leq\mu_d\\,\mathbb E_\rho[u^2].$$ For the raw permanent reward $r_p=-h\mathbf1_W$, this gives $$\frac{\\|\Pi r_p\\|_\rho^2}{\\|r_p\\|_\rho^2} \leq\frac{\mu_d}{\kappa}.$$ A small $\mu_d/\kappa$ thus rules out accurate reconstruction of the raw penalty. In the toy, the captured share decreases throughout the barrier sweep, from 0.84 for the Laplacian to less than 0.001 at $\kappa=10\lambda_{50}$. In the table, localization is averaged over $h=1,\ldots,10$. “Stated” uses the paper's reward unchanged; “centred” subtracts its mean. The last column reports the mean fitted reward outside $W$ minus the mean inside, divided by $h$. | Basis | Share of raw $r_p$ captured | Localization, stated | Localization, centred | Penalty contrast / $h$, centred | |---|---|---|---|---| | Laplacian | 0.84 | 0.05 | 0.05 | 0.83 | | ReAL, $\kappa=\lambda_{50}/3$ | 0.80 | 0.23 | 0.23 | 0.79 | | ReAL, $\kappa=3\lambda_{50}$ | 0.007 | 0.00 | 0.42 | 0.06 | | ReAL, $\kappa=10\lambda_{50}$ | 0.0007 | 0.00 | 1.00 | 0.06 | | 49 Laplacian eigenvectors + orthogonalised $r_p$ | 1.00 | 1.00 | 1.00 | 1.00 | For the strong barriers in the table, the stated reward is fitted near zero on $W$ and near $-1$ elsewhere. Its maximum lies in $W$, even at $h=0$. Centering changes this because adding a constant $a$ changes the fit by $a\Pi\mathbf1$. The Laplacian reproduces constants exactly; ReAL's fit of a constant falls towards zero on $W$. At $\kappa=10\lambda_{50}$, centering gives perfect localization while retaining only about 6% of the penalty contrast. Thus, a fitted surface can show a clear goal and deepening pits while badly understating the cost of entering $W$. Figures 1 and 8 need a common vertical scale to show how much of that cost is retained. The paper needs to establish whether its learned functions behave this way. Please report: - whether rewards are centred, standardised or shifted before computing $z$; - the Rayleigh quotients of the learned functions under $L_r-CI$, alongside $\kappa$; - the share of the raw and centred permanent reward captured by each representation; - fitted rewards on a common vertical scale, or their penalty contrast relative to the true penalty. **Weakness 2.** The evaluation does not establish that agents use the shared penalty. I computed shortest paths on the OGBench cell grids using the low-reward cells in Fig. 5. Every large-maze task has a shortest path avoiding $W$. In the medium maze, only task 4 requires a detour, of two cells. Nine of the ten tasks do not require the detours described in App. A.1. Success rates measure goal reaching, while returns combine travel time with penalty exposure. On medium-navigate, ReAL's return falls by 10.5, 21.9, 45.7 and 92.8 at $h=1$–$4$, while success remains at 97–98% (Tables 9 and 12). The tables do not show whether these losses come from longer episodes or more time in $W$. Localization is also a poor quantitative guide to success. For the Laplacian on medium-navigate, localization stays at 1.0 for $h=1$–$4$, while success falls from 52.4% to 14.8% (Fig. 4, Table 7). For ReAL at $h=4$, localization is about 0.9 on both large datasets, but success is 39.2% and 10.0% (Table 9). The training protocol deserves an ablation as well. Appendix A.2.3 adds $b\bar z_p$ to sampled conditioning vectors before normalising them, with $b\sim U[0,10]$ and $\bar z_p$ fitted to the unit penalty. The shift depends on the representation: in the toy, $\\|\bar z_p\\|$ is 0.23 for the Laplacian and 0.011 for ReAL at $\kappa=6\lambda_{50}$. At $h=0$, where the downstream reward is unchanged from standard OGBench, Laplacian success falls from 94.8% to 78.8% on large-navigate and from 63.6% to 24.8% on large-stitch; FB loses 20–53 percentage points across datasets (Tables 5–8). Section 5.2 suggests interference from training one USFA across all $h$. The sampling shift is a plausible contributor to those losses. Please: - add tasks where the shortest route crosses $W$ and a detour gives a better return, as in the reward-shaping environments of Tse et al. (2025); - report time spent in $W$ and success per task; - train all methods with and without the sampling shift; - extend Table 1 to $h=10$, matching the training range and Fig. 4. **Weakness 3.** Two simple and important baselines are missing. The most direct way to put the permanent reward into the representation is to append it as a feature. A neural predictor can be trained on the labels for the permanent reward already available during pre-training. The extra feature should be orthogonalised against the others, or the inference coefficients fitted by least squares with the feature Gram matrix. With exact $r_p$, the fit becomes $\Pi r_t+r_p$: the permanent component is represented without error. In the toy, 49 Laplacian eigenvectors plus this feature localize every tested goal at every $h$. This implements the paper's “offloading” idea directly and should be a baseline for both the Laplacian and FB. The second baseline is spectral filtering. The paper diagnoses a Gibbs phenomenon and cites Gottlieb & Shu (1997), but does not test a remedy. A heat-kernel filter, $$z_i\longmapsto e^{-\alpha\lambda_i/\lambda_d}z_i,$$ uses eigenvalues that ALLO already estimates and requires no rewards during pre-training. With $\alpha=2$–$3$, it raises the Laplacian's localization at $d=50$ from 0.15 to 0.78–0.80 in the toy, using the Fig. 2 convention of averaging over $h=0,\ldots,9$. The argument against larger bases in Sec. 3.2 is also too broad. Persistent overshoot need not prevent the goal peak from becoming dominant. In the toy, localization reaches 0.47 at $d=150$ and 0.72 at $d=200$. Appendix B.4 reports control results at $d=100$; localization results beyond $d=150$ would show whether the plateau in Fig. 2 persists. **Weakness 4.** The connection to the default representation is closer than stated. Section 4.1 argues that ReAL applies more broadly than the default representation (DR), citing the DR's terminal-state assumption and exponential reward transformation. But substituting $$\tilde r=-\tau\log(1+cf(r/\lambda))$$ into the DR definition gives $$\mathrm{DR}(\tilde r) =\left[\mathrm{diag}(e^{-\tilde r/\tau})-P\right]^{-1} =L_r^{-1}.$$ ReAL shares the DR's eigenvectors under this transformed reward. Tse et al. (2025) already use the DR without terminal states. The ReAL eigenproblem also corresponds to a continuous-time, average-cost linearly solvable control problem with passive generator $P-I$ and state cost $cf(r/\lambda)$ (Todorov, 2009). I would present ReAL as a DR variant with a bounded reward transformation and an objective for learning several eigenvectors. The claim of broader applicability in Sec. 4.1 should be revised. **Minor points.** - The motivating examples of wear and tear and energy use often depend on actions, while the formalism is state-based. - “Around 40% higher” in Sec. 1 should read “40 percentage points higher.” - The encoder hyperparameters and $c$ were selected by localization on the evaluation mazes and goals; the behaviour-cloning weight was selected by the best result per dataset. Please report sensitivity to $c$. - The reward's goal radius, the number of reward samples used to fit $z$, and the states over which the maximum is taken are not stated. These choices matter: changing the toy's goal radius from 0.5 to 1.5 changes Laplacian localization at $d=50$ from 0.10 to 0.43 under the Fig. 2 convention. - Appendix A.1 gives a success radius of 1.0, whereas OGBench uses 0.5 for the ant. If the experiments use 1.0, the comparison with Bagatella et al. (2026) is not like for like. - The large maze has four distinct goals; tasks 4 and 5 share one. One goal is close enough to $W$ that an overshoot near its boundary can count as localized under the 3.5-unit threshold. - Figure 3 does not state $c$ or $\lambda$. ReAL is a simple change to ALLO with large gains. The paper still needs to show what produces those gains and whether the agent uses the shared penalty. I would raise my score to weak accept if the revision: - identifies the mechanism through the diagnostics in Weakness 1 and revises the explanation accordingly; - shows that ReAL can reach the goal while choosing worthwhile detours around $W$; - compares against the appended-$r_p$ baseline, with ReAL remaining competitive.
Questions for authors
  1. Are downstream rewards centred, standardised or shifted before computing zz? If not, what keeps the fitted maximum out of WW at h=0h=0?
  2. What are the Rayleigh quotients under Lr−CIL_r-CI, the mass of each learned function on WW, and the captured fractions of raw and centred rpr_p?
  3. How much time do agents spend in WW, per task, method and hh?
  4. Is the success radius 1.0, as stated, or OGBench's 0.5?
  5. What goal radius, number of reward samples and set of states are used to compute localization?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Summary. We thank the reviewer for the detailed review and for building a toy model of our setting. Our responses can be summarized as follows.

First, on the mechanism behind ReAL’s gains (Q1), the possible failure mode the reviewer points out, where low-reward regions are almost completely suppressed, does not arise in our experiments. The learned ReAL representation retains on average 44% of the penalty contrast across the four antmaze datasets and h=1,…,10h = 1, \dots, 10, which is far from the near-complete suppression (6%) the reviewer’s toy produces under extreme barrier strengths.

Second, on whether agents use the shared penalty (Q4), while the reviewer's tabular toy approximation of OGBench suggests most shortest paths already avoid the low-reward regions, in practice, agents trained on the standard OGBench datasets without any low-reward region still enter these regions in at least 20% of episodes in 11 of the 20 task–dataset combinations. On these tasks, the fraction of successful ReAL episodes that cross the low-reward region falls from 63% at h=0h=0 to 29–36% for h≥6h\geq6, demonstrating meaningful low-reward avoiding behavior.

Third, we added the two baselines the reviewer proposed: appending the permanent reward as a feature, and heat-kernel spectral filtering (Appendix C). We also ran the sampling-shift ablation (response A4 to Reviewer E1q9). ReAL significantly outperforms all of them.

Finally, on the connection to the DR (Q3), the equivalence between the DR and ReAL the reviewer points out holds only for the transformed reward under a restricted setting. Further, existing DR objectives in the function-approximation setting also rely on terminal states, which ReAL does not require. We therefore maintain our claim that ReAL is more broadly applicable than the DR.

We have also corrected the success-radius typo (0.5, as in Bagatella et al., 2026) and clarified the localization protocol and the sensitivity to c.


Q1: The claimed explanation for ReAL's gains is untested.

Q1.1: “Sections 4.2 and 5.1 attribute the gains to a basis that better captures the permanent reward. Yet the operator suppresses the basis functions on the low-reward region. Strong suppression can keep the goal visible in the fitted reward while losing most of the penalty.”

A1.1: It is clear from Figure 8 that this is not the case for our experiments. We can see from the visible blue dips at the low-reward regions that the fitted reward does not lose most of the penalty, while simultaneously keeping the goal visible, as evident from the red peaks with the green marker.

Q1.2: The reviewer then shows quantitatively with a table that, in extreme cases, the fitted reward function can almost totally ignore the low-reward region.

A1.2: We would like to remind the reviewer that the hyperparameter cc controls the suppression strength, so the extreme cases are likely very avoidable in practice by hyperparameter tuning. We also measure the penalty contrast, as suggested by the reviewer, which is around 44% for the ReAL representation in our experiments, averaged over all 4 antmaze settings and h=1,…,10h = 1,\dots, 10. This is far from the extreme case, 6%, that the reviewer is concerned about.

Overall, we respectfully disagree that just because the ReAL representation can fail to capture shared reward information under extreme hyperparameter values, “the claimed explanation for ReAL’s gains is untested.”

Q2: Two simple and important baselines are missing.

A2: We include the additional baselines in Appendix C. Both of the baselines still significantly underperform the ReAL representation.

Q3: The connection to the default representation (DR) is closer than stated.

Q3.1: The reviewer substitutes a particular choice of the reward function into the definition of the DR, and shows that the reward-aware Laplacian (ReAL) is the inverse of the DR.

A3.1: The reviewer has identified a special case, where the DR and ReAL, computed for two distinct reward functions under the unstated assumption of the existence of terminal states, give the same eigenvectors.

Respectfully, we are not sure why an instance of eigenvector equivalence (not even when ReAL and DR are computed w.r.t. the same reward function) in a restricted setting invalidates our claim that ReAL is more broadly applicable than the DR.

Q3.2: “Tse et al. (2025) already use the DR without terminal states.”

A3.2: This is only true in the tabular case, when the eigenvectors of the DR are approximated in an online setting (Tse et al, 2025). In fact, in a subsequent paper extending the DR to the function approximation setting, the resulting objective directly relies on the existence of terminal states (Tse & Machado, 2026). Therefore, the fact that ReAL representation can be applied without terminal states in the function approximation case meaningfully supports our claim that ReAL applies more broadly than the DR.

AuthorsOpenReview ↗

Rebuttal (2/3)

Q4. The evaluation does not establish that agents use the shared penalty.

Q4.1: “I computed shortest paths on the OGBench cell grids using the low-reward cells in Fig. 5. Every large-maze task has a shortest path avoiding W (the low-reward region). In the medium maze, only task 4 requires a detour, of two cells. Nine of the ten tasks do not require the detours described in App. A.1.”

A4.1:

Summary: This may be true for the idealized, tabular approximation of OGBench that the reviewer operates on. In practice, we train on actual OGBench datasets with behavior cloning, and thus the behavior in the dataset heavily affects agent’s behavior. We observe that, different from what the reviewer concludes from the tabular toy, agents trained on the standard OGbench datasets regularly pass through our selected low-reward regions. Therefore, reaching the goal while avoiding the low-reward region requires meaningful behavioral change, and the agent cannot simply repeat the behavior it takes when there is no low-reward region.

Details: We train the Laplacian representation on the standard OGBench datasets, generating 200 episodes per downstream task and measuring the percentage of episodes that passes through the low-reward region.

Task 1Task 2Task 3Task 4Task 5
mn12198950
ms055931000
ln1299773
ls2915565216

We can see that the agent passes through the low-reward regions in more than half of the 20 settings above. Different from what the reviewer suggests, the low-reward regions we used in the paper are far from being irrelevant to solving the task.

In fact, downstream tasks where a standard agent almost never passes through the low-reward region actually further highlight the severity of the Gibbs phenomenon. Even if the low-reward region is totally irrelevant and far away from the optimal path to reach the goal, the approximation errors could still obscure the goal location, causing a failure to reach the goal.

To further demonstrate that the trained ReAL agents exhibit low-reward-region avoiding behavior, we generate 200 episodes for each task in each of mn, ms, ln, ls, and measure the average proportion of successful episodes that cross the low-reward region.To focus on the tasks where the low-reward region lies on the route an agent would naturally take, we average only over the 11 tasks where an agent trained without the low-reward region enters it in at least 20% of its successful episodes (see the above table). A lower percentage means that, when the agent reaches the goal, it more often does so without entering the low-reward region.

h012345678910
Mean proportion of episodes crossing low-reward region6351484542433536293335

We can see that the percentage of episodes crossing the low-reward region drops as h increases from 0 to 6, and then plateaus afterwards. This effectively demonstrates that as the penalty for the low-reward region increases, the agent reaches the goal while avoiding the low-reward region more often. Note that the avoidance effect is not extremely strong here since for OGBench, TD3+BC biases the agent’s behavior towards the behavior taken in the dataset.

Q4.2: The tables do not show whether the return loss when h increases comes from longer episodes or more time in low-reward regions.

A4.2: We compute the return decrease for the ReAL representation, comparing h=0h = 0 and h=4h = 4. We also report the difference in the number of steps spent in the low-reward region (the number for h=4 −h=4 \ - the number of h=0h=0).

Return decreaseDifference in the Number of Steps Spent in Low-reward Region
mn92.7-3.0
ms425.0-5.5
ln316.2+4.8
ls199.4-8.3

The difference in the number of steps spent in the low-reward regions is minimal. The return loss mostly comes from longer episodes.

Q4.3: “Localization is also a poor quantitative guide to success.”

A4.3: Localization rate was never meant to be a quantitative guide to success. As stated near line 367, “To measure how well different representations preserve goal information, similar to Section 3, we measure the goal localization rate.”

AuthorsOpenReview ↗

Rebuttal (3/3)

Q4.4: “The training protocol deserves an ablation as well”

A4.4: Please refer to our response A4 to reviewer E1q9. The ablations, while slightly improving the baselines, still significantly underperform the ReAL representation.


Response to minor points

  1. The ReAL representation achieves similar success rates for c=0.5,1,5c = 0.5, 1, 5, and the success rates drop significantly as cc decreases to 0.1,0.050.1, 0.05.
  2. The reward’s goal radius is 0.5. We use 50K transitions to fit zz. We compute the maximum over a 120 x 120 grid of xy positions covering the maze interior.
  3. We had a typo in Appendix A.1. Same as Bagatella et al. (2026), we use 0.5 as the success radius.
  4. For Figure 3, we use c=1c = 1 and λ=1\lambda = 1.
  5. Downstream rewards are centered before computing zz.

Conclusion. The reviewer set out three conditions for raising their score to weak accept, and we have addressed each of them. On the mechanism, we report the reward-centering protocol and the penalty contrast of the learned ReAL representation. These show that in our experiments ReAL keeps the goal localized while retaining a substantial share of the penalty, rather than discarding it by suppressing the low-reward region. On detours, we show that the low-reward regions lie on routes that standard agents actually take. ReAL agents reach the goal while avoiding these regions more often as the penalty grows. The added cost comes from longer episodes, rather than from more time spent inside them. On the appended-reward baseline, we added it to Appendix C together with heat-kernel filtering. ReAL does not merely remain competitive with these baselines but significantly outperforms them. We hope these results resolve the reviewer’s concerns, and we kindly ask the reviewer to raise their score accordingly. We are happy to answer any further questions during the discussion period.

As we are approaching the end of the discussion phase, we would like to check whether we have addressed all your concerns. We appreciate the work that goes into the review, and we understand that the reviewer might be busy with their own paper, but we would still like to make sure that the reviewer is satisfied with our answers. If the reviewer has any further concerns, we are happy to address them. We also uploaded another revision, which further incorporates the additional experiment results into the main text, with major changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.