acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Control with Gaussian Representations and Test-Time Adjustment

129 views

Abstract

Zero-Shot Reinforcement Learning (ZSRL) pretrains general-purpose policies from reward-free experience to enable deployment on unseen rewards without further learning. Successor Feature (SF)-based methods facilitate this transfer by fitting a linear reward coefficient to retrieve a pretrained policy. However, existing approaches often overlook how representation geometry affects SF evaluation, while direct policy retrieval can be suboptimal under reward misspecification and function-approximation errors. First, we introduce Gaussian Latent Dynamics-based Successor Features (GLDSF), which combines latent dynamics prediction with regularization toward an isotropic Gaussian to learn control-relevant representations with well-conditioned geometry for SF evaluation. To mitigate retrieval suboptimality, we propose a model-based test-time adjustment procedure that searches over the pretrained policy family rather than over action sequences, with no parameter updates or additional environment interaction. We provide analysis supporting this adjustment design. On the ExORL benchmark, planning-free GLDSF is competitive with the strongest state-based baseline and outperforms the strongest pixel baseline by 24.0% in aggregate. Incorporating test-time adjustment further boosts aggregate performance by 7.1% on states and 8.4% on pixels. Ablation studies further examine the effects of representation regularization, search space, and other test-time adjustment design choices.

Primary area:
reinforcement learning
Keywords:
zero-shot reinforcement learning, representation learning, test-time optimization

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 2 reviews, above 99% of the others.
Reviewer_7attOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors "introduce Gaussian Latent Dynamics-based Successor Features (GLDSF), which combines latent dynamics prediction with regularization toward an isotropic Gaussian to learn control-relevant representations with well-conditioned geometry for SF evaluation.". That is, they build successor features in the ZSRL framework on a representation based on latent dynamics in such a way that the underlying geometry is well captured by an isotropic Gaussian, which has several favorable mathematical properties. They also improve the policy retrieval mechanism over the standard SF retrieval mechanism by searching for the most appropriate policy without further policy rollout. They backup the approach with some theory and show that it outperforms other frameworks on state-based and pixel-based ExoRL benchmarks.

Critical strengths
  • Clearly distinguishing the reward coefficient w and the behavior code z that are often conflated is a useful conceptual contribution
  • the authors provide compelling evidence that regularizing the latent successor features representation towards an isotropic Gaussian has a positive effect on performance in the pixel-based case.
  • the model-based test-time adjustment mechanism is also a clever contribution
  • the experimental methodology is clean
Critical weaknesses
  • The authors do not have a reproducibility statement and they do not release the code, but they provide it with the paper
  • In Section 6.2, it seems to me that the relationship to using skills is misleading: "More generally, pretrained behaviors can serve as a basis for constructing downstream policies through skill composition"... In this work, a task, hence a reward function, induces a single choice of z, the framework does not use different z over time to perform a single episode, otherwise there is something I misunderstood.
Questions for authors
  • The fact that isotropic Gaussian regularization provides no advantage in the state-based case is unexpected after reading the methods. The authors probably slightly oversold this contribution before getting these experimental results. Helping the reader better understand why this regularization makes a difference in the pixel-based case and not in the state-based case would make the paper much stronger. Can you think of any toy environments where its effect can be better scrutinized?

  • Why not use a figure similar to Fig. 2 to replace Table 3? A figure would be more readable...

Minor points, typos:

  • it seems that θ\theta is not used, naming policies πθ\pi_\theta would be more standard than πν\pi_\nu
  • line 118: Test-time adjustment then search -> searches
  • 119 : a USF -> an USF (?)
  • Eq. (3) and line 130: aren't LfL_f and LdynL_{dyn} the same thing?
  • 141 code , -> code,
  • 360: It improves 9 of the 12 state-based dataset–domain entries and 7 of the 12 pixel-based entries. -> wasn't it 4 domains? This raises a cherry-picking alert!
  • 410: sequences , -> sequences,
  • 441: Hierarchical approach (Chandrasekar & Machado, 2026) -> A hiera...
2 comments

Weakness 1 (Reproducibility) We added the reproducibility statement.

Weakness2 (Skill composition) Thank you for noting this ambiguity. While w^\hat w is fixed per episode, GLDSF-TTA re-searches the behavior code at every step (revised Section 3.2), so the selected policy can change (closed-loop control). We revised Section 6.2 to distinguish this state-dependent switching from explicit skill composition: GLDSF-TTA selects codes in the code space at each state, emerging temporal composition but without explicitly constructing composite options or learning separate skill-level dynamics.

Question1 (Modality-dependent regularization) Thank you for this insightful suggestion. We clarify that Theorem 4.1 bounds worst-case SF error amplification (under fixed feature energy and Bellman residuals) rather than guaranteeing improved average control performance in all settings.

To better explain the modality-dependent effect, we added a controlled 2D PointMass goal-reaching study in Appendix E. It has deterministic, fully actuated dynamics and six goal locations, with rewards decaying exponentially with squared goal distance. Both modalities use identical dynamics, rewards, and pretraining trajectories; only observations differ: clean 2-dimensional physical coordinates versus 24×2424\times24 rendered pixels. The feature dimension dd is fixed at dd=16 throughout. We report here: (1) condition number (max/min eigenvalue ratio of the feature second-moment matrix); (2) effective rank (entropy-based feature dimensionality); (3) position R² (average linear-probe R² for physical coordinates); and (4) control return (discounted downstream reward of the frozen policy). For the specific information please see newly added Appendix E. All values in tables are mean ±\pm standard deviation over six seeds and rounded to one decima.

Table 4: Representative PointMass results.

ObservationMethodReturn (×100) ↑Condition Number ↓Effective Rank ↑Position Probe ↑
StateNoReg63.1±4.063.1 \pm 4.01261.06.41.0
StateIsotropy-only67.8±3.467.8 \pm 3.411.214.00.9
StateGaussian67.1±6.167.1 \pm 6.111.314.00.9
PixelNoReg45.0±4.345.0 \pm 4.3134667.01.91.0
PixelIsotropy-only58.2±4.258.2 \pm 4.215189.43.51.0
PixelGaussian64.9±2.564.9 \pm 2.540.112.11.0

The key difference is the feature geometry. NoReg retains linearly decodable positions in both modalities, but its pixel representation is much more ill-conditioned (~107× higher condition number than states; effective rank 1.9/16). Gaussian regularization alleviates this severe spectral concentration (condition number 40.1; effective rank 12.1), accompanying a substantial return increase. In contrast, NoReg state observations already have a comparatively reasonable geometry, leaving less room for additional regularization to translate into control gains. Indeed, the state-based return increases only modestly.

Table 5: Gaussian regularization coefficient sweep on pixel observations.

Coefficient α3\alpha_3Condition Number ↓Effective Rank ↑Position Probe ↑Return (×100) ↑
101.6±0.11.6 \pm 0.115.9±0.015.9 \pm 0.00.2±0.10.2 \pm 0.128.5±2.328.5 \pm 2.3
1.01.7±0.11.7 \pm 0.115.8±0.015.8 \pm 0.00.7±0.20.7 \pm 0.261.7±2.261.7 \pm 2.2
0.140.1±15.140.1 \pm 15.112.1±0.312.1 \pm 0.31.0±0.01.0 \pm 0.064.9±2.564.9 \pm 2.5
0.05668.6±171.6668.6 \pm 171.68.4±0.18.4 \pm 0.11.0±0.01.0 \pm 0.068.8±2.868.8 \pm 2.8
0.0115588.2±4473.915588.2 \pm 4473.93.8±0.13.8 \pm 0.11.0±0.01.0 \pm 0.055.9±1.955.9 \pm 1.9
0.00522140.7±4604.322140.7 \pm 4604.33.2±0.53.2 \pm 0.51.0±0.01.0 \pm 0.055.0±6.255.0 \pm 6.2
0.00138928.1±4735.738928.1 \pm 4735.73.0±0.13.0 \pm 0.11.0±0.01.0 \pm 0.048.9±14.548.9 \pm 14.5

The coefficient sweep reveals a trade-off between geometry and task information. Weak-to-moderate regularization improves conditioning, effective rank, and return, whereas excessive regularization (α₃=10) pushes condition number toward 1 and effective rank toward 16 while suppressing task information—dropping position-probe score to 0.2 and return to 28.5 (below NoReg's 45.0). Thus, effective control requires balancing geometric regularity with preserving task-relevant features, as favorable geometry alone is insufficient. More results are presented in Appendix E.

We have revised Sections 4.1 and 5.3 to clarify the scope of the theoretical result and to better understand geometry importance, respectively.

Question 2 We followed the suggestion and replaced Table 3 with Figure 2; average returns are reported in Section 5.3.

Minor: We addressed all typos, including "improves 9 of the 12 state-based…". We appreciate you pointing this out! Our original comparison between GLDSF and GLDSF-TTA included insignifaicant entries due to an oversight. To prevent overstatement, we have removed the per-entry win counts and focus on aggregate performance and task variation.

Reviewer_7attOpenReview ↗

Fine

OK, the paper looks even stronger with the new additions, I'm still in favor of accepting it

Reviewer_3y5sOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces GLDSF, a successor feature based zero shot RL algorithm. It combines a RLDP-style dynamics based representation loss as well as the TD-JEPA predictor and feature loss into a single algorithm, allowing not only for normal zero-shot reward inference but also test-time adaptation (TTA) of the latent using latent dynamics rollouts. The method follows recent advancements in self-predictive feature learning and regularizes its features using a SIGReg-style loss to be close to isotropic Gaussian, whereas prior successor feature methods opt for an orthogonality loss and projecting features onto a hypersphere. The resulting algorithm is competitive with state of the art zero-shot RL algorithms and surpasses them when adding TTA because it allows training an explicit reward model on the inference dataset overcoming the errors that arise from projecting the reward function onto latent features. Notably, the choice of latent regularization and the TTA search are well-motivated by the theoretical results in the paper.

Critical strengths

The paper introduces a state of the art zero-shot RL method that can boost its own performance through TTA search. All design choices of this algorithm seem to be well-motivated by the performance of prior work (TD-JEPA and RLDP) or the theoretical results in the paper.

Critical weaknesses

None

Questions for authors

I do not have clarifying questions to the authors, the paper is clearly understandable and self-contained. Both theorems 4.1 and 4.2 seem to suggest that the feature covariance matrix being close to the identity matrix to be an important factor in how well successor features are learned and TTA performs. While the SIGReg loss is well motivated and validated in practice, I wonder whether the approach of constraining features to the hypersphere and using orthogonality losses as done in Forward Backward and TD-JEPA might result in a better conditioned feature basis in practice. It would be interesting to see the performance of this version as an ablation in Section 5.3, i.e. a “Isotropy-only + Hypersphere projection” version.

2 comments

We thank the reviewer for this valuable suggestion. Motivated by the use of hyperspherical representations in prior work, we added an additional study in Appendix F.1 to explore fixed-norm feature normalization and its interaction with our distributional regularization.

Importantly, hyperspherical normalization and Gaussian regularization impose distinct but compatible geometric constraints. Hyperspherical normalization fixes the norm of each feature vector, whereas SIGReg matches random one-dimensional projections to standard Gaussian marginals without imposing a fixed norm. These objectives also have a complementary maximum-entropy interpretation. Among distributions supported on a fixed-radius hypersphere, the uniform distribution maximizes entropy with respect to the sphere's surface measure. Under a fixed-norm constraint, SIGReg's Gaussian projection matching can encourage approximately uniform angular coverage. This provides a complementary maximum-entropy perspective consistent with the motivation of Eq. (6). However, the two objectives are not equivalent: at finite dimensions, spherical projections are not exactly Gaussian, and neither fixed-norm normalization nor Gaussian projection matching alone guarantees the other constraint. This motivates exploring their combination rather than treating them as mutually exclusive alternatives.

Specifically, we normalize the encoder output to a hypersphere of radius d\sqrt{d}, preserving the feature-energy scale of our Gaussian target. We investigate three variants: Sphere-only, Sphere + Isotropy, and Sphere + Gaussian. All other components, including the latent dynamics model, USF, actor, and their training objectives, remain unchanged.

Table 9: ExORL performance with hyperspherical feature normalization. Entries report mean ±\pm standard deviation over six seeds; domain scores are averaged over the same four tasks as in the main experiments.

ObservationDomainNoRegIsotropy-onlyGaussianSphere-onlySphere + IsotropySphere + Gaussian
StateCheetah433±14433 \pm 14435±44435 \pm 44404±84404 \pm 84504±26504 \pm 26485±26485 \pm 26538±93538 \pm 93
StateWalker656±43656 \pm 43618±15618 \pm 15648±30648 \pm 30641±9641 \pm 9599±4599 \pm 4630±17630 \pm 17
StateQuadruped640±45640 \pm 45633±31633 \pm 31663±5663 \pm 5593±21593 \pm 21646±9646 \pm 9643±13643 \pm 13
StateAverage576562572579577604
PixelCheetah252±34252 \pm 34440±14440 \pm 14529±29529 \pm 29371±138371 \pm 138562±23562 \pm 23600±33600 \pm 33
PixelWalker243±43243 \pm 43388±15388 \pm 15437±42437 \pm 42365±34365 \pm 34401±9401 \pm 9425±28425 \pm 28
PixelQuadruped391±81391 \pm 81459±33459 \pm 33414±65414 \pm 65400±75400 \pm 75354±48354 \pm 48321±69321 \pm 69
PixelAverage295430460379439449

The results suggest that hyperspherical normalization can provide a useful additional constraint, particularly for pixel representations. Sphere-only improves the pixel-based average return over NoReg (379 vs. 295). Combining hyperspherical normalization with second-order isotropy shows competitive average return over Isotropy-only on both states (577 vs. 562) and pixels (439 vs. 430), although the gains are not uniform across domains. Importantly, Sphere + Gaussian demonstrates the compatibility of fixed-norm normalization with Gaussian regularization, achieving average returns of 604 on states and 449 on pixels, compared with 572 and 460 for the original Gaussian variant, respectively. These results suggest that the interaction between fixed-norm and distribution-level constraints can be beneficial.

We include this as an additional exploration in Appendix F.1, rather than a modification of the main method. The results illustrate that hyperspherical normalization and Gaussian regularization offer complementary geometric constraints, without establishing either as universally superior.

Dear authors, thank you for adding this extra exploration.

Comments on the paper

5 comments
Area_Chair_a4R3OpenReview ↗

Official Review

Paper summary:

The paper studies successor-feature-based zero-shot RL and makes two contributions, which combine to form a method named Gaussian Latent Dynamics-based Successor Features (GLDSF). First, GLDSF learns the state representation from latent dynamics prediction and a universal SF (USF) loss, with a regularizer that pushes the features toward an isotropic Gaussian, which the authors argue gives a well-conditioned basis for fitting unseen rewards. Second, a mechanism to choose actions based on test-time adjustment keeps the fitted reward coefficient fixed and searches over the behavior codes of the frozen policy family. Each candidate code is scored by a short rollout in the learned latent model plus an SF terminal value. The paper provides supporting bounds on the method's resulting value error. The methods is evaluated on the ExORL benchmark against several baselines.

Presentation: 2: Generally understandable, but requires substantial effort in places.

Informativeness: 2: I learned something interesting or useful.

Soundness: 3: I have no substantive concerns about the paper’s soundness.

Critical Strengths:

  • Separating the task coefficient from the behavior code lets the method search for a better policy within the frozen pretrained family. It requires no fine-tuning and no new environment data. The comparison with action-space MPC suggests this is an effective search space.

  • The planning-free model improves on the best baseline by 24% on pixels. The test-time adjustment adds gains on both states and pixels. The evaluation is reasonably broad, and ablations isolate the main design choices.

Critical Weaknesses:

  • Regarding the paper's presentation, the Introduction and Section 2 are dense, and several terms are used before they are properly explained, such as “behavior code.” For example, this sentence is difficult to follow and relies on terminology not yet introduced: “Our analysis further shows that near-isotropy controls value changes under behavior code perturbations, motivating reward-centered search, and derives a test-time value bound that separates policy-family suboptimality from candidate-evaluation error while exposing the rollout-horizon trade-off.”

  • The source of the TTA gains is not disentangled. The full method differs from TTA-0 in two ways: the latent rollout and the nonlinear reward predictor r^χ\hat{r}_{\chi} . Since r^χ\hat{r}_{\chi} is the only component that can capture reward structure outside the span of φ, a variant with H = 3 that uses the linear reward ϕ⊤w^\phi^\top \hat{w} in the rollout prefix is needed to separate these two effects. In addition, the paper emphasizes a rollout-horizon trade-off but only compares H = 0 and H = 3; a sweep over H would test this claim directly. Two related issues:

    • Details of r^χ\hat{r}_\chi are missing from Appendix D and Table 4, including its architecture, training procedure, and the size nn of the labeled set.
    • The abstract states that the procedure requires "no parameter updates", yet r^χ\hat{r}_\chi is a new network trained for each task.
  • The bound in Theorem 4.1 follows directly from Cauchy–Schwarz and a Rayleigh-quotient bound. It compares representation geometries at a fixed ϵB\epsilon_B, but the Euclidean-norm vector Bellman residual itself depends on Σρ\Sigma_\rho, so minimizing the bound does not imply minimizing the actual error. Moreover, the paper never measures how isotropic the learned features are in practice, e.g., the spectrum or condition number of Σ^\hat{\Sigma} under NoReg, Isotropy-only, and Gaussian regularization.

  • Missing related work: Combining a learned model with SFs, as in Section 3.2, is closely related to [1] and [2]. These works should at least be discussed in the related work section.

  • Attribution in Section 3.1. Eqs. 3, 4 and 5 are standard in related work and should come with citations. Could you state clearly that the contribution of this subsection is only the use of Eq. 6?

[1] Lucas N. Alegre et al.. Multi-Step Generalized Policy Improvement by Leveraging Approximate Models. NeurIPS 2023.

[2] Boyuan Chen et al. Self-Supervised Reinforcement Learning that Transfers using Random Features. NeurIPS 2023.

Questions for Authors:

  • What are the architecture and training procedure of r^χ\hat{r}_\chi, and how many labeled samples nn are used to fit it? Do the baselines have access to the same labeled set for reward inference?

  • What are the spectrum and condition number of Σ^\hat{\Sigma} for the learned features under NoReg, Isotropy-only, and Gaussian regularization? Does better conditioning correlate with downstream performance, and how does the SF Bellman residual ϵB\epsilon_B compare across these variants?

Flag For Ethics Review: No

Rating: 2: Weak rejection

Confidence: 3: I am familiar with this topic and its relevant literature.

AuthorsOpenReview ↗

Response to Weaknesses 1–2 and Question 1

We thank the reviewer for the constructive feedback. We have revised the manuscript and added new experiments to address the concerns, with full details provided in the uploaded revision.

Weakness 1 (Presentation) We revised the Introduction and Section 2 to improve readability. In the first paragraph of Introduction we introduced the reward coefficient and behavior code before using them, and in the third paragraph we clarified the value bounds associated with code perturbations and test-time adjustment. In Section 2, we first introduced the successor feature and linear inference and then used another subsection to introduced the universal successor feature and behavior code.

Weakness 2 and Question 1 (TTA component attribution, reward model and rollout horizon). We added TTA-Lin in Section 5.5 (Figure 3) to disentangle the two components. TTA-Lin retains latent rollout and the SF terminal value but replaces the nonlinear prefix reward predictor with the linear reward ϕ⊤w^\phi^\top\hat w. Compared with TTA-0, TTA-Lin improves average return by 10.6% on states and 2.4% on pixels, demonstrating the benefit of rollout-based evaluation even with linear rewards. GLDSF-TTA further improves over TTA-Lin in all three pixel domains (8.3% on average), while state results are mixed. This suggests that nonlinear reward prediction is particularly beneficial for pixel observations. One possible explanation is that learning features from pixels that adequately linearize downstream reward functions is more challenging; a nonlinear predictor can better capture residual nonlinear reward structure in the learned representation.

Table: Ablations of the reward model. Entries report mean ±\pm standard deviation over six seeds, with each domain score averaged over four evaluation tasks.

Obs.MethodCheetahWalkerQuadrupedAverage
StateTTA-0421±16421 \pm 16658±28658 \pm 28703±16703 \pm 16594594
TTA-Lin572±87572 \pm 87707±60707 \pm 60692±15692 \pm 15657657
GLDSF-TTA519±47519 \pm 47691±25691 \pm 25727±11727 \pm 11646646
PixelTTA-0577±20577 \pm 20416±34416 \pm 34351±36351 \pm 36448448
TTA-Lin573±20573 \pm 20452±34452 \pm 34352±27352 \pm 27459459
GLDSF-TTA612±17612 \pm 17477±27477 \pm 27402±81402 \pm 81497497

Appendix D.3 specifies that the nonlinear reward model uses the same 10000 labeled transitions as linear reward inference, with the pretrained encoder frozen. It is an MLP (512 hidden units, LayerNorm, ReLU, sigmoid output), trained with MSE/Adam (learning rate 3×10−43\times10^{-4}, weight decay 10−510^{-5}, batch size 256, 50 epochs), identically across domains/modalities and reset per task. Baselines use the same labeled subset for reward inference.

The Abstract now clarifies that no pretrained parameters are updated. The qualifier 'pretrained' has been added where needed elsewhere to make this distinction explicit.

We also added a horizon sweep in Appendix F.3 (Table 10), evaluating H∈{1,3,10,30}H\in\{1,3,10,30\}. All the other training setting keeps the same across HHs, including the horizon T=5T=5. H=3H=3 is the default and H=0H=0 corresponds to TTA-0.

Table: Effect of rollout horizon. Entries report mean ±\pm standard deviation over six seeds, with each domain score averaged over four evaluation tasks.

Obs.DomainH=1H=3H=10H=30
StateCheetah508±84508 \pm 84519±47519 \pm 47571±68571 \pm 68548±60548 \pm 60
Walker675±59675 \pm 59691±25691 \pm 25731±11731 \pm 11795±21795 \pm 21
Quadruped679±7679 \pm 7727±11727 \pm 11711±19711 \pm 19701±14701 \pm 14
Average621621646646671671681681
PixelCheetah570±21570 \pm 21612±17612 \pm 17627±14627 \pm 14638±17638 \pm 17
Walker437±41437 \pm 41477±27477 \pm 27480±34480 \pm 34504±17504 \pm 17
Quadruped377±16377 \pm 16402±81402 \pm 81400±73400 \pm 73383±65383 \pm 65
Average462462497497502502508508

H=1H=1 consistently underperforms H=3H=3. Interestingly, extending the horizon does not degrade overall performance; rather, average returns increase from 646 to 681 on states and from 497 to 508 on pixels at H=30H=30. While gains vary across domains—with modest drops on Quadruped—performance remains robust even at H=30=6TH=30=6T.

We conjecture that this robustness stems from two factors: (i) the learned latent dynamics generalize beyond their training horizon on ExORL distributions, and (ii) receding-horizon control replans at every real observation after executing only the first action, curbing rollout error accumulation. This aligns with Theorem 4.3, where accurate dynamics allow the γH\gamma ^{H} discount factor to better suppress terminal SF and linear-reward errors at larger H. Notably, unlike primitive action-sequence search whose dimensionality scales linearly with HH, searching over behavior codes preserves a fixed-dimensional optimization space—a potentially favorable advantage for long-horizon planning.

AuthorsOpenReview ↗

Response to Weakness 3 and Question 2

Weakness 3 and Question 2 (Theoretical qualification and empirical representation geometry). Section 4.1 now clarifies that ϵB\epsilon_B depends on the learned representation: Theorem 4.1 bounds the value error in terms of both the SF Bellman residual and representation geometry; it does not guarantee that regularization reduces the residual itself.

To address these empirical questions, Appendix E presents a 2D PointMass study (Tables 4–6; Figures 5–6) evaluating various regularizers and coefficient sweeps. The environment has deterministic dynamics, six goals with rewards decaying exponentially with squared goal distance, 24×2424 \times 24 pixel observations rendered from 2D coordinates, and a fixed feature dimension d=16d = 16.

From the empirical feature covariance Σ^\hat {\Sigma } on held-out data, we report two complementary spectral metrics instead of all 16 eigenvalues: (1) condition number κ=λmax⁡/λmin⁡\kappa = \lambda_{\max}/\lambda_{\min} (spectral imbalance) and (2) entropy-based effective rank (spectral spread). We also report position R2R^{2} (linear-probe coordinate prediction) and control return (discounted return of the frozen policy); see Appendix E for full details. We highlight the main results below (mean $\pm $ std over six seeds, omitting std for some metrics to save space; see also our response to Reviewer 7att on the same topic).

Table: Representative PointMass results.

Method (α3=0.1\alpha_3=0.1)Return (×100) ↑Condition Number ↓Effective Rank ↑Position Probe ↑
NoReg45.0±4.345.0 \pm 4.3134667.01.91.0
Isotropy-only58.2±4.258.2 \pm 4.215189.43.51.0
Gaussian64.9±2.564.9 \pm 2.540.112.11.0

NoReg exhibits severe spectral concentration, with an effective rank of only 1.9 out of 16. Gaussian regularization substantially improves conditioning and increases the effective rank, accompanying a return increase from 45.0 to 64.9. Iso-only also improves both geometry and control, although less substantially.

To further examine this relationship, we conduct coefficient sweeps.

Table: Regularization coefficient sweeps

RegularizerCoeff.Condition Number ↓Effective Rank ↑Position Probe ↑Return (×100) ↑
Iso-only101.515.90.116.8±2.716.8 \pm 2.7
1.04.214.91.061.0±5.061.0 \pm 5.0
0.115189.43.51.058.2±4.258.2 \pm 4.2
0.0161758.92.31.053.8±13.053.8 \pm 13.0
0.001135906.51.91.048.9±6.748.9 \pm 6.7
Gaussian101.615.90.228.5±2.328.5 \pm 2.3
1.01.715.80.761.7±2.261.7 \pm 2.2
0.140.112.11.064.9±2.564.9 \pm 2.5
0.0115588.23.81.055.9±1.955.9 \pm 1.9
0.00138928.13.01.048.9±14.548.9 \pm 14.5

Coefficient sweeps reveal a trade-off between geometric regularity and task-relevant information. Weak-to-moderate regularization improves geometry whereas control performance is non-monotonic. At a coefficient of 10, near-ideal conditioning and an effective rank near 16 are reached, yet returns and position decodability drop sharply. Thus, good representation geometry is insufficient unless task-relevant information is preserved.

Table: SF residual results

RegularizerCoeff.SF res. ↓GN SF res. ↓Return ↑
NoReg02.1337.345.0±4.345.0\pm4.3
Iso-only101.41.616.8±2.716.8\pm2.7
1.03.87.561.0±5.061.0\pm5.0
0.11.4201.658.2±4.258.2\pm4.2
0.011.6281.353.8±13.053.8\pm13.0
0.0011.9327.548.9±6.748.9\pm6.7
Gaussian101.61.828.5±2.328.5\pm2.3
1.02.32.761.7±2.261.7\pm2.2
0.14.318.964.9±2.564.9\pm2.5
0.013.2215.755.9±1.955.9\pm1.9
0.0011.8295.148.9±14.548.9\pm14.5

SF res. (the SF residual) alone does not consistently track return. Motivated by ϵB\epsilon_B, we estimate the Bellman expectation per held-out transition via Monte Carlo averaging over next observations (for pixel uncertainty), then average the resulting pointwise SF Bellman residual norm across transitions. While this captures the typical magnitude of the residual whose supremum defines ϵB\epsilon_B, a finite-sample average cannot certify the uniform supremum over all states, actions, and codes—which is itself impractical to estimate. Moreover, because value bounds also depend on representation geometry and control relies on task-relevant information and policy quality, raw residuals need not correlate closely with return.

Theorem 4.1 motivates Geometry-Normalized SF (GN SF) residual: SF residual/λmin⁡(Σ^)\sqrt{\lambda_{\min}(\hat\Sigma)}. Gaussian achieves lower GN SF than Iso-only in 4/5 matched coefficients. At coefficients 1.0–0.01, lower GN residuals accompany higher returns, suggesting a more informative diagnostic than raw residuals alone. But GN SF is a diagnostic, not a formal ϵB\epsilon_B bound or guarantee.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.