acceptodds
Under review as a conference paper at ICLR 2027

Geometric Time Score Estimators: Bridging Self-Predictive Representations and Generative Occupancy Models for Reinforcement Learning

Abstract

In reinforcement learning (RL), temporal reasoning is essential for estimating the density of future outcomes under a policy's occupancy measure. While self-predictive representations have been a dominant paradigm for occupancy density estimation, we find that they yield surprisingly poor estimates. On the other hand, generative occupancy models can produce remarkably accurate samples from the occupancy measure. Yet, they cannot easily recover occupancy densities, limiting their use for control despite their expressivity. In this paper, we propose , a framework that bridges the two paradigms by learning occupancy density estimators along probability paths. We analyze how GTSE implicitly induces better occupancy estimators through improved representation learning and show how to directly extract self-predictive representations from this framework. Through experiments across several locomotion and manipulation tasks, we show that GTSE improves performance over state-of-the-art methods by to in goal-conditioned RL settings, and that the representations learned using our framework transfer effectively to downstream RL tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Self-Predictive Representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oBuLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GTSE, which combines Time Score Matching with TD Learning to estimate future state occupancy density. It also provides a theoretical connection to representation learning. The results on OGBench are strong, and GTSE performs much better than several baselines.

Critical strengths
  1. The problem is important. Combining generative models with occupancy density estimation is an interesting idea.
  2. The method is reasonable. Using TD Learning with Time Score Matching is a useful contribution.
  3. The experimental results are strong. GTSE performs well on different navigation and manipulation tasks.
Critical weaknesses
  1. Missing important ablations. It is unclear how much improvement comes from TD Learning or Time Score Matching. A comparison with Time Score Matching without TD would be helpful.
  2. The theory does not fully explain the results. Higher representation rank does not always give better performance, as shown in Appendix E.2.
  3. Missing strong planning baselines. GTSE uses a policy to generate actions and then ranks them. This is similar to using a value function for action selection. A world model with value-guided short-horizon planning could do something similar. I would like to see this comparison.
  4. Inference cost is not clear. The reported time only covers density estimation, not the full action selection process.
Questions for authors

I think the idea is interesting and the results are good. However, some important ablations and baseline comparisons are missing. I am not fully convinced that GTSE offers clear advantages over existing value learning or model-based planning methods. I lean toward rejection, but I may increase my score if these concerns are addressed.

3 comments

We thank you for your review and appreciate your feedback. In our response, we provide detailed answers to the weaknesses highlighted in your review and questions you ask us: missing ablation, theory vs results, planning baseline and inference cost. Please find our response below. We also revised the paper to reflect these changes. All changes are denoted in brown color.

Missing ablation: TD vs MC

To address this concern, we compare GTSE to an ablation that uses Time Score Matching (GTSE-MC) which uses Monte-Carlo style learning, as you suggested, to Appendix E.3. We find that GTSE significantly outperforms GTSE-MC, likely because GTSE can stitch information across trajectories:

Methodscene-playscene-noisypuzzle-4x4-playpuzzle-4x4-noisy
GTSE-MC57 ± 427 ± 316 ± 22 ± 1
GTSE69 ± 655 ± 640 ± 422 ± 2

Theory vs Results

We acknowledge your concern. We want to clarify that we do not claim that higher available representation rank leads to better performance. We claim that, under a bilinear parameterization (Proposition 1, Sec 6.2), integrating the time score is equivalent to an estimator with higher available rank. In this regard, our experiments in Appendix E.2 show that the relationship between available rank and downstream performance depends on the structure of the occupancy measure of the dataset and task (Tables 10, 11 and 12), under bilinear parameterization.

In summary, our theory tries to explain what additional capacity GTSE provides and our experiments try to understand when that capacity translates to additional performance. Kindly let us know if you have any further questions.

Planning baseline

To address this concern, we compare GTSE to F-MPC (Flow-Model Predictive Control), used as a model-based planning baseline in [1]. F-MPC is a flow-matching version of Diffusion Model Predictive Control [2], and uses a flow-matching policy and learned model to propose plans which are ranked by an action-chunked value function. We tuned F-MPC by sweeping over the choice of critic (MC vs TD) and horizon (10 as used in [1] vs 32). The results added to Appendix E.4 indicate that GTSE's occupancy density estimator is able to better model long-horizon temporal effects of actions:

Methodscene-playscene-noisypuzzle-4x4-playpuzzle-4x4-noisyantmaze-large-navigate
F-MPC19 ± 318 ± 37 ± 29 ± 344 ± 2
GTSE69 ± 655 ± 640 ± 422 ± 253 ± 4

[1] Kwanyoung Park, Seohong Park, Youngwoon Lee, and Sergey Levine. "Scalable Offline Model-Based RL with Action Chunking." arXiv preprint, 2025.

[2] Guangyao Zhou et al. "Diffusion Model Predictive Control." arXiv:2410.05364, 2024.

Inference cost not clear

We have updated Table 8 with the 3rd row showing the full inference time for selecting 16 actions during inference. We find that the ranking between algorithms is roughly similar to row 2 (critic inference time) and that GTSE and its variants remain comparable in inference time to the baselines.

CRLFBTD-JEPABYOL-γDual GRTMDSMORETD-flowsNFGTSE-bGTSE-rGTSE
Training (ms) ↓3.123.665.123.774.5144.552.532.264.624.493.592.17
Inference (critic-only) (ms) ↓0.240.290.450.290.260.230.190.700.640.550.360.31
Inference (full) (ms) ↓0.770.770.930.800.670.760.701.071.180.890.860.85

We thank you again for your review. We hope that our responses adequately address your concerns and we are ready to answer any further questions.

Dear Reviewer,

Kindly let us know if our response, which consists of multiple new sets of experiments and revisions to the paper, addresses your concerns adequately. We would greatly appreciate your engagement.

Thank you! The Authors

I thank the authors for the additional experiments and analyses. The rebuttal has addressed my concerns to a reasonable extent. I have therefore raised my score to 3.

Reviewer_wCcuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper learns the time score (the derivative of log-density along a VP probability path) of the discounted occupancy measure, using a TD-style bootstrapped regression. The log-density is then recovered by integrating the time score. The authors use this theory as motivation for GTSE-b and GTSE-r, where the latter holds explicit state-action and goal representations. Empirical validation is primarily on OGBench tasks.

Critical strengths
  1. The diagnostic in Section 4 is reasonably useful. It uses ground-truth occupancy where it can be computed, shows that existing methods rank actions poorly, and ties estimation quality to control.

  2. On the hard OGBench datasets, the gains are large (and also come from 8 seeds.) Training cost is comparable to the baselines, per Table 8.

Critical weaknesses
  1. I don't think the contributions are as mechanistic as the authors claim. For example, Propsition 1 is Parseval’s identity under a bilinear assumption. It applies strictly to GTSE-b, not to the MLP-based GTSE instantiation actually employed in the experiments. Furthermore, I think the gains reported could also easily be explained by dense multi-noise targets used or the log-space regression.

  2. It feels like there was significant tuning for the method under test, where the baselines are often stuck in awkward configurations. GTSE has per-dataset tuned settings (Table 7). But baselines only get “best hyperparameters from their papers” or unspecified tuning, and they are forced into best-of-16 selection over a shared BC policy rather than their native policy extraction. Several of the baselines score 0% on pointmaze, and TD-flows diverges entirely, which to me suggest under-tuning.

Questions for authors
  1. How much tuning did each baseline receive, in comparison with GTSE’s per-dataset sweeps? What does the estimator converge to when Kmax<1?
  2. Why does GTSE trail CRL on antmaze by 20 points, and does that say something about the method beyond the low-dimensionality argument?
  3. Can you run an ablation with the same network and TD structure but a single-noise-level log-ratio target? This would separate the time-score formulation from just raw capacity.
  4. How do the baselines perform when using their native policy extraction?
5 comments

Thank you for your detailed and valuable feedback. We acknowledge your concerns regarding the source of GTSE's performance gains and hyperparameter tuning. We answer your questions and concerns below. We also revise the paper to reflect the changes, modifications are denoted in brown color.

Hyperparameter tuning

Hyperparameters for baselines

We list the common hyperparameters across all methods in Table 5 of our paper. In addition to this, we shared the goal-sampling hyperparameters for the occupancy estimator and actor per dataset in Table 7 across all baselines. We tuned each baseline as follows:

  • FB: We swept over values (0.01, 0.1, 1.0, 2.5, 10.0) for the orthonormality coefficient and used 0.01 for manipulation tasks, 2.5 for locomotion tasks.
  • TD-JEPA: We swept over values of orthonormality coefficients (0.01, 0.1, 1.0, 2.5, 10.0) for both the encoders. We used (2.5, 2.5) for manipulation tasks, (0.1, 0.1) for antmaze and (0.01, 0.01) for pointmaze.
  • BYOL-γ: Following the original paper, we use bi-directional prediction and the cross-entropy loss between normalized representations. We also find that using randomly initialized features for ψ\psi produced better performance.
  • CRL: CRL did not involve specific additional hyperparameters to tune beyond the common hyperparameters.
  • Dual GR: We used an expectile of 0.7 for all tasks except 0.9 for antmaze navigate as in the original paper.
  • TMD: We swept over ζ\zeta (0.05, 0.1), the weight on the TMD backup and invariance losses, as well as the weight between diagonal and off-diagonal pairs (0.5, 1). We used a combination of ζ=0.05\zeta = 0.05 and diagonal weight =0.5= 0.5.
  • SMORE: We swept over (0.6, 0.7, 0.9) and used an expectile of 0.7 for manipulation tasks and 0.9 for locomotion.
  • TD-flows: We compare the MC estimator against the Hutchinson trace estimator. We find that the Hutchinson trace estimator is better with m=32m = 32.
  • NF: We use the full state space for goals and 4 flow blocks as we found this to be similar in size to other baselines. Following the original paper, we use Gaussian noise with std 0.05 and 256 channels.

We updated Appendix D.2 with this additional information.

GTSE with per-dataset tuning vs single set of hyperparameters

We ran experiments to measure the performance of GTSE when using a single set of hyperparameters across all datasets in Table 2. The hyperparameters used were:

τGTSE (p_D)Npolicy (p_cur, p_traj, p_rand)K_max
0.003(0, 1, 0)30(0, 0.5, 0.5)0.925

The results are shown below and the new results with same hyperparameters are under GTSE (same tuning):

DatasetGCBCCRLFBTD-JEPABYOL-γDual GRTMDSMORETD-flowsNFGTSE-rGTSEGTSE (same tuning)
scene-play20 ± 425 ± 56 ± 231 ± 417 ± 219 ± 157 ± 213 ± 435 ± 49 ± 360 ± 569 ± 669 ± 6
scene-noisy10 ± 17 ± 32 ± 124 ± 39 ± 312 ± 77 ± 13 ± 219 ± 45 ± 237 ± 455 ± 651 ± 2
puzzle-4x4-play1 ± 12 ± 122 ± 836 ± 32 ± 117 ± 61 ± 10 ± 14 ± 21 ± 126 ± 640 ± 433 ± 2
puzzle-4x4-noisy1 ± 00 ± 06 ± 29 ± 30 ± 02 ± 30 ± 00 ± 01 ± 01 ± 09 ± 222 ± 223 ± 2
cube-triple-play5 ± 22 ± 11 ± 01 ± 10 ± 113 ± 21 ± 12 ± 316 ± 30 ± 07 ± 118 ± 118 ± 1
cube-triple-noisy2 ± 11 ± 10 ± 10 ± 00 ± 14 ± 20 ± 01 ± 18 ± 21 ± 14 ± 18 ± 18 ± 1
antmaze-large-navigate34 ± 473 ± 615 ± 829 ± 632 ± 543 ± 629 ± 435 ± 330 ± 951 ± 1678 ± 453 ± 452 ± 4
Mean10.415.77.418.68.615.76.47.716.19.731.637.936.2

The results show that the performance of GTSE degrades by only 4.5% relative to the best hyperparameters and GTSE with the same set of hyperparameters still significantly outperforms the best baseline in mean performance by approximately 2×.

GTSE vs CRL on antmaze, low dimensionality argument

Kindly refer to Appendix E.1 where we discuss this question in detail. We find that CRL outperforms GTSE with best-of-N on antmaze-large-navigate (although GTSE-rr (78 ± 4) outperforms CRL (73 ± 6)), while GTSE outperforms CRL on all other tasks (e.g., puzzle-4x4-play: 40 vs. 2; scene-play: 69 vs. 25). We believe that under the best-of-N policy extraction, which isolates the effect of the critic, this is in major part due to the occupancy measure in antmaze-large-navigate being determined by the 1st and 2nd dimensions (x-y positions). The following results in our paper support this:

  • Fig. 5 shows that the time score varies primarily in the 1st and 2nd dimensions of the state space.
  • Running GTSE after masking all but the first 2 dimensions improves performance from 53 ± 4 to 68 ± 5, which is within the CI of CRL (Appendix E.1).
  • We also saw that in a task like puzzle-4x4-play where the time score varies along multiple different dimensions (Fig. 6), GTSE performs substantially better than CRL.

What does the estimator converge to when Kmax<1K_{max} < 1?

When Kmax<1K_{max} < 1, the time score integral produces the log-density ratio between a slightly smoothed version of the occupancy density and the base distribution. We find this truncation to be beneficial in practice. Please find ablation studies with different truncation levels in Sec. 7, Fig. 3.

Weakness 1 (Claims) and Ablation

Ablation with a single noise-level target

We run ablations with a version of GTSE that uses a monolithic time score network but is trained with a single intermediate noise level 0.5. During inference, we obtain the density estimate as:

log⁡dπD(g∣s,a)p0(g)≈0.5⋅sθ(g,0,s,a)+(Kmax⁡−0.5)sθ(g,0.5,s,a)\log \frac{d^{\pi_{\mathcal{D}}}(g \mid s, a)}{p_0(g)} \approx 0.5 \cdot s_\theta(g, 0, s, a) + \left(K_{\max} - 0.5\right) s_\theta(g, 0.5, s, a) Every other detail and hyperparameter are the same as GTSE. The results are as follows:

Methodscene-playscene-noisypuzzle-4x4-playpuzzle-4x4-noisyantmaze-large-navigate
GTSE (single target)41 ± 834 ± 214 ± 25 ± 227 ± 6
GTSE69 ± 655 ± 640 ± 422 ± 253 ± 4

We hypothesize that the loss in performance is due to using just a single intermediate noise level and due to a coarse approximation of the time score integral during inference, as these are the only differences from GTSE. The results also align with our ablation in Fig. 4 that studies the effect of NN during inference.

Weakness 1

We acknowledge your concerns regarding our claims. We believe that the reported gains of GTSE come from the training procedure that uses dense multi-noise targets as well as the inference procedure which integrates the time score. We note this in Sec. 6.1 (lines 326-329) and in Sec. 6.2 (lines 359-361). Our ablation in Fig. 4 shows that increasing the number of points the time score is evaluated at during inference generally helps improve performance. The single noise level target ablation you suggested supports this and we thank you for your suggestion.

We would like to clarify that we explicitly state that Proposition 1 applies under a bilinear parameterization. We do mention using Parseval's identity as the final step in the proof of Proposition 1 in Appendix A.4, lines 851-852. We also acknowledge in Appendix E.2 that under this bilinear parameterization, the matter of whether available rank translates to better downstream performance is nuanced and dependent on the occupancy measure structure of the dataset and environment.

Furthermore, we acknowledge in Sec. 6.2, lines 371-372 that GTSE's architecture is not restricted to a bilinear architecture, which may explain why it outperforms GTSE-bb. However, as in [1], using a bilinear parameterization allows us to analyze the connection between integrating the time score during inference and available rank. We use the GTSE-bb variants primarily as an analysis tool in our paper. We replaced the word "mechanism" and will further revise the paper to make these points clearer to readers.

Regarding gains and log-space regression, we believe that if log-space regression alone could explain the control performance of GTSE, baselines such as CRL, TMD and NF that estimate the occupancy measure in the log space would be comparable to GTSE in overall performance. Instead, CRL, TMD and NF reach 15.7%, 6.4% and 9.7% average success, against 37.9% for GTSE (Table 2).

[1] Ahmed Touati and Yann Ollivier. "Learning One Representation to Optimize All Rewards." In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.

Policy Extraction

We use the best-of-N style policy extraction since the aim of our experiments is to isolate the effectiveness of the occupancy density estimator. This allows us to provide a fair comparison. However, we acknowledge your question about performance when using each baseline's native policy extraction. To investigate this, we ran additional experiments with each paper's policy extraction strategy wherever applicable.

CRL, FB, Dual GR, TMD, TD-JEPA, SMORE and NF use a DDPG+BC-style policy extraction (policy gradient with behavioral regularization). Though BYOL-γ conditions a BC policy on its representations in the paper, we extract a policy from it using DDPG+BC for fair comparison. Our TD-flows Hutchinson trace estimator is not used in another paper, and we stick to best-of-N policy extraction for it as well as for GTSE and GTSE-rr. For each baseline that uses DDPG+BC, we sweep over the BC coefficient α∈{0.1,0.5,1.0,3.0}\alpha \in \{0.1, 0.5, 1.0, 3.0\} and choose the best result:

DatasetCRLFBTD-JEPABYOL-γDual GRTMDSMORENFTD-flows (best-of-N)GTSE-r (best-of-N)GTSE (best-of-N)
scene-play18 ± 46 ± 224 ± 24 ± 135 ± 35 ± 18 ± 21 ± 135 ± 460 ± 569 ± 6
scene-noisy1 ± 12 ± 111 ± 22 ± 123 ± 21 ± 13 ± 20 ± 019 ± 437 ± 455 ± 6
puzzle-4x4-play0 ± 033 ± 437 ± 40 ± 014 ± 40 ± 00 ± 04 ± 64 ± 226 ± 640 ± 4
puzzle-4x4-noisy0 ± 02 ± 18 ± 20 ± 017 ± 20 ± 00 ± 00 ± 01 ± 09 ± 222 ± 2
antmaze-large-navigate88 ± 218 ± 352 ± 714 ± 235 ± 222 ± 322 ± 580 ± 1230 ± 978 ± 453 ± 4

In comparison to Table 2, CRL, TD-JEPA and NF improve on antmaze-large-navigate but perform worse on scene-play. While FB gains on puzzle-4x4-play, Dual GR improves overall on manipulation tasks but performs worse on antmaze. BYOL-γ, SMORE and TMD perform worse. Overall, the results are mixed, and GTSE, which uses best-of-N, still performs best on 5 out of 6 tasks considered (including pointmaze-large-diverse). On antmaze, we additionally trained GTSE with DDPG+BC-style policy extraction and observed gains (from 53 ± 4 to 78 ± 2).

We swept over α\alpha on pointmaze-large-diverse, but none of CRL, FB, TD-JEPA, BYOL-γ, Dual GR or TMD gave non-zero performance, similar to Table 1 except for CRL which gives 3 ± 6 with best-of-N. We ran TD-flows with lower τ=0.001\tau=0.001 and found that it does not diverge to NaN but still gives 0 performance. We maintain that these estimators are unable to accurately capture the occupancy measure on pointmaze and therefore perform badly despite hyperparameter tuning.

We hope our responses have addressed your concerns. If so, we would be grateful if you could consider how they strengthen our paper in your evaluation. Please let us know if you have any further questions; we are happy to answer them.

Reviewer_6i7UOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Geometric Time Score Estimators (GTSE) for estimating the behavior-policy occupancy measure in offline reinforcement learning. It learns the time derivative of log density along a Gaussian probability path using conditional time-score targets and temporal-difference bootstrapping, then integrates the estimated score for goal-conditioned action ranking (Secs. 5.1–5.2, Eqs. 3–6). A bilinear decomposition interprets the estimator through representations at different noise levels, and GTSE-r learns an explicit state-action representation (Sec. 6, Eqs. 8–9). Validation comprises occupancy-ranking diagnostics, seven goal-conditioned control datasets, frozen-representation transfer, and architectural and numerical ablations (Fig. 2; Tables 1–3, 9; Figs. 3–4).

Critical strengths
  1. The estimator is practically promising under a controlled action-selection protocol. The occupancy-estimator comparisons use a common flow-matching policy class and best-of-16 action selection, helping isolate the usefulness of their scores (Sec. 7; Table 5). GTSE reaches 37.9% mean success versus 18.6% for the strongest baseline in Table 2, with eight seeds per dataset; the scene-play improvement is particularly substantial, 69 ± 6% versus 35 ± 4%. These results provide meaningful evidence for a useful occupancy critic, even though they do not establish superiority over every method's original policy-learning pipeline.

  2. Time-score learning offers a distinct route from occupancy modeling to control. Eqs. 5–6 combine bootstrapped occupancy learning with direct log-density evaluation at a fixed goal, avoiding the reverse-flow/Jacobian calculation used by the TD-flow density baseline (Appendix D.2, Eqs. 25–26). GTSE-r also produces frozen features with higher mean performance than the compared frozen encoders and remains competitive with end-to-end FQL in Table 3. I view this combination as a substantive contribution; the small mean advantage over vanilla FQL, 62.2% versus 58.2%, is not by itself convincing evidence of a general transfer improvement.

Critical weaknesses
  1. The representation contribution is insufficiently distinguished from close prior work. The paper does not discuss directly related work, such as geometric horizon models [1]. The factorization used by GTSE-r closely parallels those proposed in [2] and [3], yet neither paper is discussed.

  2. The representation claims overreach. The abstract states that GTSE allows one to "directly extract self-predictive representations," but GTSE-r is a separately parameterized and trained model.


[1] Generalised Policy Improvement with Geometric Policy Composition. Shantanu Thakoor et al. ICML 2022.

[2] Diffusion Spectral Representation for Reinforcement Learning. Dmitry Shribak et al. NeurIPS 2024.

[3] Bridging Successor Measure and Online Policy Learning with Flow Matching-Based Representations. Haosen Shi et al. ICLR 2026.

Questions for authors
  1. How does GTSE-r differ from the methods in [2] and [3] beyond the choice of learned field and predictive distribution?

  2. Why did you choose this form of factorization? Proposition 1 retains the dependence on kk in both ϕ\phi and ψ\psi, whereas GTSE-r removes this dependence from ϕ\phi (Eq. 9).

4 comments

Thank you for the thoughtful, detailed, and constructive feedback. We understand that your main concerns relate to the representation learning claims and contributions with respect to existing work. We address these concerns below and revised the paper to reflect the changes. All changes are denoted in brown color.

Question 1 and Weakness 1

Thank you for the suggestion to improve our framing with respect to prior methods. We modified Sec. 2 to include a discussion of [1], [2] and [3]. We also ran additional empirical comparisons in Sec 7, Tables 3 and 4 to reflect this.

Differences from DSR [2] and Flow-SF [3]

Diffusion Spectral Representations [2] uses a diffusion model trained with score matching to generate samples from the distribution of the immediate next state, i.e., a transition model. As the authors remark in Sec 3.3 of [3], DSR representations are not able to encode transition dynamics across extended time horizons. Flow-SF [3], instead, extracts a state-action representation from the flow-matching occupancy model from TD-flows [4] used to generate samples from the occupancy measure. This is achieved by parameterizing the velocity field as an inner product of a vector field and a time-dependent matrix field.

In contrast, GTSE-rr is an alternate parameterization of our framework, GTSE. It estimates occupancy density rather than generating samples from it, using the exact same geometric score matching objective (Eq. 5). It only differs in that it parameterizes the score function as sθ(g,k,s,a)=ϕθ(s,a)⊤ψθ(g,k)s_\theta(g, k, s, a) = \phi_\theta(s, a)^\top \psi_\theta(g, k) (Lines 389-390).

[4] Jesse Farebrother et al. "Temporal Difference Flows." In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. arXiv:2503.09817.

Additional Experiments

While we did not find a public implementation of DSR, we compared our method to Flow-SF. The experimental protocol is the same as described in Sec. 7.

scene-play

TaskCRLFBTD-JEPABYOL-γDual GRTMDVanilla FQLFlow-SFGTSE-r
task 167 ± 658 ± 3398 ± 551 ± 2737 ± 71 ± 2100 ± 0100 ± 1100 ± 1
task 214 ± 1217 ± 1932 ± 228 ± 84 ± 30 ± 083 ± 2383 ± 781 ± 12
task 327 ± 131 ± 285 ± 1110 ± 611 ± 90 ± 099 ± 381 ± 294 ± 5
task 410 ± 50 ± 018 ± 1410 ± 47 ± 40 ± 09 ± 724 ± 2036 ± 18
task 52 ± 20 ± 00 ± 00 ± 04 ± 70 ± 00 ± 00 ± 00 ± 0
Mean24.015.246.615.812.60.258.257.262.2

puzzle-4x4-play

TaskCRLFBTD-JEPABYOL-γDual GRTMDVanilla FQLFlow-SFGTSE-r
task 12 ± 11 ± 112 ± 41 ± 11 ± 20 ± 034 ± 831 ± 1264 ± 12
task 21 ± 11 ± 15 ± 11 ± 10 ± 00 ± 016 ± 50 ± 014 ± 6
task 33 ± 10 ± 05 ± 30 ± 00 ± 00 ± 018 ± 517 ± 1160 ± 2
task 41 ± 10 ± 04 ± 20 ± 00 ± 00 ± 011 ± 310 ± 826 ± 7
task 50 ± 00 ± 02 ± 20 ± 00 ± 00 ± 07 ± 33 ± 422 ± 5
Mean1.10.25.30.20.20.017.211.936.9

The results show that GTSE-rr outperforms Flow-SF in both environments. In puzzle-4x4-play, GTSE-rr performs approx. 2.1× better than Vanilla FQL even though its representations are frozen and not updated during downstream learning, unlike Vanilla FQL. GTSE-rr also performs approx. 3× better than Flow-SF.

Question 2 and Weakness 2

Validity of representation learning claims

We understand your concern. We revised the abstract to clarify that representations are extracted under a bilinear parameterization of the GTSE framework.

Throughout our paper, we refer to GTSE as a framework (e.g., lines 20-22, lines 46-47, lines 161) and we mention that this framework can be used to learn self-predictive representations (lines 27-28). In Sec. 6.2 (lines 344-346) and Sec. 6.3 (lines 389-390), we show that a neural network trained using this framework can be parameterized to directly extract state-action and goal representations (GTSE-rr). Crucially, GTSE-rr has the same training loss and inference method as GTSE and differs only in model parameterization. We clarify that the claims in Prop. 1 hold under a bilinear parameterization in the Intro (lines 51-52) and Sec. 6.2. We also state explicitly that GTSE-rr is a variant of GTSE throughout the paper, for instance, in the Intro (lines 70-72), lines 322-323 and Sec. 6.3.

That said, the algorithm that we call GTSE in Figs. 2, 3, 4 and Tables 1 and 2 is a monolithic neural network parameterization of our framework, which we take to be the default. We will further revise our paper to make this clearer. Kindly let us know if you have any more questions regarding this.

Why choose this form of factorization for GTSE-rr?

The chosen parameterization for GTSE-rr allows us to recover explicit state-action and goal representations, as we mention in Sec. 6.3. As seen in Eq. 9, after integrating the time score, this explicitly leaves us with 2 representations that are independent of kk. In GTSE-bb, the parameterization based on Proposition 1 used for analysis in Sec. 6.2, both the state-action and goal representations are dependent on kk. GTSE-bb's time score integration does not lead to explicit kk-independent representations.

Our experiments indicate that GTSE-rr's parameterization still offers strong performance as evidenced by lines 396-399 and Table 2. We will work towards making these distinctions clearer in the paper. Kindly let us know if our response addresses your concerns.

Dear Reviewer,

Kindly let us know if the new experiments, baselines and discussion we added address your concerns about our paper. We would greatly value your engagement.

Thank you! The Authors

I thank the reviewer for the response which address my concerns. I would like to maintain the score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.