acceptodds
Under review as a conference paper at ICLR 2027

AQRL: Quasimetric Reinforcement Learning with Split State Encoding and Approximate Nearest Completion

Abstract

Goal-conditioned tasks require an agent to reach a state satisfying a given goal. Such a goal often constrains only some state coordinates, which we define as specified coordinates; the rest are free coordinates. Quasimetric Reinforcement Learning (QRL) is a goal-conditioned reinforcement learning method that performs well. It takes states as inputs to estimate state-reaching cost. However, it has two limitations. First, its actor receives sampled real states as goals during training but artificially constructed states during testing, creating a mismatch in goal inputs. Second, QRL trains its actor to reach a single specific state for each training sample, even though other states that satisfy the same goal may have lower reaching costs. To address both limitations, we propose Approximate nearest-goal Quasimetric Reinforcement Learning (AQRL) with two components: split state encoding (SSE) and approximate nearest completion (ANC). SSE encodes specified and free coordinates separately, enabling the actor to use only specified-coordinate latents as goal inputs in training and testing, resolving the mismatch. ANC searches over free-coordinate latents during training to find a completion with lower estimated reaching cost. Across fourteen online tasks with five training seeds, AQRL outperforms QRL on thirteen tasks and ranks first among all compared methods on twelve, achieving 66.1% mean success versus 24.5% for QRL and 45.3% for the strongest compared baseline.

Primary area:
reinforcement learning
Keywords:
goal-conditioned reinforcement learning, quasimetric reinforcement learning, partially specified goals, representation learning, online reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TuLcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper adress a missmatch between the training and testing distribution of goal conditioned reinforcemente learning algorithms trained from state action trajectories. These algorithms usually learn via a self supervised learning objective that assumes that states closed in time are positive pairs at short distance and push any other state, taking any state as a possible goal. This creates a missmatch since at inference usually the goal is defined as a set of possible goals for example all the states sharing the same (x, y) position but do not care about the velocity (vx,vy)(v_x, v_y) at which you reach the goal. The paper assume to have knowledge about this features split and proposed a fix that adress this problem by selecting the closest goal and optimize to that instead of considering a random goal velocity to fill the observation gap.

Critical strengths
  • The paper is sound and adresses one of the know limitation of training with observed state as goals.

  • The method is simple and showed good empirical results in an extensive set of evaluation environments.

Critical weaknesses
  • The method is trivial and potentially a well known solutions, nonetheless I'm not aware of any published work about this.

  • The method assumes to have access to the information about task-provided splits that is a strong assumption that limit the application of the approach to factored observation, and make impossible to apply the method to the interesting case of high-dimensional observations. I think a much more interesting setting for this problem would be to learn from a limited dataset of (goal, goal_desired) paired observations how to regress the goal_desired to the closest goal.

Questions for authors
  • I think the related work is missing some other references learning the temporal distance. Like:

  • HIQL: Offline Goal-Conditioned RL with Latent States as Actions

  • Foundation Policies with Hilbert Representations

  • State Representation Learning for Goal-Conditioned Reinforcement Learning

  • Learning The Minimum Action Distance

2 comments
AuthorsOpenReview ↗

Response to Reviewer TuLc: Originality, Applicability, and Related Work

Thank you for carefully reviewing our paper and for your helpful suggestions.

Response to the “Questions for Authors” section.

Thank you for pointing out these omissions in our related work. We have added the missing references and expanded the discussion of distance representations and goal-conditioned value learning in the revised manuscript. However, we are not yet clear on the specific question you would like us to address in this section. Could you please clarify it so that we can respond directly?

Response to Critical Weakness 1.

We agree that the method is simple. Simplicity alone, however, does not imply a lack of originality. Although using only specified coordinates as goal inputs is an established practice, we are not aware of prior work combining split state encoding with training-time latent completion search. Specifically, SSE in AQRL separately encodes specified and free coordinates, retaining both parts for state-to-state cost estimation while conditioning the actor's goal input only on the specified part. ANC then fixes the specified-coordinate latent and optimizes the free-coordinate latent to seek a completion with lower estimated reaching cost from the predicted next state for actor training.

Response to Critical Weakness 2.

We acknowledge that requiring a task-provided split limits the applicability of our method. Our experiments focus on vector-state observations with known splits. AQRL is difficult to apply directly to tasks where specified and free coordinates cannot be clearly distinguished, as is often the case with image observations. However, we would like to clarify that the formulation does not require low-dimensional states: it also applies to high-dimensional state vectors when the split between specified and free coordinates is known.

Regarding your suggested paired-observation setting, is your aim to address this applicability limitation? Specifically, do you mean learning a model from paired observations that directly maps a desired task goal to a closer full goal state or goal representation that satisfies the same task requirements? Would the model take specified-coordinate values as input, or directly use images or full state observations without a predefined split?

Thank you for the response. My main concern about the paper is the strong assumption about task-provided splits that limit the application of the approach, I will mantain my score for the moment and looking forward to discuss with other reviewers and AC discussion whether the paper make it to the quality bar for a top-tier conference.

Reviewer_bfaAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AQRL to improve QRL for goals that only need part of the state. It has two parts. SSE uses separate encoders for goal-related and other state features. ANC searches for a target with lower cost in latent space. The results are good on 14 tasks. AQRL gets 66.1% average success rate, much higher than QRL.

Critical strengths
  1. The results are good. AQRL beats QRL on 13 of 14 tasks. The improvement is large on some tasks.
  2. The paper points out a real problem in QRL. The goal inputs are different during training and testing. Fixing this problem is useful.
  3. The paper is easy to follow. The authors test on many tasks and report results with five seeds. The experiments are quite detailed.
Critical weaknesses
  1. I think the novelty of SSE is quite limited. Separating useful and unused goal features is a common idea. There are many simple ways to do this, such as adding a binary mask or just using the goal coordinates. I don't see why two separate encoders are needed. The paper should compare SSE with these simple methods.
  2. The comparison with QRL is not fully convincing. QRL uses zero-filled goal states during testing. As the paper says, these goals can even be invalid states. So it is not very surprising that fixing this problem helps a lot. Also, AQRL changes the network and adds an MSE loss for the dynamics model. It is hard to know which part brings the improvement. I would like to see a QRL baseline with a simple goal mask and the same other settings.
  3. I have some concerns about ANC. It searches freely in latent space. But the found target may not be a real state. The critic may give a low cost to such a target, but this does not mean the robot can actually reach it. The paper does not test this enough. It would also be useful to compare with just choosing a target from the replay buffer.
  4. The gain from ANC is not very large. Without ANC search, the average success rate is already 67.54% on the six tasks. With 32 search steps, it becomes 72.37%. This is some improvement, but 32 gradient steps also add computation. The paper does not show training time. I am not sure the extra cost is worth it.
  5. The method is tested only on tasks with known state splits. It is not clear how to use it for images or language goals. Also, the original QRL paper has discussed goal sets. I think this should be compared or discussed more.
Questions for authors

The results are good, and the problem is useful. But I am not convinced by the novelty. SSE looks like a simple fix for the goal input problem. I think simpler methods could do the same thing, but the paper does not test them. ANC is more interesting, but the improvement is not very large. It also adds more computation, and the searched states may not be valid. I think the paper needs stronger baselines and more ablations to show that these two methods are really needed.

2 comments

Thank you for your helpful review. We apologize for our late reply; we conducted additional experiments to address the reviewers' concerns.

Response to the “Questions for Authors” section.

We address the concerns in Critical Weaknesses below. Due to time limits, many additional experiments use three seeds and a subset of tasks.

Response to Critical Weaknesses 1 and 2.

We understand QRL+GoalOnly as giving the actor the current state ss and specified coordinates gIgg_{I_g} of a sampled full target gg. The critic still uses the full target during actor training.

We are unsure how to apply your suggested mask. Could you clarify its use in training and testing?

Table 1. Test success (%), mean ± sample SD; seeds 1000–1002. GoalOnly uses AQRL's residual backbone family and IQE+MSE dynamics loss.

TaskStepsQRL+GoalOnlyAQRL/0
PandaPnP400k50.19 ± 3.4453.67 ± 7.65
PointMaze-M200k95.23 ± 1.7294.00 ± 1.00
PandaSlide400k42.53 ± 4.0847.17 ± 7.29
Pusher500k37.26 ± 2.6141.37 ± 8.42

GoalOnly and SSE both remove free coordinates from the actor's goal input, which may explain their similar results (Table 1). In our implementation, the GoalOnly actor uses raw inputs, whereas AQRL/0 uses encoded representations. We hypothesize that learning features from raw inputs may make actor training harder and contribute to GoalOnly's slightly lower overall performance than AQRL/0. SSE also combines with ANC: AQRL/32 improves mean success in the six-task ablation.

Response to Critical Weakness 3.

We apologize; ANC's latent search does not guarantee physically reachable completions. Replay-8 replaces this search while retaining AQRL's backbone. It selects the lowest-cost target among the original and up to seven replay states within a task-specific goal tolerance (Table 2).

Table 2. Test success (%), mean ± sample SD; AQRL: seeds 1000–1004; Replay-8: 1000–1002; 200k steps.

TaskAQRL, main table (5 seeds)Replay-8 (3 seeds)
FetchPnP72.3 ± 15.668.33 ± 12.79
Maze2D-L73.2 ± 3.975.67 ± 4.04
Reacher-H98.5 ± 1.097.17 ± 0.29

Table 3. Initial → selected critic cost (relative reduction); seeds 1000–1002, 200k steps.

TaskAQRL/32Replay-8
FetchPnP15.85 → 10.89 (31.33%)15.24 → 10.48 (31.25%)
Maze2D-L74.44 → 67.04 (9.94%)73.68 → 66.78 (9.36%)
Reacher-H36.90 → 34.75 (5.82%)37.33 → 31.22 (16.36%)

Cost reductions are similar on FetchPnP and Maze2D-L, but larger for Replay-8 on Reacher-H. Both select lower-cost targets, which may explain their similar success (Tables 2–3). Replay selects previously observed, physically realized states and merits further comparison with ANC. Thank you for this helpful suggestion.

Response to Critical Weakness 4.

Table 4. Estimated model-update time (hours); seeds 1000–1004. The last column is gain per added hour (pp/h).

TaskStepsAQRL/0AQRL/8AQRL/16AQRL/32pp/h
AntNavigate500k7.1915.0523.9639.860.07
FetchPush200k2.866.6910.2619.860.16
PandaPnP-J400k5.8312.9921.2931.520.10
PandaPush-J400k6.0812.5819.1933.190.23
PandaSlide400k6.0112.8119.9233.000.36
Pusher500k7.0616.2525.8347.610.13

From AQRL/0 to AQRL/32, gains range from 0.07 to 0.36 percentage points per added model-update hour (Table 4). Mean success rises 4.83 points for 28.34 extra hours, or 0.17 points/hour. This modest return may still be worthwhile at a performance bottleneck.

Response to Critical Weakness 5.

AQRL requires a known specified/free coordinate split and cannot directly handle images or language when this split is unavailable.

As you suggested, we added a comparison based on the goal-set extension discussed in QRL [1, Appendix A]. Specifically, we compare our goal-marker adaptation with QRL and AQRL/0 (Table 5).

Table 5. Test success (%), mean ± sample SD; seeds 1000–1002. QRL+GoalMarker uses baseline's backbone.

TaskStepsQRLQRL+GoalMarkerAQRL/0
PandaPnP400k37.33 ± 7.2340.83 ± 4.1653.67 ± 7.65
PointMaze-M200k86.67 ± 2.0878.00 ± 5.2094.00 ± 1.00
PandaSlide400k30.33 ± 10.3734.17 ± 2.0847.17 ± 7.29
Pusher500k22.27 ± 3.167.67 ± 2.9341.37 ± 8.42

GoalMarker trails SSE-based AQRL/0 on all four tasks. It exceeds QRL on PandaPnP and PandaSlide, but is 14.60 points lower on Pusher.

[1] Wang et al. (2023), “Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning,” ICML.

Thank you for the response. I have read the rebuttal and the revised paper. SSE has no advantage over GoalOnly when compared. Furthermore, SSE has not been compared with common mechanisms such as Attention + Mask. ANC also shows no improvement compared to Replay-8. Therefore, I maintain a score of 2.

Reviewer_jQZGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies goal-conditioned RL where a goal constrains only a subset of state coordinates (specified coordinates), leaving the remaining (free coordinates) unconstrained. Building on QRL, the authors identify two limitations of QRL in this setting. First, QRL trains its actor with sampled real states as goals but tests it on artificially constructed goal states with zero-filled free coordinates, creating a train-test mismatch in goal inputs. Second, for each replay pair, QRL trains the actor toward a single sampled target state, ignoring other states that satisfy the same goal and may be cheaper to reach. To address this, the proposed AQRL leads to two modifications. Split state encoding (SSE) encodes specified and free coordinates with separate encoders, so that the actor only takes the specified-coordinate latent as its goal input in both training and testing. Approximate nearest completion (ANC) fixes the specified-coordinate latent and runs gradient steps over the free-coordinate latent to find a completion with lower estimated reaching cost, which replaces the sampled target in the actor loss during training only. Experiments show improved sample efficiency and final performance on 14 online goal-reaching tasks.

Critical strengths
  1. The proposed method is simple and intuitive. Both components are well motivated by concrete limitations of QRL in partially specified goal setting. The negative correlation between QRL’s success and the fraction of free coordinates also supports this motivation.

  2. The intuition behind ANC, that other states satisfying the same goal can be easier to reach with lower cost, is not tied to QRL. It could be transferred to the design of other goal-reaching algorithms whose critics estimate goal-reaching costs.

Critical weaknesses
  1. The presentation of this paper could be largely improved.

(a) The QRL pipeline is not presented as background, so it is hard for a reader unfamiliar with QRL to tell which parts are the contributions/modifications of this paper. I suggest adding a preliminary section on QRL and then presenting AQRL as modifications on top of it.

(b) In Figures 3 and 7, the dashed and dotted curves of the five baselines are hardly distinguishable. Figures and tables are not placed close to the text describing them. For example, Tables 2 and 3 are discussed on page 8 but appear on page 9, and the AUC results in Section 5.3 are only in Table 10 in the Appendix.

(c) Moreover, several important details needed to interpret the main results are only in the Appendix: the network architectures of AQRL versus the baselines (A.1, B.2), the goal inputs of each baseline (B.2), and the sampling of actor-training pairs. It is better for them to be included in the main body.


  1. The performance gains cannot be clearly attributed to SSE and ANC, because AQRL differs from the QRL baseline in several other aspects:

(a) Network architecture. AQRL uses Scaled-CRL’s architecture. Do the baselines use the same architecture as well? What I can infer from the Appendix is that including QRL, baselines’ networks are only described as four-layer MLPs widened to match parameter counts (Appendix B.2). If the baselines are plain MLPs, matching parameter counts does not make the comparison fair, since [1] shows that this architecture is critical for goal-conditioned RL to benefit from larger networks.

(b) Dynamics loss. AQRL adds a full-latent MSE term to QRL’s IQE-based dynamics loss, motivated by catastrophic training failures observed in IQE-only runs. This suggests the MSE term matters for training stability, but its effect is not ablated for QRL.

(c) Goal sampling. Half of the actor-training pairs use later states from the same episode as targets (Appendix A.2), while QRL samples goals from random replay transitions. So here is a difference in the Hindsight Experience Replay (HER) strategy. And from my experience, HER’s relabeled goal source is an important hyperparameter.

(d) The comparison between AQRL/0 and QRL (Lines 385-387) does not isolate the effect of SSE. For example, AQRL/0 already achieves 95.68% on AntNavigate, compared with 4.9% for QRL. Meanwhile, ANC, which I see as the more novel component, only adds 4.83 points on average over six tasks, with large variance on some of them. In addition, these six tasks are ones where AQRL/0 is already strong, and they exclude the maze tasks and FetchSlide.


  1. I am concerned about the validity of the baseline results.

(a) QRL is the most related work and the main baseline, but its performance here is sometimes lower than reported in the QRL paper. For example, on state-based FetchPush, QRL reports roughly 60% success at 200k environment steps (Figure 5 in QRL), while Table 1 reports 16.8% under the same budget. QRL is also the only baseline excluded from the reproduction checks in Appendix C.5.

(b) Goal inputs are inconsistent across baselines (Appendix B.2). On non-Fetch tasks, TD-InfoNCE, C-Learning, and GCSL receive only the task goal coordinates, while CRL receives zero-filled complete goal vectors on all tasks except AntMaze-U. Scaled-CRL [1] usually accepts specified coordinates. Conversely, TD-InfoNCE, the strongest baseline, already receives goal-only inputs on most tasks and thus partially plays the role of SSE.


References:

[1] 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (Wang et al., 2025)

Questions for authors
  1. Regarding Weakness 2, could the authors confirm whether the baselines use the same architecture backbone as AQRL? On at least a subset of tasks, could the authors report (a) QRL with AQRL’s backbone, (b) AQRL/0 without the MSE term in Eq. 7, and (c) AQRL/0 with QRL’s goal sampling?
  2. Regarding Weakness 3, could the authors explain the gap between their QRL results and those reported in the QRL paper on Fetch tasks, and why QRL is not included in Appendix C.5? Could the authors justify the goal inputs chosen for CRL and the inconsistencies when choosing the goal inputs?
  3. ANC is the more novel component under this submission’s scope, but the paper provides little insight into what the search actually does. How far does h^K\hat{h}_K move from h0h_0, and how much does L(h)L(h) decrease within 32 SGD steps? Do the searched completions stay close to the latents of replay states, or does the actor exploit critic errors in regions without data? Some failure mode analysis can shed useful insight into the proposed method.
4 comments

Response 1 of 2

Due to the length limit, we have split our response into two comments.

Thank you for your careful review and helpful suggestions. We apologize for the delay while conducting additional experiments. Due to limited time, many use three seeds on selected tasks.

Response to the “Questions for Authors” section.

Q1(a): QRL with AQRL's backbone.

The main-table baselines use plain MLPs; AQRL uses residual networks with LayerNorm and SiLU. Due to limited time, Table 1 compares the QRL backbone control on selected tasks, including AQRL/0 results outside the original six-task ablation.

Table 1. Test success (%, mean ± sample SD). All methods use the same nn seeds per task, starting at 1000. The paper uses five seeds; differences reflect the three-seed subset or rounding. AQRL/0 disables search; AQRL/32 is our main method.

TasknnQRLQRL + AQRL backboneAQRL/0AQRL/32
Pusher322.27 ± 3.1632.57 ± 6.4541.37 ± 8.4255.57 ± 19.45
Reacher-H54.12 ± 0.616.94 ± 2.2599.00 ± 0.7598.5 ± 1.0
FetchSlide328.50 ± 6.9924.33 ± 5.8029.13 ± 6.1524.70 ± 1.25
PandaPnP337.33 ± 7.2350.33 ± 2.0253.67 ± 7.6556.00 ± 3.61
PandaPnP-J34.83 ± 1.895.00 ± 3.1247.17 ± 5.0351.83 ± 4.04
PandaPush-J312.17 ± 0.5813.00 ± 1.3287.00 ± 4.0994.83 ± 2.47
PandaSlide330.33 ± 10.3726.67 ± 6.6447.17 ± 7.2952.67 ± 3.33
PointMaze-M386.67 ± 2.0891.67 ± 7.5194.00 ± 1.0093.67 ± 3.51

In Table 1, the residual backbone usually improves QRL's mean success, though often modestly, and decreases it on some tasks.

Q1(b): AQRL/0 without MSE.

In a few early failed runs, IQE prediction error stayed nearly constant while full-latent MSE rose and success remained zero. We added MSE to improve prediction stability, not specifically to raise success. Due to limited time, we tested six tasks. Table 2 shows similar success with and without MSE, suggesting our main conclusion remains unchanged on these tasks.

Table 2. Test success (%, mean ± sample SD), using matched seeds 1000–1002; the paper's AQRL/0 results use five seeds.

TaskAQRL/0AQRL/0 without MSE
AntNavigate95.50 ± 0.8795.33 ± 1.61
FetchPush86.37 ± 3.3987.33 ± 0.29
PandaPnP-J47.17 ± 5.0349.50 ± 4.44
PandaPush-J87.00 ± 4.0988.83 ± 2.47
PandaSlide47.17 ± 7.2944.00 ± 1.32
Pusher41.37 ± 8.4237.50 ± 3.91

Q1(c): Goal sampling.

Both methods use the released QRL default: 50% random replay goals and 50% future states from the same trajectory. The code link is on the first page of [1]; our code is in the supplement. Thus, the paper's AQRL/0 already uses QRL's goal sampling.

Q2.1–Q2.2: Fetch results and QRL implementation checks.

Our main-table QRL and the original Fetch setup [1, Sec. 3.4, App. C.3] differ in configuration (Table 3), which may contribute to the gap. Our QRL baseline and AQRL both use one critic, disable extra Gaussian action noise, and use adaptive entropy regularization.

Table 3. Original Fetch and our main-table QRL configurations.

SettingOriginal QRLOur QRL
Encoder/actor hidden layers2 × 5124 × 1184
State latent dimension128512
Critics21
Additional Gaussian action noiseStandard deviation 0.3None
Adaptive entropy regularizationEnabledEnabled

We apologize for omitting QRL from the reproduction checks in Appendix C.5. We ran our baseline with the original configuration on original online tasks, including image inputs (Table 4). FetchPush at 200k steps is in a similar range to [1]. These checks support our implementation's validity in these settings.

Table 4. Validation success (%) with the original QRL configuration. The first row uses seeds 1000–1002 (mean ± sample SD); the others show the final recorded evaluation for seed 1000.

TaskOur QRL baseline, original configuration
FetchPush75.33 ± 6.11
FetchPush94
FetchReach100
FetchReachImage100
FetchPushImage54

Q2.3: Baseline goal inputs.

We follow each baseline's original repository and extend its goal-input convention to other environments. For example, for FetchPush evaluation, CRL [2] uses a 25-dimensional goal vector with the desired object position in both the gripper and object-position coordinates, and zeros elsewhere. Our CRL baseline uses the same input.

Q3: Search diagnostics and replay support.

Consider a robot tasked with reaching a position; its state also includes velocity and orientation. Many full states satisfy the goal, but each QRL training pair directs the actor toward one sampled full state, with its particular velocity and orientation. Ideally, it could instead aim for a goal-satisfying state with lower reaching cost.

Continued in Response 2.

References.

[1] Wang et al., Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning, ICML 2023.

[2] Eysenbach et al., Contrastive Learning as Goal-Conditioned Reinforcement Learning, NeurIPS 2022.

Response 2 of 2 — Q3 continued

ANC approximates this in latent space by fixing the specified-goal latent and applying gradient descent to the sampled target's free-coordinate latent. For an actor-sampled action, the critic estimates the remaining cost from the predicted next-state latent to each completion. ANC retains the lowest-cost candidate for the actor loss during training. This seeks lower estimated cost; it does not guarantee the globally closest physically feasible state.

The original logs lacked the L2 distance between initial and searched latents. Due to limited time, we reran three tasks and measured it on intermediate training batches (Table 5).

Table 5. Mean per-example L2 distance between the initial free-coordinate latent h0h_0 and the retained completion h^\hat h, using seeds 1000–1002.

Task∥h^−h0∥2\lVert\hat h-h_0\rVert_2
PointMaze-M13.22
FetchPnP9.69
FetchPush10.05

Table 6 reports the estimated cost decrease on the six ablation tasks.

Table 6. Critic cost before search and for the retained completion after 32 updates in the original six-task ablation, averaged over five seeds. Reductions use unrounded values.

TaskBefore searchRetained costDecrease (absolute; relative)
AntNavigate50.4943.506.99 (13.85%)
FetchPush15.8810.405.48 (34.54%)
PandaPnP-J27.6118.079.54 (34.56%)
PandaPush-J27.2018.588.62 (31.70%)
PandaSlide63.1656.666.50 (10.29%)
Pusher34.5725.628.94 (25.87%)

Table 7 compares searched and random completions' relative proximity to goal-matched replay latents.

Table 7. L2 distance from searched and random replay completions to sampled goal-matched replay latents (mean ± sample SD; seeds 1000–1002).

TaskSearched → replayRandom → replay
PointMaze-M16.06 ± 0.8314.18 ± 0.57
FetchPnP21.37 ± 0.5727.22 ± 0.70
FetchPush19.92 ± 0.2029.42 ± 1.93

Following another reviewer's suggestion and due to limited time, we compared Replay-8 with ANC on three tasks. Replay-8 retains AQRL's backbone, randomly samples up to seven replay states within the task's goal tolerance, and selects the lowest-cost target among those and the original target. Tables 8 and 9 compare success and estimated cost reductions.

Table 8. Test success (%, mean ± sample SD). AQRL/32 uses seeds 1000–1004; Replay-8 uses seeds 1000–1002.

TaskAQRL/32Replay-8
FetchPnP72.3 ± 15.668.33 ± 12.79
Maze2D-L73.2 ± 3.975.67 ± 4.04
Reacher-H98.5 ± 1.097.17 ± 0.29

Table 9. Initial → selected critic cost (relative reduction), using seeds 1000–1002.

TaskAQRL/32Replay-8
FetchPnP15.85 → 10.89 (31.33%)15.24 → 10.48 (31.25%)
Maze2D-L74.44 → 67.04 (9.94%)73.68 → 66.78 (9.36%)
Reacher-H36.90 → 34.75 (5.82%)37.33 → 31.22 (16.36%)

Both select targets with lower estimated costs (Table 9), which may contribute to similar mean success (Table 8). Replay selects observed states and merits further comparison with ANC.

Because training is online, we expect interaction to reduce the risk of the actor exploiting critic errors in under-sampled regions. If the critic underestimates a cost, the actor is more likely to follow the predicted low-cost path, and further interaction can add data along that path. If the critic overestimates a region that is repeatedly required as a task goal, interaction can also add data there. When a region is rarely required as a goal, we expect its estimation accuracy to matter less for task performance; as the task places greater emphasis on that region, we expect data coverage there to increase. With sufficient training, we therefore do not expect the actor's exploitation of critic errors in under-sampled regions to lead to training failure.

Response to Critical Weakness 1.

Thank you for these valuable suggestions. The initial nine-page limit required us to place some implementation and evaluation details in the appendix. The discussion-stage ten-page limit gives more room but still requires choices. In the revised paper, we expanded the introduction to describe QRL's actor, critic, dynamics model and training flow; clarified actor-pair sampling in Section 4.3; moved the full AUC table into the main text; and improved the curve colors and figure/table placement.

We are sorry that the remaining space does not allow a standalone, more detailed QRL background, or the complete AQRL and baseline architectures, per-baseline goal inputs, and detailed actor-pair sampling procedure in the main text. Some figures, including Figures 2 and 4, still appear on the page after their first mention because of layout constraints. We have nevertheless placed first mentions and their figures and tables as close together as possible for readability.

I thank the authors for the additional experiments and analyses. The rebuttal has addressed my concerns to a reasonable extent. That said, these deeper analyses and additional results could have served as key evidence supporting the paper's main claims from the outset, rather than being presented as ad hoc supplementary material in the rebuttal. Nevertheless, the proposed method appears to be a natural and useful improvement upon the original QRL. I have therefore raised my score to 3.

Thank you for reviewing our response and the additional results. We appreciate your taking the new evidence into account and raising your score. We agree that presenting these analyses alongside the main results in the original submission would have made the evidence for our conclusions clearer. Your questions helped us improve the baseline comparisons and better explain the role of completion search. Thank you again for your careful review and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.