acceptodds
Under review as a conference paper at ICLR 2027

Acting in the Right Basis: Test-Time Compositional Generalisation for Offline Goal-Conditioned RL

Abstract

Robotic manipulation goals combine several factors, such as the placements of different objects. Offline data cover each factor but rarely their combination, so a goal-conditioned agent must generalise compositionally. Test-time methods improve a frozen agent, the backbone, without new data, yet they compose along time. They chain dataset states by graph search or fine-tune on retrieved sub-trajectories. When no dataset state realises the goal combination, the chain keeps a long final hop. Retrieval then teaches nothing beyond the backbone value. We compose along factors instead. Our method, GC-FACT, shows the frozen policy one single-factor target at a time. Such a target copies the current state and moves one object to its goal placement. Play data contain many such changes. This works only if the policy can carry out each target. That depends on how the goal is split into units, which we call the basis. We prove that the serial plan is near optimal when units are atomic and weakly coupled. We also prove that the relative cost of each target depends on the basis, not only on the environment. On the Lights Out puzzles, per-cell targets are unreachable or costly. Per-press targets are cheap, yet monotone descent over them stalls. A linear solve over the binary field finds the exact plan. GC-FACT lifts three frozen backbones on nine OGBench manipulation tasks. Their average success rises from 34.4, 9.3 and 5.0 percent to 60.7, 45.0 and 32.6 percent. Graph search lowers all three averages.

Primary area:
reinforcement learning
Keywords:
offline goal-conditioned reinforcement learning, test-time adaptation, factored goals, compositional generalization, robotic manipulation, frozen policy, planning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VQ6aOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GC-FACT, a training-free method for improving compositional generalization in offline goal-conditioned RL. Instead of planning through intermediate states from the dataset, GC-FACT decomposes a goal into individual factors and asks a frozen policy to complete them one at a time. The authors also study how the choice of factorization, called the basis, affects the feasibility of these subgoals.

Critical strengths
  • The paper is well presented, with clear and intuitive figures and tables.
  • Test-time goal decomposition is an important research direction, and training-free methods could be particularly useful in practice.
Critical weaknesses
  • The proposed method relies too heavily on human-designed priors. The most important parts of the method, Factorization and Basis Selection, are manually specified rather than learned. For some tasks, the authors even use known environment dynamics to construct an exact solution. This makes me question how well GC-FACT would generalize to new tasks. More importantly, if goal decomposition still requires substantial human knowledge and task-specific design, I am not sure how useful this framework is in practice. Given how capable frontier LLMs have become, I believe that simply describing the task to an LLM could already produce reasonable goal decompositions at different levels of granularity, even for tasks requiring complex reasoning. I would like to see what GC-FACT offers beyond such straightforward alternatives.

  • Most of the reported improvements come from the Lights Out puzzles, where the authors use a known transition matrix and an exact algebraic solver. The improvements on the other manipulation tasks are much smaller, and some tasks remain almost unsolved. It is therefore difficult to tell how much of the improvement comes from GC-FACT itself, rather than the additional task-specific planning knowledge.

  • The experiments are limited to state-based simulation tasks. All evaluations are conducted on OGBench, without visual observations or real-world robot experiments. These tasks have relatively simple and explicitly defined state structures, which makes manual factorization much easier. I would like to see whether the proposed method can work on more realistic goal-conditioned manipulation tasks, where the factors and their relationships are not explicitly provided.

Questions for authors

Please see Weaknesses

9 comments
AuthorsOpenReview ↗

Response to Reviewer VQ6a (1/2): human priors and the LLM alternative

Thank you for the review. You raise three concerns: reliance on human priors, the dependence of the gains on Lights Out with a known matrix, and the restriction to state-based simulation. We address the first here and the other two in (2/2); all changes are marked in blue in the revised PDF.

W1. The factorisation and the basis are specified by hand.

The structure can be read off the offline data. Play data move one unit at a time, so consecutive states reveal the units. On all four Lights Out grids, the distinct flip patterns between consecutive states are exactly the columns of the toggle matrix, each column occurs from at least 510 distinct colourings, and the moving button identifies the pressed cell. With this matrix GC-FACT stays within 1.2 points of the built-in one on five of six backbone-grid pairs and within the seed spread on the sixth. On the cube tasks, grouping observation dimensions by the transitions in which they change recovers every cube block.

A plausible decomposition is not enough. "Fix one light at a time" is the decomposition a person or a language model proposes first, and a frozen policy cannot execute it. No press flips a single light, and a single-light target is unreachable or costs more presses than the whole goal on 41 to 100% of the reachable configurations. With the same data, policy and schedule, this decomposition falls below the frozen backbone on all four grids. The data reject it before any rollout, because a single-light flip never occurs in them.

We ran your suggestion. We described each task to deepseek-chat (task, success test, frozen goal-conditioned policy) and asked for a Python function that returns the next intermediate goal; five samples per prompt. We then classified every returned function on the exact Lights Out model, without rollout:

promptunit proposedtargets costlier than the goalsolved residualspresses / optimal
no press rule (20 samples)single cell in 20/2019 to 25% (100% on 4x4, unreachable)100% on 3 grids, 0% on 4x44.6 to 10.0
with press rule (17 valid)single press in 17/170%42 to 67%1.0 to 1.3

Without the press rule, every sample proposes the per-cell basis, and following it spends 4.6 to 10.0 times the optimal number of presses. With the rule, the basis becomes actionable, but only 8 of the 17 valid samples also plan exactly; the other 9 choose presses by a local rule and stop on most residuals, as monotone descent does. On the cube tasks no sample proposes a one-object target: every target moves one or several cubes only part of the way toward the goal.

What GC-FACT offers beyond such alternatives is therefore the test, not only the decomposition. Its checks (data support, factor cost ratio, an efficient planner) classify each of these proposals before any rollout and need no knowledge of where a proposal comes from. A human, a language model or a learned model can supply candidate decompositions, and GC-FACT decides which one a frozen policy can execute and how to plan in it. Coupling the language model with these checks, so that a rejected proposal is returned with the failed check as feedback, is a direct extension we intend to pursue.

AuthorsOpenReview ↗

Response to Reviewer VQ6a (2/2): where the gains come from, and pixels

W2. Most of the improvement comes from Lights Out with a known transition matrix. You are right that the puzzles dominate the average, and the revised paper reports both groups separately (success, %):

average successGCIQLHIQLOTA
cube tasks and scene: frozen to GC-FACT33.0 to 37.311.5 to 17.57.6 to 13.9
Lights Out: frozen to GC-FACT36.1 to 90.16.6 to 79.41.8 to 55.9

Three observations separate the contribution of GC-FACT from that of extra knowledge. First, the cube tasks and scene use no transition model, and scene alone gains 4.7, 19.6 and 26.8 points. Second, on the puzzles the toggle matrix is read off the data, and GC-FACT with that matrix matches the built-in one. Third, knowledge alone does not help, because with the same factor slices per-cell targets fall below the frozen backbone on every grid and a greedy choice among exact single presses reaches at most 26.1%, whereas the F2 solve reaches 7.2 to 100.0%. The cubes stay hard for a measurable reason: with four cubes present the frozen policies finish a single-cube target in only 11 to 28% of episodes, so a chain of four targets rarely succeeds, and raising this local competence by relabelling one-object changes in the offline data as goals is the step we plan next.

W3. State observations only. We agree that this was the main limitation, and we now run GC-FACT on the pixel-based OGBench tasks with the same frozen checkpoints and no simulator access. The units are read off image data: cubes are colour clusters that rarely move between consecutive frames, and buttons are blobs that alternate between two colours, which also yields the toggle matrix. A cube target erases one cube from the current frame and pastes it at its goal position, and a press target repaints the buttons that one press toggles. An offline check against targets rendered by the simulator shows that, on visual-cube-double, the synthesised images are closer to the true single-factor targets than the current frame is, and the reader recovers every button state and the toggle matrix on the 3x3 and 4x4 grids. Results over four seeds (success, %):

pixel tasksfrozenTTGSGC-FACT
GCIQL, average of 9 tasks9.75.634.4
HIQL, average of 9 tasks47.022.662.2
OTA, average of 9 tasks50.421.359.3
visual-puzzle-4x5, GCIQL / HIQL / OTA4.8 / 16.4 / 18.40.4 / 0.4 / 1.282.8 / 84.4 / 87.6
visual-puzzle-4x6, GCIQL / HIQL2.4 / 15.60.0 / 0.080.8 / 76.4

The press basis therefore carries over from states to pixels, and TTGS again lowers nearly every pair. On the cube tasks the picture is mixed: pasted targets help OTA on visual-cube-double (58.0 to 70.0) but leave most other pairs close to or below the frozen backbone. We attribute this to the painted image, which keeps the shadows and occlusions of the current frame around a cube copied from the goal frame, a combination the pixel policy never met as a goal. Our planned remedy follows from this cause: replace copy and paste by an object-centric encoder (slot attention or DLPv2, as used by HECRL) that swaps one object slot for its goal slot and decodes a consistent image. On visual-puzzle-4x6 the frozen OTA policies fail every goal, and no planner can compose presses that the policy cannot execute, which is the local-competence bound of our failure proposition. The reader works because OGBench objects are colour-coded, and scenes without such cues will need the same object-centric encoder.

Real robots. GC-FACT changes only the goal input of a frozen policy and learns no parameters, so it adds no training risk on hardware, and its success is bounded by the local competence of the policy, which the paper names and measures.

We hope these results address your concerns and would be grateful if you would take them into account in your assessment.

References for this response.

  • deepseek-chat: DeepSeek-AI, DeepSeek-V3 technical report, arXiv:2412.19437 (API model queried on 10 November 2026).
  • OGBench: Park et al., OGBench: Benchmarking offline goal-conditioned RL, ICLR 2025.
  • TTGS: Opryshko et al., Test-time graph search for goal-conditioned RL, ICML 2026.
  • HECRL: Haramati et al., Hierarchical entity-centric RL with factored subgoal diffusion, ICLR 2026.
  • Slot attention: Locatello et al., Object-centric learning with slot attention, NeurIPS 2020.
  • DLPv2: Daniel and Tamar, DDLP: Unsupervised object-centric video prediction with deep dynamic latent particles, TMLR 2024.
  • Backbones: GCIQL (Kostrikov et al., ICLR 2022, goal-conditioned variant of OGBench), HIQL (Park et al., NeurIPS 2023), OTA (Ahn et al., NeurIPS 2025).
AuthorsOpenReview ↗

Changes (revised PDF, marked in blue)

PointLocation
W1 human priors§6.3, Tab. 16, App. F.4 (factor slices and toggle matrix read off the offline data; no single-cell flip occurs in the data); new App. F.8, Tab. 24 (decompositions proposed by deepseek-chat, checked without rollout); AI use statement
W2 gains from Lights Outabstract; last two rows of Tab. 2; §6.1; Tab. 3 (value-greedy row), Tab. 15; Tab. 16 (A read from data); §6.4, Tab. 19 (local competence on cube-quadruple)
W3 state-based only§5.3 (paragraph after Prop. 5.8); §6.3 (units read off images); App. E, paragraph "Pixel observations"; App. F.7 with Tab. 22 and Figs. 13–14; §7 (next steps)
Reviewer_nL21OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the factorization of goals into individually achievable parts based on a chosen basis in the setting of offline goal-conditioned RL within robotics (OGBench). The proposed method, GC-Fact, wraps a frozen policy, chooses a basis (hand-crafted/selected) and replaces the policy's goal conditioning with an open factor. A theoretical outline examines the conditions under which factorisation is beneficial and when it isn't. Experiments support the benefit of the proposed method in OGBench manipulation experiments, where it outperforms the baseline TTGS and the frozen base policy meaningfully on the puzzle problems. A secondary experiment on Lights-Out showcases the importance of the provided basis, i.e., only in the right basis does it outcompete the frozen base here.

Critical strengths

The proposed method has value for many robotic manipulation tasks that are inherently decomposable into approximately independent object-level goal factors; thus, the addressed issue is meaningful and follows an important solution principle. The results are strong for puzzles, although not as convincing for the cube problems. The greatest value is arguably in its reusability of existing policies via a simple wrapper on top. Genuine failures (press comparison) also provide informative insight into the limits of the method.

Critical weaknesses

The largest problem for me is that following the paper was difficult at times, despite multiple readings. Initially, the acronyms TTGS, GC-TTT looked like versions of the proposed method, since their acronyms were not connected to method names. From the methodology section onwards, it did not succinctly state the core of the method, its clear workings, required components and where they come from. Instead, forward references made reading non-linear and the methodology was directly followed by a theoretical section that, in my opinion, would have been better served as a compact prose description with concrete propositions in the appendix. Although section 4 and Algorithm 1 give the overall procedure, key workings are seemingly not explained together. Essential details about the domain knowledge appear in the appendix. A worked example would substantially improve clarity. As is, the paper reads too disjointedly in many parts.

Another worry lies with the discussion of related work, or its lack thereof. Task decomposition is a widely studied topic in planning which underlies the robotics area of Task and Motion Planning, and is directly related to the proposed decompositions in this paper. In particular, width-based goal serializations are a common solution principle, see e.g. Drexler et. al., yet this entire related section is ignored, instead of contextualizing the approach. They could serve as baselines, especially on puzzles like Lights-Out.

Lastly, the handcrafted basis is a design choice that does not invoke generality of this method and the lack of a practical way of determining this basis limits its transfer.

Questions for authors
  1. How much does the value-based selection matter beyond the decomposition and planner? Do you have comparisons against fixed or random object ordering?

  2. Could you give a worked example showing the components for a new environment, how the targets are constructed etc., and what the frozen policy receives? How could one judge a basis as suitable?

  3. What disntiguishes the contribution from goal serializations and planning sketches, such as Drexler et al.? Could you please give a comprehensive comparison?

8 comments
AuthorsOpenReview ↗

Response to Reviewer nL21 (1/2): presentation and a worked example

Thank you for reading the paper several times and for the concrete suggestions on presentation. We revised the paper along each of them; all changes are marked in blue.

W1. The paper is hard to follow.

  1. Acronyms. TTGS and GC-TTT now appear with their full names (test-time graph search, goal-conditioned test-time training) at first mention, introduced as prior methods. A new comparison table sets them apart from GC-FACT and from the planning methods you mention.
  2. The core of the method in one place. The method section now opens by listing what GC-FACT needs and where each part comes from: the frozen value and policy, the factor slices of the observation with their success tolerances, and, on Lights Out only, the effect of a press. The experiments show that the slices and the press effects can be read off the offline data; a few hard preconditions such as a locked drawer are written by hand.
  3. Linear reading. The theory section opens with a short roadmap (why time-based composition keeps a long hop, which bases a frozen policy can execute, where the guarantees lapse). Each formal result is preceded by one sentence of intuition, proofs stay in the appendix, and we removed forward references that forced non-linear reading. We kept the formal statements in the main text because the experiments test them directly, for example the per-cell versus per-press comparison.
  4. Worked examples. A new appendix traces two episodes step by step.

The method in five steps. GC-FACT wraps a frozen goal-conditioned agent and changes only the goal its policy sees. At every environment step it (i) finds the units whose value differs from the goal; (ii) builds one target per such unit, a copy of the current state with only that unit set to its goal value; (iii) drops targets whose unit cannot move yet, such as a locked drawer, or that stalled; (iv) passes the remaining target with the smallest value-derived distance; (v) passes the raw goal once it is within 48 steps.

cube-double. The observation is a 19-dimensional arm block followed by one 9-dimensional block per cube. A goal that moves both cubes leaves cubes A and B unsatisfied. GC-FACT builds one target by copying the current observation and overwriting the block of A with its goal value, and a second target for B in the same way. The frozen value scores both, and the policy receives the cheaper one, exactly as it would receive any goal observation. Once A is in place, only B remains, and its target keeps A at its goal value. If the goal stacks B on A, the precedence rule holds B back until A is placed.

Lights Out 3x3. Suppose only the centre light differs from the goal. A per-light target ("flip only this light") is the goal itself, which needs five presses, and every possible first press leaves three or four lights wrong, so greedy search stops at once. In the press basis each target is the current colouring with one press applied. Solving Ax = b over F2 gives five presses in a plus shape, and the policy sees one press target at a time while the number of wrong lights rises to five and then falls to zero.

Q2 (second part). Judging whether a basis is suitable. Three checks, none of which needs a rollout.

  1. Data support: each unit target must correspond to a change that occurs in the data. On Lights Out each single press occurs from at least 510 distinct colourings, while a single-light flip never occurs, which rejects the per-light basis immediately.
  2. No target harder than the goal: the factor cost ratio compares the hardest target with the goal. Estimating it with the frozen value is unreliable exactly where it matters: on 4x5 and 4x6 the estimate stays below one although single-light targets cost 8 to 12 presses. Check 1 is therefore the dependable one.
  3. An efficient planner: weak coupling, so that any order is near optimal, or solvable structure such as linearity over F2.
AuthorsOpenReview ↗

Response to Reviewer nL21 (2/2): planning literature and the value-based order

W2 and Q3. Task decomposition, goal serialisation and sketches. Thank you for this pointer; it sharpened the positioning. Related work now discusses goal serialisation (Lipovetzky & Geffner, 2012), planning sketches written by hand and learned (Drexler et al., 2021; 2022) and task and motion planning (Garrett et al., 2021). A new table compares them with our setting on the unit of a subgoal, how subgoals are produced, who executes them, and how executability is tested.

We also ran a serialisation baseline on Lights Out that follows SIW. It hill-climbs on the number of correct lights, and each step runs a breadth-first search over presses with novelty width k until one more light is correct. It serialises over light atoms but plans with exactly the same press model as our F2 solve. Share of reachable residuals solved (exact on 3x3 and 4x4, 3000 sampled residuals for SIW on 4x5 and 4x6):

gridmonotone descentSIW(1)SIW(k <= 2)F2 solve
3x35.3%6.5%6.9%100%
4x49.0%15.5%23.5%100%
4x50.2%0.6%0.7%100%
4x60.1%0.1%0.1%100%

Drexler et al. (2021) note that serialisation fails when the goal is not serialisable, and Lights Out over light atoms is such a case: fixing one light almost always breaks another. In the language of sketches, the press basis chooses features under which every subproblem is a single action.

The comprehensive comparison you ask for comes down to one difference. In classical planning, SIW and sketches assume a symbolic model, and an action is executable by definition. In our setting every subproblem must be executed by a frozen low-level policy trained offline on whole-state goals, and nothing guarantees that it can. Our actionability test (the factor cost ratio together with data support) states this requirement, and the per-cell basis fails it on all four grids although its subproblems are well defined. A second difference is the source of the model: on the cube tasks and scene GC-FACT uses no transition model at all, and on Lights Out the press effects are read off the offline data. A natural extension is to learn sketch-like features directly from such data.

Q1. The role of value-based selection beyond the decomposition and the planner. We compared random, fixed-index and closest-to-goal orders of the same targets, three seeds. On cube-triple every order gives the same success within 0.9 points (GCIQL 5.2 to 6.9%, OTA 4.5 to 4.7%), all far above the frozen backbones (1.9%). On scene, random and fixed orders change GCIQL by +2.0 and -0.2 points, and for OTA the closest-to-goal order improves on ours by 10.4 points. The gain therefore comes from the single-factor targets; the value function matters mainly for two switches, when to pass the raw goal and when to demote a stalled unit. On the puzzles, choosing each press greedily by the frozen value fails (at most 26.1% on any backbone and grid, against 7.2 to 100.0% for the F2 solve), which again places the gain in the basis and the planner rather than in value-based selection.

W3. The handcrafted basis limits transfer. The press effects and the cube blocks are now read off the offline data. With the identified matrix GC-FACT stays within 1.2 points of the built-in one on five of six backbone-grid pairs and within the seed spread on the sixth, and grouping dimensions by co-change recovers every cube block. What remains hand-written are the lock and stacking preconditions. We plan to learn them from interaction data, as SRICS learns an interaction graph from random-policy data, which would remove the last hand-written component.

We hope the revision now reads linearly and would be grateful if you would take it into account in your assessment.

References for this response.

  • TTGS: Opryshko et al., Test-time graph search for goal-conditioned RL, ICML 2026.
  • GC-TTT: Bagatella et al., Test-time offline RL on goal-related experience, ICML 2026.
  • SIW: Lipovetzky and Geffner, Width and serialization of classical planning problems, ECAI 2012.
  • Sketches: Drexler et al., Expressing and exploiting the common subgoal structure of classical planning domains using sketches, KR 2021 and Learning sketches for decomposing planning problems into subproblems of bounded width, ICAPS 2022.
  • TAMP: Garrett et al., Integrated task and motion planning, Annual Review of Control, Robotics, and Autonomous Systems 2021.
  • SRICS: Zadaianchuk, Martius, Yang, Self-supervised RL with independently controllable subgoals, CoRL 2021.
  • Backbones: GCIQL (Kostrikov et al., ICLR 2022, goal-conditioned variant of OGBench), HIQL (Park et al., NeurIPS 2023), OTA (Ahn et al., NeurIPS 2025).
AuthorsOpenReview ↗

Changes (revised PDF, marked in blue)

PointLocation
W1 acronyms§1 (full names of TTGS and GC-TTT); new comparison Tab. 1
W1 core of the method§4 opening (inputs and where each comes from); §4.1 (what the policy receives)
W1 linear reading§5 opening (roadmap); new lead-in before Prop. 5.5
W1 worked examplenew App. G with Fig. 15 (cube-double, Lights Out 3x3), referenced at the end of §4.2
W2 / Q3 planning literature§2, paragraph "Planning over goal atoms", and Tab. 1; SIW baseline in §6.2, App. C, Tab. 5, Fig. 6(c)
W3 handcrafted basis§6.3, Tab. 16, App. F.4 (basis read off the data); §7 (learning the preconditions)
Q1 value-based selection§6.4, Tab. 18(b), App. F.5 (target orders); Tab. 3 (value-greedy row), Tab. 15
Q2 suitability of a basisApp. G; §6.3, Tab. 16 (data support); Tab. 17 (ratio estimated with the value)
Reviewer_PNzmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies offline goal-conditioned RL where test goals combine object configurations that are covered by the data individually but rarely jointly. Instead of composing along time (graph search, test-time training), it composes along factors: at each step the frozen policy is given the current state with a single factor set to its goal value, selected by the learned value function. For Lights Out puzzles the factorisation is changed from cells to button presses, obtained by solving a linear system over F2\mathbb{F}_2. Experiments on OGBench manipulation with three backbones show large average improvements over the frozen policies and over temporal composition baselines.

Critical strengths
  • Simple, training-free method that improves three different frozen backbones on most tasks (Table 1). This indicates that the bottleneck is the goal the policy is conditioned on, and it is a useful baseline for compositional generalisation on OGBench.
  • Clean demonstration that the basis matters: Table 2 changes only the coordinates with the same checkpoint.
Critical weaknesses
  • The major gains rely on privileged structure that the baselines do not get
    • On the four puzzles, which dominate the average (e.g. HIQL puzzle-4x5 3.3 → 99.9), GC-Fact gets the exact toggle matrix AA, i.e. the full transition model, and solves the task symbolically (Cor. 5.7). The policy only executes single presses.
    • The other tasks use hand-derived factor slices (Table 7) and hand-written precedence rules (stack order, locks).
    • TTGS and GC-TTT get none of this.
  • Composing along factors at test time is not new:
    • Haramati et al. (ICLR 2026, not cited) give a frozen offline GCRL agent entity-factored subgoals (sparse changes to the current state), selected with the value function, on OGBench cube and scene variants. GC-Fact replaces their learned subgoal generator by copying the state and overwriting one factor.
    • SMORL (Zadaianchuk et al., 2021) is cited only as an object-centric representation (L109), but it already achieves per-object subgoals one at a time, with the representation learned from images. SRICS (Zadaianchuk et al., 2021) also learns the order of the subgoals, which GC-Fact hand-codes as precedence rules.
    • For puzzles, PPGS (Bagatella et al., 2021) learns the combinatorial model from pixels and plans with graph search, instead of being given AA.
  • Without the puzzles the averages have a much more modest improvement (around 5 points), as I deduce from Table 1. cube-quadruple stays at 0, cube-triple below 8%. So where compositional generalisation is hardest and no model is given, the effect is small. The abstract does not reflect this.
  • The argument against temporal composition is not supported by the paper's own data.
    • Prop. 5.2 needs m<Km<K, but Table 6 shows m=Km=K on 7 of 9 environments, so the bound is vacuous there. Still, the intro and Sec. 6.1 use it to explain why TTGS and GC-TTT do not help.
    • TTGS uses one fixed Ts=48T_s=48 everywhere and loses up to 36.1 points, while with per-task tuning Opryshko et al. (2026) report losses of at most 2 points (L412–413). So "graph search lowers all three averages" (abstract) likely reflects the configuration.
    • For GC-TTT, Prop. 5.2 only shows that the ranking of retrieved sub-trajectories depends on HH and V(sH,g)V(s_H,g) (App. B.1), not that fine-tuning "teaches nothing beyond the backbone value" (abstract, L47). GC-TTT in fact gains 5 points on puzzle-3x3 (Table 9).
Questions for authors
  1. What are the TTGS results with per-task tuned TsT_s/τ\tau as in Opryshko et al. (2026)?
  2. How does GC-Fact compare to Haramati et al. (2026), which generates factored subgoals for a frozen GCRL agent on similar OGBench tasks? Is there a baseline with the same privileged information, e.g. a fixed or random order of the single-factor targets (tests the value-based scheduling), or a scripted per-object subgoal sequence for the cubes?
  3. How many training seeds and evaluation runs go into each cell of Table 1? Please report std or CI over seeds.
  4. How sensitive are the cube and scene results to the hand-written precedence filter for GCIQL/HIQL? Table 10 only ablates it for OTA.
  5. GC-TTT on cube-single drops from 70.3 to 47.7 (Table 9), while Bagatella et al. (2026) report 0.74 → 0.75 (tuned) and 0.72 → 0.65 (fixed hyperparameters) for GCIQL on the same task. Which relevance criterion did you use on the cubes (value-based or cube-position distance as in the released code)?
  6. Is the weak-coupling assumption (Ass. 5.3(ii), for all states) ever satisfied with small ε\varepsilon on the cube tasks, e.g. when cubes are stacked?

Suggestions:

  • Learn the factor maps and the basis from data (e.g. object-centric representations as in SMORL, or locally factored models as in MoCoDA), which would make the comparison to the baselines fair.

Minor points / Details

  • Abstract and L47: "Retrieval then teaches nothing beyond the backbone value" is stronger than what App. B.1 proves.
  • Fig. 6 plots Prop. 5.2 with m=1m=1, which is not the regime of the data (Table 6).
  • Props. 5.5(ii), 5.6 and Cor. 5.7 are standard Lights Out results (Anderson & Feil, 1998). You call it the "the centre of the paper", which oversells them.
  • Thm. 5.4 bounds the plan cost under d∗d^*, but Alg. 1 uses d^\hat d, and the theorem says nothing about whether the policy can execute each target.
  • Thm. 5.4 uses transitivity of ≃\simeq (App. B.2), but the success test is a per-object tolerance, which is not transitive.
  • Fig. 8: "every environment with ρ^Φ<1\hat\rho_\Phi<1 gains" rests on five data points.
  • Table 1 caption: the highlighting rule is hard to parse; a significance test over seeds would be clearer.
  • Table 9: why is puzzle-4x4 missing for GC-TTT? Fig. 9 omits puzzle-4x6.
  • Terms like "play data", "backbone", "seed spread" are used before they are defined.
  • Missing and relevant work:
    • D. Haramati, C. Qi, T. Daniel, A. Zhang, A. Tamar, G. Konidaris. Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion. ICLR 2026. (Factored subgoals selected by value for a frozen offline GCRL agent on OGBench; closest prior work.)
    • A. Zadaianchuk, M. Seitzer, G. Martius. Self-supervised Visual Reinforcement Learning with Object-centric Representations (SMORL). ICLR 2021. (Cited, but only as a representation; per-object subgoals achieved one at a time, representation learned from images.) And: A. Zadaianchuk, G. Martius, F. Yang. Self-supervised Reinforcement Learning with Independently Controllable Subgoals (SRICS). CoRL 2021. (Learned interaction graph to order the subgoals.)
    • S. Pitis, E. Creager, A. Mandlekar, A. Garg. MoCoDA: Model-based Counterfactual Data Augmentation. NeurIPS 2022. (Locally factored dynamics for out-of-distribution object combinations in offline RL.)
    • S. Nasiriany, V. Pong, S. Lin, S. Levine. Planning with Goal-Conditioned Policies. NeurIPS 2019. (Subgoal planning on top of a goal-conditioned policy.)
    • M. Bagatella, M. Olšák, M. Rolínek, G. Martius. Planning from Pixels in Environments with Combinatorially Hard Search Spaces (PPGS). NeurIPS 2021. (Learned latent model plus graph search for combinatorial puzzles.)
6 comments
AuthorsOpenReview ↗

Response to Reviewer PNzm (1/4): privileged structure and novelty

Thank you for a detailed and technically precise review. We answer W1 and W2 here, W3 and W4 in (2/4), the questions in (3/4), and the minor points in (4/4). All changes are marked in blue in the revised PDF.

W1. The gains rely on structure that the baselines do not get. We agree that the submission supplied structure to GC-FACT that TTGS and GC-TTT did not receive, and we address this in three ways.

(a) The toggle matrix is read off the data. On all four grids the distinct flip patterns between consecutive dataset states are exactly the columns of A, and the moving button identifies the pressed cell. With this matrix, GC-FACT stays within 1.2 points of the built-in matrix on five of six backbone-grid pairs and within the seed spread on the sixth (OTA, 4x4: 24.0 versus 17.2, spread 7.1). On the cube tasks, grouping observation dimensions by co-change recovers every cube block.

(b) Information alone does not produce the puzzle gains. The per-cell basis uses only the factor slices, the same information you list for the cube tasks, and it falls below the frozen OTA backbone on all four grids:

OTA, success (%)3x34x44x54x6
frozen4.02.50.70.5
per-cell targets2.90.50.00.0
press targets, monotone descent3.24.17.51.7
press targets, frozen-value choice3.14.14.13.6
press targets, F2 solve6.816.699.6100.0

The gain appears only when presses are the units and the F2 solve plans them. On Lights Out GC-FACT therefore acts as a task-level planner over a frozen low-level executor, and the method section now says so explicitly. The policies already execute single presses (exact press targets produce 95 to 98% of the OTA steps); what they lack is the plan.

(c) The baselines with the same information. Restoring already-placed objects in each TTGS waypoint, which gives TTGS the factor slices, recovers at most 6.2 points. GC-TTT with the environment's cube-position relevance changes cube-single by +0.4 and cube-double by -4.4 points.

Rules. Removing all hand-written rules changes cube-triple by at most 0.7 points on all three backbones; their effect concentrates on scene, where they encode lock preconditions (details under Q4).

W2. Novelty with respect to HECRL, SMORL, SRICS and PPGS. Thank you for pointing us to Haramati et al. (2026). We now discuss all four papers and compare them in a new table, with each entry checked against the original paper.

  • HECRL trains a diffusion model that proposes entity-factored subgoals with sparse changes, and composes it post hoc with a separately trained entity-centric value-based agent. Its Algorithm 1 keeps the candidates with V(s, c) above a competence radius, where c is a candidate subgoal, and selects the candidate with the highest V(c, g). GC-FACT writes each target explicitly, needs no generator, wraps unmodified GCIQL, HIQL and OTA, and tests whether a unit is actionable. With a perfect generator (all n single presses, all inside the competence radius), the HECRL selection rule reaches 0.3 to 26.1% across backbones and grids, against 7.2 to 100.0% for the F2 solve. On puzzle-3x3 it lowers GCIQL from 95.9 to 20.0%, because a press often helps only several presses later.
  • SMORL trains an online policy on single-object goals from a learned object-centric representation and, at evaluation, cycles through the goal objects under an independence assumption. SRICS learns an interaction graph from random-policy data and orders subgoals by depth in it. Our precedence rules encode such an order by hand, and learning it as SRICS does is the next step named in the conclusion of the revised paper.
  • PPGS plans action sequences by breadth-first search in a latent graph learned from pixels and has no goal-conditioned policy. GC-FACT keeps the frozen offline policy as the controller and changes only the goal it sees.
AuthorsOpenReview ↗

Response to Reviewer PNzm (2/4): W3 and W4

W3. Without the puzzles the improvement is modest. You are right, and the abstract now says so. The main table reports both group averages:

average success (%)GCIQLHIQLOTA
cube tasks and scene, frozen33.011.57.6
cube tasks and scene, GC-FACT37.3 (+4.3)17.5 (+5.9)13.9 (+6.3)
Lights Out, frozen36.16.61.8
Lights Out, GC-FACT90.1 (+54.0)79.4 (+72.9)55.9 (+54.1)

Scene, which uses no transition model, gains 4.7, 19.6 and 26.8 points. On cube-quadruple every method stays near zero, and a new probe explains why. With all four cubes present, the frozen policies finish a single-cube target in only 11 to 28% of episodes. A chain of four such targets then succeeds in at most 0.03 to 0.43% if the targets fail independently, and the observed success is 0.0 to 0.1%. This is the local-competence cap of our failure proposition, not a failure of the decomposition. A remedy is to raise local competence by relabelling the one-object changes in the offline data as goals, which we leave to future work.

W4(a). The hop bound needs m < K. We agree that the bound is vacuous on all data, where m = K on seven environments. Its proof only uses that the path lies in a fixed set, so we now state it for the set of states that a planner may route through. For TTGS this set is its 4000 sampled vertices. On these vertex sets (four seeds), m < K holds for 23 of the 40 goals in every seed:

taskgoals with m < Kmean bound
cube-triple4/50.32
cube-quadruple5/50.45
puzzle-4x4 / 4x5 / 4x64/5, 5/5, 5/50.08 / 0.12 / 0.13
cube-double, scene0/5vacuous

On cube-double and scene the vertex sets contain goal states, and the paper no longer uses the bound there. The loss on these tasks has a measured cause instead: TTGS hands the policy a dataset state on 95 to 99% of the steps, and on 6 to 44% of the steps the waypoint moves an object that is already in its goal placement.

W4(b). The TTGS configuration. We checked this carefully, and it changed how we report the baseline.

  • Our setting is the TTGS default: 4000 uniformly sampled vertices, one plan per episode, tau = 24 and T_s = 48. These are also the default arguments of the original code.
  • For the record, Opryshko et al. (2026) report changes of 0 to +2 points with GCIQL on scene, cube-triple, puzzle-4x5 and puzzle-4x6, not losses. Our sentence about their tuning was ambiguous, and it now states their setting and numbers explicitly.
  • Because our numbers disagree with the reported ones, we ran the original code unchanged on our seed-0 checkpoints. It reproduces the loss on all five tasks we checked.
  • We then selected (tau, T_s) per task on the evaluation goals, a choice that favours the baseline, and evaluated the chosen setting on three seeds:
taskselected (tau, T_s)frozenTTGS tunedpaired changeGC-FACT
scene(24, 192)53.246.9-6.356.9
puzzle-3x3(12, 48)95.999.1+3.299.4
puzzle-4x4(48, 48)22.730.0+7.392.7
puzzle-4x5(24, 192)13.712.3-1.579.1
puzzle-4x6(24, 192)11.54.9-6.589.3

On cube-quadruple tuning leaves 0.0%, and on cube-single, cube-double and cube-triple the best setting in the selection run exceeded the backbone by at most 2.4 points. Larger T_s hands the raw goal to the policy more often, which moves TTGS toward the backbone rather than above it.

W4(c). "Retrieval teaches nothing beyond the backbone value." Agreed; the statement was stronger than the proof. The abstract and introduction now say only that retrieval can rank the data by the backbone value at the unseen goal, and the proof notes that fine-tuning can still change the policy. GC-TTT does gain 5.0 points on puzzle-3x3.

AuthorsOpenReview ↗

Response to Reviewer PNzm (3/4): questions Q1 to Q4

Q1. TTGS with per-task tuned T_s and tau. See W4(b): tuned per task, TTGS changes success by -6.5 to +7.3 points; its best case, 30.0% on puzzle-4x4, compares with 92.7% for GC-FACT.

Q2. Comparison with HECRL, and baselines with the same information. For the HECRL selection rule, see W2. For the order of the single-factor targets, which isolates the value-based schedule, we ran random, fixed-index and closest-to-goal (the HECRL rule) orders with the same targets. Success (%), with the change from the greedy order of the same seeds in parentheses, three seeds:

orderGCIQL cube-tripleGCIQL sceneOTA cube-tripleOTA scene
frozen backbone1.953.11.923.1
greedy on value (ours)6.056.54.745.2
random6.3 (+0.3)59.2 (+2.0)4.7 (+0.0)49.5 (+4.3)
fixed index6.9 (+0.9)57.0 (-0.2)4.5 (-0.1)52.4 (+7.2)
closest to goal (HECRL)5.2 (-0.8)54.2 (-3.0)4.7 (+0.0)55.6 (+10.4)

A scripted order on scene (unlock, cubes, sliders, buttons) gives 59.6 (+2.4) with GCIQL and 56.0 (+10.8) with OTA. Every order stays well above the frozen backbone, so the single-factor targets, not the schedule, carry the gain. On scene the closest-to-goal and scripted orders beat our greedy order for OTA by about 10 points; the paper now reports this. Lock coupling is strongest there, and an order that accounts for measured coupling is a direct improvement we plan to make.

Q3. Seeds and evaluation runs. The main table keeps its original runs, one to eight evaluations from up to six training seeds per cell, with the standard deviation over these runs; the setup now states this. Every new experiment runs the frozen backbone and the planner in the same process with identical reset seeds over three or four training seeds and reports paired differences with 95% bootstrap intervals over seeds. A new forest plot shows these paired intervals for the whole OTA block, and two evaluations of the same checkpoint differ by up to 2.4 points.

Q4. Sensitivity to the hand-written rules for GCIQL and HIQL. We ran the rule ablation on GCIQL and HIQL, and now also on OTA. Success (%), change from the reported configuration of the same seeds in parentheses, three seeds:

variantGCIQL c-doubleGCIQL c-tripleGCIQL sceneHIQL c-doubleHIQL c-tripleHIQL sceneOTA c-tripleOTA scene
frozen39.41.853.15.64.934.72.320.6
reported55.85.556.510.49.155.54.746.4
no precedence-6.8-0.1-1.5-1.2-0.3-2.4-0.1*-2.1*
no patience+0.0-0.5-1.2+0.5+1.1-5.7+0.0*-5.2*
no rules-8.6-0.3-17.6+0.0+0.7-23.5+0.1-22.1

(*) from the earlier OTA component ablation over up to six seeds. Removing the buttons-last rule alone lowers scene by 4.1, 5.1 and 9.3 points with GCIQL, HIQL and OTA.

Three conclusions follow. On cube-triple the rules change success by at most 0.7 points with any backbone. On cube-double the stack order matters for GCIQL (-6.8), yet without any rule GCIQL still exceeds its backbone by 7.8 points. On scene the rules matter jointly because they encode the lock preconditions. For OTA the loss concentrates on the goal that unlocks and relocks both sliders, whose success falls from 96.7 to 6.7%: the buttons toggle the locks, so pressing them before the sliders are aligned relocks a slider that still has to move. Learning such preconditions from interaction data, as SRICS learns its interaction graph, would remove this hand-written component.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): the common concern and the new evidence

We thank the three reviewers for careful and constructive reviews. All three ask, in different words, how much of the gain comes from structure that we supply (factor slices, precedence rules and, on Lights Out, the toggle matrix) and whether the baselines received a fair chance. We answer with new experiments; every change in the revised PDF is marked in blue.

1. The structure can be read off the offline data, or written by a language model. Play data move one unit at a time, so consecutive dataset states reveal the units.

  • Lights Out: on all four grids the distinct flip patterns between consecutive states are exactly the n columns of the toggle matrix A. Each column occurs from at least 510 distinct colourings, the moving button identifies the pressed cell, and no transition flips a single cell. The data therefore reject the per-cell basis before any rollout.
  • GC-FACT with A read from the data versus the built-in A (success, %):
backbone, gridbuilt-in AA from data
GCIQL, 3x399.298.8
GCIQL, 4x494.095.2
OTA, 4x417.224.0
HIQL, 4x5100.099.2
OTA, 4x599.2100.0
OTA, 4x699.699.6
  • Cube tasks: grouping observation dimensions by the transitions in which they change recovers every 9-dimensional cube block.
  • New: everything task-specific fits in a declarative description of 51 to 431 numbers per task. A planner that reads only this description, with no task-specific code, stays within two points of our implementation on 13 of 14 backbone-task pairs. Reading the environment's source code, DeepSeek-V4.1-Flash (thinking mode) writes this description exactly on all 21 task-sample pairs; from data statistics alone, without the task name or code, it passes every check within two rounds of feedback.

2. The baselines run in their own default setting, and with the same information.

  • TTGS uses its default (4000 random vertices, one plan per episode, tau = 24, T_s = 48). The original code of TTGS, run unchanged on our checkpoints, reproduces the loss on all five tasks we checked. The cause is the waypoints themselves: on the cube tasks and scene the policy receives a dataset state on 95 to 99% of the steps, and every plan contains an edge longer than tau. Such a waypoint differs from the current state in several objects at once, which the frozen policy cannot execute.
  • Tuning (tau, T_s) per task on the evaluation goals keeps TTGS between -6.5 and +7.3 points of the backbone. Restoring objects that a waypoint disturbs recovers at most 6.2 points.
  • GC-TTT with the default setting reported in the original paper (cube-position relevance, tuned cube hyperparameters) changes cube-single by +0.4 points, matching the reported result, and lowers cube-double by 4.4 points.

3. The hand-written rules and the value-based order are not the source of the gain. Removing all rules changes cube-triple by at most 0.7 points on any backbone. On scene the rules encode the lock preconditions, and removing them costs 17.6 to 23.5 points; declaring the locks as gates in the task description recovers them on both backbones we tested (within 1.6 points). Random, fixed and closest-to-goal orders change cube-triple by at most 0.9 points on GCIQL and OTA.

4. Neither the frozen value nor a language model finds the right basis on its own.

  • Given all n single presses as candidates, a frozen-value choice (the selection rule of HECRL with a perfect generator) reaches at most 26.1% on any backbone and grid, against 7.2 to 100.0% for the F2 solve.
  • deepseek-chat proposes single-cell targets on all 20 Lights Out samples when the prompt omits the press rule. With the rule, only 8 of 17 valid samples plan exactly. Our rollout-free checks classify all of these proposals.
  • As the author of the task description, whose output the same checks verify, a language model succeeds instead (item 1).
  • With the same press model, complete search (breadth-first or greedy best-first) also solves Lights Out, so the failures come from the per-cell units or from monotone descent; the F2 solve gives the minimum-weight plan in cubic time.

5. Scope is stated plainly. Without a transition model (cube tasks and scene) GC-FACT raises the averages of GCIQL / HIQL / OTA from 33.0 / 11.5 / 7.6 to 37.3 / 17.5 / 13.9. On the puzzles it raises them from 36.1 / 6.6 / 1.8 to 90.1 / 79.4 / 55.9. The main table reports both groups. On pixel observations of colour-coded objects the same units can be read off the images, and GC-FACT raises the pixel averages of GCIQL / HIQL / OTA from 9.7 / 47.0 / 50.4 to 34.4 / 62.2 / 59.3, mostly on the puzzles. Real-world scenes remain future work: we have no physical robot, and the BridgeData V2 digital twins (ManiSkill3, SimplerEnv) offer only single-primitive tasks so far.

New reference: DeepSeek API documentation (model deepseek-flash, DeepSeek-V4.1-Flash), accessed 16 November 2026.

AuthorsOpenReview ↗

General response (2/2): theory, statistics, positioning, future work

6. Theory, corrected where reviewers found gaps.

  • The hop bound now holds for the states a planner routes through, since its proof only uses that the path lies in a fixed set. On the TTGS vertex sets (four seeds of 4000 vertices), m < K holds for 23 of the 40 goals in every seed. The bound then averages 0.32 on cube-triple and 0.45 on cube-quadruple. On cube-double and scene the vertex sets contain goal states, so the paper no longer uses the bound there.
  • The near-optimality theorem no longer uses transitivity of the success tolerance: the target set now keeps already satisfied factors in their goal class. Stacking and locks are hard preconditions that violate weak coupling for every finite epsilon. A new remark shows that merging each such chain into one unit restores the theorem for the coarser factorisation.
  • The F2 solve is credited as the classical linear-algebra solution of Lights Out (Anderson & Feil, 1998), and our contribution is to read these facts as properties of a basis for a frozen policy.

7. Statistics. The main table keeps its original runs (one to eight evaluations from up to six training seeds), and the setup now says so. Every new experiment pairs the frozen backbone and the planner in one process with identical reset seeds over three or four training seeds. On cube-single, where GC-FACT is the identity, the paired difference is +0.1 points (95% interval [-1.4, +1.6]), and two evaluations of one checkpoint differ by at most 2.4 points, our noise floor.

8. Positioning. Related work now covers HECRL, SMORL, SRICS, MoCoDA, LEAP, PPGS, goal serialisation (SIW), planning sketches and task and motion planning (TAMP). A new table compares them on the unit of a subgoal, its source, its executor and its executability test, with each entry checked against the original paper. The distinguishing requirement of our setting is that a frozen offline policy must execute every subproblem, which the factor cost ratio measures and the data-support check tests before any rollout.

9. Limitations and how we plan to remove them.

limitationroot causeplanned remedy
pixel cube targets help only occasionallya cube pasted from the goal frame keeps the shadows and occlusions of the current frame, a combination the pixel policy never met as a goalrender object targets with an object-centric encoder (slot attention or DLPv2, as in HECRL) that replaces one slot by its goal slot and decodes a consistent image
hand-written stack ordernot in the task description (locks now are, as gates)learn an interaction graph from offline data, as SRICS does
cube-quadruple stays near zerofrozen policies finish a single-cube target in only 11 to 28% of episodesraise local competence by relabelling one-object changes in the offline data as goals
nonlinear combinatorial unitsthe F2 solve needs a linear press algebrasearch a learned latent graph of unit effects, as PPGS does for puzzles
greedy order is suboptimal on scenelock coupling is strongorder targets by measured coupling, which the closest-to-goal rule approximates (+10.4 points for OTA)

10. What has not changed. GC-FACT keeps one configuration for all tasks and backbones and learns no parameters, and no new experiment reverses a result of the main table.

References for this response.

  • TTGS: Opryshko et al., Test-time graph search for goal-conditioned RL, ICML 2026.
  • GC-TTT: Bagatella et al., Test-time offline RL on goal-related experience, ICML 2026.
  • HECRL: Haramati et al., Hierarchical entity-centric RL with factored subgoal diffusion, ICLR 2026.
  • SMORL: Zadaianchuk et al., Self-supervised visual RL with object-centric representations, ICLR 2021.
  • SRICS: Zadaianchuk et al., Self-supervised RL with independently controllable subgoals, CoRL 2021.
  • MoCoDA: Pitis et al., Model-based counterfactual data augmentation, NeurIPS 2022.
  • LEAP: Nasiriany et al., Planning with goal-conditioned policies, NeurIPS 2019.
  • PPGS: Bagatella et al., Planning from pixels in environments with combinatorially hard search spaces, NeurIPS 2021.
  • SIW: Lipovetzky and Geffner, Width and serialization of classical planning problems, ECAI 2012.
  • Sketches: Drexler et al., Expressing and exploiting the common subgoal structure of classical planning domains using sketches, KR 2021, and Learning sketches for decomposing planning problems into subproblems of bounded width, ICAPS 2022.
  • TAMP: Garrett et al., Integrated task and motion planning, Annu. Rev. Control Robot. Auton. Syst. 2021.
  • Anderson and Feil, Turning lights out with linear algebra, Mathematics Magazine 1998.
  • Backbones: GCIQL (Kostrikov et al., ICLR 2022), HIQL (Park et al., NeurIPS 2023), OTA (Ahn et al., NeurIPS 2025).
  • deepseek-chat: DeepSeek-AI, DeepSeek-V3 technical report, arXiv:2412.19437.
  • Slot attention: Locatello et al., NeurIPS 2020. DLPv2: Daniel and Tamar, TMLR 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.