acceptodds
Under review as a conference paper at ICLR 2027

Compiling Task-Event Circuits for Offline Goal-Conditioned Reinforcement Learning

Abstract

Goal-conditioned policies often reach nearby goals yet fail when a task requires composing many distinct changes. We present the Algebraic Task Quotient (ATQ), a compiler that extracts repeated task changes from offline replay. ATQ represents task progress with persistent variables, which we call registers. It learns how recurring task-level events update these registers and compiles the learned rules into a task-event circuit. Each path through the circuit represents a possible sequence of task changes. At test time, a planner selects an event sequence, a policy executes one local request, and feedback updates the circuit state. On all eight OGBench Puzzle tasks, ATQ reaches 95.3–100.0% success with eight independently trained GCIQL policies. Under the same policies and test conditions, Test-Time Graph Search reaches 0.2–44.1% on the 4x4–4x6 layouts, whereas ATQ reaches 95.3–100.0%. ATQ also exceeds the cited published scores on all 4x5 and 4x6 tasks. On the larger puzzles, 98.4–100.0% of its requests apply an event in a register configuration where that event was absent from compiler training data. Episodes composed entirely of these new combinations still succeed 95.3–100.0% of the time. Beyond Puzzle, ATQ raises success in navigation, object manipulation, and soccer, with gains of up to 27.2 percentage points. These results show that replay can reveal event rules that support composition beyond search over recorded states.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, goal-conditioned reinforcement learning, planning, operator learning, compositional generalization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YCceOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates long-horizon planning in Offline Goal-Conditioned Reinforcement Learning. It addresses a specific challenge where agents know how to execute individual local actions but struggle to compose them into long and complex tasks. To tackle this, the authors propose ATQ, or Algebraic Task Quotient, which adds a task-level planning layer on top of existing reinforcement learning policies. Under the offline goal-conditioned setup, the agent learns entirely from pre-collected interaction trajectories and receives both the current state and target goal to output control actions.

ATQ operates through a four-step framework. First, it identifies key task-relevant information to retain from offline trajectories. Second, it learns transition rules describing how state changes occur when specific events happen. Third, it generates an event-level plan at the task layer. Fourth, it executes a step, observes the result, and replans dynamically.

The primary evaluation is conducted on a robotic button-pressing Puzzle task, where pressing a button toggles multiple board lights to reach a specified target pattern. Experiments demonstrate substantial performance gains when comparing GCIQL combined with ATQ against both standard GCIQL and GCIQL augmented with Test-Time Graph Search.

Critical strengths

The primary value of this paper lies in extracting repeatable task transitions from offline interaction trajectories into explicit event rules and combining them with existing goal-conditioned policies via a high-level planner, thereby enhancing performance on long-horizon, complex tasks. By decoupling task-level decision-making from low-level action execution, this architecture offers an effective solution to the challenge where agents possess local skills but struggle with long-horizon tasks. Remarkably, without modifying the underlying policy, the proposed method increases the success rate on Puzzle 4×6-play from 12.5% to 99.4%, demonstrating that well-designed task representations and planning mechanisms can substantially improve existing policies.

The paper also presents compelling validation regarding compositional generalization and methodological effectiveness. The authors evaluate state-event combinations unobserved during compiler fitting and demonstrate that these unseen pairs still sustain high task success rates. This indicates that, under stable event dynamics and suitable model candidates, the system can leverage rules identified from limited data to infer novel operational sequences without relying on complete trajectory coverage. Furthermore, ablation experiments—including event effect shuffling, greedy planning, and feedback interventions—verify the respective importance of correct event semantics, multi-step planning, and closed-loop error correction, providing a thoroughly grounded explanation for the source of performance gains.

Critical weaknesses

The proposed method relies heavily on hand-crafted structural priors, and its capacity for automatic discovery as well as its generalizability require clearer definition. Candidate variables, operator types, and execution interfaces across different environments are all manually pre-designed, while the primary benchmark task, Puzzle, inherently features an XOR structure particularly suited for algebraic solving. Consequently, the current evaluation does not fully isolate the individual contributions of automated task-structure extraction, event-model selection, algebraic solving, and feedback replanning. The authors should consider adding a simplified baseline that directly learns XOR masks for planning under the same front-end and execution mechanisms to clarify the incremental value of the full ATQ framework. Additionally, the theoretical section mostly provides abstract conditions and a unified formalization; as the authors acknowledge that the quotient construction inherits classic residual-language abstraction ideas, its explicit contribution to actual algorithm design and performance warrants further elaboration.

Two points in the experimental design require further clarification. First, TTGS employs a default open-loop setting without replanning, whereas ATQ dynamically replans based on execution feedback. As a result, the performance gap in the main experiments conflates the effects of representation modeling with those of feedback mechanisms, meaning it cannot be attributed solely to the event-rule model itself. Second, the identification of "unseen combinations" is restricted to the dataset used for compiler fitting, whereas the underlying policy utilizes a more complete offline dataset; hence, the current evidence primarily supports compositional generalization at the compiler level. Incorporating a comparison where both methods are allowed to replan, along with a test setup where specific combinations are held out from both the compiler and the low-level policy, would significantly strengthen the attribution of contributions and the credibility of system-level generalization claims.

Regarding applicability, while the paper reports strong results on Puzzle, performance gains on other tasks are inconsistent, with visible degradation observed in certain HumanoidMaze benchmarks. This indicates that the effectiveness of the high-level event model critically depends on whether the abstraction preserves essential information and whether the low-level policy can reliably execute the subgoals generated by the planner. Currently, the explanation for these failures lacks thorough diagnostic analysis. It is recommended to further disentangle the root causes—such as event-model errors, executability misjudgments, improper subgoal construction, or low-level control failures—and evaluate the method on tasks featuring context dependencies, irreversible operations, or sequential constraints. These additions will help clarify whether ATQ's advantages can generalize beyond clean, highly structured puzzle tasks to broader long-horizon decision-making problems.

Questions for authors

See Critical Weaknesses

6 comments

Response to Reviewer YCce

We are deeply grateful for the time and expertise you devoted to such a thorough and constructive evaluation of our manuscript. Your insightful comments guided us in designing a comprehensive set of new experiments that directly address each of your concerns, all of which we will incorporate into the revised manuscript.

ATQ needs a task representation through registers that can distinguish the states relevant to the task, but supplying registers does not supply event effects, execution conditions, or a solution.

W1. "The proposed method relies heavily on hand-crafted structural priors... The authors should consider adding a simplified baseline that directly learns XOR masks for planning under the same front-end and execution mechanisms to clarify the incremental value of the full ATQ framework. Additionally, the theoretical section mostly provides abstract conditions and a unified formalization... its explicit contribution to actual algorithm design and performance warrants further elaboration."

Supplied structure, direct-XOR baseline, and theory. The submitted Theorem 1 gives one-step consistency conditions under which registers preserve event applicability, transitions, and goal completion along executable sequences. Theorem 2 characterizes the coarsest task memory preserving successful event sequences (Section 2.1), yielding Corollary 1's goal-relative residual formulation for Puzzle (Section 2.4). Proposition 2 explains when replay determines predictions at unseen configurations within an operator family (Section 2.3; Appendix A.4). These results justify composition when their conditions hold, rather than guaranteeing arbitrary unseen combinations. New direct-XOR, Boolean-without-named-XOR, and physical-mask controls all recover the original policy-evaluation model on 8/8 Puzzle datasets. Thus, the Puzzle gain comes from learning and composing concrete event rules, not an advantage over direct fitting of the correct operator family.

W2. "First, TTGS employs a default open-loop setting without replanning, whereas ATQ dynamically replans based on execution feedback... Second, the identification of 'unseen combinations' is restricted to the dataset used for compiler fitting, whereas the underlying policy utilizes a more complete offline dataset... Incorporating a comparison where both methods are allowed to replan, along with a test setup where specific combinations are held out from both the compiler and the low-level policy, would significantly strengthen the attribution of contributions and the credibility of system-level generalization claims."

Feedback and generalization. First, in addition to TTGS's native drift-triggered replanning, we forced TTGS to replan every 50 environment steps; it actually replans in at least 99.8% of episodes on each 4×5 and 4×6 dataset, yet the large performance gap remains (0.15–2.55% for TTGS versus 78.55–88.05% for fixed-set ATQ and 97.85–100.00% for replanning ATQ on 4×5 and 4×6). Second, we excluded a two-register combination from both policy and compiler training, including relabeled goals, and retrained both from scratch. ATQ achieves 95.08–100.00% on official goals requiring that combination, compared with 0.00–0.08% for GCIQL and 0.00–0.42% for replanning TTGS.

W3. "...performance gains on other tasks are inconsistent, with visible degradation observed in certain HumanoidMaze benchmarks... It is recommended to further disentangle the root causes—such as event-model errors, executability misjudgments, improper subgoal construction, or low-level control failures—and evaluate the method on tasks featuring context dependencies, irreversible operations, or sequential constraints."

Failure mechanisms and learning beyond supplied registers. First, new HumanoidMaze giant controls separately test policy connection, waypoint distance, target selection, graph support, and intervention timing. Preserving both LAVL policy stages and delaying intervention recovers 24.45 and 29.55 percentage points relative to the submitted method (84.75% and 85.30%), restoring LAVL-level performance without policy retraining. Second, in a new Scene intervention (robot-arm tasks involving buttons, a lockable drawer, a sliding window, and a cube with sequential preconditions), using learned cross-register event conditions improves success by 8.60 and 6.75 percentage points while keeping registers, event effects, and policies fixed.

The three follow-ups (Parts 1–3) provide the matched tables, protocols, and case analyses.

Part 1: Supplied Structure, Direct-XOR Baseline, and Role of Theory

We sincerely appreciate your constructive comments on supplied structure, the direct XOR baseline, and how our theoretical results inform the algorithm.

1. Supplied registers and the direct-XOR baseline

A task representation through registers specifies which aspects of task state are recorded, such as whether a light is on (Section 2.1), but does not specify which lights each switch changes or which switches solve a new goal. As in a digital circuit, simple stored values and elementary logic operations support complex computations through composition over time. Likewise, simple register variables support long-horizon behavior when their changes are learned and composed to reach a goal.

ATQ supplies a family-specific language for variables and updates, using replay to determine concrete event rules and execution targets (Section 2.2; Appendix B.1, Tables 3–4). These structural assumptions beyond receiving a goal observation are central to the method.

Following your suggestion, we fitted a direct XOR baseline using the same register extractor, event identities, execution prototypes, and planning interface. Across all eight Puzzle datasets, it recovered the original executable model used in the submitted policy evaluations. Removing the explicitly named XOR candidate (fitting general Boolean updates) and supplying physical Lights-Out masks also recovered the same model:

Operator constructionResult across the eight Puzzle datasets
ATQ candidate selectionSelects XOR on 8/8 datasets
Direct XOR fitting from replayRecovers the same executable model on 8/8
Explicit XOR candidate removedGeneral Boolean fitting recovers equivalent COPY/NOT updates on 8/8
Physical Lights-Out masksMatches the learned executable model on 8/8

The XOR-removed fits achieve 100% held-out whole-register successor accuracy on the four play datasets and 99.51–99.94% on the noisy datasets; all four constructions have identical accuracy under this evaluation. These accuracies use the held-out 20% of training replay, with events identified from pre-event effector positions. Appendix C.1 reports a separate structural-compiler audit on official validation replay with different event grounding. Executable-model equivalence is checked against the original policy-evaluation banks, verifying equality of all model fields used by the executor rather than treating identical models as independent rollouts.

This clarifies the contribution: on Puzzle, candidate-family selection does not improve performance over directly fitting the correct family. The benefit lies in recovering reusable event laws and composing them with the frozen policy.

2. How the theory informs the method

Theorems 1 and 2 provide a theoretical basis for the compositional view behind the circuit analogy (Section 2.1; Appendices A.1–A.2):

  • One-step consistency (Theorem 1): An exact deterministic task-event circuit factoring source transitions through the register extractor exists if and only if states assigned the same register values (for a fixed goal) agree on one-step goal completion, event applicability, and next register values (Eq. 2). When these local conditions hold, outcome preservation extends by induction to executable event sequences. This motivates fitting and validating operators on one-step transitions, though finite-replay validation does not by itself guarantee that Eq. (2) holds globally.
  • Predictions at unseen configurations (Proposition 2): Section 2.3 explains when observed transitions determine predictions at new configurations within an operator family (Appendix A.4, Proposition 2). When the true effect class is identified and the global XOR structure holds, one exact transition fixes that event's mask (me=r⊕r′m_e = r \oplus r'). Alongside event-type coverage (Appendix C.5), this helps explain the empirical recovery of the full operator bank from eight replay episodes (Figures 2 and 3).
  • Continuation quotient and Puzzle planning (Theorem 2 & Corollary 1): Theorem 2 uses the classical continuation-language construction to characterize the coarsest merging of task configurations preserving which future event sequences reach the goal. Under Puzzle's XOR structure, Corollary 1 (Section 2.4) shows that the goal-relative state is the residual q=r⊕gq = r \oplus g, planning reduces to min⁡∥z∥0\min \|z\|_0 subject to Mz=q(mod2)Mz = q \pmod 2 over F2\mathbb{F}_2 (Eq. 16), and the valid configuration space forms 2ρ+1[ρ<n]2^\rho + \mathbf{1}[\rho < n] equivalence classes—one per reachable residual in im⁡M\operatorname{im} M and, when ρ<n\rho < n, one merged class for all unreachable residuals (Eq. 17; Table 7).

Part 2: Replanning in Both ATQ and TTGS, and Shared-Exclusion Controls

Thank you for asking us to separate feedback effects and test generalization beyond both learners' training data.

1. Replanning in both ATQ and TTGS

The submitted event-substitution test assesses ATQ's recovery (Section 3.2, Figure 2; Appendix C.7). We now compare TTGS without replanning, with its released drift-triggered replanning, and with replanning every 50 environment steps. ATQ either replans from observed registers or keeps its initial event set fixed. The latter retains local event detection and retries, removing global plan revision rather than all feedback.

Puzzle datasetTTGS no replanningTTGS native replanningTTGS every 50 stepsATQ fixed event setATQ with replanning
3×3 play98.25 ± 0.5998.20 ± 0.6198.05 ± 0.52100.00 ± 0.00100.00 ± 0.00
3×3 noisy98.05 ± 0.5098.05 ± 0.5098.15 ± 0.52100.00 ± 0.00100.00 ± 0.00
4×4 play25.65 ± 1.6125.90 ± 1.5716.85 ± 0.3578.85 ± 1.9198.45 ± 0.50
4×4 noisy44.15 ± 1.7544.15 ± 1.7548.25 ± 1.6094.55 ± 1.6195.20 ± 1.47
4×5 play1.80 ± 0.581.95 ± 0.610.65 ± 0.1878.55 ± 1.84100.00 ± 0.00
4×5 noisy2.30 ± 0.532.30 ± 0.532.55 ± 0.4388.05 ± 3.2097.85 ± 0.57
4×6 play0.60 ± 0.250.70 ± 0.150.15 ± 0.1180.00 ± 3.1499.60 ± 0.15
4×6 noisy0.25 ± 0.150.20 ± 0.150.20 ± 0.1585.00 ± 1.3798.85 ± 0.51

All entries are new success percentages (mean ± SEM over eight frozen GCIQL policy seeds, five official tasks, 50 episodes per task). Checkpoints and reset seeds are shared; TTGS variants use the same 4,000-node graphs.

Native replanning triggers in 36.6% and 67.6% of episodes on 4×5-play and 4×6-play, and never on the noisy datasets. The periodic control replans in at least 99.8% of episodes on each 4×5 and 4×6 dataset. It improves TTGS on 4×4-noisy, but TTGS remains below 3% on 4×5 and 4×6 under both settings. Fixed-set ATQ substantially exceeds both replanning TTGS variants, and ATQ's own replanning adds 9.80–21.45 percentage points on these datasets. Thus, feedback contributes, but the tested replanning mechanisms alone do not close the gap.

2. Combinations excluded from both policy and compiler training

The submitted experiment measures compiler-level novelty (Section 3.2; Appendix C.3, Table 11). We removed every training transition whose source or successor had the top-left and bottom-right lights both on, and excluded this pattern from all relabeled training goals. We refitted the compiler, including its register extractor, event rules, and execution prototypes, and trained eight new GCIQL policies per dataset on the retained replay. The compiler recovered the 20-bit and 24-bit registers within the supplied binary candidate family. TTGS used the same filtered replay with native replanning.

The following table reports the three official tasks per board whose goals contain the excluded pattern, selected by goal content rather than outcomes. Their initial states do not contain the pattern.

Puzzle datasetGCIQLTTGS with replanningATQ (refitted extractor & rules)
4×5 play0.08 ± 0.080.17 ± 0.1798.00 ± 1.08
4×5 noisy0.00 ± 0.000.42 ± 0.2595.08 ± 1.11
4×6 play0.00 ± 0.000.00 ± 0.00100.00 ± 0.00
4×6 noisy0.00 ± 0.000.00 ± 0.0097.00 ± 1.30

For the same trained models, the averages over all five official tasks, including the two whose goals do not contain the pattern, are:

Puzzle datasetGCIQLTTGS with replanningATQ (refitted extractor & rules)
4×5 play12.95 ± 0.791.90 ± 0.3398.65 ± 0.63
4×5 noisy14.00 ± 1.941.90 ± 0.4096.75 ± 0.71
4×6 play12.85 ± 0.871.30 ± 0.44100.00 ± 0.00
4×6 noisy15.05 ± 1.350.40 ± 0.1998.00 ± 0.77

3. Equal replay access without pattern exclusion

All methods use the original 60% compiler-fit episodes, without pattern exclusion, and are evaluated on all five official tasks. TTGS uses native replanning.

Puzzle datasetGCIQLTTGS with replanningATQ
4×5 play10.80 ± 0.902.30 ± 1.0898.00 ± 0.83
4×5 noisy18.85 ± 0.232.85 ± 0.5698.55 ± 0.45
4×6 play10.90 ± 0.940.85 ± 0.2199.20 ± 0.23
4×6 noisy17.10 ± 0.690.55 ± 0.4497.45 ± 0.69

The two data controls use separate cohorts of eight newly trained GCIQL policies per dataset. Their tables report success percentages (mean ± SEM across policy seeds, 50 episodes per task). Shared exclusion tests a two-register combination absent from both policy and compiler training, complementing the full register–event pair audit in Table 11.

Reviewer_FsatOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Algebraic Task Quotient (ATQ), a method for recovering reusable event rules from offline RL data and composing it to solve goals through a goal-conditioned policy. They introduce a representation with terminology from the sequential digital circuits domain: registers storing progress of a task and circuits being a collection of rules describing how events change the registers. A planner uses the circuit to find a sequence of events to reach the goal. The main contribution of this paper is this method for learning and using the reusable event data from offline replay, as well as providing a rigorous analysis and success on diverse set of tasks, including puzzle, navigation and others from OGBench.

Critical strengths

S1. The paper provides rigorous experiments across diverse tasks, demonstrating the generality of their approach.

S2. The paper is well written, providing a helpful formalization in section 2 and algorithmic description of their approach.

Critical weaknesses

W1. The method seems to rely on a predefined candidate library of representations and rules. Although it learns rules from the offline data, it is unclear how well it would work in environments where this library cannot be specified.

Questions for authors
  1. How well would ATQ generalize to environments where the appropriate task representations cannot be specified?
3 comments

Response to Reviewer Fsat

We would like to express our sincere gratitude for your positive evaluation of our work and your insightful question regarding generalization. Your insightful comment provides a valuable opportunity to clearly distinguish between what the registers must capture and what ATQ actively learns from replay, and motivated two new experiments that we will incorporate into the revised manuscript.

Q1. How well would ATQ generalize to environments where the appropriate task representations cannot be specified?

1. Scope of the candidate library and when it must be extended

ATQ does not require the concrete registers or event rules to be specified in advance. Rather, it searches within a supplied candidate family (e.g., complementary binary coordinates in Puzzle) and uses replay to select the concrete registers, event effects, and execution targets (Section 2.2).

  • When the candidate family is sufficient: Theorem 1 establishes that exact composition holds whenever the extracted registers preserve goal completion, event applicability, and one-step task transitions, while Theorem 2 characterizes the coarsest state merging that preserves successful continuations (Section 2.1). Within an expressive operator family, Proposition 2 shows when replay determines event laws at unseen register configurations (Section 2.3; Appendix A.4).
  • When the candidate family cannot capture a necessary task distinction: Fixed candidate extractors that merge states with different event applicability or outcomes will fail the Theorem 1 consistency checks on held-out development transitions (Section 2.2). In such environments, the candidate library must be extended with richer state features or paired with learned latent representations.

2. Specifying registers does not specify the event rules (New Scene experiment)

Even when appropriate registers are specified, knowing the current register values (e.g., whether a drawer is open) does not reveal which events change them or under what conditions an event is applicable (e.g., that the drawer must be unlocked before opening, and open before inserting a cube).

To isolate the value of learning these rules over fixed registers, we conducted a new intervention on Scene (robot-arm manipulation of buttons, a drawer, a window, and a cube). Holding the register extractor, event effects, replay prototypes, and eight frozen HIQL policies fixed, we removed only the learned cross-register conditions (requiring locks to be released before moving the drawer/window and the drawer to be open before inserting the cube):

Scene datasetOriginal HIQL policyATQ without cross-register conditionsATQ with learned conditions
Play37.80 ± 2.5947.90 ± 2.2456.50 ± 2.23
Noisy26.10 ± 1.3931.30 ± 0.5438.05 ± 1.14

Values are success percentages (mean ± SEM over 8 seeds, 5 official tasks, 50 episodes per task). Learning the cross-register applicability conditions improves success by 8.60 and 6.75 percentage points across all eight seeds in both datasets, demonstrating that supplying suitable registers still leaves a critical rule-learning problem for ATQ to solve.

3. Generalizing to register patterns unseen by both compiler and policy (New Puzzle experiment)

To further test how ATQ generalizes when only the broad candidate family is supplied and specific state combinations are entirely missing from experience, we removed all transitions and relabeled goals where the top-left and bottom-right lights are simultaneously on from both compiler and GCIQL policy training on the larger Puzzle boards. We then refitted the register extractor and event rules from scratch and trained eight new GCIQL policies per dataset:

Puzzle datasetGCIQLTTGS with replanningATQ (refitted extractor & rules)
4×5 play0.08 ± 0.080.17 ± 0.1798.00 ± 1.08
4×5 noisy0.00 ± 0.000.42 ± 0.2595.08 ± 1.11
4×6 play0.00 ± 0.000.00 ± 0.00100.00 ± 0.00
4×6 noisy0.00 ± 0.000.00 ± 0.0097.00 ± 1.30

On the three official evaluation tasks per board whose goals require this excluded pattern, ATQ successfully recovers the 20-bit and 24-bit registers from the remaining replay and achieves 95.08–100.00% success. We will incorporate these tables and analyses into the revised manuscript.

Dear Reviewer Fsat,

Please excuse this brief follow-up as the discussion period draws to a close. We are grateful for your careful review and have posted a detailed response to your question, together with new generalization experiments on Scene and Puzzle. We would greatly appreciate your feedback on whether our response addresses your concern, and we would be happy to provide any further clarification.

Respectfully,

The Authors

Thank you for answering my question and providing these additional experiments. I will maintain my positive score.

Reviewer_9rxDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Algebraic Task Quotient (ATQ). In summary, ATQ does three things: a compiler emits a task-event circuit; a planner searches this circuit; and a linker converts the next event into a local goal for the policy. Figure 1 gives the same loop: compiler → event circuit → planner → linker → policy. The authors do many experiments on OGBench. Within different tasks, the Puzzle experiments are the most complete and strong.

Besides, the paper shows two theorems. They first borrow the terms register and circuit from sequential digital circuits. Then, they give Theorem 1 which says when the circuit may ignore the robot arm and keep only the task state. Theorem 2 says how far those patterns can be merged.

Overall, this is a work with theorems, new algorithms, and well-designed experiments. However, the writing can be polished. For example, more insights can be added. Meanwhile, an example can be used to better explain the ideas. The strongest experiment on Puzzle may require other ablations.

Critical strengths

On Puzzle, the paper tests operator composition outside compiler training data. On the 4×4–4×6 layouts, 98.4–100.0% of requests apply an event in a register configuration where that event was absent from compiler training data. Episodes composed entirely of these new combinations still succeed 95.3–100.0% of the time (Table 11). A replay graph with more than 15,000 states connects only 3.5–5.0% of sampled 4×4 start–goal pairs, while the compiled circuit plans all sampled pairs. These results support the claim that ATQ can apply a learned event rule in a task configuration that replay never paired with that event.

Critical weaknesses
  1. OGBench defines Puzzle as Lights Out: pressing a button toggles that button and its neighbors. ATQ includes XOR in the candidate library, selects XOR on all eight datasets, and plans by solving Eq. (16), a minimum-Hamming-weight linear system. No ablation removes XOR from the candidate library, so the experiments do not separate operator selection by replay from a menu that already contains the published effect class.

  2. I think that the experimental analysis could require more insights. For example, in Table 2, HumanoidMaze giant navigate and giant stitch fall by 20.8% and 26.1%. The paper explains these declines as a mismatch between planned events and the executor’s reliable local goals, without separating operator error, guards, linker targets, and replanning. Also, the gains of up to 27.2% outside Puzzle are not accompanied by the same intervention tests.

Questions for authors
  1. In the sentence “ATQ can apply a learned event rule in a task configuration that replay never paired with that event,” does “event rule” denote the selected operator family, or the per-event guard and operator? XOR is selected on all eight Puzzle datasets, while each accepted event has its own mask and operator. Is the claim that this per-event operator is applied at a register configuration absent from compiler training data?

  2. Could you add more comparisons on Puzzle? For example, remove XOR from the candidate library. Or replace the learned masks with the published Lights Out rule, in which each switch toggles itself and its neighbors, and then solve Eq. (16).

7 comments

Response to Reviewer 9rxD

We are deeply grateful for the time and care you devoted to reviewing our manuscript, as well as for your insightful and constructive suggestions. Your thoughtful questions directly inspired a new set of control experiments that allow us to much more clearly articulate the learned event rules and their broader role beyond Puzzle, all of which we will incorporate into the revised manuscript.

Q1. In the sentence “ATQ can apply a learned event rule in a task configuration that replay never paired with that event,” does “event rule” denote the selected operator family, or the per-event guard and operator? XOR is selected on all eight Puzzle datasets, while each accepted event has its own mask and operator. Is the claim that this per-event operator is applied at a register configuration absent from compiler training data?

An event rule denotes the per-event guard and operator (i.e., the specific mask for each switch), not the operator-family label “XOR.” Confirming your reading, our composition claim is that this learned per-event operator is applied at a register configuration that was never paired with that specific event in compiler training data. Registers provide the task-state representation. Knowing their values does not already specify how each event changes them or which conditions permit those changes.

The submitted theory formalizes this distinction. Theorem 1 gives conditions under which registers preserve goal completion, event applicability, and task-level transition results. Theorem 2 characterizes the coarsest grouping of states that preserves successful future event sequences (Section 2.1). These are conditional guarantees about event semantics, and the submitted Puzzle experiments provide empirical evidence of composition beyond recorded register–event pairs (Section 3.2; Appendix C.3).

Q2 & W1. Could you add more comparisons on Puzzle? For example, remove XOR from the candidate library. Or replace the learned masks with the published Lights Out rule, in which each switch toggles itself and its neighbors, and then solve Eq. (16).

We performed both requested controls on all eight Puzzle datasets. Removing the named XOR candidate recovered equivalent COPY/NOT rules through general Boolean fitting, and replacing the learned masks with the physical Lights-Out masks also matched the learned executable models on all 8/8 datasets. Direct XOR fitting gave the exact same result. All controls retain the register extractor and event identities, and Boolean rules are converted to the same algebraic planning form after equivalence verification. These are exact comparisons against the original banks used in the submitted policy evaluations, not additional policy rollouts. They show recovery of concrete per-event rules from replay (including without a named XOR candidate) rather than an advantage over a learner already given the correct operator family.

W2. I think that the experimental analysis could require more insights. For example, in Table 2, HumanoidMaze giant navigate and giant stitch fall by 20.8% and 26.1%. The paper explains these declines as a mismatch between planned events and the executor’s reliable local goals, without separating operator error, guards, linker targets, and replanning. Also, the gains of up to 27.2% outside Puzzle are not accompanied by the same intervention tests.

New diagnostic experiments directly address both points beyond Puzzle:

  • Intervention test outside Puzzle (Scene): In Scene (robot-arm manipulation involving buttons, a lockable drawer, a sliding window, and a cube), holding the registers, event effects, and frozen HIQL policies fixed while including the learned cross-register guard conditions improves success by 8.60 and 6.75 percentage points on Play and Noisy over removing those conditions.
  • Disentangling the HumanoidMaze giant declines: We conducted controlled ablations separating policy-stage interface, waypoint lookahead, state-matched linker targets, replay-edge support, and intervention timing. Preserving both stages of the existing LAVL policy and delaying ATQ interventions recover 24.45 and 29.55 percentage points on Giant Navigate and Giant Stitch over the submitted interface (84.75% and 85.30%), returning performance to the original LAVL policy's level without retraining.

The following replies provide the concrete illustrative example, controlled comparisons, and detailed failure analysis.

Additional Response 1 (Meaning of an event rule and the role of registers)

Thank you for raising this important question regarding the precise meaning of an event rule.

An event rule is the guard and update for a particular event. In Puzzle, guards are globally enabled, and the learned quantity is each event's mask (Section 2.4, Equation (15)). The composition claim is that this event-specific operator is applied at a register configuration never paired with that event in compiler training (Appendix C.3, Table 11).

By task representation through registers, we mean which task-state information is recorded in the registers. An event rule specifies how a particular event changes those values and when it may apply (Section 2.1). As in a digital circuit, simple stored values and elementary updates can support much more complex behavior through their composition over time. Supplying the registers is therefore different from supplying the event rules or a solution sequence.

For illustration, suppose a three-bit register changes from 000 to 101 after an event. An XOR candidate gives that event the mask 101, specifying which bits it flips. It then predicts that applying the same event at 011 produces 110. Replay determines the mask and its association with an execution target, and held-out transitions test whether the rule is stable across contexts. The planner combines such rules to reach a new goal.

Thus, the unseen unit is the complete source register vector paired with the event identifier. We hope the example makes this definition more concrete.

Theorem 1 states when the chosen registers preserve event applicability and the task-level results of executable event sequences. Theorem 2 characterizes how states can be merged while preserving which future event sequences succeed (Section 2.1; proofs in Appendices A.1–A.2). These results require the relevant conditions to hold; the submitted Puzzle experiments demonstrate successful composition of learned rules beyond the register–event pairs recorded in compiler training (Section 3.2).

Additional Response 2 (Removing XOR and using physical Lights-Out masks)

We are grateful for your constructive suggestion to include these two control studies. We performed both on all eight Puzzle datasets. Removing the explicitly named XOR candidate recovered equivalent Boolean updates, and replacing the learned masks with the physical Lights-Out masks also gave the same executable model. To isolate operator learning, the controls share the register extractor, event identities, execution prototypes, and planning interface.

Operator constructionResult across the eight Puzzle datasets
ATQ candidate selectionSelects XOR on 8/8 datasets
Direct XOR fitting from replayRecovers the same executable model on 8/8
Explicit XOR candidate removedGeneral Boolean fitting recovers equivalent COPY/NOT updates on 8/8
Physical Lights-Out masksMatches the learned executable model on 8/8

The XOR-removed fits achieve 100% held-out whole-register successor accuracy on the four play datasets and 99.51–99.94% on the noisy datasets. All four constructions have identical accuracy under this evaluation. These accuracies use the held-out 20% of training replay, with events identified from pre-event effector positions. Appendix C.1 reports a separate structural-compiler audit on official validation replay with different event grounding. Executable-model equivalence is checked against the original banks used in the submitted policy evaluations.

After verifying that the recovered Boolean functions are equivalent to the XOR updates, we convert them to the same algebraic planning form. Thus, the control removes the explicitly named XOR candidate while retaining binary registers and testing recovery of the event law.

Lights-Out is the switch puzzle in which pressing a switch toggles a specified set of lights. A mask records which lights that switch changes. For the physical-mask control, we construct these effects from the environment's Lights-Out rule and align them with the same execution requests. All three controls match the original policy-evaluation banks in every field used by the executor, including effects, support, and execution prototypes. These are exact executable-model comparisons, not additional independent policy rollouts.

The compiler can therefore recover the correct concrete event rules from replay, including through a Boolean parameterization without a named XOR candidate. On these Puzzle tasks, direct XOR fitting already recovers the same model, so candidate-family selection is not the source of an additional performance gain. The contribution demonstrated here is learning and composing reusable event rules, rather than outperforming a specialized learner supplied with the correct family.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.