acceptodds
Under review as a conference paper at ICLR 2027

Symbol and Rule Discovery in Continuous Control Through Progressive Inductive Abductive Learning

Abstract

We study the discovery of symbols and rules from continuous-control trajectories, preserving characteristics of motion dynamics while supporting discrete reasoning. Jointly learning continuous factors, value boundaries, and temporal rules creates a large and coupled optimization problem. We address this problem by DynaAbd, a multi-stage progressive inductive abductive learning (inductive ABL) approach. In the first two stages, contrastive spectral initialization and one-step dynamics modeling organize continuous motion into cyclic and dependent factors, capturing initial features in motion dynamics. In the final stage, we introduce an inductive ABL strategy for fine-grained dynamics modeling. Logical induction and abduction refine the symbol discovery and build the corresponding rules based on multi-step logical consistency. Alternating gradient optimization fine-tunes the symbol value boundaries. Our method is experimentally evaluated in polar motion systems and MuJoCo locomotion tasks. The results show that DynaAbd is able to discover latent driving and dependent symbols hidden inside behavior dynamics, meanwhile, discovering the symbolic rules generalized across environments to accelerate policy learning. Our code and data will be open-sourced upon formal release.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
neuro-symbolic learning, continuous control, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_nDgEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the representation learning problem of discovering latent symbolic factors/rules from cyclic motion trajectories. The proposed approach, DynaAbd, is a two-stage approach that first provides initial estimates of cyclic / dependent coordinates, and second models one-step dynamics to refine the coordinates. The approach is evaluated in a simulated polar motion setting and Mujoco locomotion tasks. The main results are that DynaAbd can recover the underlying motion structural factors, and the learned symbolic representations can accelerate downstream policy learning.

Critical strengths

The paper is generally clear and provides enough detail to fully specify the method. The empirical improvement of DynaAbd over baseline representation learning methods and ablations is substantial (Table 1).

Critical weaknesses

First, while the paper argues why the problem is hard, it insufficeintly argues why this problem is important / how the developed approach could be broadly relevant or impactful beyond the specific studied problem. The method is specific to learning symbolic descriptions of cyclic motion trajectories, as the identified structural factors (Driver, Dependent), are specific to cyclic motion. Additionally, the empirical benefits for downstream learning over a generic autoencoder are relatively modest (Table 3), producing improvements with overlapping confidence intervals with the AE baseline on all tasks. While this is a representation learning problem, I do not think it is a broad enough problem to be interesting to the ICLR community.

Second, I am unsure why the method needs to use positive/negative trajectory quality labels (as specified in the problem formulation). It is also unclear whether any of the baselines receive the benefit of these labels.

Questions for authors

Which of the baselines could use the positive/negative trajectory quality labels?

5 comments

We would like to thank the reviewer for the detailed comments and suggestions. Thank you for your recognition of the method description and the factor-recovery improvements over the baselines and ablations.

W1.1
Reviewer concern: First, while the paper argues why the problem is hard, it insufficiently argues why this problem is important / how the developed approach could be broadly relevant or impactful beyond the specific studied problem.

  • Why this problem matters. Explicit task structure has been useful for reward shaping and task decomposition in Reward Machines and SPECTRL, and for symbolic planning (Toro Icarte et al., 2022; Jothimurugan et al., 2019; Konidaris et al., 2018). This motivates discovering interpretable structure from continuous trajectories that can serve as guidance for policy learning. We have added this connection to the Introduction (lines 33–37). Applying the discovered structures to higher-level reasoning could build on these methods and would benefit from further research.

W1.2
Reviewer concern: The method is specific to learning symbolic descriptions of cyclic motion trajectories, as the identified structural factors (Driver, Dependent), are specific to cyclic motion.

  • Definitions and initial focus. In response to this concern, we have extended the Driver–Dependent formulation to nonperiodic motion: Driver locates the movement stage, and Dependent describes the response at that stage. The Introduction (lines 51–54) connects these roles to dynamical movement primitives and phase-conditioned control in DeepMimic (Ijspeert et al., 2013; Peng et al., 2018). We initially focused on cyclic motion because it provides a well-defined phase for controlled evaluation and a compact rule that can be reused across cycles (Section 3, lines 124–126; Section 4, lines 146–148; Section 6, lines 478–483).

  • Nonperiodic evidence. To test these roles beyond cyclic motion, we added InvertedPendulum with reversible stage transitions and increasing, decreasing, or steady responses. The table below gives its downstream results at 100,000 interactions. Area under the learning curve (AUC) is the integral of native evaluation return divided by the training budget, measuring average performance during learning.

InvertedPendulum under reference-state initialization (RSI; Peng et al., 2018) at 100,000 interactions (manuscript Table 4, lines 452–454). Mean ± sample standard deviation over ten paired seeds, averaging three target variants within each seed.

MetricSACSAC + Driver + Shuffled DependentSAC + Driver + Dependent
Return973.70 ± 83.181000.00 ± 0.001000.00 ± 0.00
AUC575.34 ± 69.61611.75 ± 82.10659.47 ± 44.53

The learned Driver–Dependent guidance improves mean AUC from 575.34 to 659.47. The qualitative example in Figure 5 gives a physical interpretation: P0–P2 mainly correspond to rightward cart translation with leftward pole rotation, and P3–P4 to the opposite combination. Together, these experiments provide evidence that the same stage–response roles can extend to nonperiodic motion with the adaptations described in Appendix C.3 (lines 1204–1250); the physical interpretation appears in Section 5.3 (lines 467–470), with its visualization protocol in Appendix C.5 (lines 1298–1312).

W1.3
Reviewer concern: Additionally, the empirical benefits for downstream learning over a generic autoencoder are relatively modest (Table 3), producing improvements with overlapping confidence intervals with the AE baseline on all tasks.

  • Original uncertainty. We agree that the original autoencoder results gave limited evidence of a consistent downstream benefit. Reconstruction training may change information accessibility or discard information, while reconstruction importance can differ from control relevance. Suppressing less relevant information could help early learning, and losing useful control information could have the opposite effect. We view this as a possible explanation for the task-dependent results.

  • Revised evidence. We have revised Section 5.2 (lines 421–431) to preserve full observations and use the discovered factors and rules directly as intrinsic reward under changed physical parameters. Native-reward soft actor-critic (SAC), SAC + Driver + Shuffled Dependent, and SAC + Driver + Dependent share policy settings, ten paired seeds, and reference-state initialization (RSI), which starts episodes from reference-trajectory states. Shuffled Dependent retains Driver dynamics and Dependent magnitudes while reassigning phase-dependent Dependent directions. All tasks use 100,000 interactions. For convenience, we include the complete return and time-averaged AUC results below.

Downstream learning under reference-state initialization (RSI) at 100,000 interactions (manuscript Table 4, lines 441–454). Mean ± sample standard deviation over ten paired seeds, after averaging target environments within each seed. Return and AUC are in native-return units.

TaskMetricSACSAC + Driver + Shuffled DependentSAC + Driver + Dependent
HopperReturn1804.42 ± 263.272113.02 ± 263.582178.75 ± 315.51
AUC892.62 ± 110.511134.64 ± 112.041213.43 ± 95.38
SwimmerReturn242.51 ± 41.47271.42 ± 48.98311.08 ± 17.23
AUC114.76 ± 23.48173.73 ± 30.17198.80 ± 16.56
Walker2dReturn1177.15 ± 178.961619.83 ± 243.731929.78 ± 242.42
AUC675.07 ± 33.03903.11 ± 70.74975.07 ± 63.39
HalfCheetahReturn437.76 ± 184.89452.95 ± 188.12513.92 ± 162.73
AUC241.53 ± 43.43232.50 ± 43.61257.63 ± 59.70
InvertedPendulumReturn973.70 ± 83.181000.00 ± 0.001000.00 ± 0.00
AUC575.34 ± 69.61611.75 ± 82.10659.47 ± 44.53

SAC + Driver + Dependent improves mean final return and AUC over SAC by 26.61% and 34.98%, averaging relative gains equally across the five tasks. Mean AUC also exceeds the Shuffled Dependent control in each task. These additional ten-seed experiments provide evidence under the revised policy-learning protocol, with cross-seed variation reported in the table (Section 5.2, lines 421–431 and 456–459; Appendix C.2, lines 1144–1184; evaluation protocol, lines 1253–1266).

W2.1
Reviewer concern: Second, I am unsure why the method needs to use positive/negative trajectory quality labels (as specified in the problem formulation).

  • Purpose of quality labels. Dependent is intended to capture the response associated with high-quality motion at a given Driver stage. Predictable transitions also occur in low-quality trajectories, so dynamics prediction alone can favor an easy-to-predict factor that misses this distinction. Quality labels help direct Dependent discovery toward the relevant response. We have added this motivation to the Introduction (lines 54–57); Related Work discusses how insufficient negative-example coverage can leave concepts indistinguishable under task supervision and permit reasoning shortcuts (lines 100–102; Marconato et al., 2023).

  • How they are used. Dependent discovery uses quality information in its contrastive initialization, positive/negative classification loss, and negative-class separation during continuous fitting. These objectives distinguish responses associated with different trajectory qualities while the factor values, boundaries, and rules are learned (Sections 4.1–4.3, lines 160–161, 182–183, 211–215, and 262–263).

  • Why this ablation is informative. We remove the objectives that distinguish positive and negative examples during Dependent discovery and withhold quality labels from its initialization, while fixing Driver to the full model at each stage. The remaining dynamical and logical supervision can still fit motion regularities. This isolates whether quality information helps select the intended response factor from other dynamically explainable factors. We evaluate the recovered Dependent by its absolute Pearson correlation with true radial increments on the same held-out data (Section 5.1, lines 311–314 and 373–375; Appendix B.2, lines 1036–1038).

Dependent quality-label ablation (manuscript Table 1, lines 348–368). Mean ± sample standard deviation over ten seeds; Driver is fixed to the full model at each stage. The four-task mean is computed within seed first.

TaskDynaAbd Dependent correlationDynaAbd− Dependent correlation
Polar30.97 ± 0.000.05 ± 0.03
Polar40.92 ± 0.050.29 ± 0.12
Polar50.96 ± 0.010.10 ± 0.08
Polar100.97 ± 0.010.09 ± 0.07
Mean0.95 ± 0.010.13 ± 0.05

Without Dependent quality supervision, mean correlation falls from 0.95 ± 0.01 to 0.13 ± 0.05. With the same Driver available, the large loss in factor recovery supports the role of positive/negative information in reducing dynamical shortcuts: representations that explain predictable motion but fail to capture the intended phase-dependent response (Table 1, lines 348–368; Appendix B.2, lines 1036–1038).

W2.2
Reviewer concern: It is also unclear whether any of the baselines receive the benefit of these labels.

  • Matched supervision. We have added SINDy-AE+ and mechanism sparsity+ with the same positive/negative trajectory labels, together with supervised autoencoding (SAE) and supervised contrastive learning (SupCon), which use the labels directly. All methods share the observations and data partitions. We discuss the closely related baseline question in Q1 below, where we include the full comparison and the original versions for reference (Section 5.1, lines 316–323 and 370–375; Appendix B.2, lines 999–1038).
Reviewer_g9qzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to extract symbolic, interpretable rules from continuous robot trajectories by exploiting cyclic patterns in locomotion data. Specifically, they propose a three stage framework: first, using spectral analysis to learn a linear dynamics model (EDMD) on the trajectory, specifically the "Driver", trackign position in the motion cycle, and the "Dependent", tracking phase; second, refining both parameters for predictability; finally, use Prolog to learn rules based on discretizing the Driver and Dependent, then picking the best out of the top three rules which explain the trajectory.

The method is tested on both synthetic tasks with 3,4,5 phases, with high recovery correlation relative to baselines like VQ-VAE. They also test this on Mujoco continuous control (locomotion) tasks, which improved SAC learning over the first 30k steps slightly.

Critical strengths
  1. Clearly motivated, strong intuition method which brings spectral analysis to creating symbolic representations of behaviors. If we could accurately and repeatedly extract symbolic representations of robot trajectories without compromising representation power, it would be a interesting result for the field.
  2. Strong ablations demonstrating performance improvements over reasonable baselines like VQ-VAE.
Critical weaknesses
  1. Assumptions of the Mujoco tasks are fairly strong, i.e. a recurring cycle gait which doesn't change over time. In general the method is quite hard-coded, e.g. five discrete intervals; there are only 30 possible direction programs.
  2. Mujoco results are inconclusive. In particular, the sample SD is largely overlapping between AE and AE+rule, and the Walker2d Raw (i.e. no autoencoder) setting has the highest mean. It would greatly strengthen the paper if the main positive results included these tasks and not just the synthetic datasets.
Questions for authors
  1. Could you expand in more detail on how you'd extend the method to multiple gaits and/or time-varying (drifting) gaits? How would one extend this to general locomotion, whole-body control, or potentially manipulation?
6 comments

We would like to thank the reviewer for the detailed comments and suggestions. We appreciate your positive assessment of the method’s motivation and the ablation evidence for its improvement over the baselines.

W1.1
Reviewer concern: Assumptions of the Mujoco tasks are fairly strong, i.e. a recurring cycle gait which doesn't change over time.

  • Broader motion setting. We consider two related extensions here: nonperiodic motion and changes in motion patterns. We have added nonperiodic InvertedPendulum, using ordered Driver stages that allow reversals and increasing, decreasing, or steady Dependent responses. Figure 5 relates its stages to cart translation and pole rotation (Section 3, lines 124–126; Section 5.3, lines 467–470; Appendix C.3, lines 1204–1250).

  • Changing motion. We have also expanded Section 6 (lines 485 and 526–532) to discuss how time- or context-dependent factor maps and separate rule groups might handle changing or distinct motion modes. We outline these possibilities in Q1.1 below.

W1.2
Reviewer concern: In general the method is quite hard-coded, e.g. five discrete intervals; there are only 30 possible direction programs.

  • Expanded phase count. We have added Polar10 to test a larger setting directly. It uses ten phases and eleven clauses, expanding the candidate space from 30 to 1,022 programs. The first table reports absolute Pearson correlations of learned and true increments: signed circular phase increments for Driver and radial increments for Dependent. The second reports joint program-and-label search under a common ten-second budget.

Final factor recovery (manuscript Table 1, lines 342–368). Mean ± sample standard deviation over ten seeds.

TaskDriver correlationDependent correlation
Polar30.88 ± 0.010.97 ± 0.00
Polar41.00 ± 0.000.92 ± 0.05
Polar50.99 ± 0.000.96 ± 0.01
Polar100.95 ± 0.000.97 ± 0.01

Joint program-and-label Prolog search (manuscript Table 3, lines 433–439). Ten seconds per task; search-time mean ± sample standard deviation across ten seed means.

TaskPhasesCandidate HClausesPure search time (s)
Polar353060.25 ± 0.03
Polar453060.26 ± 0.03
Polar553060.60 ± 0.49
Polar10101,022111.88 ± 0.06

On Polar10, the final Driver/Dependent correlations are 0.95/0.97, and mean pure search time is 1.88 seconds. The added Polar10 experiment tests factor recovery and inference cost at a larger phase count (Table 1, lines 342–368; Section 5.1, lines 415–418; Table 3, lines 433–439; Appendix B.3, lines 1087–1092).

W2
Reviewer concern: Mujoco results are inconclusive. In particular, the sample SD is largely overlapping between AE and AE+rule, and the Walker2d Raw (i.e. no autoencoder) setting has the highest mean. It would greatly strengthen the paper if the main positive results included these tasks and not just the synthetic datasets.

  • Original uncertainty. We agree that the original autoencoder results gave limited evidence of a consistent benefit. Reconstruction training can change the accessibility of information or lose some of it, and information important for reconstruction may differ from information needed for control. This offers a possible explanation for the task-dependent results, while the rule loss constrains only selected motion attributes.

  • Revised evaluation. We have revised Section 5.2 (lines 421–431) to retain full observations and use the discovered factors and rules as intrinsic reward under changed physical parameters. Native-reward soft actor-critic (SAC) and the two guided conditions share policy settings, ten paired seeds, and reference-state initialization (RSI; Peng et al., 2018), which starts episodes from reference-trajectory states. Shuffled Dependent retains Driver dynamics and Dependent magnitudes while reassigning phase-dependent Dependent directions. The table reports final return and time-averaged area under the learning curve (AUC), the integral of native evaluation return divided by the 100,000-interaction budget.

Downstream learning under reference-state initialization (RSI) at 100,000 interactions (manuscript Table 4, lines 441–454). Mean ± sample standard deviation over ten paired seeds, after averaging target environments within each seed. Return and AUC are in native-return units.

TaskMetricSACSAC + Driver + Shuffled DependentSAC + Driver + Dependent
HopperReturn1804.42 ± 263.272113.02 ± 263.582178.75 ± 315.51
AUC892.62 ± 110.511134.64 ± 112.041213.43 ± 95.38
SwimmerReturn242.51 ± 41.47271.42 ± 48.98311.08 ± 17.23
AUC114.76 ± 23.48173.73 ± 30.17198.80 ± 16.56
Walker2dReturn1177.15 ± 178.961619.83 ± 243.731929.78 ± 242.42
AUC675.07 ± 33.03903.11 ± 70.74975.07 ± 63.39
HalfCheetahReturn437.76 ± 184.89452.95 ± 188.12513.92 ± 162.73
AUC241.53 ± 43.43232.50 ± 43.61257.63 ± 59.70
InvertedPendulumReturn973.70 ± 83.181000.00 ± 0.001000.00 ± 0.00
AUC575.34 ± 69.61611.75 ± 82.10659.47 ± 44.53

SAC + Driver + Dependent raises mean final return and AUC over SAC by 26.61% and 34.98%, averaging relative gains equally across the five tasks. Its mean AUC also exceeds the Shuffled Dependent control in each task. These additional ten-seed experiments provide evidence under the revised policy-learning protocol, with cross-seed variation reported in the table (Section 5.2, lines 421–431 and 456–459; Appendix C.2, lines 1144–1184; evaluation protocol, lines 1253–1266).

Q1.1
Reviewer concern: Could you expand in more detail on how you'd extend the method to multiple gaits and/or time-varying (drifting) gaits?

  • Multiple gaits. One approach we would like to investigate is clustering trajectories into relatively consistent motion modes or skills, learning a factor–rule description for each, and then studying how higher-level coordination could connect them. Separating modes may also reduce the number of phases that need to be searched jointly. We discuss this idea in Section 6 (lines 528–532).

  • Time-varying or drifting gaits. For gradual changes, a time- or context-dependent factor map might adapt the continuous representation while retaining a shared program. Clearly distinct modes may instead benefit from separate maps and programs. We have included these possibilities in Section 6 (lines 485 and 526–532) as directions for extending the current fixed-map model.

Q1.2
Reviewer concern: How would one extend this to general locomotion, whole-body control, or potentially manipulation?

  • Richer control systems. We agree that a single Dependent gives only a partial description for whole-body control or manipulation. One possible next step is to identify relatively independent motion components, learn separate Dependents that may share a Driver, and investigate higher-level rules for combining their relations. Section 6 now discusses this direction toward a fuller motion representation (lines 480–483).
Reviewer_vEVfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose DynaAbd, which extracts a small symbolic description of cyclic motion from trajectories labeled as coming from a better or worse policy.

There are two factors for the description: 1. Driver: A cyclic factor that shows the phase of a cycle. 2. Dependent: A scalar factor that goes up or down depending on the phase.

There are 3 stages within the pipeline. The first step is spectral initialization, which is a contrastive variant of EDMD over a linear + quadratic feature dictionary that initializes both factors above. Then the factors are refined with differentiable next-step prediction losses with a term that separates good and bad trajectories. The third part is inductive ABL, where each factor is cut into five intervals. Prolog-based abduction picks a program, which is a sign vector saying whether the Dependent increases or decreases in each of the five phases. The factor projections and interval boundaries are adjusted at the same time. Then a small MLP adds a continuous residual check on top.

There are two experiments. On synthetic PolarCycle tasks with known phase and radius, the paper reports factor correlations and transition acceptance/rejection rates. On Hopper, HalfCheetah and Walker2d, the rules supervise an autoencoder whose frozen latent is fed to SAC, and the paper reports normalized AUC over the first 30k steps.

Critical strengths

The problem is well motivated. Turning continuous control trajectories into phase-level symbols and rules is a useful goal for interpretability and for connecting low-level control with higher-level reasoning. The paper also identifies a real difficulty clearly: factors, groundings and rules have to be learned jointly, and that coupled search is hard to start from scratch.

The initialization ablation is convincing. The same later stages reach only about 0.1–0.2 correlation from random or VQ-VAE initialization. From the spectral start they reach about 0.9 on the Driver. This supports the paper’s central design argument that a structured starting point is what makes the joint search work.

Critical weaknesses
  1. The rule space is very small. Background knowledge fixes that each phase has a single up or down direction, so only few programs are possible, and the structure of one cycle and dependent is chosen by hand. So the claim of rule discovery is not supported by the experiments as presented.

  2. he RL improvement is weak, and it does not use the discovered factors. The rule supervision improves over the plain autoencoder by only 6–11%. That is small compared with the variation across seeds (standard deviations in Table 3). On Walker2d, the 95% confidence interval includes zero, and the rule-based model is even worse than using raw observations (Table 3; Appendix C.3). The paper only reports the first 30k training steps (Table 3; Section 5.2). It also does not test whether any extra training target would give the same gain.

Questions for authors

How do the policy-learning results change if the teacher uses DynaAbd’s own discovered factors instead of the predefined joint angles?

Why does the autoencoder hurt on Walker2d but help a lot on HalfCheetah?

5 comments

We would like to thank the reviewer for the detailed comments and suggestions. We appreciate your recognition of the motivation for symbol and rule discovery and the experimental evidence supporting spectral initialization.

W1
Reviewer concern: The rule space is very small. Background knowledge fixes that each phase has a single up or down direction, so only few programs are possible, and the structure of one cycle and dependent is chosen by hand. So the claim of rule discovery is not supported by the experiments as presented.

  • Larger rule space. We agree that the original 30-program space offered a limited test of rule discovery. In response, we added Polar10, which uses ten phases and eleven clauses and expands the candidate space to 1,022 programs, or approximately 34 times its original size.

  • Rule recovery. All ten Polar10 seeds recover the correct aligned number and cyclic order of direction switches. To make this comparison easier to assess, we include recovery and search-time results for all four tasks below. The observation-to-factor mapping, boundaries, and phase-dependent directions are learned together.

Rule recovery after complete Coarse+Fine refinement over ten seeds (manuscript Table 2, lines 400–410). AIRA is mean ± sample standard deviation; accuracy is the proportion of seeds recovering the correct switch structure.

TaskPhysical sectorsLearned phasesAIRA (%)Rule accuracy (%)
Polar33595.24 ± 0.27100.00
Polar44596.29 ± 0.51100.00
Polar55592.24 ± 0.34100.00
Polar10101089.54 ± 0.49100.00

Joint program-and-label Prolog search (manuscript Table 3, lines 433–439). Ten seconds per task; search-time mean ± sample standard deviation across ten seed means.

TaskPhasesCandidate HClausesPure search time (s)
Polar353060.25 ± 0.03
Polar453060.26 ± 0.03
Polar553060.60 ± 0.49
Polar10101,022111.88 ± 0.06

Polar10 reaches 100.00% rule accuracy and 89.54% AIRA, with mean joint program-and-label search time of 1.88 seconds. The added Polar10 experiment tests rule recovery and inference cost in a larger candidate space (Section 5.1, lines 395–418; Tables 2–3, lines 400–410 and 433–439).

AIRA calculation. Angular Interval Rule Agreement (AIRA) measures the percentage of a PolarCycle circle covered by the correct increasing/decreasing rule. After a fixed phase alignment and Dependent sign alignment determined on separate alignment data, its definition is:

AIRA=∑i,jℓ(Ii∩I^j) 1[si=s^j]2π×100%.\mathrm{AIRA}=\frac{\sum_{i,j}\ell(I_i\cap\widehat I_j)\,\mathbf{1}[s_i=\widehat s_j]}{2\pi}\times100\%.

Here Iᵢ and Îⱼ are the true and learned phase intervals, sᵢ and ŝⱼ are their increasing (+1) or decreasing (−1) directions, and ℓ is angular length. The numerator adds the lengths where the two directions agree. For example, correct rules over 90% of the circle give AIRA = 90%. Rule accuracy separately counts seeds that recover the correct number and cyclic order of direction switches after neighboring intervals with the same direction are merged. The definitions appear at lines 294–304, with alignment and merging at lines 1076–1079.

W2
Reviewer concern: RL improvement is weak, and it does not use the discovered factors. The rule supervision improves over the plain autoencoder by only 6–11%. That is small compared with the variation across seeds (standard deviations in Table 3). On Walker2d, the 95% confidence interval includes zero, and the rule-based model is even worse than using raw observations (Table 3; Appendix C.3). The paper only reports the first 30k training steps (Table 3; Section 5.2). It also does not test whether any extra training target would give the same gain.

  • Why we revised the downstream test. We agree that the original autoencoder results gave limited evidence of a downstream benefit. Reconstruction-based encoding can change the accessibility of control information or discard some of it, while a rule loss constrains only selected attributes. These changes make the benefit of rule supervision difficult to isolate. We therefore retain the full observation and compute an intrinsic reward directly from DynaAbd’s discovered factors, boundaries, rules, and motion models in the revised experiment. Q1 below explains how the discovered factors are used.

  • How we test the extra-target explanation. To examine whether an extra training target could give the same gain, we compare learned Dependent direction assignments with shuffled assignments while retaining Driver guidance in both conditions. The control keeps the learned encoder, boundaries, Driver dynamics, and Dependent magnitude predictions, and matches the auxiliary-reward scale on fitting data. It randomly reassigns the phase-dependent Dependent directions and fixes them during policy training; the reassigned directions are used in both rule checking and signed Dependent prediction. This comparison tests whether the learned stage–response relationship adds value when Driver guidance is already present (Appendix C.2, lines 1159–1184).

  • Evaluation. Section 5.2 (lines 421–431) compares native-reward soft actor-critic (SAC), SAC + Driver + Shuffled Dependent, and SAC + Driver + Dependent on five tasks with changed physical parameters. All conditions share policy settings, ten paired seeds, and reference-state initialization (RSI; Peng et al., 2018), which starts episodes from reference-trajectory states. Each task is trained for 100,000 interactions. We report native task return at the endpoint and time-averaged area under the learning curve (AUC), the integral of evaluation return divided by 100,000, in native-return units.

Downstream learning under reference-state initialization (RSI) at 100,000 interactions (manuscript Table 4, lines 441–454). Mean ± sample standard deviation over ten paired seeds, after averaging target environments within each seed. Return and AUC are in native-return units.

TaskMetricSACSAC + Driver + Shuffled DependentSAC + Driver + Dependent
HopperReturn1804.42 ± 263.272113.02 ± 263.582178.75 ± 315.51
AUC892.62 ± 110.511134.64 ± 112.041213.43 ± 95.38
SwimmerReturn242.51 ± 41.47271.42 ± 48.98311.08 ± 17.23
AUC114.76 ± 23.48173.73 ± 30.17198.80 ± 16.56
Walker2dReturn1177.15 ± 178.961619.83 ± 243.731929.78 ± 242.42
AUC675.07 ± 33.03903.11 ± 70.74975.07 ± 63.39
HalfCheetahReturn437.76 ± 184.89452.95 ± 188.12513.92 ± 162.73
AUC241.53 ± 43.43232.50 ± 43.61257.63 ± 59.70
InvertedPendulumReturn973.70 ± 83.181000.00 ± 0.001000.00 ± 0.00
AUC575.34 ± 69.61611.75 ± 82.10659.47 ± 44.53
  • What the comparison shows. With learned Dependent directions, mean AUC is higher than the shuffled control in all five tasks, by 6.94%–14.43%. Mean final return is higher in four tasks, with both conditions reaching 1000.00 on InvertedPendulum. Since both conditions already include Driver guidance and auxiliary supervision, this provides evidence that the learned phase-dependent Dependent directions contribute additional useful information.

Relative to SAC, the full guidance also gives higher mean final return and AUC in all five tasks. Equal-task averages of the relative improvements are 26.61% for final return and 34.98% for AUC. Together, these comparisons assess both the overall downstream benefit and the added value of learned Dependent directions. The table reports cross-seed variation, and the revised training and evaluation protocols appear in Section 5.2 (lines 421–431 and 456–459) and Appendix C.2 (lines 1144–1184 and 1253–1266).

Q1
Reviewer concern: How do the policy-learning results change if the teacher uses DynaAbd’s own discovered factors instead of the predefined joint angles?

In the original downstream experiment, the learned factors selected predefined phase and joint-angle functions, which then supplied the teacher (original submission, Appendix C.1, lines 981–993). The revised generalization experiment directly uses DynaAbd’s discovered Driver and Dependent factors. The learned factor map, boundaries, program, and continuous motion model are frozen and used to compute the intrinsic reward from observed transitions; the policy retains the full observation. The revised Table 4 therefore evaluates policy guidance from the discovered factors and rules (Section 5.2, lines 421–431; Appendix C.2, lines 1144–1157).

Q2
Reviewer concern: Why does the autoencoder hurt on Walker2d but help a lot on HalfCheetah?

Our interpretation is that autoencoder training prioritizes reconstruction loss, whereas information important for reconstruction can differ from information useful for reinforcement learning. Losing control-relevant information may hurt learning; suppressing less relevant information may help early training. Different task requirements could therefore account for the contrasting results. We offer this as a possible explanation. The revised experiment retains the full observation and supplies rule guidance through intrinsic reward, reducing the influence of this encoding uncertainty (Section 5.2, lines 421–431 and 456–459; Appendix C.2, lines 1144–1184; evaluation protocol, lines 1253–1266).

Reviewer_jPyeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper trains a model for cyclic motions like locomotion, where each complete motion cycle passes through several different phases. The model predicts the current phase (the Driver) and the level of a second quantity (the Dependent), and learns a rule that says whether this quantity rises or falls in each phase. After training the model in three stages, they did experiments to try to show that the trained model predicts meaningful phases and levels. They also show that these are critical pieces of information for an autoencoder to learn to predict, which is helpful for training RL in the latent space.

Critical strengths
  1. A well-structured decomposition of a hard joint problem. Learning continuous factors, symbol boundaries, and rules together is a large, coupled optimization. The paper breaks it into three coarse-to-fine stages, each with a clear role: spectral initialization, one-step dynamics modeling, and abductive refinement with Prolog. This matters to my assessment because the decomposition is what makes the problem tractable.

  2. Experiments designed to test each component. The synthetic PolarCycle tasks have known ground-truth phases and directions, so recovery can be checked directly. The ablations remove or replace each stage, Figure 3 tracks recovery after each stage, and Table 2 isolates what the continuous checks add over the discrete rule alone. This matters because it shows where the method’s performance comes from, instead of resting only on downstream RL numbers.

Critical weaknesses
  1. Unclear motivation for this particular abstraction. The paper motivates symbol and rule discovery with interpretability, a link to higher-level reasoning, and faster policy learning, but only the last is tested. It is also not clear why one cyclic phase variable plus one rising-or-falling quantity is the right description for control. Locomotion depends on many other quantities, and the authors themselves note in Section 6 that their motion descriptions may omit control-relevant quantities. This matters to my assessment because without a clearer use case, it is hard to judge the value of the discovered symbols beyond this specific setup.

  2. Narrow scope and simple evaluation settings. The method assumes a single recurring cycle, one Dependent quantity, and a fixed up/down grammar, which the authors acknowledge. This limits it to simple periodic motions, and it is unclear how it would extend to multiple gaits, non-periodic behavior, or long-horizon tasks. The evaluation reflects this scope. It uses synthetic 2D loops and three standard MuJoCo locomotion tasks (Hopper, HalfCheetah, Walker2d), with no higher-dimensional bodies such as Humanoid and no real-world data. This matters to my assessment because it leaves the practical significance of the approach uncertain.

Questions for authors
  1. How should the number of symbolic phases be chosen? It is fixed at five throughout, and every PolarCycle task has five or fewer true phases. What happens when the true number of phases is larger than the number of slots, and how would a user set this number when the true value is unknown?
  2. How does the pipeline scale to motions with more phases, for example 10 or 20? The number of possible direction programs grows exponentially with the number of phases, and each Prolog query is limited to three seconds. Does the search remain feasible, and do the earlier stages still provide a good enough starting point?
6 comments

We would like to thank the reviewer for the detailed comments and suggestions. We appreciate your recognition of the three-stage decomposition of the joint learning problem and the experiments designed to assess each component.

W1.1
Reviewer concern: Unclear motivation for this particular abstraction.

  • Modeling basis. To help explain this choice, we have clarified the basis for the Driver–Dependent abstraction. Dynamical movement primitives separate a canonical phase system from phase-modulated motion dynamics (Ijspeert et al., 2013), while DeepMimic conditions control on reference-motion phase (Peng et al., 2018). These precedents motivate our use of Driver to locate the movement stage and Dependent to describe how a motion quantity evolves at that stage. We now explain this rationale in the Introduction (lines 51–54) and define the roles in Section 3 (lines 124–126).

  • Why initially focus on cyclic motion. We initially focused on cyclic motion because cycle phase gives Driver a clear meaning for controlled evaluation, and a compact program can describe one cycle and recur throughout a trajectory. We have made this choice explicit in Section 4 (lines 146–148) and Section 6 (lines 478–483). The InvertedPendulum experiment discussed in W2.2 extends these roles to nonperiodic motion.

W1.2
Reviewer concern: The paper motivates symbol and rule discovery with interpretability, a link to higher-level reasoning, and faster policy learning, but only the last is tested.

  • Interpretability. We have added qualitative examples to make the learned descriptions easier to assess. In Hopper, Dependent decreases in P2–P4 correspond to landing and forward foot rotation through push-off, while increases in P5/P1 correspond to upward foot rotation after take-off. The negative trajectory shows smaller oscillations and different phase durations. For InvertedPendulum, the stages are associated with predominant combinations of cart translation and pole rotation. These examples appear in Section 5.3 (lines 463–470) and Figures 4–5, with the visualization protocol in Appendix C.5 (lines 1298–1312).

  • Higher-level reasoning. We view connecting the discovered descriptions to symbolic planning and task decomposition as a promising next step. This would require additional methodological and experimental work, drawing on approaches such as Konidaris et al. (2018) and Toro Icarte et al. (2022).

W1.3
Reviewer concern: It is also not clear why one cyclic phase variable plus one rising-or-falling quantity is the right description for control. Locomotion depends on many other quantities, and the authors themselves note in Section 6 that their motion descriptions may omit control-relevant quantities.

  • Current scope. We agree that one Driver and one scalar Dependent capture only part of the motion. Our reason for using increasing/decreasing rules is that they summarize local trends of continuously evolving physical quantities, while continuous fitting describes variation within a stage. The policy retains its full observation so that it can also use the other quantities needed for control (Section 5.2, lines 421–431; Section 6, lines 478–483).

  • Possible extension. A fuller description may require additional Dependents for relatively independent motion components. Section 6 now discusses several Dependent factors, organized by a shared Driver with clauses for each component (lines 480–483). Combining these descriptions through higher-level rules is one possible direction to investigate. We regard this as a direction for further work; W2.2 below discusses its relevance to more complex bodies.

W2.1
Reviewer concern: The method assumes a single recurring cycle, one Dependent quantity, and a fixed up/down grammar, which the authors acknowledge. This limits it to simple periodic motions, and it is unclear how it would extend to multiple gaits, non-periodic behavior, or long-horizon tasks.

  • Nonperiodic motion. We have extended the stage–response formulation to nonperiodic motion, using ordered Driver stages that allow reversals and increasing, decreasing, or steady Dependent responses. Driver locates the movement stage, and Dependent describes the response at that stage. We have added the revised definitions in Section 3 (lines 124–126), with the nonperiodic adaptation in Appendix C.3 (lines 1204–1250). The corresponding experiments are described in W2.2 below.

  • Multiple gaits and long-horizon motion. For multiple gaits or long-horizon motion containing distinct behaviors, we think a useful next step would be to segment trajectories into relatively consistent motion modes or skills and learn factors and rules for each. Higher-level coordination could then connect these descriptions. We have added this idea to Section 6 (lines 528–532) as a possible extension that would need further investigation.

W2.2
Reviewer concern: The evaluation reflects this scope. It uses synthetic 2D loops and three standard MuJoCo locomotion tasks (Hopper, HalfCheetah, Walker2d), with no higher-dimensional bodies such as Humanoid and no real-world data.

  • Expanded evaluation. We have added nonperiodic InvertedPendulum and Swimmer to the MuJoCo evaluation, alongside Polar10 in the controlled experiments. The revised study covers four controlled tasks and five MuJoCo tasks with 22 target variants, testing factor and rule recovery, inference cost, physical interpretation, and policy guidance under changed physical conditions (experimental setup, lines 272–286; Sections 5.1–5.3, lines 395–431 and 456–470; target counts, lines 1253–1255). The recovery and scaling results are given below in Q1–Q2.

  • Nonperiodic evidence. The InvertedPendulum experiment evaluates the adapted Driver–Dependent roles on three target variants with the same 100,000-interaction budget. Time-averaged area under the learning curve (AUC) is the integral of native evaluation return divided by the training budget.

InvertedPendulum under reference-state initialization (RSI; Peng et al., 2018) at 100,000 interactions (manuscript Table 4, lines 452–454). Mean ± sample standard deviation over ten paired seeds, averaging three target variants within each seed.

MetricSACSAC + Driver + Shuffled DependentSAC + Driver + Dependent
Return973.70 ± 83.181000.00 ± 0.001000.00 ± 0.00
AUC575.34 ± 69.61611.75 ± 82.10659.47 ± 44.53

The higher mean AUC provides additional evidence of improved performance during learning in this setting, while both guided conditions reach the maximum final return. Figure 5 relates the learned stages to cart translation and pole rotation. The policy results and qualitative interpretation appear in Sections 5.2 (lines 421–431 and 452–459) and 5.3 (lines 467–470), with the nonperiodic protocol in Appendix C.3 (lines 1204–1250).

  • Higher-dimensional bodies. We agree that richer bodies such as Humanoid and real-world motion would be valuable next steps for assessing practical reach. For a more complex body, one Dependent still describes only part of the motion. One idea is to learn several factor–rule systems for relatively independent components and then investigate how to combine their local descriptions. We discuss this possibility in Section 6 (lines 480–483) as a route toward a fuller representation.

Q1
Reviewer concern: How should the number of symbolic phases be chosen? It is fixed at five throughout, and every PolarCycle task has five or fewer true phases. What happens when the true number of phases is larger than the number of slots, and how would a user set this number when the true value is unknown?

  • How we currently choose the count. We currently select the phase count manually. Polar3, Polar4, and Polar5 use five learned phases for three, four, and five physical sectors, respectively. All ten seeds recover the correct switch structure in each task, showing that boundary adjustment can accommodate extra phase capacity in Polar3/4. We have specified these settings in the revised experimental setup (lines 278–282).

  • When the count is too small. If the available phases cannot express the required direction switches, rule accuracy can decrease. This is why, when a rough estimate is available, we suggest allowing a modest buffer above it. The Polar3/4 results provide evidence for accommodating such a buffer.

  • When the count is unknown. A small pilot sample could be used before the main abduction stage to compare several candidate counts. Held-out positive acceptance and negative rejection could measure how well the rules explain the data; a threshold on their improvement could guide the choice, followed by a few additional phases as a buffer. This is a possible extension for selecting the count that would need to be evaluated.

The recovery results supporting the phase-capacity discussion are shown below, including Polar10 with ten learned phases. Rule accuracy is the proportion of seeds recovering the correct number and cyclic order of direction switches, after merging neighboring intervals with the same direction.

Rule recovery after complete Coarse+Fine refinement over ten seeds (manuscript Table 2, lines 400–410). AIRA is mean ± sample standard deviation; accuracy is the proportion of seeds recovering the correct switch structure.

TaskPhysical sectorsLearned phasesAIRA (%)Rule accuracy (%)
Polar33595.24 ± 0.27100.00
Polar44596.29 ± 0.51100.00
Polar55592.24 ± 0.34100.00
Polar10101089.54 ± 0.49100.00

AIRA calculation. Angular Interval Rule Agreement (AIRA) additionally measures the percentage of the circle covered by the correct increasing/decreasing rule. After fixed phase and Dependent-sign alignments determined on separate alignment data,

AIRA=∑i,jℓ(Ii∩I^j) 1[si=s^j]2π×100%.\mathrm{AIRA}=\frac{\sum_{i,j}\ell(I_i\cap\widehat I_j)\,\mathbf{1}[s_i=\widehat s_j]}{2\pi}\times100\%.

Iᵢ and Îⱼ denote the true and learned phase intervals, sᵢ and ŝⱼ their increasing (+1) or decreasing (−1) directions, and ℓ angular length. Thus, correct rules over 90% of the circle give AIRA = 90%. Definitions appear at lines 294–304, with alignment and merging at lines 1076–1079.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.