- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the representation learning problem of discovering latent symbolic factors/rules from cyclic motion trajectories. The proposed approach, DynaAbd, is a two-stage approach that first provides initial estimates of cyclic / dependent coordinates, and second models one-step dynamics to refine the coordinates. The approach is evaluated in a simulated polar motion setting and Mujoco locomotion tasks. The main results are that DynaAbd can recover the underlying motion structural factors, and the learned symbolic representations can accelerate downstream policy learning.
Critical strengths
The paper is generally clear and provides enough detail to fully specify the method. The empirical improvement of DynaAbd over baseline representation learning methods and ablations is substantial (Table 1).
Critical weaknesses
First, while the paper argues why the problem is hard, it insufficeintly argues why this problem is important / how the developed approach could be broadly relevant or impactful beyond the specific studied problem. The method is specific to learning symbolic descriptions of cyclic motion trajectories, as the identified structural factors (Driver, Dependent), are specific to cyclic motion. Additionally, the empirical benefits for downstream learning over a generic autoencoder are relatively modest (Table 3), producing improvements with overlapping confidence intervals with the AE baseline on all tasks. While this is a representation learning problem, I do not think it is a broad enough problem to be interesting to the ICLR community.
Second, I am unsure why the method needs to use positive/negative trajectory quality labels (as specified in the problem formulation). It is also unclear whether any of the baselines receive the benefit of these labels.
Questions for authors
Which of the baselines could use the positive/negative trajectory quality labels?
5 comments
W1.3
W1.3
Reviewer concern: Additionally, the empirical benefits for downstream learning over a generic autoencoder are relatively modest (Table 3), producing improvements with overlapping confidence intervals with the AE baseline on all tasks.
-
Original uncertainty. We agree that the original autoencoder results gave limited evidence of a consistent downstream benefit. Reconstruction training may change information accessibility or discard information, while reconstruction importance can differ from control relevance. Suppressing less relevant information could help early learning, and losing useful control information could have the opposite effect. We view this as a possible explanation for the task-dependent results.
-
Revised evidence. We have revised Section 5.2 (lines 421–431) to preserve full observations and use the discovered factors and rules directly as intrinsic reward under changed physical parameters. Native-reward soft actor-critic (SAC), SAC + Driver + Shuffled Dependent, and SAC + Driver + Dependent share policy settings, ten paired seeds, and reference-state initialization (RSI), which starts episodes from reference-trajectory states. Shuffled Dependent retains Driver dynamics and Dependent magnitudes while reassigning phase-dependent Dependent directions. All tasks use 100,000 interactions. For convenience, we include the complete return and time-averaged AUC results below.
Downstream learning under reference-state initialization (RSI) at 100,000 interactions (manuscript Table 4, lines 441–454). Mean ± sample standard deviation over ten paired seeds, after averaging target environments within each seed. Return and AUC are in native-return units.
| Task | Metric | SAC | SAC + Driver + Shuffled Dependent | SAC + Driver + Dependent |
|---|---|---|---|---|
| Hopper | Return | 1804.42 ± 263.27 | 2113.02 ± 263.58 | 2178.75 ± 315.51 |
| AUC | 892.62 ± 110.51 | 1134.64 ± 112.04 | 1213.43 ± 95.38 | |
| Swimmer | Return | 242.51 ± 41.47 | 271.42 ± 48.98 | 311.08 ± 17.23 |
| AUC | 114.76 ± 23.48 | 173.73 ± 30.17 | 198.80 ± 16.56 | |
| Walker2d | Return | 1177.15 ± 178.96 | 1619.83 ± 243.73 | 1929.78 ± 242.42 |
| AUC | 675.07 ± 33.03 | 903.11 ± 70.74 | 975.07 ± 63.39 | |
| HalfCheetah | Return | 437.76 ± 184.89 | 452.95 ± 188.12 | 513.92 ± 162.73 |
| AUC | 241.53 ± 43.43 | 232.50 ± 43.61 | 257.63 ± 59.70 | |
| InvertedPendulum | Return | 973.70 ± 83.18 | 1000.00 ± 0.00 | 1000.00 ± 0.00 |
| AUC | 575.34 ± 69.61 | 611.75 ± 82.10 | 659.47 ± 44.53 |
SAC + Driver + Dependent improves mean final return and AUC over SAC by 26.61% and 34.98%, averaging relative gains equally across the five tasks. Mean AUC also exceeds the Shuffled Dependent control in each task. These additional ten-seed experiments provide evidence under the revised policy-learning protocol, with cross-seed variation reported in the table (Section 5.2, lines 421–431 and 456–459; Appendix C.2, lines 1144–1184; evaluation protocol, lines 1253–1266).
W2
W2.1
Reviewer concern: Second, I am unsure why the method needs to use positive/negative trajectory quality labels (as specified in the problem formulation).
-
Purpose of quality labels. Dependent is intended to capture the response associated with high-quality motion at a given Driver stage. Predictable transitions also occur in low-quality trajectories, so dynamics prediction alone can favor an easy-to-predict factor that misses this distinction. Quality labels help direct Dependent discovery toward the relevant response. We have added this motivation to the Introduction (lines 54–57); Related Work discusses how insufficient negative-example coverage can leave concepts indistinguishable under task supervision and permit reasoning shortcuts (lines 100–102; Marconato et al., 2023).
-
How they are used. Dependent discovery uses quality information in its contrastive initialization, positive/negative classification loss, and negative-class separation during continuous fitting. These objectives distinguish responses associated with different trajectory qualities while the factor values, boundaries, and rules are learned (Sections 4.1–4.3, lines 160–161, 182–183, 211–215, and 262–263).
-
Why this ablation is informative. We remove the objectives that distinguish positive and negative examples during Dependent discovery and withhold quality labels from its initialization, while fixing Driver to the full model at each stage. The remaining dynamical and logical supervision can still fit motion regularities. This isolates whether quality information helps select the intended response factor from other dynamically explainable factors. We evaluate the recovered Dependent by its absolute Pearson correlation with true radial increments on the same held-out data (Section 5.1, lines 311–314 and 373–375; Appendix B.2, lines 1036–1038).
Dependent quality-label ablation (manuscript Table 1, lines 348–368). Mean ± sample standard deviation over ten seeds; Driver is fixed to the full model at each stage. The four-task mean is computed within seed first.
| Task | DynaAbd Dependent correlation | DynaAbd− Dependent correlation |
|---|---|---|
| Polar3 | 0.97 ± 0.00 | 0.05 ± 0.03 |
| Polar4 | 0.92 ± 0.05 | 0.29 ± 0.12 |
| Polar5 | 0.96 ± 0.01 | 0.10 ± 0.08 |
| Polar10 | 0.97 ± 0.01 | 0.09 ± 0.07 |
| Mean | 0.95 ± 0.01 | 0.13 ± 0.05 |
Without Dependent quality supervision, mean correlation falls from 0.95 ± 0.01 to 0.13 ± 0.05. With the same Driver available, the large loss in factor recovery supports the role of positive/negative information in reducing dynamical shortcuts: representations that explain predictable motion but fail to capture the intended phase-dependent response (Table 1, lines 348–368; Appendix B.2, lines 1036–1038).
W2.2
Reviewer concern: It is also unclear whether any of the baselines receive the benefit of these labels.
- Matched supervision. We have added SINDy-AE+ and mechanism sparsity+ with the same positive/negative trajectory labels, together with supervised autoencoding (SAE) and supervised contrastive learning (SupCon), which use the labels directly. All methods share the observations and data partitions. We discuss the closely related baseline question in Q1 below, where we include the full comparison and the original versions for reference (Section 5.1, lines 316–323 and 370–375; Appendix B.2, lines 999–1038).
W1.1&W1.2
We would like to thank the reviewer for the detailed comments and suggestions. Thank you for your recognition of the method description and the factor-recovery improvements over the baselines and ablations.
W1.1
Reviewer concern: First, while the paper argues why the problem is hard, it insufficiently argues why this problem is important / how the developed approach could be broadly relevant or impactful beyond the specific studied problem.
W1.2
Reviewer concern: The method is specific to learning symbolic descriptions of cyclic motion trajectories, as the identified structural factors (Driver, Dependent), are specific to cyclic motion.
Definitions and initial focus. In response to this concern, we have extended the Driver–Dependent formulation to nonperiodic motion: Driver locates the movement stage, and Dependent describes the response at that stage. The Introduction (lines 51–54) connects these roles to dynamical movement primitives and phase-conditioned control in DeepMimic (Ijspeert et al., 2013; Peng et al., 2018). We initially focused on cyclic motion because it provides a well-defined phase for controlled evaluation and a compact rule that can be reused across cycles (Section 3, lines 124–126; Section 4, lines 146–148; Section 6, lines 478–483).
Nonperiodic evidence. To test these roles beyond cyclic motion, we added InvertedPendulum with reversible stage transitions and increasing, decreasing, or steady responses. The table below gives its downstream results at 100,000 interactions. Area under the learning curve (AUC) is the integral of native evaluation return divided by the training budget, measuring average performance during learning.
InvertedPendulum under reference-state initialization (RSI; Peng et al., 2018) at 100,000 interactions (manuscript Table 4, lines 452–454). Mean ± sample standard deviation over ten paired seeds, averaging three target variants within each seed.
The learned Driver–Dependent guidance improves mean AUC from 575.34 to 659.47. The qualitative example in Figure 5 gives a physical interpretation: P0–P2 mainly correspond to rightward cart translation with leftward pole rotation, and P3–P4 to the opposite combination. Together, these experiments provide evidence that the same stage–response roles can extend to nonperiodic motion with the adaptations described in Appendix C.3 (lines 1204–1250); the physical interpretation appears in Section 5.3 (lines 467–470), with its visualization protocol in Appendix C.5 (lines 1298–1312).