acceptodds
Under review as a conference paper at ICLR 2027

From Redundancy to Minimality: Fixed-Point-Guided Hierarchical Reduction of Learned Piecewise-Linear Dynamics

Abstract

Understanding a nonlinear dynamical system from time series requires not only reproducing its trajectories, but also identifying a simple representation that preserves its essential dynamical structure. Almost-linear recurrent neural networks (AL-RNNs) are piecewise-linear RNNs in which only a subset of units use ReLU nonlinearities, while the remaining units are linear, so that nonlinear capacity is explicitly controlled by the number of ReLU units. Their activation patterns define linear regions, represented as symbols, whose observed transitions form a symbolic transition graph. However, directly training AL-RNNs with few ReLU units to realize minimal dynamical representations can be unreliable. We ask whether an AL-RNN with more ReLU units can instead be trained first and systematically reduced to a minimal dynamical representation. We introduce a fixed-point-guided hierarchical reduction procedure that progressively linearizes selected ReLU units, thereby merging neighboring linear regions and their corresponding graph nodes while preserving distinct symbols containing fixed points (FPs). The resulting reduction tree defines a hierarchy of progressively simpler candidate models. Each reduced candidate is initialized from the parent parameters and retrained under guidance from the parent dynamics, providing a practical route from structural reduction to dynamical realization. We also prove that reproducing distinct fixed points requires at least FP-containing symbols, providing a certificate of symbol-level minimality when this bound is attained. On the 3-scroll Chua system, direct training with the theoretical minimum of three ReLU units achieves high-fidelity minimal realizations in only 20% of seeds, whereas our learn-reduce-retrain strategy increases the seed-macro success rate to approximately 71% at the same final nonlinear capacity. These results show that redundant nonlinear capacity can serve as a scaffold for discovering and realizing minimal dynamical representations.

Primary area:
learning on time series and dynamical systems
Keywords:
recurrent neural networks, nonlinear dynamics, learning dynamical systems, piecewise-linear dynamics, fixed-point analysis, symbolic dynamics, chaotic dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aCcwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes learning a larger almost-linear RNN, then removing nonlinearities using fixed-point structure and retraining the smaller model to match the parent’s dynamics. It derives a conditional lower bound on the number of linear regions needed to preserve the target fixed points. On the three-scroll Chua system, this approach raises the reported success rate for learning accurate minimal models significantly.

Critical strengths

The fixed-point lower bound provides an interesting and principled criterion for model simplification. The authors decently guide the reader through the paper, ranging from an introduction of almost linear RNNs to the fixed point criterion.

Critical weaknesses
  • While the guidance through the paper is decent, it gets harder to follow in particular towards the end. I would therefore recommend the authors to polish the presentation, making it easier to follow for not just machine readers, but also human readers.
  • The experimental validation is limited to one clean, fully observed dynamical system. Does the method, empirically, generalize to other systems?
Questions for authors
  • Can the authors ablate the fixed-point guided complexity reduction to f.e. a random control or other reduction methods?
1 comment
AuthorsOpenReview ↗

Response to Reviewer aCcw

Note:

  • The supplementary ZIP includes a PDF copy of this response with identical content, formatted for easier reading (rebuttal_aCcw.pdf).
  • It also includes the revised manuscript with all changes from the original submission highlighted in yellow (ICLR2027_AL_RNN_rebuttal_all_changes_highlighted.pdf).

Response to Reviewer aCcw

Thank you for your comments. We have expanded the evaluation beyond 3-scroll Chua to include 4- and 5-scroll Chua, the Dadras 4-wing system, and 2- and 3-bit flip-flop memory. We also reorganized the presentation and added controlled comparisons to test which ReLU units should be linearized.

1. Presentation and readability

Reviewer comment

While the guidance through the paper is decent, it gets harder to follow in particular towards the end. I would therefore recommend the authors to polish the presentation, making it easier to follow for not just machine readers, but also human readers.

Response

To make the expanded experiments easier to follow, we present the attractor-reconstruction and memory results in a consistent format (Section 4). Figures 2 and 3 follow the same progression: illustrations of each task’s goal, the trade-off between direct-training performance and symbolic complexity that motivates our approach, and the results of reduction-guided retraining. This gives readers a consistent progression through the experiments and makes comparisons across task families easier to follow. Detailed training procedures and evaluation definitions are collected in Appendices C and D, keeping the main results focused on the findings.

To make the reduction process easier to follow, we added step-by-step visualizations showing how successive ReLU linearizations merge symbolic regions and simplify their transition graphs (Figures A6 and A8).

For the added computational-cost comparison, we use a flowchart to explain the search procedures and stacked bars to show their cost components (Figures A12 and A13; Appendix G).

2. Experimental breadth

Reviewer comment

The experimental validation is limited to one clean, fully observed dynamical system. Does the method, empirically, generalize to other systems?

Response

We expanded the evaluation from 3-scroll Chua to six tasks, adding 4- and 5-scroll Chua, Dadras 4-wing, and 2- and 3-bit flip-flop memory.

Reduction-guided retraining improves minimal-realization success across all six tasks. On the added autonomous systems, success increases from 3% to 49% for 4-scroll Chua, 3% to 8% for 5-scroll Chua, and 0% to 15% for Dadras (Section 4.1; Figure 2(c)). On memory tasks, success increases from 48% to 92% for 2-bit memory and from 29% to 78% for 3-bit memory (Section 4.2; Figure 3(c)). Within the Chua and memory families, direct training generally becomes less reliable at finding minimal models as the number of target fixed points increases. Reduction-guided retraining retains an advantage in these harder settings.

The expanded evaluation also reveals a computational benefit (Table A4; Figure A13). On 4- and 5-scroll Chua and both memory tasks, direct training first finds minimal models above the parent-estimated lower bound capacity. Our method avoids a separate capacity sweep and reduces the estimated average cost of finding the first minimal model by approximately 2.7–4.3-fold relative to parent-informed direct search, even after including parent training and reduction.

3. Control for fixed-point-guided selection

Reviewer comment

Can the authors ablate the fixed-point guided complexity reduction to f.e. a random control or other reduction methods?

Response

We added matched random ReLU-linearization controls on 3- and 4-scroll Chua and both memory tasks (Appendices E.5 and F.3). We exclude randomly drawn sets that coincide with that parent's selected candidates, so these off-tree controls directly compare alternative choices of units while matching parent initialization, remaining capacity, and retraining settings.

With gradual linearization and no auxiliary guidance, replacing off-tree controls with tree-selected sets raises minimal-realization success from 21% to 80% on 3-scroll Chua and from 14% to 71% on 4-scroll Chua (Figure A5). Memory-state recovery similarly rises from 45% to 92% for 2-bit memory and from 23% to 97% for 3-bit memory (Figure A10). The advantage persists with parent-state guidance and abrupt linearization. Both protocols also show better fixed-point correspondence for tree-selected sets. These controls identify fixed-point-guided ReLU selection as a key contributor to the improvement.

Reviewer_WJxkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a reduction procedure for almost-linear recurrent neural networks (AL-RNNs). Autonomous trajectories define an activation-symbol transition graph, and admissible fixed points are computed analytically within affine regions. The procedure deletes activation bits while keeping distinct fixed-point symbols distinguishable. It maps reduced structures back to networks by replacing selected ReLUs with identity mappings. Retraining starts from parent parameters, uses the parent's full latent trajectory as a teacher signal, and periodically synchronizes the student state.

Proposition 1 gives a conditional capacity bound: under assumptions including isolated fixed points in the interiors of visited regions and nonsingular regional fixed-point equations, reproducing QQ distinct target fixed points requires at least QQ visited symbols and ⌈log⁡2Q⌉\lceil\log_2 Q\rceil ReLUs. Experiments use a three-scroll Chua system with latent dimension 20 and observation dimension 3. They compare reductions from ten to three ReLUs against direct three-ReLU training and examine guidance variants and fidelity thresholds.

Critical strengths

Strength 1: The reduction constraint has an interpretable mathematical basis

Section 2.2 and Proposition 1 give a conditional capacity bound for representations preserving the target fixed points. Section 3.1, Eq. (13), and Appendices B.1 to B.3 retain distinctions between the parent's FP symbols during projection and derive a valid monotone pruning rule for these projections. This connects a concrete structural search to the capacity argument. The paper explicitly treats quotient graphs as candidates whose dynamical realization is tested by retraining.

Strength 2: Reductions map to concrete networks and reuse parent information

Section 3.2 and Appendices C.3 to C.7 specify parameter copying, ReLU linearization, teacher-trajectory construction, synchronization, and guidance losses. Candidates retain the latent dimension, change the nonlinear structure through a fixed mask, and retrain from parent parameters. These definitions give the reduction and retraining procedure a clear implementation path.

Strength 3: Controls support improved success rates for the complete pipeline and parent-trajectory guidance

Under the paper's high-fidelity criterion, Figure 2(c) reports approximately 71% versus 20% success for reduced/retrained versus directly trained models with three ReLUs. Figure A1 compares guidance variants on identical candidate structures and parent initializations. In this 141-candidate ablation, which includes 15 four-ReLU candidates, the parent-balanced high-fidelity rates are approximately 73% with full-state guidance and 14% without parent-trajectory guidance. These controls support the complete pipeline and the role of trajectory guidance after selection.

Critical weaknesses

Weakness 1: The success criterion does not verify correspondence with the target fixed points

Section 2.2 defines preservation through correspondence with the target fixed points, and Proposition 1 uses target reproduction as a premise. Section 3.3 and Appendix D.3, Eqs. (14)-(15) and (A38)-(A40), instead test five admissible fixed points in visited regions, five visited symbols, and a distributional threshold. Neither these tests nor Appendix E.4's chain-like symbolic organization quantify pointwise correspondence with the Chua targets in Appendix C.1. This leaves a premise for interpreting the models as faithful minimal realizations unverified. The introduction motivates fixed points through their local dynamical roles. If skeleton preservation is intended to include these roles, the local types of matched points also need validation.

Weakness 2: The fidelity metric does not explicitly account for out-of-range generated mass

Appendix D.2 constructs a histogram with 30 bins per dimension over the target trajectory's coordinate ranges. It excludes generated samples outside those ranges, adds 10−510^{-5} to each cell, and normalizes. The excluded fractions are not reported. Eqs. (15) and (A40) classify high fidelity using Estsp<2.0E_{\mathrm{stsp}} < 2.0, so the criterion depends on the smoothed, normalized in-range histogram without explicitly penalizing out-of-range probability mass. Figure A2 provides representative trajectories but does not quantify this omission's effect on the overall success rate. Reporting excluded fractions and accounting for this mass would establish whether the classifications and method comparison are robust.

Weakness 3: The independent benefit of structural selection has not been isolated

Section 3.1 selects reduced structures by separating the parent's FP symbols; actual dynamics are established through retraining, as Section 3.2 and Appendix A.3 explain. Appendix E.2 and Figure A1 fix the selected structures and parent initialization while varying teacher information, guidance losses, and synchronization. They support guided retraining, but do not establish an advantage for the proposed selection over other masks retaining the same number of ReLUs. A comparison with random masks of the same capacity, matching parent parameters, teacher guidance, and training budget, would test the central structural addition.

Minor comments

  1. Section 4.2 reports 26 parents with a five-symbol candidate, including 25 with a three-ReLU, five-symbol candidate. Appendix E.2 attributes the additional four-ReLU candidates to two parents whose trees reach five symbols only at four ReLUs. These statements require at least 27 parents with five-symbol candidates. Please reconcile the counts using per-parent candidate records and state whether the parent-balanced rates need updating.
  2. Figure A3 reports approximately 73% success at threshold 2.0, whereas the main capacity-matched comparison reports approximately 71%. Appendix E.3 should specify the candidate pool used for this threshold scan, as Appendix E.2 does for the guidance ablation.
Questions for authors
  1. For models classified as high-fidelity minimal successes, what are the one-to-one matching errors in observation space between the five fixed points in visited regions and the Chua targets? Please give a tolerance and the fraction meeting it. If skeleton preservation includes local dynamics, how are matched stability types and extra latent directions assessed?
  2. What fractions of autonomous samples fall outside the target histogram range for directly trained models, reduced/retrained models, and parents? How do the high-fidelity classifications and method comparison change when these probability masses are explicitly included?
  3. With the same parameters copied from the parent, teacher guidance, and training budget, how does the proposed selection compare with random masks retaining the same number of ReLUs? Please apply the same correspondence check with the target fixed points to both groups when comparing success rates.
3 comments
AuthorsOpenReview ↗

Response to Reviewer WJxk (1/3)

Note:

  • The supplementary ZIP includes a PDF copy of this response with identical content, formatted for easier reading (rebuttal_WJxk.pdf).
  • It also includes the revised manuscript with all changes from the original submission highlighted in yellow (ICLR2027_AL_RNN_rebuttal_all_changes_highlighted.pdf).

Response to Reviewer WJxk

Thank you for identifying these issues. We have added direct checks of fixed-point correspondence and local dynamics, matched controls for ReLU selection, and an analysis of excluded histogram mass. The expanded evaluation covers 3-, 4-, and 5-scroll Chua, and Dadras 4-wing attractors, and 2- and 3-bit flip-flop memory systems.

W1. Fixed-point correspondence and local dynamics

Reviewer comment

Section 2.2 defines preservation through correspondence with the target fixed points, and Proposition 1 uses target reproduction as a premise. Section 3.3 and Appendix D.3, Eqs. (14)-(15) and (A38)-(A40), instead test five admissible fixed points in visited regions, five visited symbols, and a distributional threshold. Neither these tests nor Appendix E.4's chain-like symbolic organization quantify pointwise correspondence with the Chua targets in Appendix C.1. This leaves a premise for interpreting the models as faithful minimal realizations unverified. The introduction motivates fixed points through their local dynamical roles. If skeleton preservation is intended to include these roles, the local types of matched points also need validation.

Response

The new analysis verifies strong recovery of both fixed-point positions and local stability types on Chua systems. Across all retrained models, joint recovery reaches 99.4%, 96.1%, and 78.9% for 3-, 4-, and 5-scroll Chua, respectively, compared with 17.8% for Dadras (Figure A3; Appendix E.4).

These direct correspondence checks support recovery of the Chua fixed-point structure, while identifying discrepancies in Dadras that fixed-point counts alone miss (Figure 2(c); Appendix E.3). The all-five-point check for high-fidelity successes and the assessment of extra latent directions are reported in Q1 below.

For memory tasks, we likewise verify fixed-point positions and stability types. Minimal-achieving models recover all memory corners, while retrained models recover a larger fraction of the reference saddle arrangement than their parents (Figure 3(c)).

W2. Histogram truncation and high-fidelity classification

Reviewer comment

Appendix D.2 constructs a histogram with 30 bins per dimension over the target trajectory's coordinate ranges. It excludes generated samples outside those ranges, adds 10−51{0}^{-5} to each cell, and normalizes. The excluded fractions are not reported. Eqs. (15) and (A40) classify high fidelity using $E_{\mathrm{stsp}}<2.0$, so the criterion depends on the smoothed, normalized in-range histogram without explicitly penalizing out-of-range probability mass. Figure A2 provides representative trajectories but does not quantify this omission's effect on the overall success rate. Reporting excluded fractions and accounting for this mass would establish whether the classifications and method comparison are robust.

Response

Neither correcting the histogram boundaries nor accounting for excluded probability mass changes the high-fidelity classification of any checked 3-scroll model. This additional classification check, reported here, supports the original comparison; Q2 below gives the success rates.

For this robustness check, we recomputed the errors after correcting the bin-indexing error and accounting for generated probability mass outside the target range. For retrained 3-scroll models, fixing the indexing error increases the median error by approximately 0.014; additionally accounting for out-of-range mass increases it by a further 0.007. These corrections therefore have only a small effect on the reported errors (Appendix H). The excluded fractions for parents, retrained models, and directly trained models are reported in Figure A14(a), while Figure A14(b) compares the original and mass-corrected errors.

The original threshold ($E_{\mathrm{stsp}}<2.0$) was specific to 3-scroll Chua. With the expanded benchmark set, a single fidelity threshold would not provide a comparable standard across tasks, while choosing separate thresholds would introduce additional task-specific choices. We therefore report structural success and reconstruction or task error separately in the revised manuscript (Figures 2(c) and 3(c); Appendix D.3).

AuthorsOpenReview ↗

Response to Reviewer WJxk (2/3)

W3. Structural selection versus other linearization sets

Reviewer comment

Section 3.1 selects reduced structures by separating the parent's FP symbols; actual dynamics are established through retraining, as Section 3.2 and Appendix A.3 explain. Appendix E.2 and Figure A1 fix the selected structures and parent initialization while varying teacher information, guidance losses, and synchronization. They support guided retraining, but do not establish an advantage for the proposed selection over other masks retaining the same number of ReLUs. A comparison with random masks of the same capacity, matching parent parameters, teacher guidance, and training budget, would test the central structural addition.

Response

Linearization sets selected from the reduction tree achieve higher success and fixed-point positional recovery than matched off-tree sets under both tested retraining protocols (Figures A5 and A10). The comparison holds parent initialization, remaining ReLU count, and retraining settings fixed. Off-tree sets are randomly sampled outside each parent's selected candidate collection, preventing overlap between the selected sets and the controls (Appendices E.5 and F.3).

The advantage holds with parent-state guidance and abrupt linearization, as well as with gradual linearization without auxiliary guidance. Thus, the improvement is attributable to which ReLU units are linearized, beyond using the same parent and retraining procedure. Q3 below gives the numerical comparisons.

M1. Parent counts: 26 versus at least 27

Reviewer comment

Section 4.2 reports 26 parents with a five-symbol candidate, including 25 with a three-ReLU, five-symbol candidate. Appendix E.2 attributes the additional four-ReLU candidates to two parents whose trees reach five symbols only at four ReLUs. These statements require at least 27 parents with five-symbol candidates. Please reconcile the counts using per-parent candidate records and state whether the parent-balanced rates need updating.

Response

The original Appendix E.2 incorrectly described both parents contributing four-ReLU candidates as reaching five symbols only at that capacity. The original candidate records show that one contributes 12 four-ReLU candidates only, whereas the other contributes three four-ReLU and five three-ReLU candidates. Thus, 26 parents contribute five-symbol candidates, including 25 with three-ReLU candidates; only one parent contributes exclusively four-ReLU candidates (Table A2; Appendix B.5).

We have removed the incorrect description. Rechecking the original aggregation confirms that the error affected only the wording: the calculations used the correct candidate records, and the original parent-balanced success rates require no correction.

M2. Approximately 71% versus 73% and the threshold-scan pool

Reviewer comment

Figure A3 reports approximately 73% success at threshold 2.0, whereas the main capacity-matched comparison reports approximately 71%. Appendix E.3 should specify the candidate pool used for this threshold scan, as Appendix E.2 does for the guidance ablation.

Response

The original threshold scan used all 141 five-symbol candidates, including 15 with four ReLUs, whereas the main capacity-matched comparison used only the 126 three-ReLU candidates. The approximately 73% and 71% rates therefore refer to different candidate pools, which the original Appendix E.3 did not clearly distinguish.

The revised manuscript explicitly distinguishes the candidate pools used across analyses. Table A2 in Appendix B.5 summarizes the selected cohorts, and each analysis identifies the candidates included in its comparison. For example, the main 3-scroll comparison uses the 126 three-ReLU candidates (Figure 2(c)), whereas the guidance ablation includes all 141 candidates (Figure A7; Appendix E.6).

AuthorsOpenReview ↗

Response to Reviewer WJxk (3/3)

Q1. Matching errors, joint recovery, stability types, and extra latent directions

Reviewer question

For models classified as high-fidelity minimal successes, what are the one-to-one matching errors in observation space between the five fixed points in visited regions and the Chua targets? Please give a tolerance and the fraction meeting it. If skeleton preservation includes local dynamics, how are matched stability types and extra latent directions assessed?

Response

All 136 models satisfying the original high-fidelity criterion among the 141 models retrained with the revised protocol recover all five 3-scroll Chua targets in both position and local stability type. All six qualifying direct P = 3 models also pass: the fraction meeting the joint requirement is therefore 100% in both groups. Under one-to-one matching in observation space, the largest positional error is 0.145 times the minimum target separation, below the tolerance of 0.25 times that separation. This complete five-point verification is additional evidence reported here.

We assess local stability by comparing the numbers of unstable directions in the target dynamics and the model’s full 20-dimensional Jacobian. After matching three model eigenvalues to the target spectrum, we separately examine the remaining 17 directions (Figure A4; Appendix E.4). Across the 3-, 4-, and 5-scroll Chua systems, the additional latent modes are predominantly contractive: extra unstable directions occur at no more than 1.3% of position-matched fixed points, and no near-neutral modes were identified under the numerical checks (Appendix E.4).

The revised manuscript also reports joint position-and-local-type recovery across all retrained models, including unsuccessful ones: 99.4%, 96.1%, and 78.9% for 3-, 4-, and 5-scroll Chua, versus 17.8% for Dadras (Figure A3; Appendix E.4). These checks reveal discrepancies on Dadras that fixed-point counts alone miss.

For memory tasks, minimal-achieving models recover all prescribed memory corners as stable fixed points, and retrained models recover a larger fraction of the reference saddle arrangement than their parents (Figure 3(c)).

Q2. Excluded fractions and mass-corrected classifications

Reviewer question

What fractions of autonomous samples fall outside the target histogram range for directly trained models, reduced/retrained models, and parents? How do the high-fidelity classifications and method comparison change when these probability masses are explicitly included?

Response

For 3-scroll Chua, accounting for out-of-range mass leaves the checked high-fidelity classifications and method comparison unchanged. The excluded fractions for parents, retrained models, and directly trained models are shown in Figure A14(a), Appendix H; Figure A14(b) shows the resulting error correction.

At the original threshold $E_{\mathrm{stsp}}<2.0$, the original-protocol high-fidelity rate remains 73.45% across all 141 candidates. For models retrained with the revised protocol (combined guidance with gradual linearization), applying the original high-fidelity criterion yields parent-balanced success rates of 80.23% for the 126 three-ReLU candidates and 83.56% for all 141 candidates, compared with 20% for direct training at P = 3. None of these rates changes after either histogram correction. These results are from an additional robustness analysis reported here.

Q3. Same-capacity random control and target correspondence

Reviewer question

With the same parameters copied from the parent, teacher guidance, and training budget, how does the proposed selection compare with random masks retaining the same number of ReLUs? Please apply the same correspondence check with the target fixed points to both groups when comparing success rates.

Response

We compare tree-selected linearization sets with random off-tree controls matched for parent initialization, remaining ReLU count, retraining settings, and training budget. Controls exclude each parent’s selected candidate collection to prevent overlap. Within each task, fixed-point recovery is assessed over all produced models using the same one-to-one correspondence criterion for both groups.

For attractor reconstruction, selected sets achieve 71–82% minimal-realization success on 3- and 4-scroll Chua, compared with 11–21% for off-tree controls. Fixed-point positional recovery also favors the selected sets under both tested retraining protocols (Figure A5; Appendix E.5).

For memory tasks, selected sets likewise achieve higher memory-state recovery and higher position- and type-verified recovery of the reference fixed-point arrangement under both protocols (Figure A10; Appendix F.3).

Together, these controls identify the reduction tree’s selection of ReLU units to linearize as a central contributor to the gains in structural success and fixed-point recovery.

Reviewer_nEANOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a procedure for obtaining compact piecewise linear dynamical representations with almost linear recurrent neural networks (AL-RNNs). The authors first train a parent model, represent its visited ReLU activation patterns as a symbolic transition graph, and then construct a hierarchy of quotient graphs by replacing selected ReLUs with identity activations. The theoretical component observes that, under nonsingularity, isolation, interiority, and visitation assumptions, a linear region contains at most one fixed point. Therefore, representing QQ distinct fixed points requires at least QQ fixed point containing symbols and at least ⌈log⁡2Q⌉\lceil\log_2 Q\rceil ReLU units. On a three-scroll Chua system with five fixed points, direct training with the minimum P=3P=3 ReLU units achieves high-fidelity minimal realizations for 6/30 seeds. The proposed pipeline starts from parents with P=10P=10 ReLUs and reports an approximately 71% parent-balanced high-fidelity minimal-realization rate after reduction and retraining.

Critical strengths
  • The paper identifies a meaningful distinction between representational sufficiency and optimization: a minimal architecture may exist but is difficult to learn directly.
  • The proposed reduction has an intuitive dynamical interpretation. Linearizing a ReLU removes a switching boundary, while the fixed-point constraint protects part of the parent’s local dynamical structure.
  • The manuscript is clearly written.
Critical weaknesses
  1. Unclear why we need this additional expense. The paper describes the parent as a “large” model, but all reported models retain the same latent dimension M=20M=20. The parent has P=10P=10 ReLU coordinates, while the reduced model has P=3P=3; the other seven coordinates are made linear rather than removed. Thus, “large” means higher nonlinear switching capacity, not necessarily a substantially larger parameter count or state dimension. The resulting procedure is not a computationally economical way to obtain a minimal model as it requires: training many parents, generating a 400,000 step teacher trajectory, constructing the reduction hierarchy, retraining many reduced candidates, repeatedly evaluating long autonomous rollouts. The paper doesn't demonstrate that the procedure is more efficient than allocating the same compute to repeated direct training, continuation methods, or simpler parent-to-student distillation.

  2. Cannot find end-to-end computational cost. Table A1 reports only reduction-tree construction. But this table does not tell us about the dominant costs of parent training and candidate retraining (in wall clock time).

  3. Scalability of the proposed procedure is not convincing, thus, it leaves us with question on practicality of the proposed algorithm.

  4. Limited empirical evaluation. There is only single model considered, which is 3-scroll Chua system. This system has five known, well seperated fixed points, which may be a favorable example for this paper. However, it should have at least one or two more qualitatively different systems to support the paper's broader claims.

  5. Weak Novelty. The paper does not adequately compare its method against the most relevant neighboring ideas, such as but not limited to: ordinary pruning, geometric pruning for dynamical reconstruction, and random ReLU linearization.

Questions for authors
  1. What is the total end-to-end compute required by the pipeline, including parent training and all candidate retraining?
  2. Do the learned fixed points match the target fixed points in position, stability type, and eigenspectrum?
  3. How quickly do the number of admissible candidates and total retraining cost grow with PparentP_{\mathrm{parent}}?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nEAN (1/3)

Note:

  • The supplementary ZIP includes a PDF copy of this response with identical content (rebuttal_nEAN.pdf).
  • It also includes the revised manuscript with changes from the original submission highlighted in yellow (ICLR2027_AL_RNN_rebuttal_all_changes_highlighted.pdf).

Response to Reviewer nEAN

Thank you for highlighting the need to distinguish structural simplification from computational savings and to evaluate the selection rule and recovered dynamics more directly. We have expanded the evaluation beyond 3-scroll Chua to include 4- and 5-scroll Chua, the Dadras 4-wing system, and 2- and 3-bit flip-flop memory. We also added matched linearization controls, retraining ablations, fixed-point correspondence and local-dynamics analyses, and computational-cost comparisons across all six tasks.

1. What is reduced, and why incur the additional training expense? — Weakness 1

Reviewer comments

Thus, “large” means higher nonlinear switching capacity, not necessarily a substantially larger parameter count or state dimension.

Unclear why we need this additional expense.

The paper doesn't demonstrate that the procedure is more efficient than allocating the same compute to repeated direct training, continuation methods, or simpler parent-to-student distillation.

Response

Higher capacity here means more nonlinear ReLU units at a fixed latent dimension. Our objective is to find symbolically minimal dynamics, rather than reduce parameter count (Section 2.1).

The additional parent-training expense helps identify which reduced models to train, rather than only estimating a starting capacity (Figure A12). Naive direct search (Protocol D1) trains models at successive capacities. Parent-informed direct search (Protocol D2) estimates a fixed-point-based capacity lower bound, but direct training first succeeds above that bound on 4- and 5-scroll Chua and both memory tasks. Our method (Protocol O) uses the learned parent dynamics to construct reduction candidates and retrains them in ascending quotient-symbol count, avoiding a separate capacity sweep.

The cost comparison includes preparation for each method (Table A4; Figure A13; Appendix G). Our method's estimated expected first-success costs remain approximately 94–105 h across the four attractor tasks. Protocol D2 costs approximately 111, 331, and 429 h for 3-, 4-, and 5-scroll Chua. On Dadras, our method finds a minimal model at approximately 103 h, whereas Protocol D2 finds none in the evaluated capacity sweep costing approximately 734 h. The benefit extends to input-driven memory: our method costs 2.03 and 1.96 h for 2- and 3-bit memory, compared with 5.51 and 7.47 h for Protocol D2 (Figure A13(b)).

The matched controls test whether selecting units through the reduction tree improves performance when both groups are retrained using the same parent as a teacher (Figures A5 and A10; Appendices E.5 and F.3). With parent-state guidance and abrupt linearization, our selected reductions attain 82% and 73% minimal-realization success on 3- and 4-scroll Chua, versus 16% and 11% for alternative unit sets, with parent initialization, remaining capacity, and retraining settings matched. The advantage also holds under gradual linearization without auxiliary guidance and on memory tasks.

AuthorsOpenReview ↗

Response to Reviewer nEAN (2/3)

2. Runtime, all-candidate expense, and comparison with repeated direct training — Weakness 2 and Question 1

Reviewer comments

Cannot find end-to-end computational cost. Table A1 reports only reduction-tree construction. But this table does not tell us about the dominant costs of parent training and candidate retraining (in wall clock time).

What is the total end-to-end compute required by the pipeline, including parent training and all candidate retraining?

Response

The table below reports the estimated cost of producing every selected retrained model, including parent training and reduction. Teacher-cache construction is also included for the attractor tasks. This additional accounting is reported here; it differs from the first-success costs in Table A4.

TaskSelected candidatesTotal estimated cost (summed run-hours)
3-scroll Chua141633.5
4-scroll Chua102456.7
5-scroll Chua10118.3
Dadras26182.9
2-bit memory525.44
3-bit memory374.44

Although retraining costs 25–28% more per run in the controlled 4-scroll Chua measurements, our method achieves lower estimated first-success costs on 4- and 5-scroll Chua and both memory tasks, even after including parent training and reduction (Table A4; Figure A13). Thus, the benefit lies in the search for a minimal model, rather than faster individual training.

Controlled sequential measurements on 4-scroll Chua separate the costs of individual stages (Appendix G.1). Direct training took 5,650–5,826 s, retraining 7,228–7,304 s, and parent training 5,703 s. Reduction, including trajectory generation, graph construction, search, and analysis figures, took 47–87 s. Generating and storing the reusable parent trajectory took about 34 s per parent. Memory unit costs come from separate uncontended measurements, also reported in Appendix G.1.

The search-cost estimates include preparation costs for both methods and combine observed model outcomes with per-run cost estimates (Appendices G.2–G.3). Reported hours sum estimated individual run durations rather than elapsed calendar time, and exclude final success-verification time.

3. Scalability and growth of the candidate/retraining budget — Weakness 3 and Question 3

Reviewer comments

Scalability of the proposed procedure is not convincing, thus, it leaves us with question on practicality of the proposed algorithm.

How quickly do the number of admissible candidates and total retraining cost grow with P_parent ?

Response

The admissible search grows rapidly: increasing the parent's ReLU count from P = 10 to 14 raises the median number of admissible reduction-tree nodes from 476 to 10,616 and median tree-construction time from 0.2 to 10.2 s (Table A1; Appendix B.4). Exhaustive enumeration scales combinatorially with the number of nonlinear units P, rather than directly with the latent dimension. The representational capacity also increases with P: P ReLU units can define up to 2^P symbolic regions. Increasing P generally improves reconstruction or task performance in our experiments (Figures 2(b) and 3(b)), and parents with P = 10 suffice to obtain symbolically minimal models through reduction and retraining on all six tasks. Thus, although exhaustive search becomes expensive at large P, the parent capacities used in these experiments remain within a computationally manageable range.

Our goal is to find a minimal model, not to retrain every admissible candidate in the reduction tree. For example, the memory experiments retain at most four distinct candidates per parent, prioritizing the smallest quotient-symbol count and then the fewest remaining ReLUs (Table A2; Appendix B.5). Retraining cost therefore depends on the selected candidates and their per-run costs, rather than requiring retraining of the entire tree. The first-success search in Appendix G additionally stops once a minimal model is found. We have not measured total retraining cost as a function of parent capacity.

AuthorsOpenReview ↗

Response to Reviewer nEAN (3/3)

4. Relationship to pruning and the role of the structural selection — Weakness 5

Reviewer comment

The paper does not adequately compare its method against the most relevant neighboring ideas, such as but not limited to: ordinary pruning, geometric pruning for dynamical reconstruction, and random ReLU linearization.

Response

We use off-tree random linearization controls to avoid overlap with the selected candidate collection. Unrestricted random sampling may draw a linearization set already selected by the reduction tree, so we exclude those sets when sampling the controls reported in Appendices E.5 and F.3. Each control matches the parent initialization, remaining ReLU count, and retraining settings, while changing which units are linearized.

On Chua, minimal-realization success favors our selected sets under both protocols (Figure A5):

Retraining protocol3-scroll Chua: selected / alternative4-scroll Chua: selected / alternative
Gradual linearization, no auxiliary guidance80% / 21%71% / 14%
Abrupt linearization, guidance toward the parent's full state82% / 16%73% / 11%

Fixed-point positional recovery also favors the selected sets under both protocols. The memory experiments show the same advantage in recovery of memory states and the reference fixed-point arrangement (Figure A10). Thus, the selection rule contributes beyond parent initialization, remaining capacity, and retraining settings.

We do not benchmark weight-pruning methods, including geometric pruning, because connection sparsity is not our optimization target. We seek to reduce nonlinear switching capacity and symbolic complexity. The matched ReLU-linearization controls directly test whether the reduction tree selects units that can be linearized while preserving the target dynamics.

5. Evaluation beyond a single autonomous system — Weakness 4

Reviewer comment

There is only single model considered, which is 3-scroll Chua system. This system has five known, well seperated fixed points, which may be a favorable example for this paper. However, it should have at least one or two more qualitatively different systems to support the paper's broader claims.

Response

The revision now evaluates six tasks: 3-, 4-, and 5-scroll Chua, Dadras 4-wing, and 2- and 3-bit flip-flop memory (Sections 4.1–4.2). The memory tasks require persistent states and input-driven switching.

Minimal-realization success improves across the tasks (Figures 2(c) and 3(c)):

TaskDirect trainingReduction-guided retraining
3-scroll Chua23%82%
4-scroll Chua3%49%
5-scroll Chua3%8%
Dadras 4-wing0%15%
2-bit memory48%92%
3-bit memory29%78%

These results show that the benefit of reduction-guided retraining extends beyond a single favorable attractor to systems with more fixed points and to input-driven memory. In particular, our method enables minimal realizations on challenging tasks where direct training yields few or no successful models, as illustrated by 4-scroll Chua and Dadras.

6. Fixed-point positions, local type, eigenspectra, and extra latent directions — Question 2

Reviewer question

Do the learned fixed points match the target fixed points in position, stability type, and eigenspectrum?

Response

The new analysis (Figure A3; Appendix E.4) confirms strong recovery of both fixed-point positions and local stability types on Chua. Joint recovery reaches 99.4%, 96.1%, and 78.9% for the retrained 3-, 4-, and 5-scroll models, respectively. Dadras is less faithfully recovered, at 17.8%.

The additional latent dimensions are predominantly contractive in the Chua models. Extra unstable directions occur at no more than 1.3% of position-matched fixed points, and no near-neutral modes were identified under the numerical checks (Figure A4). Among retrained models achieving minimal realization, matched-eigenvalue discrepancies are smaller for Chua than for Dadras (Figure A4(c)).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and responses

We thank the reviewers for their feedback. The revision provides four main additions addressing their concerns:

  • Broader validation. We expanded evaluation from 3-scroll Chua to six tasks, adding 4- and 5-scroll Chua, Dadras, and 2- and 3-bit flip-flop memory. Reduction-guided retraining improves minimal-realization success across all six (Figures 2 and 3).

  • Benefit of structural selection. With parent initialization, remaining capacity, and retraining settings matched, linearization sets selected from the reduction tree outperform random off-tree controls in both structural success and fixed-point recovery (Figures A5 and A10).

  • Cost of finding a minimal model. In searches reconstructed from the observed runs, our method reduces estimated expected first-success costs by approximately 2.7–4.3-fold relative to parent-informed direct search on 4- and 5-scroll Chua and both memory tasks, including parent training and reduction. On Dadras, our method finds a minimal model while direct search finds none within the evaluated runs (Table A4; Figure A13).

  • Correspondence and robustness. New checks show strong recovery of fixed-point positions and local stability types on Chua (Figures A3–A4). We quantify excluded histogram mass and its effect on reconstruction error (Figure A14); accounting for this mass leaves every checked 3-scroll high-fidelity success determination unchanged (response to Reviewer WJxk, Q2).

We would appreciate the reviewers’ and Area Chair’s consideration of these additional results and clarifications in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.