acceptodds
Withdrawn from ICLR 2027

Infinitesimal Operator Discovery for Continuous-Depth Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continuous-depth neural networks replace a finite stack of layers with a learned vector field, creating a natural mathematical view of deep computation but obscuring which infinitesimal transformations are used after training. This work introduces Infinitesimal Operator Discovery (IOD), a diagnostic framework that analyzes the local variational dynamics of a trained continuous-depth model by projecting its Jacobian action onto constrained structural coordinates: diagonal modulation, topology-aware local mixing, low-rank global mixing, sparse nonlocal mixing, and an explicit residual that quantifies unexplained dynamics. The coordinates are deliberately structural rather than full architectural blocks, so descriptive reconstruction is separated from causal evidence. The audit combines residual fit, null baselines, repeated random controls, decision-boundary interventions, dictionary ablations, solver checks, and rank/radius sensitivity. On a controlled structured benchmark spanning 10 task families and 80 task-seed replications, IOD obtains low residual energy (mean 0.0138), operator profiles that are distinct from untrained, label-shuffled, and random-matched controls, and consistent causal support for low-rank global operators. Local operators are also task-relevant in most settings, while sparse nonlocal and residual coordinates remain negligible in the present benchmark. Boundary-focused interventions have larger effects than high-margin interventions, indicating diagnostic value for decision-sensitive examples. These results support a focused contribution: continuous-depth models can be made more transparent by analyzing their infinitesimal operator structure through a conservative, causal, and reproducible diagnostic coordinate system.

Primary area:
learning theory
Keywords:
continuous-depth networks, Jacobian analysis

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_goa3OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Infinitesimal Operator Discovery, a post-hoc diagnostic for trained continuous-depth models. It sequentially decomposes local vector-field Jacobians into diagonal, topology-local, low-rank global, sparse nonlocal, and residual coordinates. It combines reconstruction budgets with null models, matched-random intervention comparisons, boundary-sensitive diagnostics, and numerical/dictionary checks. On ten controlled task families with eight seeds each, it reports mean residual energy 0.0138 and the strongest intervention support for low-rank global coordinates. The intended contribution is a structural diagnostic coordinate system, rather than identification of architectural modules or a new predictive model.

Critical strengths
  1. The paper addresses a useful interpretability problem for continuous-depth models and clearly distinguishes structural coordinates from identifiable modules. It explicitly warns that low reconstruction residual is not causal evidence and discusses basis and topology dependence (pp. 2–3, §3.1; p. 7, §5.6; p. 8, §6).
  2. The reported audit is broader than a reconstruction study: it includes untrained and label-shuffled baselines, repeated scale-matched random intervention controls, eight seeds per task, boundary comparisons, and solver/dictionary sensitivity checks. The principal task-level and component-level means are internally consistent across the tables (pp. 3–7, 10–11; Tables 1–7).
Critical weaknesses
  1. I could not reconstruct how the local intervention in Eq. (7) produces the loss increase used in Eq. (8). The original predictor is defined in Eq. (1), and §6 appropriately acknowledges that component removal is approximated through local Jacobian actions. A local perturbation or tangent approximation could be sufficient for this diagnostic aim; no globally integrable component or explicit vector-field factorization is required. However, the PDF does not specify the perturbation/state update and readout used to obtain the intervened loss, or whether this is a finite perturbed-prediction loss or a linearized surrogate. This leaves the central reported behavioral quantity insufficiently defined (p. 3, §3.2; p. 8, §6; Appendices B–I).
  2. The PDF provides the task families, 80 task-seed design, sequential projection framework, scale-matched random comparator, support threshold, and rank/radius/solver sweeps. It does not provide the concrete task generators and model/training dimensions/settings, default projection budgets and sampling settings, intervention grid and maximum-stable selection rule, random-control sampling/repetition details, or confidence-interval construction and per-task intervals. These particular omissions prevent independent interpretation of the loss comparisons and verification of the 10/10 and 8/10 support counts. The residual sweeps are useful but do not themselves establish that the causal support counts persist across those settings (pp. 3–5, 7–12).
Questions for authors
  1. Could you give the actual update/readout equations or short pseudocode from Eq. (7) to ΔL_m, including the perturbation initialization/scale and where the intervention is applied? Please specify whether the reported quantity is a finite perturbed-prediction loss or a local surrogate, and interpret the task-relevance claim accordingly.
  2. What are the intervention grid and maximum-stable selection rule, the random-matching norm/distribution and repetition count, and the seed-level CI method/confidence level? The per-task intervals underlying the support counts would make the stated threshold rule directly checkable.
  3. Could you point to the exact task/model/training and Jacobian-sampling/default-projection configuration, together with the order-sensitivity results mentioned in §3.1? A precise location in the stated anonymized supplement would suffice; the PDF gives the audit outline but not these instantiated settings.
Reviewer_A1g2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Infinitesimal Operator Discovery (IOD), a framework for interpreting continuous-depth neural networks by decomposing their Jacobian into diagonal, local, low-rank global, sparse nonlocal, and residual components. The authors evaluate the framework on 10 structured tasks and use causal interventions to investigate which components contribute to model predictions.

Critical strengths
  • Studying the Jacobian through different structural operators is an interesting and potentially informative approach to understanding the internal dynamics of neural networks. -The combination of Jacobian decomposition and causal interventions provides useful insights into the importance of different operators for model predictions.
Critical weaknesses

-The proposed framework is evaluated only on continuous-depth neural networks, which limits its demonstrated applicability. It is unclear whether the method is inherently restricted to continuous-depth architectures or could also be applied to ordinary discrete-depth neural networks. If the latter is possible, the paper provides no experiments investigating this setting or demonstrating the specific advantages of using continuous dynamics. -The writing and presentation could be significantly improved. The paper contains many short, scattered paragraphs, making the discussion feel fragmented and difficult to follow. This affects readability and makes it harder for readers to understand the overall motivation, methodology, and contributions.

Questions for authors

Is IOD inherently limited to continuous-depth neural networks, or could the same Jacobian decomposition be applied to ordinary discrete-depth architectures such as MLPs, ResNets, or Transformers? If it can be applied to these architectures, what specific advantages does the continuous-depth formulation provide?

Reviewer_wkXKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This method tries to diagnose a continuous-step model by investigating its Jacobian. It projects the Jacobian into different structural coordinates, such as diagonal, local, low-rank, and so on. It ran experiments on 10 task families. It found that low rank carries most of the behavior, followed by local decomposition.

Critical strengths

None

Critical weaknesses
  • This paper is so hard to understand. It's poorly written. It takes a lot of effort to read. The contributions are unclear. It took me effort to find the takeaways from the paper. And I couldn't find much. Most of the text is just an experimental report, listing numbers and confidence intervals without much insight.
  • There are few details about their models and their tasks. They only list the names of the tasks. I'm not an expert in this domain. I don't know if these are general, widely known tasks. But I can't tell what the 10 tasks are exactly from the names. And the paper never specified what models they are using: what architecture, what training target, etc.
  • And in the introduction, they mentioned many methods that can be seen as continuous depth models (continuous normalizing flows, latent dynamics for irregular time series, neural controlled differential equations, graph diffusion models, implicit depth models, and continuous-time Transformer). I don't know which model was used in the paper, or whether the findings hold for other kinds of continuous depth models. A clarification on this would make it more sound.
Questions for authors

Other than the problems in Weaknesses, here are more questions:

  • Could the authors summarize the contributions of this paper? The contribution paragraph in the introduction describes only what the paper did, not its useful findings. In other words, there is a question for reviewers: Is the paper informative? What can we learn from this paper?
  • Figures 6, 7, and 8 appear to repeat those in the main text.
Reviewer_CatfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Infinitesimal Operator Discovery (IOD), a diagnostic for trained neural ODEs that decomposes sampled Jacobians of the learned vector field into diagonal, local, low-rank, sparse and residual components, and tests each component’s task relevance through interventions compared against size-matched random perturbations. Experiments on 10 synthetic tasks report small residuals and the most causal support for the low-rank and local components.

Critical strengths

Interpreting what trained continuous-depth models compute is a relevant question. The audit design also appears to be careful in intent, as it keeps an explicit residual and compares against null models and size-matched random interventions. However, these do not outweigh the weaknesses below.

Critical weaknesses

W1. Insufficient detail to reproduce or assess the results.

Key details are missing, including the model architecture and hidden dimension, the task definitions, the training setup, the dictionary settings and how each projection is computed. The appendix mostly restates the main text rather than providing these details.

W2. Weak evidence and limited significance.

  • The energy profile (Fig. 2a) is broadly similar across tasks, and the causal results do not recover designed sparse structure (sparse components are supported on 0 of 10 tasks, including sparse edge XOR), while low-rank has the largest effect on every task. Table 4 also lists the expected structure without reporting the structure actually found. With low-rank alone raising the residual by only 0.035 (Table 3) and no order-sensitivity results reported, it is unclear whether the profiles reflect task structure or the chosen dictionary.
  • The interventions modify the Jacobian's action rather than the model, and the paper does not explain how this enters the forward computation or specify the intervention grid and what "maximum stable" strength means, so it is hard to tell what the causal effects measure.
  • The evaluation uses only small synthetic tasks, with no demonstrated practical use.
Questions for authors
  1. What are the model architecture, hidden dimension, task definitions and dictionary settings used, and how is each projection computed?
  2. How is the intervention implemented in the model's forward computation?
  3. Can the authors show IOD on a real task where it yields some actionable insight?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.