- Rating
- 2
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Learnable Lens Networks (LLN), an architecture that alternates linear propagation and coordinate-wise Gaussian refraction in an augmented position–angle space. The paper characterizes the invertibility, volume preservation, and symplecticity of the implemented latent transport and evaluates prediction error and costs on simulated dynamical systems.
Critical strengths
-
The propagate–refract construction assigns computational roles to the position and angle states, making the architecture straightforward to understand and analyze.
-
The paper provides an explicit inverse and analyzes volume preservation, the conditions for symplecticity, and the resulting reciprocal singular-value pairing.
Critical weaknesses
-
The core transport mechanism of LLN is closely related to established reversible additive coupling and symplectic shear constructions (e.g., RevNets and SympNets). The coordinate-wise Gaussian lens appears to provide an incremental parameterization of this existing framework.
-
Table 1 reports accuracy for selected deployments but FLOP ratios for a separate matched-width configuration. These counts also exclude exponential/tanh costs, leaving the practical efficiency of the evaluated models unclear.
-
The provided manuscript does not specify the Gaussian center locations, bandwidth, or lens-coefficient initialization sufficiently to reproduce the transport. This also makes it difficult to assess the sensitivity of the results to the fixed dictionary.
Questions for authors
-
How does LLN’s end-to-end inference latency compare with the baselines?
-
Could the authors specify these settings and report sensitivity to the center range and bandwidth?
5 comments
W2 and Q1: Practical efficiency and end-to-end inference latency
Thank you for identifying this presentation issue. To complement those counts, we provide measured end-to-end inference latency, which includes actual execution of the Gaussian exponential and angle-initialization tanh operations.
To further assess LLN's practical efficiency relative to closely related structure-preserving architectures, we extend the latency comparison beyond MLP baselines to include G-SympNet [1] and R-VPNet [3]. Comparing these implementations on the same reported hardware and inference settings directly tests whether LLN offers a latency advantage over related symplectic and volume-preserving models, rather than only over generic dense networks.
Table R1. End-to-end inference latency (ms; lower is better). D=40, H=512, batch size 1, FP32, and three initialization seeds. Bold marks the lowest reported latency in each hardware/thread column. This is a separate runtime benchmark, not a timing measurement of every manuscript Table 1 deployment.
| Model | L20 GPU0 (ms) | L20 GPU1 (ms) | CPU, 1 thread (ms) | CPU, 8 threads (ms) |
|---|---|---|---|---|
| LLN | 0.200 | 0.203 | 0.896 | 0.566 |
| Plain MLP | 0.434 | 0.439 | 5.159 | 5.128 |
| Scaled ResMLP (Res-S) | 0.437 | 0.439 | 5.111 | 5.086 |
| G-SympNet [1] | 0.410 | 0.415 | 0.937 | 0.951 |
| R-VPNet [3] | 0.596 | 0.599 | 1.182 | 1.201 |
| HNN-RK4 | 1.203 | 1.209 | 24.690 | 25.045 |
| NeuralODE-RK4 [2] | 0.759 | 0.761 | 17.418 | 17.365 |
| FNO-1D [4] | 4.281 | 4.287 | 652.930 | 221.861 |
LLN has the lowest reported latency in all four settings. On L20 GPU0, it takes 0.200 ms, compared with 0.434 ms for Plain MLP, 0.437 ms for Scaled ResMLP, 0.410 ms for G-SympNet, and 0.596 ms for R-VPNet. These correspond to approximately 2.17×, 2.19×, 2.05×, and 2.98× comparator-to-LLN latency ratios. The related-family comparators are referenced to SympNets [1], VPNets [3], NeuralODEs [2], and FNO [4]; the labels identify the implementations evaluated here.
These measurements support practical efficiency for the tested implementations and configuration. We hope the above response addresses your concerns.
- Jin et al. (2020). SympNets: Intrinsic Structure-Preserving Symplectic Networks for Identifying Hamiltonian Systems. Neural Networks, 132, 166–179.
- Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. (2018). Neural Ordinary Differential Equations. Advances in Neural Information Processing Systems (NeurIPS).
- Zhu, A., Zhu, B., Zhang, J., Tang, Y., and Liu, J. (2022). VPNets: Volume-preserving neural networks for learning source-free dynamics. Journal of Computational and Applied Mathematics, 416, 114523. DOI: 10.1016/j.cam.2022.114523.
- Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2021). Fourier Neural Operator for Parametric Partial Differential Equations. International Conference on Learning Representations (ICLR).
W3 and Q2: Gaussian-dictionary settings and sensitivity
We thank the reviewer for this suggestion. We now report the dictionary settings and provide an experiment that varies the center range and bandwidth independently to assess their effects on prediction accuracy.
Dictionary settings. Each lens uses Gaussian basis functions . The default dictionary places centers uniformly in , including both endpoints, and uses bandwidth . Thus, for and for . The centers and bandwidth remain fixed during training. The coefficients are initialized independently from and learned separately for each layer, coordinate, and basis function.
The center range determines coverage in latent position coordinates, rather than the physical input domain, while the bandwidth controls the response width and overlap between neighboring basis functions. With the default range, the center spacing is , giving . This rule reduces the bandwidth as more centers are introduced while maintaining overlapping responses. We use it as a default configuration, without assuming that it is optimal for every system.
Experimental setup. We fixed , , and , and evaluated all nine combinations of
with centers uniformly spaced in . Both factors therefore cover half, the default, and twice their default values. Bandwidth was varied independently of the center range, allowing us to examine their individual effects and interaction.
Within each system, the parameter count, training procedure, and optical total step scale were unchanged. All runs used 20 epochs of Adam, batch size 1024, zero weight decay, no learning-rate schedule, and one-step MSE training. Evaluation used 500-step autoregressive cumulative normalized MSE.
The results show that the preferred dictionary depends on the system. For example, with bandwidth fixed at , increasing from 2.5 to 5 to 10 reduces the mean error on ReactionDiff-32 from 0.00705 to 0.00374 to 0.00268. The same change does not benefit ReactionDiff-48, whose corresponding errors are 0.00574, 0.01592, and 0.01371. A larger center range is therefore not uniformly better.
The results also show an interaction between range and bandwidth. On Burgers-48, at , reducing from to increases the mean error from 0.20251 to 2.41695. However, the same narrow bandwidth paired with gives the lowest mean in the grid, 0.10865. This illustrates why the two parameters should be examined jointly.
The default configuration is within a factor of two of the lowest grid mean on eight of the ten systems; the exceptions are Lorenz-28 and ReactionDiff-48, with ratios of approximately 2.51 and 3.94.
We will include these settings and the full sensitivity table in the revision. The results suggest that LLN has further room for improvement through validation-based tuning of the dictionary settings. This does not change our main conclusions, which are supported by the reported configurations and do not rely on the dictionary being optimal or insensitive to these choices.
W1: Relation to reversible coupling and symplectic networks
Thank you for raising this distinction. We agree that additive coupling, invertibility, and symplectic shear composition are established ideas; our contribution is the particular parameter-efficient and inspectable transport architecture, not a new general principle of reversibility or symplecticity. LLN combines fixed linear propagation with a learned coordinate-wise refraction field:
\theta_{l+1}=\theta_l+\phi_l(y_{l+1}),\qquad L=T/D.$$ The implemented ordering is a drift–kick symplectic-Euler-type composition. This connects LLN to reversible coupling and SympNets [1]. The design fixes the propagation map, separates displacement from directional correction, and uses a learned initial direction. Its structural properties should not be presented as uniquely Gaussian. The Gaussian parameterization makes each directional correction explicitly decomposable. For coordinate j, $$\phi_{l,j}(y)=\sum_{k=1}^{K}a_{l,j,k} \exp\left[-\left(\frac{y-c_k}{b}\right)^2\right].$$ Here cₖ specifies "where" a basis response is concentrated, b specifies "how" broadly it acts, and $a_{l,j,k}$ specifies its signed strength. For a given trajectory, each term's contribution to the angle update can be evaluated separately, then related to subsequent displacement. This provides a concrete way to inspect local refraction and layerwise transport, rather than relying on optical terminology alone. The precise settings are given in W3/Q2 below. With fixed centers and bandwidth, each layer learns HK coefficients rather than a dense H×H transformation, yielding an O(DHK) rather than O(DH²) depth-dependent parameter term. This is a compact parameterization when K is small relative to H. 1. Jin et al. (2020). [*SympNets: Intrinsic Structure-Preserving Symplectic Networks for Identifying Hamiltonian Systems*.](https://arxiv.org/abs/2001.03750) Neural Networks, 132, 166–179.