- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes OPINE, an architecture involving POD projection as a bottleneck layer between invertible nonlinear networks, and provides a methodological and theoretical perspective, coupled with demonstrations on synthetic data and turbulent data examples. The paper begins with problem framing and related work in the first two sections, introduces the method in section 3, with numerical results in section 4 and concluding remarks in section 5.
Critical strengths
- This paper is very well written, the text and figures are complementary and present the material in a clarifying way. I really enjoyed reading it.
- I think the structure the paper follows is excellent, and that the objectives and ablations are clearly stated (I have not read the appendix in detail however.) The use of POD as the only bottleneck mechanism pairs with the use of exactly-invertible networks, to produce a `good' hierarchical compression model for high-dimensional data.
- The step for training `through' the eigendecomposition is methodologically elegant, and that presentation is clear.
- The examples lead from toy synthetic settings to more complicated turbulent settings, showcasing the methodology; the authors note that the point is not to outperform an unconstrained autoencoder.
Critical weaknesses
- Overall, it is clear to me that, if one allows for a sufficient number of POD modes in the bottleneck layer, the invertible networks are near lossless, and would recover a good reconstruction, even if the latent space is not particularly meaningful; this is not only true for turbulent flows, its true for any dataset, it would also be true for image data sets of other sorts. Thus, as the bottleneck widens, the method is reasonably made for reconstruction. Is it possible that this comes at a price, and does not give good reduced coordinates for subsequent inference (of, e.g., a simple dynamical system)?
- Training invertible networks is often hard (due to stiffness) and costly; additionally, stacking a larger number of blocks does not automatically yield a more expressive model that is easy to optimize. I would be interested in understanding how expensive training OPINE is, compared to a comparable CNN or autoencoder.
Questions for authors
- Following the first bullet from above, a subsequent simulation demonstrating, as a proof of concept, that the reduced coordinates (especially in a small-coordinate-limit) are good latent coordinates for a reduced-order-model to evolve would strengthen the motivation for using the method. If the authors can address that firs bullet in another way, I will consider that evidence as well.
- I would like to see a comparison of the training cost between the proposed methods.
- Can the authors clarify the remark regarding training for a fixed ? I do not fully understand how performance degrades when one trains OPINE on a fixed and then evaluates on , vs. training and evaluating on . Is there a loss that can simultaneously ask for good performance for several values of D?
4 comments
Response to Reviewer CTnk (Part 2/2)
Q3. Training at and evaluating at , and a loss over several . We agree that our remark in the paper was too brief. A model trained at is evaluated at by keeping the first of its eigenvalue-ordered POD modes:
These reconstructions are exact nested projections, but the map is trained only through the projection at , so they are less accurate than a model trained at . How much depends on the system (Tables 6 and 7): on the minimal channel the prefixes are within a few percent of separately trained models, on Kolmogorov flow they are about twice as inaccurate, and on the synthetic families the gap can be much larger at small . The new Table D3 shows the general pattern: the further lies above , the closer the prefix comes to POD. Our sentence "one training run thus gives accurate models of every size" overstated this, and we have corrected it.
To the reviewer's second question: yes, a loss can ask for several at once, and we added one as an option (Appendix D.5):
where each is the inverse of the POD training NMSE at , so that every counts on the same scale ( is written in the paper). All prefixes come from one encoding, at the cost of one inverse pass per . It involves a trade-off: it makes the intermediate prefixes several times more accurate, at some cost in accuracy at and in training time (Tables D3 and D4). On KS, one model trained this way is more accurate, averaged over all , than ordered autoencoders trained with the same loss, although autoencoders trained separately at each small remain more accurate.
We thank the reviewer again for a review that changed how we present the method.
Follow-up on Revised Experiments and Remaining Concerns
Dear Reviewer CTnk,
Thank you for your constructive suggestions. We wanted to briefly follow up during the discussion period. In the revision, we have added the reduced-order dynamics experiment you suggested, quantified the training-cost trade-off, and introduced and evaluated a multi-rank training objective.
We would appreciate your feedback on whether these additions address your main concerns and whether they warrant an updated assessment.
Thank you for your time and consideration.
Response to Author Rebuttal
I thank the authors for their detailed responses and their willingness to engage with my comments. The new material and answers largely resolve my concerns, and I will be raising my score to a 3.
Response to Reviewer CTnk (Part 1/2)
We thank the reviewer for the careful review and for the kind words about the writing and about training through the eigendecomposition. The main question, whether accurate reconstruction comes at the price of poor reduced coordinates, is the right one to ask of this method, and we spent much of the revision on it. New results are in Section 4.7 and Appendix D of the revised PDF, marked in red.
Weakness 1 and Q1. Does the reconstruction come at the price of poor reduced coordinates? Our short answer is that we found no such price, but also no dynamical advantage. At small D the OPINE coordinates are meaningful and support a reduced-order model as well as POD and autoencoder coordinates do; they do not make the dynamics more predictable. What they add is a transformed field of the size of the data that keeps all of its information and can be inspected directly. In more detail:
Small D. The reviewer is right that a wide bottleneck makes reconstruction easy. This is why we compare all methods at small, equal D; in OPINE the rank-D projection is always the only lossy step. The gain comes from that projection, not from the map alone: with the same map, keeping the first D entries of z is usually no more accurate than POD, whereas the POD projection of z is much more accurate (Tables 1 and 3). On Kuramoto–Sivashinsky (KS), the error drops sharply between D=6 and D=8 (Table 2), the inertial-manifold dimension estimated by Linot & Graham (2020).
Meaning of the coordinates. They are ordinary POD coordinates of the transformed variable, ranked by variance, and the reconstructions at all D are exactly nested. Decoded to physical space, the leading directions look like POD modes, with the same shape, ordering and pairing (Figure D4 for KS and Burgers, Figure 5 for Kolmogorov flow); the nonlinear map makes the responses to positive and negative amplitudes differ.
A reduced-order model, as you suggested. We trained the same model ηt+1=ηt+gϕ(ηt) in POD, OPINE and MLP-autoencoder coordinates on KS, with the same architecture search, training and seeds (Section 4.7, Appendix D.9). All long rollouts are stable. At D=8 the model in OPINE coordinates reproduces the joint distribution of (ux,uxx) better than the models in POD and autoencoder coordinates, and as well as models with D=16. We want to be clear about the limits of this result: its short-time error is lower only because the forecast starts from a more accurate reconstruction, and its forecasts do not stay accurate longer than in POD coordinates.
What OPINE offers instead. We think this is another highlight of the method, which could be useful. Because the map is invertible, the transformed variable z=fθ(x) is a field of the same size as the data that keeps all of its information (similar to using vorticitiy or streamfunction to express the same field), and the reduction is an ordinary POD of that field. One can therefore look at exactly what the reduced model works with. The new Figures D5 and D6 show z, aligned with the data, next to the physical field. On KS at D=8 the two are almost indistinguishable. On Kolmogorov flow, z keeps the large vortices but moves variance from the energetic scales to smaller ones, and less so as D grows (Appendix D.8).
Weakness 2 and Q2. Training cost. On the turbulent flows OPINE costs more to train than the CNN autoencoder: on one V100 GPU a minimal-channel run takes about 14 h against 1.2 h, with several times the memory, and on Kolmogorov flow training takes about 2.5 times as long (Table D1). On KS and Burgers a run takes about a minute, somewhat longer than a tuned MLP autoencoder and much less than a tuned CNN autoencoder (Figure D1).
On stiffness and depth: with the coupling log-scales bounded and the damped update, we observed no divergence in any OPINE run, and a stack with seven times fewer weights is nearly as accurate as the larger one (Table 5). One OPINE model also gives every smaller D (Q3), which matters for the total cost when several dimensions are needed.