- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors are proposing UniPDE-Bench that is evaluation protocol for learned reconstruction of partially observed PDE systems. Instead of relying only on reconstruction error, the authors analyze the singular-value decomposition of a noise-whitened, state-normalized observation Jacobian. The resulting singular vectors identify joint variations of the hidden state that are relatively well or poorly constrained by observations. Using this geometric reference, the paper evaluates recovery accuracy along retained directions, agreement between confidence rankings and observation sensitivity.
Critical strengths
1- The paper highlights an important limitation of conventional reconstruction benchmarks that is low error does not imply that the measurements uniquely constrain the reconstructed state.
2- Separating recovery capability, geometric agreement, and selective prediction quality is a strength that I like (illustrated in Table 2). The framework avoids equating poor uncertainty ranking with poor state reconstruction.
3- The use of a complete right singular-vector basis is appropriate, particularly when the observation dimension is smaller than the hidden-state dimension.
4- The experiments include broad ranges. It includes controlled inverse problems, six neural estimator families, five learning configurations, five random seeds, and ID/OOD evaluation.
Critical weaknesses
1- The main geometric construction is based on a normalized Jacobian, singular-value decomposition, and threshold-based subspace partitioning. These are established tools in inverse problems and observability analysis (although usage is useful, but combination of well known existing tools).
2- The proposed sCRS automatically assigns zero to non-abstaining predictors, regardless of their reconstruction accuracy. The penalty choices and fixed operating points require stronger justification and sensitivity analysis (that I could not find in appendix).
3- The retained subspace depends on the selected threshold, state metric, and linearization point. Basically we need either ablation or sensitivity analysis (that I dont find in your main nor appendix) to establish whether its main conclusions remain stable under different choices.
4- The retained subspace dimensions vary substantially across sensor configurations, directly affecting recovery scores and therefore, differences in these scores may reflect changes in the evaluation space rather than degradation in model performance (any idea to alleviate?).
5- The framework penalizes claims along poorly observed directions even when predictions are accurate and statistically reliable. To me it remains unclear whether geometric abstention improves practical predictive reliability compared with conventional uncertainty-based selection or not.
6- The experiments focus on three simulated PDE systems with only 17 state coefficients, where rank deficiency is often guaranteed by the observation dimensions. Larger and more realistic inverse problems could strengthening applicability of your approach (I am of course not deciding based on num of experiments, nor ask you to do so in rebuttal as the time is limited. I rather want intuitive explanation about how such extension is possible and later be reflected as limitation in main body).
7- The evaluation would benefit from clearer experimental specifications and broader sensitivity studies covering recovery thresholds, numerical discretization, and confidence-selection criteria.
Questions for authors
I tried to embed my questions in the weaknesses. Please see them above.