- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks what nearby PDE could explain a solver's output. It ranks physically scaled changes to coefficients, forcing, boundaries, and geometry. When causes overlap, the method returns an explanation set. A separate residual grid chooses a candidate for re-solving. I like the workflow, though the strongest guarantees need closer scrutiny.
Critical strengths
The ambiguity treatment is the best part for me. Indistinguishable causes remain together in the output. The exact-collinearity controls make that behavior concrete.
I appreciate the matched residuals, dictionaries, and independent re-solves. The paper also separates repeated solver seeds from independent PDE instances. That makes the empirical comparisons more credible.
Critical weaknesses
The dictionary still carries a lot of prior knowledge. A small remainder means the declared mechanisms explain the field. I'd want real debugging cases before treating that as evidence of the actual fault.
The certified method sometimes says nothing. Every RFM certificate cell has zero accepted diagnoses. That's a serious practical limit, even though abstaining is the honest response.
The search costs substantially more without winning consistently. Median inference takes 0.0467 seconds versus OMP's 0.0019 seconds. I'd struggle to justify roughly 25 times the cost in transfer settings where OMP is stronger.
Questions for authors
Does the nonlinear FDM advantage survive numerically stabilized BIC and stopping rules? Please explain why adding noise raises OMP accuracy from 36.5% to 98.8%. That could materially change my assessment.
Are explanation-set scores calibrated per task or per independent PDE? I'd want one randomized task per PDE or a valid grouped-calibration argument. The selective-risk certificate doesn't automatically establish explanation-set coverage.
How does Theorem 1 handle nested supports and noisy admissibility boundaries? Please distinguish constrained fitting from fitting first and rejecting afterward. I'd need the guarantee aligned with the implemented procedure.
Where does set-based action selection outperform OMP with the same validation layer? I'd want an unplanned solver-fault case where retaining several explanations changes the intervention.
2 comments
Thank you for the detailed response and the revised manuscript. The constrained fitting procedure and updated recovery proof address my concerns about physical admissibility, and using one randomized task per independent PDE resolves my concern about the calibration unit. The stabilized BIC experiments also explain the original noiseless FDM anomaly. OMP with the same validation still matches the reported repair success rates, so the practical advantage remains unclear. I will raise my score in light of the substantive revisions and additional evidence.
Official Comment by Authors
Thank you for recognizing the ambiguity treatment and matched evaluation. Your comments helped strengthen the comparisons and clarify their limits.
W1: Dictionary prior knowledge and debugging evidence.
Thank you for emphasizing application evidence. A small remainder means compatibility within the dictionary. New public-source tests cover FiPy's face-coefficient cutoff, scikit-fem's plane-stress formula and FiPy's flux/gradient configuration, alongside SciPy. Direct reproductions and controlled adaptations are labeled separately. FiPy stabilization is proposed, not upstream-merged. Dictionaries/action catalogues still encode prior knowledge. These are retrospective tests, not blind production faults (Appendix O.1, p. 48).
W2: RFM abstention.
Thank you for highlighting this limit. We retain original abstention and align the set score with the existing point null gate before independent study B. With mean-five width-384 RFM, 512 certificate and 384 test PDEs, certified acceptance is 357/384 at all three noise levels. A joint 108-rule audit gives risk upper bounds 6.72/6.70/7.30%, with 18/17/18 accepted-test errors. OLS with the same score has comparable acceptance. Low-fidelity physical-score rules still fail certification. This modified pipeline requires higher fidelity and does not retroactively certify the old rule (Table 39, pp. 45–46).
W3: Search cost.
Thank you for requesting complete costs. We retain the 24.6-fold legacy search ratio and tight-budget failures (Table 34, p. 41). New small-dictionary RFM search medians are 2.97/1.15 ms for structured/OMP. Set/path validation plus re-solving take 94.57/92.53 ms, including assembly and search, excluding input/reference construction (Table 42, p. 47). Matching physical-score OLS has lower search cost. We recommend selection by regime and budget (Sec. 6, pp. 9–10).
Q1: Stabilized baselines and the noise anomaly.
Thank you for requesting this check. Stabilized BIC removes the original noiseless FDM advantage. The legacy baseline refits each extension (OLS-style), although labeled OMP. Near-roundoff RSS makes logarithmic BIC over-select. Noise avoids this regime rather than improving identifiability. Three floors raise 210/576 to 576/576, while noisy cells remain 569/576 (eight tasks on 72 PDEs). Actual stopping tests on 512 fresh PDEs give 100% noiseless accuracy for stabilized OLS and structured selection (Table 26, p. 35 and Table 32, p. 39). New complete-path correlation OMP with the same physical score gives 306/384 versus structured 360/384 in M5, but OLS also reaches 360/384 (Table 35, p. 42).
Q2: Calibration unit and coverage target.
Thank you for this distinction. Each A/B study has 128/256/512/384 independent tuning/calibration/certification/test PDEs, one pre-randomized task each. Profiles/noise are paired. Scores, generators and gates are fixed before calibration. High-fidelity B coverage is 95.31–95.57%, mean size 1.00. Low-fidelity coverage is 90.10–90.63%, mean size 1.39–1.61 (Table 37, p. 44). Missing truths receive infinite excess scores, without deletion. Legacy pooling is descriptive. Per-PDE maxima target simultaneous task coverage (Table 31, p. 38). Risk certification does not establish set coverage.
Q3: Nested supports and noisy admissibility.
Thank you for identifying this gap. Fits now optimize within joint physical bounds. Theorem 1 (p. 5) uses feasible residual images. Nested supports can have zero separation, requiring an actual penalty preference. Non-null output also must pass the fixed null gate. On 128 paired PDEs, the closest-boundary/highest-noise joint case reduces true-support exclusions from 54 to 0 and raises correct diagnoses from 70 to 119 (Table 30, p. 37). The old rejection rule additionally needs an interior margin accounting for noise amplification and ridge bias (Proposition 6, p. 32). Old fits do not inherit the new guarantee.
Q4: Sets versus OMP with the same validation.
Thank you for requesting matched interventions. The new external tests do not establish superiority over full OMP with the same validation. In the flux adaptation both repair 24/24, with OMP cheaper (0.0668 versus 0.3725 s). In SciPy extensions, sets rescue structured point selection from 16/24 to 24/24, while OMP also repairs 24/24 (Table 43, p. 49 and Table 45, p. 50). RFM repair-success counts also match. Identical complete sets/validation give identical actions. The supported benefit is retaining and testing alternatives, with blind production debugging still open.
Thank you again for your careful review and constructive suggestions.