- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The goal is to choose a learned inverse solver not based on the immediate output, but the downstream task accuracy. This paper gives a review of this phenomenon in a setting where motion is estimated using a CNN from complex channel-state information, where down the road the velocity is used for an output image. Paper provides no novelty. The contribution is analysis of this phenomenon, which is evident in the literature of inverse problems, diagnostics, and boundaries.
Critical strengths
NA
Critical weaknesses
-
This challenge is not new. It is obvious that upstream accuracy is not always consistent with downstream task accuracy, something well-studied in decision-focused learning and TaskMet. The whole contribution or the message of the paper is to report ambiguity-aware error alongside state error and downstream error, standard in pose estimation for example.
-
The mechanism is textbook radar.
-
The paper presents empirical results showing a coherence-regularised learned estimator settles on alias branches. Then, the model ranking is reversed in one chosen sampling setting. No loss, no evidence, no metric is presented.
-
This paper is out of scope for ICLR. What is presented is a general evaluation mechanism using one synthetic physical system. No other task is evaluated or presented.
-
There is no method, no theorem or no novel phenomenon is presented. Paper has no novelty and contribution.
Questions for authors
NA
3 comments
The authors fails to address any of the concerns raised.
-
There is no revision uploaded. The paper at the time of the response by the authors is the same as the uploaded one. Therefore, " The revision adds a five-seed confirmation on 5,000 new scenes, a width sweep of the same architecture, hardware-like perturbations, an energy-matched amplitude intervention, and 30 new runs with a direct imaging-loss baseline." all these promised experiments don't exist for the reviewers to assess.
-
"No loss, no evidence and no metric": It is not clear what is the contribution of the paper! Is it a new Loss, a new Metric, new evidence contrary to what is expected to happen? No new insight is given. To be perfectly clear, paper mentions that selecting by state accuracy can choose the worse image producing model, however, the one experiment that tests selection shows that the velocity RMSE picks the test-best imager in every seed. How can this be taken as a new loss or metric for choosing? or even The experiments shows against what the paper warns? Examples are:
- "....yet they are commonly trained, validated, and selected by state-space accuracy rather than downstream performance." (Abstract)
- "Such models are usually trained, validated, and selected with a state-space distance d_z (ˆ z, z), whereas deployment cares about dy (G(ˆ z), y)." (lines 048–049)
- "when coherent imaging is the endpoint, it alone can choose the worse image-producing model." (lines 463–464)
- "selecting among eight candidates per seed on 512 labelled selection scenes by velocity RMSE, folded range-rate RMSE, or direct image error chooses the same model, with zero paired sealed-test image difference and zero regret to the hindsight test-best candidate" (lines 373–376)
Result point contrary to what is motivated in introduction or the abstract. While the experiments does not support what is being motivated, no new loss or metrics is given, what is really the significance and novelty? "We do not propose a universally superior loss, a universal ambiguity metric, or a clutter-robust imaging system."
-
Where is appendix O exactly? There is only up to J!
-
Where does the paper exactly cite "symmetry-aware pose metrics"?
-
What paper promises is a general evaluation principle for learned inverse models, but test is only on "one controlled physical instantiation", no generalization to the cited systems, As an application paper, where no new insight or novelty ingredient is given, it falls below the line.
Revised PDF now uploaded; where each point is addressed
Thank you for the quick follow-up, and our apologies: the revised PDF had not been uploaded when we posted our responses. It is now on the paper page (33 pages). Below we list where each of your points is addressed, so that it can be checked directly.
Where the new material is. Appendix O (pages 32-33) is the direct imaging-loss baseline, and Section 5.5 (page 9) summarises it. Appendix L (page 28) contains the width sweep and the hardware-like perturbations, Appendix M (page 31) the five-seed confirmation on 5,000 new scenes, and Appendix N (page 32) the energy-matched amplitude intervention, summarised in Section 5.4 (pages 7-8). The symmetry-aware pose metrics (Hodan et al.) are cited in Related Work, page 3, with the reference on page 11.
On the selection result, we agree with you that the original text and the selection experiment were inconsistent, and we have corrected the text rather than the result. All four passages you quote overstated the claim and are no longer in the paper:
- "commonly trained, validated, and selected by state-space accuracy" (old abstract): removed;
- "usually trained, validated, and selected with a state-space distance" (old lines 048-049): removed;
- "it alone can choose the worse image-producing model" (old lines 463-464): removed;
- the selection paragraph (old lines 373-376) is rewritten in Section 5.5, page 8.
The selection experiment itself is unchanged and is now reported as a null result in four places: the last sentence of the abstract (page 1), the third contribution, "it distinguishes diagnostic agreement from selection utility" (page 3), Section 5.5 (page 8), and the conclusion (page 9). Section 5.5 also explains why it gives a null result. Picking one model from the pool only depends on the top of each ranking, and in that pool the best candidate (the residual CNN with alpha = 0.1) was best on both criteria, so every rule chose it.
The claim of the revised paper is narrower and is what the experiments show. When two learned estimators are compared, for example a coherence-regularised and a state-only CNN, velocity RMSE can order them the opposite way from image localisation. This holds on 3,000 sealed scenes for two architectures (Table 1, page 5; Figure 2, page 6) and reproduces with five seeds on 5,000 new scenes (Appendix M). Such pairwise comparisons are how training objectives are chosen in practice. The new imaging-loss experiment (Appendix O) shows that training directly on a differentiable image loss does not avoid the problem: that loss also selects distant sampled branches and localises worse than the coherence penalty. The contribution is therefore an evaluation finding and the conditions under which it appears, not a new loss or metric, and the paper says so.
On generality, we agree that a single physical system limits how far the result extends. The revised paper states this as a limitation in the introduction (page 3) and the conclusion (page 9), rather than claiming a general principle for all learned inverse models.
Response to Reviewer Xemj
Thank you for the review. Several statements in it do not match the content of the paper, and we would like to correct them.
The review states that no loss, no evidence and no metric is presented. The paper defines a coherence-regularised training objective with a sweep over its weight (Section 3.2), a folded path-rate metric (Section 3.4), and reports paired results on sealed test sets with confidence intervals. The revision adds a five-seed confirmation on 5,000 new scenes, a width sweep of the same architecture, hardware-like perturbations, an energy-matched amplitude intervention, and 30 new runs with a direct imaging-loss baseline.
The review also states that the reversal is shown in one chosen sampling setting. The paper reports four sampling rates (15, 31, 62 and 124 Hz). The 31-Hz setting was not chosen after the fact: it is where the textbook branch spacing predicts the effect in advance, and the other rates serve as contrasts. At 62 and 124 Hz the secondary error mode disappears, as predicted (Figure 3).
On novelty: we agree that a gap between upstream and downstream accuracy is expected in general, and the paper cites decision-focused learning, TaskMet and symmetry-aware pose metrics. The natural remedy suggested by that literature is to train on the downstream task directly. We tested this (Appendix O). A differentiable image-localisation loss also pushes velocity estimates onto distant sampled branches, yet it localises worse than the coherence-regularised model in every main comparison, by 0.036 to 0.140 m. That endpoint supervision does not fix the problem is, in our view, not obvious from the prior work. The paper also shows at the scene level where the RMSE penalty and the localisation gain come from, and which measurement content switches the reversal on and off.
On the radar mechanism: the ambiguity itself is textbook, and we describe it as such. We use the textbook branch spacing as a prediction and test it against the learned errors. The observed first-branch medians (0.81 and 1.67 m/s at 15 and 31 Hz) sit at 0.96 of the predicted spacing. What radar theory does not tell us is whether and when a trained network lands on those branches, and how this changes model rankings. That is what the experiments measure.
On scope: the paper is an empirical study of how learned estimators should be evaluated and compared, which is an ML question, studied in a physical system where the mechanism can be controlled. We do not claim a new algorithm or theorem. A single physical system limits generality, and we state this as a limitation.
Given that the central factual points of the review do not correspond to the paper, we would ask the reviewer and the area chair to take this into account.