- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper is asking a question about fourier light field microscopy (FLFM): whether a model that gives a better reconstruction of the 2D camera image is also better for reconstructing the underlying 3D structure. This is an important problem because in many settings researchers don't actually have access to the 3D structure and 2D is all there is for benchmarking and evaluation.
The authors propose an evaluation framework, SCOPE-FLFM, with two parts. The first organize candidate models into comparable groups: by architecture family, by training run, and by checkpoint. The second separates three kinds of evidence: fit to the camera views given to the model, prediction of camera views withheld from it, and agreement with known synthetic 3D objects.
Using a synthetic dataset and a small pool of learned models, the paper reports four findings:
- Agreement between scores is high when all models are pooled but weak among closely related models.
- The model selected depends on which score is used.
- Adding a uniform background makes a normalized image-similarity score go up while agreement with the true 3D object goes down.
- A small architectural modification (Calibra-LF) improves some metrics and worsens others.
I should say at the outset that I am not an expert in light-field microscopy. I read this paper as a member of the general ICLR audience. My comments are therefore about whether the paper communicates its contribution to that audience and whether it fits the venue. I do not question the correctness of the optics or the experiments.
Critical strengths
-
The title asks a clear question. "Does better reprojection mean better 3D reconstruction?" is easy to grasp, and the underlying concern, choosing models with a proxy score when the real target is not observable, is relevant well beyond microscopy (off the top of my head: self-supervised MRI and CT reconstruction, and most representation learning methods based on auto-encoders)
-
The experimental protocol appears careful. Selection rules are fixed before final evaluation, duplicate checkpoints are not counted twice, and the authors are explicit about the limits of their claims (App. A.24).
-
Results are reported openly. Unfavourable results, including for the authors' own variant, are reported rather than hidden under the carpet.
Critical weaknesses
- fit to ICLR: As far as I can tell, the paper does not propose a new learning method, representation, training objective or theoretical result. Its contribution is an evaluation protocol for one specific imaging modality, tested on a few closely related FLFM reconstruction networks. The paper doesn;t draw a lesson for the broader ML community. The question it studies is general: when can a self-supervised, measurement-space score be trusted to select models for an inverse problem? The same question comes up in MRI, CT, computational photography and image restoration. Yet the paper is framed entirely in FLFM terms, and its related work is almost exclusively about microscopy hardware and applications. The paper seems aimed at the computational microscopy community, which would probably be better served by a journal or conference in that area. To be interesting for ICLR, I think the paper would need to (a) state the general ML problem up front, (b) position itself against the ML literature on proxy-metric validation and model selection without labels, and ideally (c) show the phenomenon in more than one inverse problem.
- Density and opacity of the text:
- I found the main text very hard to follow, and I believe most ICLR readers would too. Some examples: Within the first few pages the reader hears about Comparable-Candidate Stratification (CCS), Contract-Separated Evidence Validation (CEV), "contracts", "evidence scope", "candidate scope", the family/run/state/file hierarchy, "rule aliases", Q_obs, Q_hold and Q_obj, endpoint-oracle and prospective regret, and "historically accessible groups". Most of these name fairly ordinary ideas: grouping models, holding out data, comparing against ground truth. The new vocabulary makes the paper feel more complex than the ideas behind it.
-Hard to parse sentences: "Prospectivity concerns this rerun on historically accessible groups" (Sec. 3.3), "files may duplicate weights, and states within a run retain trajectory dependence" (Sec. 3.1), "Cohort-dependent regrets diagnose choices and are unranked across rules" (Table A7), "Population diagnostics remain unranked" (Tables A5, A6). I had to reread these several times, and I am still unsure what some of them mean. -too much reliance on the appendix -Figure 2 is overloaded - it litteraly packs everything making it unreadable at normal reading size
-introduction doesn't set up the problem- much of the first few paragraphs list biological applications of FLFM, not the ML question.
-Meanwhile the actual problem setting (what we observe, what we reconstruct, what reprojection actually means) is never laid out. the abstract is almost unreadable for the non-specialist
I would strongly encourage a rewrite. It should open with a plain-language problem statement and a simple diagram of the pipeline (camera image → reconstruction → reprojection; compare against the camera, against withheld views, against the true object). It should use far fewer coined terms and give a self-contained account of the main experiment in the main text.
- Findings feel somewhat unsurprising:
- The title question. The answer is "not necessarily", and the paper says so in the intro: "Different fluorescence distributions in depth can produce similar projections." If many 3D objects can produce the same camera image, fitting the camera image well cannot guarantee recovering the right object. This is standard the world of inverse problems and not a new finding.
-Pooled vs. close-candidate agreement. Two scores can agree on ranking very different models (a strong family vs. a weak family) and still disagree when ranking near-identical checkpoints. That is a general property of correlations computed over mixed populations. The authors state that in App A.1
- The background counterexample. The image-similarity score rises when background is added because each image is normalized by its own maximum, which hides the change in absolute intensity. The paper notes that an unnormalized error measure moves in the expected direction in the same experiment. This reads to me more as a caution about one particular metric than as a general finding about reprojection-based selection.
-The selected model depends on the score used. This is almost true by definition when different scores measure different things.
None of these is wrong and doing the experiments to spell them out is useful. But I did not come away feeling like I learned something. If the authors feel that practicioners need to know about these results, they should spell it out, giving examples of recent peer-reviewed studies where people "get it wrong" and would give different results by using the authors' prescriptions.
Questions for authors
- Can you state, in two or three sentences and without the CCS/CEV terminology, what a practitioner should do differently after reading this paper?
- Which of your findings do you consider genuinely surprising, and to whom? Is there published work in FLFM or related modalities whose conclusions would change under your protocol?
- Do you expect the same phenomena in other self-supervised inverse problems (e.g., MRI or CT with measurement-domain validation)? If so, why frame the paper only in FLFM terms?
The problem in plain words
A Fourier light-field microscope records one camera image per acquisition, and a network turns that image into a volume. Where no real volume exists for a specimen, the only check available at deployment is a reprojection, the predicted volume pushed back through the instrument and compared with the camera. Everything a practitioner can see at the moment of choice lives in measurement space, so the question is whether a score computed there orders candidates the way the unobservable object would, and that ordering turns on who is in the pool.
The single-family pool
One reconstructor family, 48 training runs, one checkpoint per run, no two readings sharing a saved file. Your reading, that the pooled-versus-local gap is a general property of correlations computed over mixed populations, predicts one coefficient here. The pool returns two, on opposite sides of zero, with disjoint intervals.
T1. One family, one checkpoint per run, with the withheld score's upper bound below the supplied score's lower bound.
Supplied-view fit tracks recovered structure at +0.583, lower bound +0.340. Withheld-view prediction reads −0.042, upper bound +0.243, and the intervals are disjoint on candidates whose only difference is the training run. A pool with one population holds no family separation to exploit, and it still separates the two scores by more than half a correlation unit. We expected the opposite sign on every run, since withheld-view prediction is the score with less access to the answer, and the sign came out the other way. Across populations the withheld score agrees with the object at 0.823 on the 23-state mixed pool and at −0.042 with the family fixed, while the supplied score reads 0.831 and +0.583.
The price of a frozen choice
Freezing a choice on groups 20–29 and reading object Pearson on the known volumes costs 0.0721 under the supplied-view rule and 0.0890 under the withheld-view rule, against the known-volume optimum of the same 48 runs.
T2. Frozen selection, object Pearson given up against the known-volume optimum.
The withheld-view cost is positive in 99.8 percent of resamples, and the supplied-view cost reaches zero at its lower end. A definitional dependence produces no frozen deployment choice that is optimal for one target and 0.1542 away from optimal on the other, and the disagreement sits in the state choice inside one family, which is the decision a checkpoint rule makes.
Literature positioning
Miller et al. and Baek et al. establish that a label-free proxy tracks target accuracy across a pool whose members differ widely. SSDU, ZS-SSL, Noise2Self, Noise2Void and Laine et al. establish that a loss on withheld measurements stands in for a reference when choosing a stopping epoch or a hyperparameter, always inside a single training setting. Varma and Simon and Cawley and Talbot establish that scoring a model on its reporting data biases the number, and prescribe nesting and freezing, which we follow. D'Amour et al. and Wirgin establish that equally good fits can differ downstream, and each of these lines stops at the mechanism, with none reporting the size of the gap between a measurement score and an object score on a trained pool at fixed family, which is the pair in T1.
Population and ranking
The same coefficient moves with the population it is computed over, at 0.691 pooled across 23 states, 0.285 on the ten signed-plus-Calibra states, −0.100 on the five signed states, 0.886 on the six positive HYPER states, and 1.000 on the seven XLFMNet states. A coefficient that changes under a tightening of the population is measuring the population.
The practitioner rule
First, compute the ranking inside the candidate population you choose from, since a coefficient over a mixed pool is carried by family separation. Second, report supplied-view fit, withheld-view prediction and known-object agreement as three separate numbers, since one combined number averages a gain against a loss. Third, select on one target and verify on another, since a freeze on groups 20–29 read on 30–49 gives different deployment choices for different targets.
Other inverse problems
A CT or MRI reconstruction scored by its own measurement residual carries the same structure, and what has to be rebuilt is the whole path the score is computed through, the acquisition model, the projector, the noise contract and the known object.