- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that topological errors in thin-structure segmentation should be judged by their functional consequence for a specific task, not by pixel overlap or by a fixed topology loss. It defines a six-dimensional interpretable edit signature, learns task-conditioned weights over these signatures from held-out preferences or functional outcomes, and then freezes or cross-fits the learned utility into segmentation training. The central methodological point is a two-stage protocol that keeps utility estimation separate from segmentation training, so that segmentation gains cannot be used to validate the utility itself.
Critical strengths
- The problem framing is clear and relevant: overlap metrics and fixed topology losses do conflate error detection with task-dependent consequence.
- The two-stage protocol directly addresses the circular-validation risk, which is a real weakness of “learn the loss from the final segmentation result” approaches.
Critical weaknesses
- Core evidence is missing. Rank/conditioning, bootstrap intervals, matched-overlap AUROC, identical-ordering, and collinearity results are not shown. These are precisely the evidence needed for H1/H4, so the paper currently supports predictive association, not identification.
- Only one dataset is reported. DRIVE is the only results table; the crack benchmark appears only in the protocol. This is a serious gap for a paper whose main claim is task-conditioned utility: without at least two tasks with different orderings, the "task-conditioned" part is not directly demonstrated.
- No multi-task or cross-task evidence. The paper never shows two tasks producing distinct utility orderings, nor the killer experiment with identical true orderings. The central scientific claim is therefore asserted rather than tested.
- Limited technical novelty. The method is a linear softmax weighting over six hand-defined features plus a Bradley–Terry likelihood.
- Insufficient implementation detail. The task descriptor q, the feature map ϕ, the extractor algorithm and its hyperparameters, and the preference-collection procedure are not specified, which makes the method hard to reproduce or assess.
- Statistical reporting is thin. The protocol promises at least five seeds and bootstrap intervals, but the table reports only point estimates; there are no confidence intervals, effect sizes, or multiple-comparison corrections.
Questions for authors
- The paper states that rank and conditioning, bootstrap-interval, matched-overlap AUROC, identical-ordering, and collinearity results are required but not displayed. Can you provide these results, and do they support H1 and H4?
- What exactly is the task descriptor q, and what does ϕ(q) encode? Is it a one-hot task label, hand-crafted features, or a learned representation?
- How are the six edit signatures extracted? Which skeleton/correspondence algorithm and hyperparameters are used, and how sensitive are the results to this choice?
- What are the crack-benchmark results? If they are not available, why does the protocol declare them?
- How many preference labels and annotators were used, and what is the inter-annotator agreement?
- Why is only DRIVE reported when the protocol declares two benchmark families?