- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes way to link ordinal ratings across graders using fixed gauge for symmetry invariance and series of heuristic checks based on Fisher information and graph connectedness.
Critical strengths
The ordinal linking problem between datasets seems underexplored in images, which appears to be the intended setting in the paper, although the experiments in the paper do not make clear if the OAI/MOST dataset is setup to be cross-dataset or just cross-reader. The proposed method scores favorably to the oracle threshold although missing experimental setup details make it unclear if this is an impressive result.
Critical weaknesses
The paper appears unfinished in places and it is unclear what insights or broader value to the ML community this paper brings. The writing contain many nonstandard and undefined phrases: "measurement invariant latent construct", "amortized encoding", "neural reparameterization direction", "scalar shared construct assumption", etc. The only dataset evaluated is specific to x-rays of knee osteoarthritis and I would like to see more evaluation on datasets of more interest to the AI community such as MOSAIQ-500K-Bench or UPIQ: Unified Photometric Image Quality for ordinal rating.
Questions for authors
-
What is the neural model? The paper is missing experimental details of the image uncoder, "designated reference population", metric definitions such as OAI QWK.
-
Why choose the OAI dataset of more relevant ML benchmarks?
-
Can you compare to any modern prior work in ordinal rating calibration/linking? What about the following?
-
Can you explain more the novel insight of the proposed method and why this task is difficult? It seems that ordinal rating calibration has a long history from psychometrics, inter-rater reliability, cumulative link models, etc. How is your setting different and what exactly is the task inputs and assumptions.