- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce the Jacobian Lens Distance (JLD), a perceptual similarity metric derived from a pretrained vision encoder. By leveraging the network's Jacobian, the authors identify which directions in the early-layer feature space most strongly influence the encoder's output and use these directions to construct a distance metric between images. The authors then demonstrate that JLD outperforms several existing perceptual metrics on established benchmarks. Moreover, unlike methods explicitly trained for perceptual similarity, JLD is resolution-agnostic while remaining computationally efficient.
Critical strengths
-
The authors introduce a novel perceptual metric derived from a pretrained DINOv2 encoder and demonstrate that it outperforms several existing metrics on established perceptual similarity benchmarks. These results provide empirical support for the hypothesis that feature directions which strongly influence the encoder's output are also informative about human perceptual sensitivity.
-
The authors provide an insightful geometric analysis of their metric in pixel space, illustrating how its sensitivity varies across different image distortions and regions.
-
The paper is well written, clearly structured, and easy to understand and follow.
Critical weaknesses
-
The authors discuss perceptual similarity broadly, without clearly distinguishing between different types of similarity. Their evaluations primarily focus on low-level distortions, such as noise, blur, and compression, whereas metrics like DreamSim target higher-level variations, including style, viewpoint, and pose. Since the paper does not evaluate these higher-level aspects of perceptual similarity, the authors should better qualify their claims of superiority over existing perceptual metrics.
-
The authors show that existing perceptual metrics suffer from performance degradation when evaluated at resolutions different from those used during training. However, it is unclear why the images cannot simply be resized to the training resolution before computing similarity. I can see this being problematic when downsampling removes important fine details or small objects, but the authors do not investigate such cases. Without such an evaluation, the practical advantage of JLD's resolution robustness remains unclear.
Questions for authors
-
Have you evaluated JLD on benchmarks that focus on higher-level perceptual variations, such as DreamSim's NIGHTS dataset? If not, do you expect JLD to perform similarly well on these variations, given that the current evaluations primarily focus on low-level distortions?
-
Have you evaluated JLD on images where downsampling to a lower resolution would remove important visual information, such as fine details or small objects? Such experiments could help demonstrate the practical advantage of JLD's resolution robustness over metrics that typically operate at a fixed, lower resolution.
3 comments
Response to the Authors
Thanks you for addressing my comments. I will update my score accordingly. Nice job!
Thank You
Dear Reviewer dcqA,
We appreciate your prompt response, and we are happy to see that you are satisfied with our rebuttal.
Response to Reviewer dcqA
We sincerely thank the reviewer for the thoughtful review. We are glad that the reviewer finds the paper well written and easy to follow, the pixel-space geometric analysis insightful, and the results supportive of our hypothesis that output-influential feature directions are informative about human sensitivity. We address the questions bellow.
W1 / Q1
We thank the reviewer for this distinction. JLD is designed as a full-reference distance for low-level perceptual differences, the image-quality setting of TID2013, CSIQ, LIVE, KADID and PIPAL, and our claims are made there. Following the reviewer's suggestion we evaluated on NIGHTS: test split, 1,824 triplets, 2AFC accuracy, DreamSim's protocol (without any retraining).
With no training on human judgements, JLD is above both LPIPS variants and below DreamSim, which is trained on NIGHTS itself. This is the ordering we expect. The lens acts on early, local patch features, so it is specialised to low-level differences. NIGHTS varies pose, layout and semantics, which is DreamSim's target. On BAPPS 2AFC, which sits between the two regimes and is LPIPS's training set, JLD scores 0.658 (note that without training) against 0.668 to 0.691 for LPIPS, DISTS and DreamSim.
Following the suggestion, we will update our mansucript with this discussion on NIGHTS dataset.
**W2 / Q2: **
We thank the reviewer for this practical question. Resizing is a reasonable remedy, and our main comparison already does it to every baseline that is release. In Table 1, DISTS, DeepDC and DeepWSD are evaluated at their released 256-pixel short side and DreamSim at 224 × 224, while JLD can run at native resolution. The Table 1 result therefore does not depend on any resolution argument.
Section 5.4 asks a different question, how each metric behaves when it is applied directly at the given resolution. On TID2013, from a 256 to a 512-pixel short side, the lens term moves from 0.850 to 0.845. LPIPS-VGG, which has no resize rule, falls from 0.757 to 0.661, and DISTS falls from 0.815 to 0.717 when its resize is not applied (with it, 0.818 as in Table 1). We will state this protocol next to the examples in our manuscript to a better read.
The advantage of JLD is that it needs no such step. Resizing to 256 pixels discards content above that resolution, which is the fine-detail case the reviewer describes, and the error map it returns is at the reduced size. JLD has no fixed input size and returns a patch-level map at native resolution, which is extremely important in modern high-resolution compression and restoration applications.
On Q2: Following the reviewer's suggestion, we ran this test on rated high-resolution content: the 120 clips of AVT-VQDB-UHD-1 test 1 (4K sources with human opinion scores), scored on 1920 × 1080 frames, 32 frames per clip. SRCC with the opinion scores:
On this content the resize costs DISTS 0.074 and LPIPS 0.041. JLD at native resolution (0.903) is above DISTS, LPIPS and DreamSim at their reduced input sizes, and each difference is significant under a paired bootstrap. JLD itself changes by only 0.008. Applied natively, DISTS is the highest here (0.928). So the preprocessing that suits DISTS depends on the content, while JLD is used one way, at native resolution, and is accurate in both cases. This is the practical advantage we claim, and we will add the experiment to the paper.
Thank you again for the constructive review. We believe the qualified claims and the NIGHTS results make the paper stronger and more useful for the ICLR community. If these answers address the concerns, we would be grateful if the reviewer would consider raising the score. We are happy to answers more questions during the discussion phase.