acceptodds
Withdrawn from ICLR 2027

DINOv3 Perceptual Loss with Token and Correlation Matching for Spike Image Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spike image reconstruction aims to recover visually meaningful images from asynchronous spike streams, but remains challenging in neuromorphic vision. Pixel-wise losses often fail to capture perceptual similarity, resulting in overly smooth or unnatural images. Perceptual supervision is therefore particularly important for this task, as it provides the structural, semantic, and texture information that cannot be directly inferred from raw spike signals. We propose a DINOv3 perceptual loss specifically designed for spike image reconstruction through complementary token and correlation matching. Built on self-supervised DINOv3 representations, class-token matching preserves global semantic consistency, patch-token matching retains fine-grained local details, and correlation matching maintains spatial structure and texture coherence. Together, these components provide comprehensive perceptual supervision at both the token and correlation levels. The proposed loss is used only during training and introduces no additional inference cost. Extensive experiments demonstrate that the proposed DINOv3 perceptual loss consistently improves CNN-, Mamba-, and Transformer-based reconstruction models and achieves state-of-the-art performance on the Spike-REDS and Spike-X4K benchmarks.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Spike Image Reconstruction, DINOv3 Perceptual Loss, Token and Correlation Matching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bdVEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a DINOv3-based perceptual loss for spike image reconstruction. It uses a frozen vision transformer to match class tokens, patch tokens, and normalized patch-token Gram matrices between reconstructed and reference images. The loss is applied to several reconstruction architectures and evaluated on paired synthetic benchmarks and real spike datasets.

Critical strengths
  1. The approach is practical and easy to integrate into existing reconstruction models. It improves the reported results for CNN, Mamba, and Transformer architectures without adding inference-time parameters or computation.

  2. The authors provide useful ablations of the loss design. Table 4 evaluates all seven nonempty combinations of the three terms, and the full objective achieves the best reported NIQE on both real datasets.

Critical weaknesses

The main weakness is the limited methodological novelty. The proposed loss combines class-token and patch-token MSE with Gram-matrix matching using a frozen DINOv3 backbone. These are conventional feature-matching objectives, and normalized patch Gram matching is already used in DINOv3’s Gram anchoring. Although the combination improves reconstruction performance, the paper does not clearly establish a new methodological contribution beyond adapting these existing techniques to spike image reconstruction.

Questions for authors
  1. What is the main methodological contributions?

  2. How much of the improvement comes from DINOv3’s pretrained representations vs the proposed loss combination? Would the same objective using DINOv2 provide comparable gains?

Reviewer_hfvEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a DINOv3 perceptual loss for spike image reconstruction. The method uses a frozen DINOv3 backbone to extract global features and local features from reconstructed images and ground truth images. It combines three loss components: a global loss for semantic consistency, a patch loss for local feature alignment, and a Gram loss for modeling inter-patch relationships.

Critical strengths

The paper addresses the limitations of pixel objectives in spike image reconstruction and explores the use of pre-trained visual representations to improve perceptual quality. The proposed loss can be incorporated into existing reconstruction architectures without changing their structure. Since DINOv3 is used during training rather than inference, the method has the potential to improve reconstruction quality without adding inference time computational overhead.

Critical weaknesses
  1. Using pre-trained visual representations as perceptual losses is an established direction. The paper needs to better distinguish its contribution from existing DINO-based perceptual losses and Gram objectives. A Gram matrix captures relationships between patch features but does not explicitly preserve their spatial coordinates. Similar inter-patch relationships do not necessarily imply correct geometric arrangement. The paper should better demonstrate how the proposed Gram term improves structural fidelity rather than merely feature correlations.
Questions for authors
  1. What is the precise methodological contribution beyond applying a pre-trained DINOv3 backbone as a perceptual feature extractor? How does the proposed formulation differ from existing DINO perceptual losses and Gram objectives? Since the Gram matrix does not explicitly encode patch coordinates, how do the authors ensure that the Gram loss preserves spatial layout rather than only inter-patch feature correlations? Have the authors tested spatially shuffled patches or other controlled structural perturbations?
Reviewer_HjbZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a DINOv3-based perceptual loss for spike image reconstruction. The proposed objective combines three components: global class-token matching, local patch-token matching, and Gram-based inter-patch correlation matching. The loss is applied during training using a frozen DINOv3 backbone and introduces no additional inference overhead. Experiments on several spike reconstruction benchmarks show improvements over the original reconstruction models, with a particularly noticeable PSNR gain on Spike-X4K.

Critical strengths
  1. The proposed approach is straightforward and can be integrated into existing spike reconstruction architectures without modifying their inference pipelines.
  2. The combination of global, local, and relational feature supervision is intuitively reasonable for image reconstruction.
  3. The authors conduct evaluations across multiple architectures and provide ablations on the individual loss components, weighting coefficients, and input resolution. The improvement on Spike-X4K is encouraging.
Critical weaknesses
  1. The core method consists of applying a pretrained DINOv3 feature extractor and combining class-token, patch-token, and Gram-matrix matching objectives. While the combination is reasonable, feature-space perceptual supervision and Gram-based matching are well-established techniques. The paper does not sufficiently clarify the methodological novelty beyond replacing a conventional perceptual network with a stronger pretrained representation. More importantly, comparing primarily against a VGG-based perceptual loss is insufficient to establish the contribution of DINOv3 or the proposed objective design. Comparisons with DINOv2, CLIP/ViT-based perceptual losses, and conventional feature-matching objectives under matched settings would be much more informative. In particular, it is important to distinguish improvements arising from stronger pretrained representations from those attributable to the specific three-component loss formulation.

  2. According to Table 1, incorporating the proposed loss improves SwinSF from 39.34 to 39.39 dB PSNR, from 0.9806 to 0.9810 SSIM, and from 0.0191 to 0.0181 LPIPS. Although these changes are consistently favorable, their magnitudes are small, and the paper does not provide results across multiple training seeds or measures of uncertainty. The gain on Spike-X4K is more substantial (39.61 to 40.58 dB PSNR), but the large discrepancy between the improvements on the two benchmarks remains unexplained. Without controlled repeated experiments, it is difficult to assess the robustness of the reported gains.

  3. The cross-architecture results in Table 3 are encouraging, but the baseline performance differs substantially across architectures. In particular, Spk2ImgMamba achieves only 15.15 dB PSNR, compared with approximately 38–39 dB for Spk2ImgNet and SwinSF. Although adding the proposed loss improves Spk2ImgMamba to 17.60 dB, the resulting reconstruction quality remains far below that of the other architectures. Such a substantial discrepancy warrants clarification regarding checkpoint selection, implementation details, preprocessing, and training convergence. Improvements over an unusually weak baseline do not necessarily demonstrate effectiveness across well-trained architectures.

  4. The authors argue that the Gram loss captures structural relationships rather than absolute spatial positions and is less sensitive to local mismatches. However, the proposed loss compares corresponding entries of two spatial patch-correlation matrices. Such an objective is not generally invariant to spatial permutations or displacements, and its claimed robustness to local mismatches is not demonstrated. Moreover, patch-token matching already encourages the preservation of relationships among features. While the ablation results suggest that Gram matching is beneficial, the paper provides limited analysis of the additional information it contributes beyond direct patch-feature alignment.

  5. Although the proposed method does not increase inference FLOPs, computing DINOv3 representations and dense patch-correlation matrices introduces additional training overhead. The Gram matrix has quadratic complexity in the number of patches. Training time, peak GPU memory, and the impact of image resolution should be reported. Additionally, evaluation on real-world datasets primarily relies on NIQE. Further perceptual metrics, human preference evaluation, or downstream task analysis would strengthen the claim that the proposed loss consistently improves perceptual quality.

Questions for authors
  1. Can the authors compare the proposed objective against DINOv2- and CLIP/ViT-based perceptual losses under identical training protocols, including matched feature-matching and Gram-matching configurations?
  2. Were the SwinSF baseline and the proposed variant trained using identical optimization schedules, initializations, and computational budgets? What is the exact total training loss, including the pixel-wise reconstruction component?
  3. Why is the baseline PSNR of Spk2ImgMamba substantially lower than those of the other reconstruction architectures? Were all methods evaluated with consistent input preprocessing, output normalization, and dataset splits?
  4. Can the authors report mean performance and standard deviation over multiple training seeds, particularly for the relatively small gains on Spike-REDS?
  5. What are the additional training-time and peak-memory costs of the proposed objective, especially at higher resolutions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.