- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PsychoNet, which combines Phasor Blocks for constructing complex-valued features with Deep Visual Coding (DVC), a learnable frequency-sub-band filtering module inspired by psychovisual image coding. The framework replaces deeper spatial processing with frequency-domain representations. Classification experiments show competitive accuracy with fewer layers in larger configurations. Salience visualizations and a proposed Distinct Part Rate (DPR) metric suggest more spatially distinct object-focused activations than ResNet baselines.
Critical strengths
- The combination of learned complex-valued features and structured spectral filtering provides an interesting architectural approach. Component ablations support the usefulness of DVC and spectral branching for classification.
- The paper investigates representations beyond accuracy, combining visualizations with quantitative localization analysis. It also transparently reports practical runtime and memory limitations, making the architectural tradeoffs easier to assess.
Critical weaknesses
- The evidence does not yet establish the claimed semantic abstraction or transparent reasoning. DPR measures multiple salient regions within an object mask, rather than correspondence to annotated semantic parts or their causal importance to predictions. Higher DPR could therefore reflect fragmented activations rather than more meaningful representations. The psychovisual interpretation remains suggestive.
- The scaling comparison does not isolate the benefit of spectral processing from width, resolution, and architectural changes. Shallow, widened spatial baselines would strengthen this claim. Moreover, fewer layers do not translate into practical efficiency: Psycho-Eff-B approximately matches ResNet-101 accuracy and FLOPs but achieves roughly half its throughput (Table A.3). The small ImageNet accuracy gains also lack reported run-to-run uncertainty.
Questions for authors
- Can you validate the part-based interpretation using semantic part annotations or targeted feature/region interventions? How sensitive is the DPR advantage to its thresholds and the salience method?
- How does PsychoNet compare with shallow, widened spatial networks under matched parameters and FLOPs, and with a comparable global spectral-filtering baseline?
- Are the ImageNet improvements, particularly the 0.25-point gain from full Phasor Blocks in Table 3, consistent across training seeds? Does removing these blocks also reduce DPR under the same evaluation protocol?
2 comments
Response 2/2 - Spectral processing from width, resolution, and architectural changes
Spectral processing from width, resolution, and architectural changes
We thank the reviewer for asking whether PsychoNet's accuracy at reduced depth comes from its spectral processing rather than width or other factors. We address this with new shallow-wide ResNet baselines and extra seeds, followed by throughput.
"The scaling comparison does not isolate the benefit of spectral processing from width, resolution, and architectural changes. Shallow, widened spatial baselines would strengthen this claim."
Q2 "How does PsychoNet compare with shallow, widened spatial networks under matched parameters and FLOPs, and with a comparable global spectral-filtering baseline?"
We have trained shallower and widened ResNets at similar layer count, parameter count and FLOPs as Psycho-Eff-B/L (see Global Response, 3. Shallow-wide ResNet baselines), and Psycho-Eff-B/L outperform them on CIFAR-10, CIFAR-100 and ImageNet-100. Regarding resolution, the base PsychoNet models keep 14×14 features for DVC (Sec. 4.2), which is why we designed the Psycho-Eff variants to downsample to 7×7 like ResNet at matched FLOPs. The shallow-wide ResNets are matched to these Eff models. For global spectral filtering, Table 3 replaces DVC's sub-bands with a single global filter, similar to GFNet's, within the same architecture (Models B, D), and DVC performs better.
"The small ImageNet accuracy gains also lack reported run-to-run uncertainty."
Q3a "Are the ImageNet improvements, particularly the 0.25-point gain from full Phasor Blocks in Table 3, consistent across training seeds?"
Every model in Table 1 now has three runs on CIFAR-10, CIFAR-100 and ImageNet-100 and, as each ImageNet-1K run takes at least two days, two runs on ImageNet-1K (GR Table 6). On ImageNet-1K, Psycho-S/B/L/H and Psycho-Eff-S remain ahead of their ResNet baselines, while Psycho-Eff-B/L are level with ResNet-101/152 within one standard deviation.
The Table 3 ablations are single runs, so the 0.25-point gain cannot be tested for significance. As it is close to the 0.13-point difference between our two Psycho-L (Model A) runs, we consider it suggestive rather than conclusive. The clearer contribution of the Phasor Blocks is the part-based representation shown above.
"Moreover, fewer layers do not translate into practical efficiency: Psycho-Eff-B approximately matches ResNet-101 accuracy and FLOPs but achieves roughly half its throughput (Table A.3)."
At matched FLOPs and parameter counts, Psycho-Eff-S/B/L's current inference throughput is indeed lower than their ResNet counterparts' (Table A.3). We believe this is largely because complex-valued operations are less optimised than real-valued ones in current hardware and deep learning frameworks, and improving PsychoNet's runtime efficiency is an important direction for future work.
Response 1/2 - Part-based representations
We thank the reviewer for their careful review, and for their interest in our psychovisual approach. We address their two concerns, the evidence for part-based representations (Weakness 1) and the scaling comparison (Weakness 2), and answer each of their questions below.
Evidence for part-based representations
We thank the reviewer for asking whether PsychoNet's salient regions reflect semantic object parts rather than fragmented activations, which we address with new part-annotation results and our ablation models (Global Response, 1).
We have validated the part-based interpretation against human-labelled semantic part masks from PartImageNet (see Global Response, 1. Quantitative evaluation of object parts), using two metrics. (1) Annotated DPR, a stricter version of the original DPR metric, only counts an image if its salience peaks fall on at least two different labelled parts, rather than just anywhere on the subject. (2) A linear part-segmentation probe predicts the labelled part masks from a block's frozen features, without salience maps. At relative depths 0.59–0.73, Psycho-B/L score higher than ResNet-101/152 on both metrics. This supports that their salience peaks and features follow semantic object parts rather than arbitrary fragments.
Yes. We evaluated the Table 3 ablation models on the original DPR under the same protocol (ImageNet-S50), as well as on annotated DPR and the linear part-segmentation probe on PartImageNet (see Global Response, 1.3 Phasor Block ablation). Replacing the Phasor Blocks with ResBlocks reduces all three metrics at relative depths 0.59–0.73, both with and without the Spectral Branches (Models A vs. C and B vs. D).
We varied the prominence threshold δ (0.1 and 0.3, default 0.2) and the on-object threshold τ (0.3 and 0.7, default 0.5) one at a time, on both ImageNet-S50 and PartImageNet. Psycho-B's advantage over ResNet-101 at the mid-to-late depths holds under every setting, for both DPR on ImageNet-S50 and the new annotated DPR on PartImageNet. Regarding the salience method, the linear part-segmentation probe (see Q1a) uses neither salience maps nor thresholds, and also favours Psycho-B over ResNet-101 at relative depths 0.59–0.73.
Table: DPR on ImageNet-S50 and annotated DPR on PartImageNet under different thresholds, Psycho-B vs. ResNet-101, averaged over relative depths 0.59–0.73. Δ = Psycho-B − ResNet-101. All differences have p < 1e-24 (paired Wilcoxon signed-rank test).
We agree that our submitted evidence was largely qualitative, and we will narrow these claims in the revised abstract (see Global Response) and conclusion. The semantic structure we now claim is at the level of object parts, which we measure directly against PartImageNet part masks, as above. For DVC, a new inference sparsity experiment (see Global Response, 2. Deep Visual Coding) shows that its output keeps most of its accuracy with 99.5% of its frequency coefficients removed, whereas its input loses accuracy sharply beyond 90%. This suggests that DVC's learned filters concentrate task-relevant information into a small set of frequencies, similar to how the original psychovisual codes preserve only the perceptually salient ones. We therefore view DVC as organising the part features by scale and filtering their frequencies, and PsychoNet's abstraction as formed jointly by its Phasor Blocks and DVC. Regarding transparent reasoning, the Phasor Blocks already offer some transparency, as they decompose objects into parts in a way similar to human vision, which is thought to recognise objects by their parts. However, we have so far found salience methods difficult to adapt to DVC's frequency-domain features, so understanding how the model uses these parts in its decisions remains future work.