acceptodds
Under review as a conference paper at ICLR 2027

Deep Psychovisual Image Representations

Abstract

Psychovisual models suggest human vision decouples low-level feature extraction from higher cognition by first forming intermediate abstractions. In contrast, deep learning-based vision models routinely extract and aggregate features using homogeneous stacks of spatial layers, rendering their decision-making processes opaque. In this paper, we propose Deep Visual Coding, a learned frequency-domain representation inspired by 1990s image codes that quantised perceptually salient frequencies, which together with complex-valued image representations produces psychovisual-style abstractions. This approach enables the first psychovisual-based deep learning framework, utilizing data-driven spectral filters that learn to encode task-relevant semantic structures within distinct frequency sub-bands. Salience analyses reveal that our psychovisual models extract highly interpretable object parts compared to the amorphous regions produced by regular Convolutional Neural Networks (CNNs). Furthermore, we find that our models are less depth dependent than CNNs for model scaling, since our complex-valued representations and learned abstractions subsume the role of the deep spatial layers. Together, these findings demonstrate that psychovisual coding provides a promising path toward more efficient and transparent vision models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
psychovisual, visual codes, frequency domain, Fourier transform, representation learning, deep learning, vision, human vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1vJaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PsychoNet, which combines Phasor Blocks for constructing complex-valued features with Deep Visual Coding (DVC), a learnable frequency-sub-band filtering module inspired by psychovisual image coding. The framework replaces deeper spatial processing with frequency-domain representations. Classification experiments show competitive accuracy with fewer layers in larger configurations. Salience visualizations and a proposed Distinct Part Rate (DPR) metric suggest more spatially distinct object-focused activations than ResNet baselines.

Critical strengths
  1. The combination of learned complex-valued features and structured spectral filtering provides an interesting architectural approach. Component ablations support the usefulness of DVC and spectral branching for classification.
  2. The paper investigates representations beyond accuracy, combining visualizations with quantitative localization analysis. It also transparently reports practical runtime and memory limitations, making the architectural tradeoffs easier to assess.
Critical weaknesses
  1. The evidence does not yet establish the claimed semantic abstraction or transparent reasoning. DPR measures multiple salient regions within an object mask, rather than correspondence to annotated semantic parts or their causal importance to predictions. Higher DPR could therefore reflect fragmented activations rather than more meaningful representations. The psychovisual interpretation remains suggestive.
  2. The scaling comparison does not isolate the benefit of spectral processing from width, resolution, and architectural changes. Shallow, widened spatial baselines would strengthen this claim. Moreover, fewer layers do not translate into practical efficiency: Psycho-Eff-B approximately matches ResNet-101 accuracy and FLOPs but achieves roughly half its throughput (Table A.3). The small ImageNet accuracy gains also lack reported run-to-run uncertainty.
Questions for authors
  1. Can you validate the part-based interpretation using semantic part annotations or targeted feature/region interventions? How sensitive is the DPR advantage to its thresholds and the salience method?
  2. How does PsychoNet compare with shallow, widened spatial networks under matched parameters and FLOPs, and with a comparable global spectral-filtering baseline?
  3. Are the ImageNet improvements, particularly the 0.25-point gain from full Phasor Blocks in Table 3, consistent across training seeds? Does removing these blocks also reduce DPR under the same evaluation protocol?
2 comments
AuthorsOpenReview ↗

Response 1/2 - Part-based representations

We thank the reviewer for their careful review, and for their interest in our psychovisual approach. We address their two concerns, the evidence for part-based representations (Weakness 1) and the scaling comparison (Weakness 2), and answer each of their questions below.

Evidence for part-based representations

We thank the reviewer for asking whether PsychoNet's salient regions reflect semantic object parts rather than fragmented activations, which we address with new part-annotation results and our ablation models (Global Response, 1).

"DPR measures multiple salient regions within an object mask, rather than correspondence to annotated semantic parts or their causal importance to predictions. Higher DPR could therefore reflect fragmented activations rather than more meaningful representations."

Q1a "Can you validate the part-based interpretation using semantic part annotations or targeted feature/region interventions?"

We have validated the part-based interpretation against human-labelled semantic part masks from PartImageNet (see Global Response, 1. Quantitative evaluation of object parts), using two metrics. (1) Annotated DPR, a stricter version of the original DPR metric, only counts an image if its salience peaks fall on at least two different labelled parts, rather than just anywhere on the subject. (2) A linear part-segmentation probe predicts the labelled part masks from a block's frozen features, without salience maps. At relative depths 0.59–0.73, Psycho-B/L score higher than ResNet-101/152 on both metrics. This supports that their salience peaks and features follow semantic object parts rather than arbitrary fragments.

Q3b "Does removing these blocks also reduce DPR under the same evaluation protocol?"

Yes. We evaluated the Table 3 ablation models on the original DPR under the same protocol (ImageNet-S50), as well as on annotated DPR and the linear part-segmentation probe on PartImageNet (see Global Response, 1.3 Phasor Block ablation). Replacing the Phasor Blocks with ResBlocks reduces all three metrics at relative depths 0.59–0.73, both with and without the Spectral Branches (Models A vs. C and B vs. D).

Q1b "How sensitive is the DPR advantage to its thresholds and the salience method?"

We varied the prominence threshold δ (0.1 and 0.3, default 0.2) and the on-object threshold τ (0.3 and 0.7, default 0.5) one at a time, on both ImageNet-S50 and PartImageNet. Psycho-B's advantage over ResNet-101 at the mid-to-late depths holds under every setting, for both DPR on ImageNet-S50 and the new annotated DPR on PartImageNet. Regarding the salience method, the linear part-segmentation probe (see Q1a) uses neither salience maps nor thresholds, and also favours Psycho-B over ResNet-101 at relative depths 0.59–0.73.

Table: DPR on ImageNet-S50 and annotated DPR on PartImageNet under different thresholds, Psycho-B vs. ResNet-101, averaged over relative depths 0.59–0.73. Δ = Psycho-B − ResNet-101. All differences have p < 1e-24 (paired Wilcoxon signed-rank test).

SettingDPR Psycho-BDPR ResNet-101ΔAnn. DPR Psycho-BAnn. DPR ResNet-101Δ
Default (δ = 0.2, τ = 0.5)0.530.27+0.260.520.15+0.37
δ = 0.10.580.30+0.280.590.19+0.41
δ = 0.30.460.23+0.230.420.11+0.31
τ = 0.30.550.29+0.270.520.15+0.37
τ = 0.70.490.25+0.240.470.14+0.33

"The evidence does not yet establish the claimed semantic abstraction or transparent reasoning... The psychovisual interpretation remains suggestive."

We agree that our submitted evidence was largely qualitative, and we will narrow these claims in the revised abstract (see Global Response) and conclusion. The semantic structure we now claim is at the level of object parts, which we measure directly against PartImageNet part masks, as above. For DVC, a new inference sparsity experiment (see Global Response, 2. Deep Visual Coding) shows that its output keeps most of its accuracy with 99.5% of its frequency coefficients removed, whereas its input loses accuracy sharply beyond 90%. This suggests that DVC's learned filters concentrate task-relevant information into a small set of frequencies, similar to how the original psychovisual codes preserve only the perceptually salient ones. We therefore view DVC as organising the part features by scale and filtering their frequencies, and PsychoNet's abstraction as formed jointly by its Phasor Blocks and DVC. Regarding transparent reasoning, the Phasor Blocks already offer some transparency, as they decompose objects into parts in a way similar to human vision, which is thought to recognise objects by their parts. However, we have so far found salience methods difficult to adapt to DVC's frequency-domain features, so understanding how the model uses these parts in its decisions remains future work.

AuthorsOpenReview ↗

Response 2/2 - Spectral processing from width, resolution, and architectural changes

Spectral processing from width, resolution, and architectural changes

We thank the reviewer for asking whether PsychoNet's accuracy at reduced depth comes from its spectral processing rather than width or other factors. We address this with new shallow-wide ResNet baselines and extra seeds, followed by throughput.

"The scaling comparison does not isolate the benefit of spectral processing from width, resolution, and architectural changes. Shallow, widened spatial baselines would strengthen this claim."

Q2 "How does PsychoNet compare with shallow, widened spatial networks under matched parameters and FLOPs, and with a comparable global spectral-filtering baseline?"

We have trained shallower and widened ResNets at similar layer count, parameter count and FLOPs as Psycho-Eff-B/L (see Global Response, 3. Shallow-wide ResNet baselines), and Psycho-Eff-B/L outperform them on CIFAR-10, CIFAR-100 and ImageNet-100. Regarding resolution, the base PsychoNet models keep 14×14 features for DVC (Sec. 4.2), which is why we designed the Psycho-Eff variants to downsample to 7×7 like ResNet at matched FLOPs. The shallow-wide ResNets are matched to these Eff models. For global spectral filtering, Table 3 replaces DVC's sub-bands with a single global filter, similar to GFNet's, within the same architecture (Models B, D), and DVC performs better.

"The small ImageNet accuracy gains also lack reported run-to-run uncertainty."

Q3a "Are the ImageNet improvements, particularly the 0.25-point gain from full Phasor Blocks in Table 3, consistent across training seeds?"

Every model in Table 1 now has three runs on CIFAR-10, CIFAR-100 and ImageNet-100 and, as each ImageNet-1K run takes at least two days, two runs on ImageNet-1K (GR Table 6). On ImageNet-1K, Psycho-S/B/L/H and Psycho-Eff-S remain ahead of their ResNet baselines, while Psycho-Eff-B/L are level with ResNet-101/152 within one standard deviation.

The Table 3 ablations are single runs, so the 0.25-point gain cannot be tested for significance. As it is close to the 0.13-point difference between our two Psycho-L (Model A) runs, we consider it suggestive rather than conclusive. The clearer contribution of the Phasor Blocks is the part-based representation shown above.

"Moreover, fewer layers do not translate into practical efficiency: Psycho-Eff-B approximately matches ResNet-101 accuracy and FLOPs but achieves roughly half its throughput (Table A.3)."

At matched FLOPs and parameter counts, Psycho-Eff-S/B/L's current inference throughput is indeed lower than their ResNet counterparts' (Table A.3). We believe this is largely because complex-valued operations are less optimised than real-valued ones in current hardware and deep learning frameworks, and improving PsychoNet's runtime efficiency is an important direction for future work.

Reviewer_8j4POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a psychovisually inspired network architecture for image classification. The contributions are two modules: Deep Visual Coding (DVC) and Phasor Blocks, which replace the later stages of a ResNet (keeping the first two stages for feature extraction) to make up PsychoNet.

  • The Phasor Blocks convert real-valued spatial features into complex-valued ones by generating imaginary components with depthwise-convolution blocks, then refine them with complex convolutions. This breaks conjugate symmetry of the signal in the Fourier domain, producing features that can exploit the full expressivity of the complex domain.

  • The DVC takes the FFT of these complex features, splits the spectrum into three nested frequency bands, and applies learned element-wise complex filters with a channel-wise softmax in each band. This is what the authors denote as a data-driven version of Barba et. al's psychovisual coding, which quantizes perceptually salient frequency bands. Classification is done directly from the frequency-domain features using a complex linear layer.

Models are evaluated on CIFAR-10/100, ImageNet-100 and ImageNet-1K against ResNet-50/101/152/270 and ConvNeXt-S, including FLOP-matched “efficient” variants. Findings indicate that PsychoNet matches or slightly exceeds baseline accuracy with fewer layers, which is interpreted as reduced dependence on depth compared to baseline. Given enough data, the DVC filter resembles psychovisual codes. Finally, ablations suggest the spectral band split improves accuracy, while Phasor Blocks mainly improve representation quality rather than accuracy

  • Distinct Part Rate (DPR) is a introduced metric defined as the fraction of images whose salience map (KPCA-CAM) contains at least two topographically distinct peaks lying mostly within the ground-truth object mask. It is used to quantify how part-focused (vs. diffuse, whole-object) the intermediate representations are. It is computed on Phasor Block activations and compared with ResNet 101 at similar relative depth. The authors report a substantially higher DPR for Psycho-B at three mid-to-late blocks, with the largest gains for animal and vehicle classes.
Critical strengths
  • Connecting complex valued/frequency based representations to the 1990s psychovisual coding literature (Saadane, Barba et al.) is a novel and compelling idea in deep learning and may be a fruitful direction for brain inspired reasoning using abstract representations such as objects, relations, and categories.

  • The experimental section is rigorous in evaluating various model configurations at different parameter scales across different datasets. Efficientvariants are also explored and FLOP matched to baselines. Ablations are conducted to isolate model behaviour. Results are honestly reported (including non-favourable ones such as wall clock speed (and kernels on complex valued networks are listed as a limitation).

  • There is an attempt at quantifying interpretability using the DPR metric on the CAM derived features. The metric is well defined, with statistical tests used for significant results.

Critical weaknesses
  • The authors conclude PsychoNet is less depth-dependent because it matches baseline accuracy with fewer layers. In the experiments, the scaling on the Psychonet Models is on width rather than depth while matching parameters of deeper ResNets. Wide Resnets (Zagoruyko & Komodakis, 2016) were proposed to show that shallow & wider resnets were in fact capable of matching performance of their narrower & deeper counterparts. In order for the paper to isolate the DVC as the mechanism accounting for the reduced depth, experiments with a width-scaled ResNet baseline (e.g., WRN-50-2 or a custom wide ResNet at a similar layer count + matched parameters) should be conducted.

  • Quantitative results lack statistical support. The results displayed in the tables are single seed, making it difficult to distinguish significant perfromance improvements from seed variance. This is especially the case for table 3, which spans 0.14-0.58pp. In addition, in table 1, ResNet 270 appears undertrained or poorly tuned to the dataset C100. I would ask the authors to report results over at least 3 seeds for Table 3 and the key rows of Table 1, and to retrain the CIFAR baselines with a tuned recipe or report their train/test curves.

  • Given accuracy parity, interpretability is the paper’s main selling point. However, the claim are not sufficiently supported. DPR measures part decomposition, which by design benefits Psychonet as it cannot by itself show that part decomposition is preferable to whole-object localization. Second, DPR is computed on the real-component activations of the Phasor Blocks, which the paper uses as the input to the abstraction step. The output of DVC, which the introduction presents as the psychovisual abstraction, is evaluated only qualitatively. I would ask that the authors justify why part decomposition is a desirable property, ideally by showing what it enables downstream, provide a quantitative evaluation of DVC’s output as an abstraction, and include a Phasor-without-DVC ablation at ImageNet-1K scale to isolate DVC’s causal role.

  • The paper claims to be the first data-driven frequency-domain approach to high-level representation learning. This overlooks closely related implementations exploring the role of the frequency domain in Learned frequency selection and filtering (e.g. Deep Frequency Filtering by Lin et al., 2023, Learning in the Frequency Domain by Xu et al., 2020). More importantly, the paper cites learned frequency-filter methods (GFNet, FFC) but does not compare against them. A parameter-matched comparison, e.g., replacing DVC with a GFNet-style global filter block, would clarify the particular role/benefit of the DVC’s band decomposition and frequency-domain classification.

Questions for authors
  • What does part the decomposition afforded by the model enable downstream? Does it better match human behaviour in prediction? Are the representations more robust/can they lead to better generalization?

  • If DVC’s output is the abstraction, what measurable property should it have that the Phasor features lack?

  • A d-channel complex map carries the same information as a 2d-channel real map. Does breaking conjugate symmetry help beyond this doubling? Comparing against real features with 2× channels would clarify this.

4 comments
AuthorsOpenReview ↗

Response 1/3 - Depth-dependence, multiple seeds

We thank the reviewer for their detailed review and positive view of connecting complex-valued and frequency focused vision representations to psychovisual literature. Below we provide a section by section response to each comment and question.

Depth-dependence and width-scaled ResNet baselines

We thank the reviewer for suggesting the width-scaled ResNet baseline. Shallow-wide ResNets matched to Psycho-Eff-B/L in layers, parameters and FLOPs remain behind Psycho-Eff-B/L on CIFAR-10, CIFAR-100 and ImageNet-100 (Global Response, 3.), which suggests that PsychoNet's Phasor Blocks and DVC together take on the role of depth (Global Response, revised abstract).

Multiple seeds, Table 3 and ResNet-270

We thank the reviewer for these suggestions. Every model in Table 1 now has three runs on CIFAR-10, CIFAR-100 and ImageNet-100 (GR Table 6). As each ImageNet-1K run takes at least two days, we completed two runs per model on ImageNet-1K (GR Table 6) and could not repeat the Table 3 ablations. With single runs, the Table 3 gaps cannot be tested for significance, so we compare them with the difference between our two Psycho-L (Model A) runs, 0.13 points. The Spectral Branch gaps (0.58 and 0.48 points for A vs. B and C vs. D) are several times this difference, while the Phasor Block gaps (0.25 and 0.15 points for A vs. C and B vs. D) are too close to it to separate, consistent with their main contribution being interpretable object part-focused features rather than accuracy, which our part metrics now support (Global Response, 1.3 Phasor Block ablation). We will revise our ablation discussion to reflect this.

We also retrained ResNet-270 and Psycho-H on CIFAR-100 with a longer schedule, on a single seed. ResNet-270 improves substantially but remains below Psycho-H, which reaches its best accuracy within 90 epochs. We will add these results to the appendix.

Table: Longer-training re-run of ResNet-270 and Psycho-H on CIFAR-100 (single runs, best epoch).

ModelLayersCIFAR-100 Top-1 (%)
ResNet-270 (orig.)27650.87 (90 ep)
ResNet-270 (+longer)27678.64 (300 ep)
Psycho-H (orig.)9379.89 (90 ep)
Psycho-H (+longer)9378.91 (300 ep)
AuthorsOpenReview ↗

Response 2/3 - Interpretability

Interpretability

To strengthen the support for our interpretability claims, we have added quantitative evaluations of both Phasor Blocks and DVC (Global Response, 1. Quantitative evaluation of object parts and 2. Deep Visual Coding).

"DPR measures part decomposition, which by design benefits Psychonet as it cannot by itself show that part decomposition is preferable to whole-object localization."

Q1 "What does the part decomposition afforded by the model enable downstream? Does it better match human behaviour in prediction? Are the representations more robust/can they lead to better generalization?"

Our new experiments test PsychoNet against human-labelled object parts from PartImageNet, through annotated DPR (Global Response, 1.1 Annotated DPR) and a linear part-segmentation probe that uses no salience maps (Global Response, 1.2 Linear part-segmentation probe). On both, the mid-to-late Phasor Blocks of Psycho-B/L that our salience maps showed to be most part-focused (Fig. 7b) align more closely with the labelled parts than ResNet blocks at matched depth. We see part decomposition as preferable for interpretability because parts are the units humans commonly use to describe objects (e.g. head, wing, tire), so part-focused features show which parts a model relies on, which whole-object localisation does not. Together with DVC, part-focused features may also contribute to PsychoNet's reduced depth dependence: Psycho-Eff-B/L outperform shallow-wide ResNets of the same depth and budget, most clearly on CIFAR-100, where the shallow-wide ResNets also fall behind the deeper ResNets (Global Response, 3.). The effects of part decomposition on human-behaviour alignment, robustness and generalisation remain important future work.

"...DPR is computed on the real-component activations of the Phasor Blocks..."

Q3 "A d-channel complex map carries the same information as a 2d-channel real map. Does breaking conjugate symmetry help beyond this doubling?"

The Phasor Blocks generate new imaginary components from real-valued image features. These carry information complementary to the real components: in our KPCA-CAMs, the real components capture distinct object parts and the imaginary components the broader object (Fig. 5). This is why both DPR variants, built on KPCA-CAM, use the real components. Replacing the Phasor Blocks with real-valued ResBlocks at a similar parameter count (Table 3, Models A vs. C and B vs. D) lowers both annotated DPR and the probe (Global Response, 1.3 Phasor Block ablation).

"The output of DVC, which the introduction presents as the psychovisual abstraction, is evaluated only qualitatively."

Q2 "If DVC's output is the abstraction, what measurable property should it have that the Phasor features lack?"

Our new sparsity analysis gives a quantitative property of DVC's output (see Global Response, 2. Deep Visual Coding). It keeps most of its accuracy with 99.5% of its frequency coefficients removed, whereas the same removal at its input, the Phasor features, lowers accuracy sharply beyond 90%. This suggests that DVC learns from training alone to select a small set of task-relevant frequencies, as hand-crafted psychovisual codes do for perceptually salient ones.

"...include a Phasor-without-DVC ablation at ImageNet-1K scale to isolate DVC's causal role."

We thank the reviewer for this suggestion. We trained Psycho-L with DVC replaced by a spatial global average pool and the same complex-valued linear classifier, using the Table 3 recipe. It reaches 79.54% top-1, falling slightly short of Psycho-L (79.85%).

In light of these results, we now present PsychoNet's abstractions as formed jointly by its two components: Phasor Blocks produce object part-focused features, and DVC encodes them by scale through sparse frequency selections (see Global Response, revised abstract). We acknowledge that DVC's representations are harder to visualise than Phasor Block features, as salience methods do not adapt easily to the frequency domain, and this remains future work.

AuthorsOpenReview ↗

Response 3/3 - Frequency domain literature

Frequency domain literature

"The paper claims to be the first data-driven frequency-domain approach to high-level representation learning. This overlooks closely related implementations... (e.g. Deep Frequency Filtering by Lin et al., 2023, Learning in the Frequency Domain by Xu et al., 2020)." ... "More importantly, the paper cites learned frequency-filter methods (GFNet, FFC) but does not compare against them."

We thank the reviewer for pointing us to these works, and apologise for the confusion. Our claim refers to explicitly exploring the frequency domain as a representation space for high-level features, as distinct from works that explore frequency-domain architectural components. In PsychoNet, DVC does this by encoding the part features of the Phasor Blocks in the frequency domain at the end of the network. Xu et al. (2020) select frequency channels of the input image to reduce its size for a standard CNN, which fits the efficiency-focused uses of the frequency domain discussed in our related work. Lin et al. (2023) are closer to our approach, using learned frequency filters to select domain-transferable frequencies, similar to how DVC selects task-relevant ones. However, their filtering serves domain generalisation and is not framed as representation learning. We will add both works to Sec. 2, and revise the claim in our introduction to refer to the frequency domain as a representation space for high-level features.

GFNet and FFC are performance-driven architectural components, repeated throughout the network for feature extraction, whereas DVC is applied once at the end, so we did not compare against them directly. Within our architecture, removing the Spectral Branches reduces DVC to a GFNet-like global filter and lowers accuracy (Table 3, A vs. B, C vs. D).

Thank you for the extensive revisions. The shallow-wide ResNet baselines, multi-seed results and PartImageNet evaluations are valuable, and the Phasor-vs-ResBlock ablation now provides convincing evidence that Phasor Blocks produce part-aligned features.

I think my concerns regarding the motivation for the DVC still stand:

  • Accuracy gap: The no-DVC ablation (79.54% vs. 79.85%, single runs) is a 0.31-point difference. This is comparable to the Phasor Block gaps (0.25 and 0.15 points) that the authors consider too close to their run-to-run difference (0.13 points) to separate.
  • Representational effect: Table A.12 shows part structure is largely unchanged without DVC's Spectral Branches, and the sparsity test looks like compression rather than abstraction.

Because DVC is central to the psychovisual framing of the paper, its contribution remains under-supported, so I'm maintaining my score. I do want to note that I find the idea of connecting learned frequency-domain representations to psychovisual coding genuinely interesting. I think the work could be strong with clearer evidence that the DVC drives the observed benefits.

Reviewer_mSbKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether psychovisual coding, a technique from image compression, can be used in deep networks for classification. It introduces Deep Visual Coding (DVC), the Phasor Block architecture and the PsychoNet framework, on the hypothesis that perceptually salient frequency selections may capture the high-level semantic abstractions used by human vision. The authors claim (i) the first psychovisual deep learning framework, (ii) features that pick out interpretable object parts, unlike the amorphous regions of standard CNNs, and (iii) less dependence on depth when scaling, because the complex-valued representations take over the role of deep spatial layers.

Critical strengths

The link between psychovisual coding and deep networks is new and well motivated, and it could open a new, information-efficient class of architectures. Across several depth and parameter levels, PsychoNet matches deeper ResNets with fewer layers at roughly the same parameter count.

Critical weaknesses

1. The depth-efficiency claim is not separated from width and parameter count. PsychoNet matches deeper ResNets with fewer layers at a similar parameter count, but Wide ResNets show the same trend with even better accuracy, so the gain may come from width rather than from psychovisual coding. Parameters grow only with the number of stacked Phasor Blocks, not in the Fourier module, so the results may say more about complex-valued networks than about psychovisual coding. Scattering networks already combine a fixed, complex-valued wavelet front end with few learned layers. Hybrid scattering networks (Oyallon et al., 2017) put a CNN on top of a scattering transform, and more recent work keeps the wavelet filters fixed and learns only 1×1 channel mixing, with a complex modulus or phase collapse as the only nonlinearity, reaching ResNet-18 accuracy on ImageNet (Zarka et al., 2021; Guth et al., 2022). This line of work is not discussed, and it is the natural baseline for separating the contribution of psychovisual coding from that of a complex-valued frequency decomposition. The design of the Phasor Block is not justified beyond producing complex numbers. "Efficient" is never defined (fewer layers, parameters, FLOPs, lower latency or memory), and fewer layers does not by itself show less dependence on depth (Fig. 7a). There are no error bars or results over seeds, and one comparison with another model type is not explained.

2. The interpretability claims are qualitative. Most of them rest on activation maps and feature visualizations that are not quantified and are not always striking. The paper says the learned frequencies resemble the hand-crafted scheme (Fig. 4) without showing what the hand-crafted map looks like or how this was judged. It is unclear whether the Phasor Block activations (Fig. 5) encode semantic object parts or simply respond to distinctive visual features, and whether Fig. 6 shows higher-level abstraction rather than a consequence of the frequency selection. In the ablation study, the feature maps look as good as or better than those of the full model. The only quantitative test (Appendix C.4) gets one line; it should be applied to all networks and moved into the main text. The conclusions should be toned down: the models are on par with the baselines at the same parameter count, not better.

References: Oyallon, Belilovsky & Zagoruyko (2017), Scaling the scattering transform: Deep hybrid networks, ICCV. Zarka, Guth & Mallat (2021), Separation and concentration in deep networks, ICLR. Guth, Zarka & Mallat (2022), Phase collapse in neural networks, ICLR.

Questions for authors
  1. In Fig. 4, is the frequency structure learned or predefined?
  2. How does PsychoNet compare with Wide ResNets at matched parameter count, over several seeds?
  3. What does the Phasor Block contribute beyond complex-valued activations? An ablation separating complex-valued processing from psychovisual coding would help.
  4. How does PsychoNet compare with a hybrid or learned scattering network (e.g. Zarka et al., 2021; Guth et al., 2022) at matched parameter count?
  5. Which notion of efficiency is meant: layers, parameters, FLOPs, latency or memory?
  6. Could the authors describe in more detail how the weights are trained and how each block is built? (clarification)
  7. What is the comparison with the other model type meant to show? (clarification)
3 comments
AuthorsOpenReview ↗

Response 1/3 - Scattering Networks

We thank the reviewer for their detailed review. We address each weakness and question in turn below.

Scattering Networks

We thank the reviewer for bringing Scattering Networks to our attention. Their use of wavelets to decompose images into frequency sub-bands has some similarities to our adaptation of psychovisual codes. They belong to an important line of work that brings multiscale transforms such as wavelets, curvelets and contourlets into neural networks, which our background already discusses, and we will extend this part of the background to include the works the reviewer suggested (Oyallon et al., 2017; Zarka et al., 2021; Guth et al., 2022).

However, while the idea of decomposing signals into frequency sub-bands is similar, PsychoNet differs in how it uses the frequency domain and what it uses it for. Scattering networks apply fixed wavelet filters to extract features, either as a front end below a CNN (Oyallon et al.) or at every layer with only learned 1×1 channel mixing (Zarka et al.; Guth et al.). PsychoNet instead follows a psychovisual pipeline: Phasor Blocks first extract complex-valued, part-focused features, and DVC is then applied once, at the end of the network, to directly encode them. DVC adapts the radial sub-bands of hand-crafted psychovisual codes (Saadane et al., 1998), which preserve the perceptually salient frequencies of an image, but learns its filters within each sub-band and applies them to high-level features rather than the raw image. We therefore see scattering networks as complementary to our approach rather than a direct baseline.

AuthorsOpenReview ↗

Response 2/3 - Depth efficiency, width and complex-valued processing

Depth efficiency, width and complex-valued processing

We separate PsychoNet's depth efficiency from width and parameter count with new shallow-wide ResNet baselines (Global Response, 3), and clarify the purpose of the Phasor Blocks with a new quantitative evaluation of their features and our Table 3 ablation (Global Response, 1 and 1.3).

Q2 "How does PsychoNet compare with Wide ResNets at matched parameter count, over several seeds?"

We trained two shallow-wide ResNets matched to Psycho-Eff-B/L in layer count, parameters and FLOPs, with the same recipes, on CIFAR-10, CIFAR-100 and ImageNet-100 over three seeds. Psycho-Eff-B/L outperform both on all three datasets (Global Response, 3. Shallow-wide ResNet baselines), which supports PsychoNet's reduced depth dependence. We did not use the original Wide ResNets (Zagoruyko & Komodakis, 2016), as their sizes and training recipes do not match our models.

Q5 "Which notion of efficiency is meant: layers, parameters, FLOPs, latency or memory?"

We thank the reviewer for pointing this out. "Efficient" refers to our Psycho-Eff variants, which downsample further than the base PsychoNet models to reduce their FLOPs to those of ResNet-50/101/152 at matched parameter counts, and we will make this explicit in Sec. 4.2. Psycho-Eff-B/L also use even fewer layers than the base models, 38% and 46% fewer than ResNet-101/152 (Table 1). Throughput, latency and peak memory currently favour ResNet (Appendix C.3, Table A.3), as complex-valued operations lack the mature optimisations of real-valued layers in current deep learning libraries, and optimising them is important future work.

"Parameters grow only with the number of stacked Phasor Blocks, not in the Fourier module, so the results may say more about complex-valued networks than about psychovisual coding."

Q3 "What does the Phasor Block contribute beyond complex-valued activations? An ablation separating complex-valued processing from psychovisual coding would help."

As the reviewer notes, the added parameters in our larger models come mainly from widening Phasor Blocks, since our models have multiple Phasor Blocks while DVC is a single, final filtering stage. Beyond producing complex values, the Phasor Blocks build features organised around object parts. Our new PartImageNet experiments show that, at the mid-to-late Phasor Blocks, these features align more closely with human-labelled object parts than ResNet features at matched depth (see Quantitative interpretability evidence below). This supports interpretability, as parts are the units humans commonly use to describe objects (e.g. head, wing, tire). Our Table 3 ablation separates this from psychovisual coding. Replacing the Phasor Blocks with real-valued ResBlocks lowers this alignment, whether or not DVC's psychovisual sub-bands are kept (Models A vs. C, B vs. D; Global Response, 1.3 Phasor Block ablation). DVC, in turn, carries out the final high-level processing of these features, and its psychovisual sub-bands improve accuracy both with and without Phasor Blocks (Table 3, Models A vs. B and C vs. D).

"There are no error bars or results over seeds"

Every model in Table 1 now has three runs on CIFAR-10, CIFAR-100 and ImageNet-100. As each ImageNet-1K run takes at least two days, we could add only one further run per model on ImageNet-1K within the rebuttal period, giving two runs each. All results are reported as mean ± SD (Global Response, 4. Extra seeds).

"One comparison with another model type is not explained."

Q7 "What is the comparison with the other model type meant to show?"

We believe the reviewer refers to the comparison between PsychoDW and ConvNeXt-S. PsychoDW replaces the remaining regular convolution blocks in its Phasor Blocks with ConvNeXt-style depthwise convolution blocks, showing that our framework can be adapted to more modern convolutional architectures and approach their accuracy. PsychoDW leads ConvNeXt-S on CIFAR-10/100 but is slightly behind on ImageNet-100 and ImageNet-1K (Table 1; see Global Response, 4. Extra seeds).

AuthorsOpenReview ↗

Response 3/3 - Quantitative interpretability evidence, Questions

Quantitative interpretability evidence

We have added quantitative evaluations for interpretability of both our Phasor Blocks and DVC, and will present them in a new results subsection. For Phasor Blocks, we extended the original DPR evaluation (App. C.4) to all three model pairs and tested them against human-labelled object parts from PartImageNet (He et al., 2022). Annotated DPR counts salience peaks only when they fall on different labelled parts (e.g. head, body, tail), and a linear part-segmentation probe on frozen features uses no salience maps (Global Response, 1.1–1.2). At the mid-to-late Phasor Blocks, Psycho-B/L score higher than ResNet-101/152 on both metrics, which supports that their activations (Fig. 5) cover several distinct semantic parts of an object.

"The paper says the learned frequencies resemble the hand-crafted scheme (Fig. 4) without showing what the hand-crafted map looks like or how this was judged." … "whether Fig. 6 shows higher-level abstraction rather than a consequence of the frequency selection."

Q1 "In Fig. 4, is the frequency structure learned or predefined?"

The frequency structure in Fig. 4 is partly predefined and partly learned. DVC predefines radial sub-bands, following the psychovisual coding of Saadane et al. (1998) shown in Fig. 2 (left) and App. B.1, but the filtering within each sub-band is learned end-to-end, so the sparse frequency selection emerges from training. We judged its resemblance to the hand-crafted scheme qualitatively in the paper, and our new inference sparsity analysis now supports it quantitatively (Global Response, 2. Deep Visual Coding). DVC's output keeps most of its accuracy with 99.5% of its frequency coefficients removed, whereas its input, the Phasor features, loses accuracy sharply beyond 90%. This suggests that DVC does more than split features into predefined sub-bands, as its learned filters concentrate the task-relevant information of the part features into a small set of frequencies, much as psychovisual coding does for perceptually salient ones. In Fig. 6, the object parts come from the Phasor Blocks, while their division across sub-bands is an intended part of our design. In our revised framing, the abstraction is the representation both components form together: part-focused features, encoded by DVC as a sparse, multi-scale frequency code (Global Response, revised abstract).

"In the ablation study, the feature maps look as good as or better than those of the full model."

We believe this refers to the comparison of KPCA-CAMs in Fig. 8, and would be grateful if the reviewer could confirm. If so, we apologise that its caption is unclear as Model A is Psycho-L (the full model), while Model C is ablated with Phasor Blocks replaced by ResBlocks. We will revise the caption accordingly. We have also quantified this comparison with our new PartImageNet metrics (see Global Response, Phasor Block ablation), on which Model C scores markedly lower than Psycho-L on both annotated DPR and the linear probe.

"The conclusions should be toned down: the models are on par with the baselines at the same parameter count, not better."

We thank the reviewer for this suggestion, and will revise our conclusion to state that PsychoNet matches ResNet baselines at the same parameter count and FLOPs.

Questions

Q4 is addressed under Scattering Networks, Q2, Q3, Q5 and Q7 under Depth efficiency, width and complex-valued processing, and Q1 under Quantitative interpretability evidence, where they are quoted.

Q6 "Could the authors describe in more detail how the weights are trained and how each block is built?"

All weights, including the DVC filters, are trained jointly from scratch with the cross-entropy loss, using the recipes in Appendix C.1–C.2 (Tables A.1–A.2). Figure A.8 (Appendix D.3) expands the Phasor Blocks of Figure 3 into detailed diagrams with their main design choices, and Tables A.7–A.19 give the full per-block configuration of every model. We would be happy to expand on any step that remains unclear.

References

Guth, F., Zarka, J. & Mallat, S. (2022). Phase Collapse in Neural Networks. ICLR.

He, J., Yang, S., Yang, S., Kortylewski, A., Yuan, X., Chen, J.-N., Liu, S., Yang, C., Yu, Q. & Yuille, A. (2022). PartImageNet: A Large, High-Quality Dataset of Parts. ECCV.

Oyallon, E., Belilovsky, E. & Zagoruyko, S. (2017). Scaling the Scattering Transform: Deep Hybrid Networks. ICCV.

Saadane, A., Sénane, H. & Barba, D. (1998). Visual Coding: Design of Psychovisual Quantizers. Journal of Visual Communication and Image Representation.

Zagoruyko, S. & Komodakis, N. (2016). Wide Residual Networks. BMVC.

Zarka, J., Guth, F. & Mallat, S. (2021). Separation and Concentration in Deep Networks. ICLR.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Global Response (1/6)

We thank all three reviewers for their careful reading and constructive feedback. Their comments have helped us balance descriptions of our contributions across DVC and Phasor Blocks as the representations PsychoNet learns are a joint contribution of both. We provide additional quantitative evidence that Phasor Blocks produce object part-focused features, in which we evaluate PsychoNet and its Table 3 ablations against human-annotated object parts (PartImageNet). The results support a clear division of roles, where Phasor Blocks produce features organised around object parts, while DVC encodes these features by scale and improves accuracy overall. We also present the width-matched ResNet baselines and multi-seed tables that all reviewers requested. We are in the process of uploading a revised manuscript with the abstract, experiments section and conclusion revised to reflect the roles of both components, the new quantitative results in the main text, and the remaining changes listed at the end of this response.

Our revised abstract reads as follows:

Psychovisual theory suggests human vision decouples low-level feature extraction from higher cognition by first forming intermediate abstractions. In contrast, deep vision models typically extract and aggregate features through homogeneous stacks of layers, rendering their decisions opaque. We propose the first psychovisual deep learning framework, whose two components jointly form psychovisually inspired abstractions: convolutional Phasor Blocks, which build multi-channelled complex-valued features, and Deep Visual Coding (DVC), which processes them in the frequency domain. Unlike other frequency domain based network architectures, DVC uses the frequency domain as a high-level representation space of Phasor Block features, adapting psychovisual coding, a 1990s image code that quantised perceptually salient frequencies. Salience analyses and quantitative evaluation on human-annotated parts show that Phasor Blocks extract features organised around object parts, unlike the amorphous regions of regular CNNs. DVC then encodes these part features by scale through sparse selections of frequency sub-bands that emerge from training, reminiscent of the hand-crafted psychovisual codes that inspired it. Our models are also less depth dependent than CNNs, suggesting that complex-valued features and DVC's multi-scale encoding together subsume the role of deeper layer stacks. Overall, these findings position psychovisual processing as a promising path toward more interpretable vision models.

AuthorsOpenReview ↗

Global Response (2/6)

1. Quantitative evaluation of object parts

All three reviewers asked whether our interpretability evidence shows semantic object parts and whether our quantitative test could be validated against part annotations. To address these, we evaluate PsychoNet against the human-labelled object parts of PartImageNet [1] with two complementary experiments. We extend DPR (App. C.4) into annotated DPR, which checks whether salience peaks fall on distinct labelled object parts rather than just anywhere on the subject. A linear part-segmentation probe then checks whether part identity can be read from the features directly, without salience maps. In both experiments, the mid-to-late Phasor Blocks of Psycho-B/L that our KPCA-CAM visualisations showed to be most part-focused (Fig. 7b) score higher than ResNet blocks at matched depth, matching our qualitative findings. We present each experiment, and the Table 3 ablations that identify which component is responsible, in detail below.

PartImageNet provides human-labelled part masks for 158 ImageNet classes across 11 super-categories, spanning animals and vehicles (e.g. head, tail, wing, tire). We ran all experiments on three model sizes, Psycho-S/B/L against ResNet-50/101/152, extending the original DPR evaluation, which covered only Psycho-B and ResNet-101.

1.1 Annotated DPR

In App. C.4, we introduced the Distinct Part Rate (DPR) to quantify the pattern seen in our KPCA-CAM visualisations, where Phasor Blocks highlight several separate object parts while ResNet blocks highlight a single amorphous region. DPR is the fraction of images whose salience map contains at least two distinct, prominent peaks lying inside the object. Annotated DPR goes further, counting an image only if its peaks fall on at least two different labelled parts, so that fragments of a single part are not counted as separate parts (GR Table 1). As in the paper, both variants are computed on the real component of the Phasor features, which our KPCA-CAM visualisations (Fig. 5) showed to focus on object parts, while the imaginary component carries the broader subject.

GR Table 1: Annotated DPR by relative network depth (block depth as a fraction of total network depth), each Psycho-S/B/L vs. ResNet-50/101/152 at matched depth. Δ = Psycho − ResNet; p = paired Wilcoxon signed-rank test p-value; – = no block at this depth.

rel. depthPsycho-SResNet-50ΔpPsycho-BResNet-101ΔpPsycho-LResNet-152Δp
0.31––––0.3210.296+0.0250.150.3070.324−0.0170.32
0.38––––0.2680.324−0.0561e-030.2230.321−0.0985e-08
0.450.3140.277+0.0374e-020.2620.310−0.0484e-030.2640.276−0.0130.47
0.520.4050.325+0.0804e-060.3490.194+0.1552e-170.3760.163+0.2132e-31
0.59––––0.6230.201+0.4223e-820.5980.106+0.4922e-104
0.630.2440.272−0.0280.12––––––––
0.66––––0.4810.139+0.3429e-700.3880.174+0.2141e-31
0.73––––0.4430.107+0.3368e-670.4160.239+0.1773e-21
0.740.2380.199+0.0402e-02––––––––
0.80––––0.1730.099+0.0738e-080.1970.159+0.0381e-02
0.88––––0.1110.061+0.0501e-050.1250.067+0.0587e-07

From relative depth 0.52 onwards, Psycho-B/L place their peaks on different labelled parts far more often than ResNet-101/152, while the ResNets are level or slightly ahead at the earliest depths. This matches our original DPR results on ImageNet-S50 (App. C.4), where Psycho-B's advantage was also concentrated at relative depths 0.59–0.73 and ResNet-101 was slightly ahead at 0.38–0.45. Psycho-S shows no consistent advantage over ResNet-50 on annotated DPR, which may be due to its smaller size and fewer Phasor Blocks (five, against nine in Psycho-B/L). The part structure measured here in the real component is consistent with the split seen in Fig. 5, suggesting that the Phasor Blocks use their real and imaginary channels to carry complementary information. We will add annotated DPR to the revised paper, with the key results in the main text and full details and results in the appendix.

We have also extended the original ImageNet-S50 DPR evaluation to the Psycho-S/ResNet-50 and Psycho-L/ResNet-152 pairs, which we will also add to the appendix.

AuthorsOpenReview ↗

Global Response (3/6)

1.2 Linear part-segmentation probe

In addition to DPR, we also evaluated performance on a segmentation linear probing task over the labelled object part regions, as a test for how much network features align with part information that does not depend on salience maps. For each block of a frozen network, we attach the minimal segmentation head from Long et al. (2015) [2], comprising just a 1×1 convolution and bilinear upsampler, to predict a semantic part mask. We train on the PartImageNet training split (19,460 images) and report mean IoU on its test split (2,408 images; separate from the validation images used for DPR), restricting each image's prediction to the parts of its own super-category so that the score measures part segmentation rather than object recognition. Phasor Block features are read as their real and imaginary components stacked.

GR Table 2: Linear part-segmentation probe (test mIoU, %) by relative network depth, each Psycho-S/B/L vs. ResNet-50/101/152 at matched depth. Δ = Psycho − ResNet; p = paired Wilcoxon signed-rank test p-value over per-image IoU (2,408 test images); – = no block at this depth. Psycho-S is evaluated at fewer depths as it is a shallower model.

rel. depthPsycho-SResNet-50ΔpPsycho-BResNet-101ΔpPsycho-LResNet-152Δp
0.31––––19.923.3−3.42e-7020.928.7−7.86e-191
0.38––––29.731.7−1.93e-1936.934.5+2.52e-57
0.4525.015.6+9.4<1e-30032.635.6−3.18e-2536.938.9−2.02e-20
0.5240.821.1+19.7<1e-30038.239.4−1.20.2141.542.6−1.13e-17
0.59––––46.641.9+4.71e-9248.244.2+4.06e-62
0.6343.130.3+12.8<1e-300––––––––
0.66––––47.344.3+3.04e-3648.246.5+1.62e-06
0.73––––49.145.0+4.05e-3849.447.0+2.55e-13
0.7445.337.3+8.01e-190––––––––
0.80––––47.546.0+1.59e-0546.947.3−0.43e-03
0.88––––42.946.1−3.24e-8342.747.2−4.53e-137

At relative depths 0.59–0.73, where annotated DPR also favours PsychoNet, Psycho-B/L segment the labelled parts better than ResNet-101/152 (+3.0 to +4.7 and +1.6 to +4.0 mIoU), while the ResNets are ahead at the earliest depths and at the last block before DVC (0.88). Psycho-B/L's best blocks also exceed the best ResNet blocks at any depth (49.1 vs. 46.1 and 49.4 vs. 47.3). Unlike on annotated DPR, Psycho-S leads ResNet-50 at every block, by +8.0 to +19.7. We will add this experiment to the revised paper, with the key results in the main text and full details and results in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.