acceptodds
Withdrawn from ICLR 2027

S2E: Dense Functional Differentiation in Feed-Forward Networks via Prototype-Guided Subspace Modulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) architectures improve model capacity via specialized experts and input-dependent routing, but incur parameter duplication, routing complexity, and irregular computation, motivating a fundamental question: can functional differentiation emerge within a single dense feed-forward network (FFN) without independent expert parameters, token dispatch, or sparse activation? We propose Semantic Subspace Expert (S2E), a dense FFN that partitions a shared hidden representation into contiguous subspaces and achieves dense functional differentiation via prototype-guided adaptive modulation. Modulation weights are computed from two complementary signals: global cosine similarity for subspace-level alignment and periodic coordinate-level alignment for fine-grained feature correspondence. Experiments across ImageNet-1K classification, out-of-distribution benchmarks, and COCO detection/segmentation show S2E consistently improves the accuracy–efficiency trade-off with fewer parameters and FLOPs. On DeiT-T, S2E achieves 74.9% Top-1 accuracy with 3.9M parameters and 0.91G FLOPs, outperforming the FFN-4× baseline by 2.7 points with lower costs. Controlled ablations validate the contributions of contiguous subspace organization and dual-signal modulation, confirming shared projection yields higher accuracy and greater subspace diversity than independent per-subspace projections. Further analyses verify non-redundant, category-dependent functional preferences across subspaces while they remain fully active.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Transformers, Feed-Forward Networks, Dense Functional Differentiation, Representation Specialization, Mixture-of-Experts, Prototype-Guided Modulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ezTQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes S2E, a dense feed-forward network (FFN) architecture that aims to achieve functional differentiation through prototype-guided subspace modulation. The method partitions hidden representations into multiple subspaces and dynamically modulates their features using similarity-based weighting. Experiments on ImageNet classification, distribution-shift benchmarks, and COCO detection and segmentation report improvements in predictive accuracy and computational efficiency.

Critical strengths
  1. Promising accuracy–efficiency trade-off. The paper reports improved classification accuracy across multiple vision backbones while reducing FLOPs and, in several configurations, parameter counts.

  1. Broad experimental coverage. The evaluation covers multiple architectures, distribution-shift benchmarks, object detection, and instance segmentation. The paper also includes ablation studies and subspace intervention analyses.
Critical weaknesses
  1. Unverifiable references and concerns about experimental reliability. The arXiv identifier (2206.02351) provided for G-FFN: Grouped Feed-Forward Network for Efficient Vision Transformers actually corresponds to The Synergy of Electromagnetic Effects and Thermophysical Properties of Metals in the Formation of Laser Induced Periodic Surface Structures, a paper in optics. Similarly, the identifier (2303.07975) provided for Dynamic Gated Feed-Forward Networks for Vision Transformers corresponds to Software-based Security Approach for Networked Embedded Devices. Neither cited work could be verified using the provided titles and author information. Notably, G-FFN is included as an important baseline in Table 2, with specific experimental results. These issues raise serious concerns about the reliability and reproducibility of the corresponding baseline results. The authors should provide verifiable references, baseline implementations, and original experimental records.

  1. Substantial inconsistencies between the supplementary code, the described method, and the reported results. The paper claims to modify only the Transformer FFN, whereas the supplementary Swin-T/S configurations set the attention window size to 56, compared with the standard Swin window size of 7. The resulting attention computation is also inconsistent with the reported 3.9G FLOPs for the entire Swin-T+S2E model. Furthermore, the implementation introduces spatial convolution and normalization components that are not adequately described in the main method, and contains missing imports, configuration entries, and training configuration files. Consequently, the provided supplementary materials do not allow reliable reproduction of the reported results. The authors should clarify these discrepancies and provide executable code and original experimental records that correspond to the reported experiments

  1. Insufficient evidence for the claimed novelty and functional differentiation mechanism. S2E essentially combines feature grouping, learnable prototypes, and input-dependent dynamic feature modulation. Although the representation-diversity, category-selectivity, and subspace-ablation analyses demonstrate differentiated responses, these observations do not establish a fundamentally new functional differentiation mechanism distinct from conventional FFNs or dynamic gating architectures. The paper also lacks sufficient comparisons with simple dynamic-gated FFN baselines under matched parameter and computational budgets, and does not establish a clear causal relationship between the claimed functional differentiation and the observed performance improvements.
Questions for authors
  1. Could the authors provide verifiable references for G-FFN and Dynamic Gated FFN and explain why the cited arXiv identifiers correspond to unrelated papers? Could the authors also provide the G-FFN implementation and original experimental records to substantiate the results reported in Table 2?

  1. Could the authors explain the discrepancies between the attention window configurations, additional architectural components in the supplementary code, and the method described in the paper? Could they provide executable code and evaluation records consistent with the reported FLOPs, accuracy, and latency?

  1. Compared with conventional dynamic-gated FFNs under matched parameter and computational budgets, does S2E demonstrate a distinct functional differentiation mechanism? What evidence establishes that the claimed differentiation is responsible for the observed performance improvements?
Reviewer_ZQGzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

S2E partitions a shared FFN hidden representation into (K) subspaces and associates each with a learnable prototype. A token-dependent centroid is compared with the prototypes using global cosine similarity and coordinate-level periodic correspondence, producing modulation weights that update all subspaces while preserving fully dense execution.

The paper evaluates S2E on ImageNet-1K, distribution-shift benchmarks, and COCO, together with component ablations, representation-diversity analyses, subspace interventions, and category-selectivity analyses. The reported results show improved accuracy–efficiency trade-offs and measurable non-redundant, category-dependent subspace behavior.

Critical strengths
  1. Strong controlled accuracy–efficiency results. The matched-capacity comparison is particularly compelling: the compact S2E configuration uses 3.9M parameters and 0.90G FLOPs and improves over the matched-width FFN-2× baseline by 3.4 Top-1 points. It also outperforms the standard FFN-4× while using substantially fewer parameters and FLOPs.

  2. Broad evaluation with meaningful mechanism-oriented analyses. S2E is evaluated across multiple backbone families, distribution-shift benchmarks, and downstream detection/segmentation tasks, with consistent accuracy–efficiency improvements in the reported settings.

Critical weaknesses
  1. The strongest functional-differentiation analyses lack a configuration-matched standard-FFN control. The paper directly compares S2E with dimension-matched groups from a standard FFN at the representation level. However, the stronger causal and category-dependent diagnostics—individual suppression, grouped masking, selectivity indices, and preferred/non-preferred class sensitivity—are reported only for S2E, primarily under the (K=48,r=16) configuration. This matters because the paper itself notes that standard FFN hidden dimensions can already exhibit structured and specialized responses. The manuscript therefore establishes that measurable functional differentiation exists within S2E, but does not directly establish how much stronger its causal or category-dependent differentiation is than that of an equivalently partitioned standard FFN. This limits the comparative claim that S2E meaningfully enhances functional differentiation beyond ordinary dense-FFN behavior.

  2. Training-run uncertainty is not reported. The manuscript does not state whether the reported classification and detection results are averaged over multiple independent training runs, nor does it report run-to-run standard deviations or confidence intervals. This is less concerning for large effects such as the +3.4-point matched-width DeiT-T improvement, but it matters for smaller reported differences, such as the +0.6-point improvement on Swin-S. The manuscript provides detailed training protocols and carefully controls system-level measurements, but direct evidence of training-run variability is not reported.

Questions for authors
  1. Could the authors apply the same causal and category-selectivity diagnostics used for S2E to a configuration-matched standard FFN? In particular, since the intervention/selectivity analyses use (K=48,r=16), a natural control would be the standard DeiT-T FFN-4× with hidden dimension 768, partitioned diagnostically into 48 groups of width 16, followed by the same individual/grouped masking and category-selectivity analyses. Because a standard FFN has no native subspace boundaries, it would also be useful to verify that the conclusion is robust to alternative channel partitions rather than relying on a single contiguous grouping. Such a comparison would clarify whether, and to what extent, S2E strengthens functional differentiation beyond what naturally emerges in a standard dense FFN.

  2. Could the authors report results across multiple independent training runs for the key DeiT-T comparisons together with mean and standard deviation or confidence intervals? The manuscript does not state whether the main predictive results are based on multiple independent runs, and run-to-run uncertainty is not reported.

  3. Please clarify how the “preferred classes” of each subspace are defined in the functional-selectivity analysis. For example, are they the single class with maximum mean activation, a top-(m) set of classes, or classes selected using a threshold? It is also unclear whether the same evaluation samples are used both to identify preferred classes and to measure the larger accuracy drop after subspace suppression. If so, the reported preferred/non-preferred sensitivity gap may be affected by selection bias. A held-out split, where preferred classes are identified on one subset and suppression sensitivity is evaluated on another, would provide stronger evidence that the observed category preference generalizes. The manuscript does not provide direct evidence on this point.

Reviewer_DpsMOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission present S2E: Dense Functional Differentiation, which replaces the FFN in the traditional Transformer with a dense block that splits a shared hidden projection into K subspaces. Each subspace is nudged toward a learnable prototype, with token-dependent weights computed from cosine-plus-periodic similarity, so specialization arises without separate experts or routing. On DeiT-T it reports 74.9% top-1 with fewer parameters and FLOPs than FFN-4× (72.2%). It shows gains across 10 backbones, robustness benchmarks and COCO, and offers diversity, intervention and selectivity analyses as evidence of functional differentiation.

Critical strengths

The submission is easy to follow, and the underlying idea is simple and practical. However, this simple idea shows benefits in a broad set of evaluations. The submission shows evaluations on many different backbones, robustness benchmarks, dense prediction, and latency/memory reporting.

Critical weaknesses

I'm not very familiar with this field, although I can use Agents to find some weakness, I'd rather to be honest here that I'm not able to find reasonable weakness in this submission.

Questions for authors

Looks like there is a contradiction.

In section 4.4, Subspace intervention analysis: the mean drop per subspace is 0.72%, yet the drops on preferred and non-preferred classes are 0.181% and 0.056% (from Table 6). Both are well below the overall mean, which is not arithmetically possible if those two subsets make up the full evaluation.

Why?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.