- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed Parallel Block Interaction (PBI), a universal module to increase the feature width of a vision model using limited extra cost. Instead of processing all channels with one wide block, it divides them among narrower and independently parameterized blocks. Each block processes its own private channels with a small set of public channels. Through pub-to-pub, priv-to-pub, and priv-to-priv communication, PBI tries to recover useful interaction loss due to grouping, so that the saved computation can then support a wider representation.
The main results are consistent across different architectures, backbones, datasets, and tasks. Although these results remain modest relative to the original backbones, they can support the main motivation of PBI and demonstrate that PBI can work for different model types (CNN, transformer, Mamba). Therefore, PBI can be a plausible and potentially useful architectural contribution.
Critical strengths
-
PBI is a genuinely architecture-agnostic mechanism. PBI operates on the channel interface. Concretely, PBI partitions channels into groups, runs the unmodified block per group, and exchanges a small number of shared states. In this way, PBI can be applied to convolutional, attention, and space-space blocks without touching their internal architectures. The cross-family results (including ConvNext, DeiT, Swin, VMamba) support this claim, and the design is easy to reproduce.
-
The motivation is very clean and well supported by the cost analysis/calculation. The saving is easy to verify by inference time and memory measurement (Figure 2)
-
The empirical coverage is board. Three architectures are examined and tested across different tasks, i.e., ImageNet-1K, COCO/ADE20K, and ImageNet-21K pretraining is substantially more than a single-backbone/task demonstration.
-
The component ablation is informative. Pub-to-pub is correctly identified as the dominant contributor. The interactions are complementary, and the paper compares against channel shuffle and pointwise mixing. Tables A8/A9 transparently explain why configurations with public channels can have fewer parameters/FLOPs.
Critical weaknesses
-
I think my personal biggest concern is there is no matched-budget dense-widening control on a real backbone. The central claim ("better than widening") is insufficiently tested at equal compute. Table 1 compares PBI to the original narrow model (Swin-T: 29M/4.5G 33M/5.2G, i.e., +14% parameters and +16% FLOPs for +1.2 points), and Table 4 compares at equal representation width, not equal compute. Eq. A32 gives the conversion, and the 3-layer ViT at 9.3G corresponds to a dense width of 1130, so the control is cheap. Without it, the gain (+1.17 over the dense backbone) cannot be separated from simply spending more capacity.
-
The conceptual contribution is relatively thin. Each ingredient has a close precedent: grouped channel processing for cheaper width (ResNeXt: Xie et al., 2017; ShuffleNet: Zhang et al., 2018), a shared latent that modules read from and write to (Shared Global Workspace: Goyal et al., 2021), multi-branch blocks (Inception: Szegedy et al., 2016; vision MoE: Riquelme et al., 2021; Xue et al., 2022), and compact cross-branch channel mixing (SCHEME: Sridhar et al., 2023). These work are mostly cited in related work, but a justification on this paper’s contribution is expected. Also, it is natural to compare the performance with Shared Global Workspace/SCHEME at matched budget.
-
Some results are not convincing. For example, Dense + 3072 in Table 4 shows NaN seems to be a problem of training setup rather than an limitation of dense widening. Meanwhile, Appendix E.3 states that the pointwise-mixing baseline uses "no additional LayerNorm, activation, or residual connection", which explains why it collapses to 72.86%. Therefore, Table 6 does not fairly represent that public sharing is the better interaction mechanism.
-
The elaboration on why PBI is working is suboptimal. My concern is, CKA, Grad-CAM, and classifier-weight magnitudes are correlational, and weight magnitude ignores activations. More specifically, the private-to-public interaction is motivated as an anti-oversmoothing measure, yet it supplies the same updated public state to every branch. It is unclear why identical shared input should reduce inter-branch similarity. Results of inter-branch CKA with/without Pr Pu are maybe helpful.
Questions for authors
Questions are basically corresponding to the weaknesses above.
-
Why is there no matched-budget dense-widening control? Or it is already represented in existing experiments? A single run with the same recipe would directly test the central claim. Without it, how should we attribute the performance gain to PBI rather than to extra capacity?
-
What is the empirical results of SCHEME-style grouped channel mixing and a Shared-Global-Workspace-style shared latent at matched parameters/FLOPs?
-
What training recipe produced NaN for Dense + 3072, and was it tuned (LR, initialization, normalization, drop path)? Would a tuned dense model train, and at what accuracy? And was the pointwise-mixing baseline re-run with LayerNorm and a residual connection?
-
How does the priv-to-pub update reduce oversmoothing when the updated public state is identical across branches? Please report inter-branch similarity (e.g., CKA with/without Pr Pu)?
I am open to reconsidering the final assessment if these questions can be addressed, especially for 1 and 4.
4 comments
Response to Reviewer hjSU (part2)
W3/Q3. Training instability in dense widening and fairness of the pointwise-mixing baseline
A3. QK normalization stabilizes equal-budget dense widening, while Pre-LN and residual mixing improve the pointwise-mixing baseline. PBI remains more accurate than both.
Dense widening. The original width-3072 dense run (Table 4) encountered NaN gradients that corrupted its weights. Filtering did not stabilize training, with accuracy falling from about 30% to 5%. We observed large QK dot products and concentrated attention distributions. Per-head QK LayerNorm stabilized Dense and MoE at both 9.3G and 11.8G (Table 5). Dense then reaches 67.0% and 70.8%, compared with 74.9% and 76.5% for PBI.
Pointwise mixing. We agree that the original baseline lacked normalization and a residual connection. We added Pre-LN and residual mixing and tested GELU under the same four-block Swin-T, 100-epoch setup (Table 8, Appendix E.4).
| Interaction method (four-block Swin-T) | Params (M) | FLOPs (G) | Top-1 (%) |
|---|---|---|---|
| No additional within-stage interaction | 33.87 | 5.203 | 77.51 |
| Channel shuffling | 33.87 | 5.203 | 78.37 |
| Original pointwise linear mixing | 42.51 | 6.591 | 72.86 |
| Pointwise mixing + Pre-LN + residual | 42.52 | 6.605 | 79.47 |
| Pointwise mixing + Pre-LN + residual + GELU | 42.53 | 6.605 | 79.05 |
| Public-channel sharing | 31.77 | 4.985 | 81.60 |
The improved mixer rises from 72.86% to 79.47%, supporting your concern about the original optimization setup. Public-channel sharing still achieves 2.13 percentage points higher accuracy than this improved baseline, with fewer parameters and lower reported FLOPs. We have updated the manuscript with the new experiment results.
W4/Q4. The role of Priv-to-Pub with identical public inputs and its effect on inter-block CKA
A4. Priv-to-Pub introduces input-dependent shared information before the first layer of each stage. It is not intended to directly minimize inter-block similarity. Although all blocks receive the same public state, they retain different private features and independent block parameters.
We agree that our original explanation did not adequately clarify the role of Priv-to-Pub when all blocks receive the same public update. With Pub-to-Pub alone, information from private features enters the shared public state after the first layer's block computations and aggregation. Priv-to-Pub instead introduces information from private features at stage entry, making input-dependent shared context available from the first layer. Each block processes this shared information together with its own private features using independent parameters. We have revised the manuscript to clarify this mechanism and its role in cross-block information exchange, and removed the previous claim on over-smoothing properties to avoid misinterpretation.
Public and private CKA. We measured inter-block CKA on the same 10,000 ImageNet-1K validation images using four-block Swin-T models, with Pub-to-Pub and Priv-to-Priv enabled and Priv-to-Pub toggled off or on. CKA was computed before public aggregation and averaged over the six block pairs (Table 10).
| Feature type | Mean CKA, 12 layers (off → on) | Final-layer CKA (off → on) |
|---|---|---|
| Private channels | 0.5522 → 0.4876 | 0.6359 → 0.6365 |
| Public channels | 0.6040 → 0.7601 | 0.8257 → 0.9996 |
| Full features | 0.5802 → 0.6976 | 0.6371 → 0.9907 |
Private CKA decreases in 8 of 12 layers, with a lower mean but almost no change at the final layer. Public and full-feature CKA increase. Thus, higher similarity in shared public features does not necessarily indicate high similarity among private representations.
Input-dependent shared information. Because CKA and visualizations are correlational, we also performed inference interventions with fixed model weights. Across all 50,000 ImageNet-1K validation images, replacing each image's Priv-to-Pub output with that of another image reduces Top-1 accuracy from 82.44% to 72.49% or 72.47%, while the mean correct-class logit gap falls from 3.38 to 2.30 or 2.31 (Table 11). On a 10,000-image subset, mismatching only the final stage reduces accuracy from 82.54% to 76.30% (Appendix E.6). These results show that predictions depend on shared information matched to the input.
Training ablation. With Pub-to-Pub and Priv-to-Priv enabled, adding Priv-to-Pub improves Top-1 accuracy from 82.24% to 82.47% (Table 7), providing complementary evidence of its contribution.
The rebuttal substantially addresses my primary concern through matched-budget dense-widening comparisons, including a standard 12-layer ViT and a 300-epoch evaluation. The corrected pointwise-mixing baseline also strengthens the empirical evidence. I am revising my assessment upward toward weak acceptance.
Thank you for your encouraging feedback. We sincerely appreciate your constructive suggestion, particularly on the computational budget comparisons and pointwise-mixing baseline, which strengthen our experimental evaluation. We will continue refining the manuscript to improve its clarity and presentation. Thank you again for your time and careful consideration.
Response to Reviewer hjSU (part1)
Thank you for your careful review and constructive suggestions. We address the four concerns below, including the new matched-budget comparisons and analyses of the interaction mechanisms.
W1/Q1. Matched-budget dense-widening controls and whether gains come from added capacity
A1. PBI outperforms dense widening on both the 12-layer and 3-layer ViTs at comparable computational budgets. These new comparisons directly examine whether the gains can be explained by additional computational capacity alone.
12-layer ViT. Starting from the ViT-S design, we compared dense widening, MoE widening, dense deepening, and PBI widening using the same ImageNet-1K training recipe for 100 epochs. The results are reported in Table 6, with configurations and training details in Appendix E.
At approximately 49M parameters and 10 GMAC, PBI exceeds dense widening by 1.4 percentage points and dense deepening by 2.9 points. MoE has similar computation but more parameters (81M), and remains 0.6 points below PBI.
3-layer ViT. We also added Dense and MoE controls at the 9.3G and 11.8G budgets of the PBI models (Table 5). At 9.3G, the dense width is 1120, close to your estimate of 1130.
Dense and MoE use per-head QK normalization at both budgets. PBI exceeds dense widening by 7.9 and 5.7 percentage points, respectively. Unlike the equal-representation-width comparison in Table 4, these controls guarantee that all the methods are compared under the same computational budgets and provide more direct evidence for the effectiveness of PBI's expansion strategy.
W2/Q2. Differences from SCHEME and Shared Global Workspace and matched-budget comparisons
A2. Unlike SCHEME's grouped feature mixing and SGW's separate shared workspace, PBI enables information exchange among complete backbone blocks through shared public channels while preserving their private representations. Our budget-control comparisons further show that PBI outperforms both SCHEME-style and SGW-style adaptations.
SCHEME uses channel covariance attention to mix grouped FFN features, while Shared Global Workspace (SGW) introduces a separate shared state accessed through read and write operations. PBI instead maintains separate private feature streams across successive backbone layers and exchanges information between blocks through shared public channels. This design preserves the original internal convolution, attention, or state-space operators.
We compared three interaction mechanisms in a two-block, 12-layer ViT, including SCHEME-style CCA, SGW-style shared workspace, and PBI-S (ViT with PBI interactions). All models were trained on ImageNet-1K for 100 epochs (Table 9).
At comparable parameter and computational budgets, PBI-S exceeds the SCHEME-style and SGW-style adaptations by 6.7 and 6.5 percentage points, respectively.