- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the information discarded during non-injective aggregation processes. By constructing null-space residuals and learnable complementary branches, the authors incorporate member-level information into graph message passing and visual token merging. The study includes experiments on graph classification, node classification, tree-neighborhood memory, and semantic segmentation, while providing comparisons involving raw member inputs and post-aggregation processing. Although the construction of linear residuals is well-grounded, their individual contribution, actual computational cost, and robustness across different settings remain to be fully substantiated.
Critical strengths
Addressing the issue of information loss during the neural network aggregation process from the perspective of null space offers a clear theoretical motivation and a versatile methodology. It yields significant performance improvements in both computer vision and graph learning tasks, with particularly notable results in high-compression scenarios.
Critical weaknesses
- In Table 14, the mIoU for the 3.4× compression setting (using ToMe) improves from a baseline of 18.31 to 49.12 with "RawSource," and further to 49.82 with "NSR." Most of this gain stems from the full complementary branch, rather than being attributable solely to null-space projection.
- The gating mechanism described in Section 3 takes the residual, member, and aggregated representations as inputs. Existing ablation studies do not disentangle the effects of the adaptive gating from those of the residual encoding; thus, it remains unclear at which stage task-relevant information is actually filtered.
- Experiments involving token compression are compared under identical compression ratios; however, NSR introduces additional parameters and computational operations. Consequently, these comparisons reflect the benefits of the added branch rather than an accuracy advantage under a fixed total computational budget.
- The computational complexity analysis in Appendix B excludes overheads such as indexing and padding, while the spatial branch retains original positional encodings. In the absence of empirical latency and peak VRAM usage measurements, the link between reduced computational complexity and actual resource efficiency remains unverified.
- Semantic segmentation experiments rely exclusively on Tiny-DeiT/16 paired with a linear decoding head. This is insufficient to demonstrate the generalizability of the proposed mechanism across backbones of varying scales or decoders capable of spatial reconstruction.
- In Appendix A, segmentation results are defined based on the final checkpoint from a single training run. In Table 14, the observed advantage of using residual inputs is only 0.20–0.24 mIoU points—a margin too small to be reliably distinguished from random fluctuations.
- In the tree-neighborhood memory experiments, NSR incorporates non-linear processing at the member level prior to aggregation. The study lacks a control condition with equivalent-capacity member processing, making it impossible to isolate whether the performance gains are truly attributable to the null-space construction.
- The ZINC experiments in Section 4 do not utilize edge (bond) types, while the graph-branch ablation in Appendix G is limited to the "Roman-empire" dataset. There is no direct evidence to substantiate the claims regarding the contributions to molecular tasks or the effective utilization of chemical relational information.
- The diagnostic quantity in Appendix E is constructed using ground-truth labels rather than learned latent features; while it can characterize the degree of label mixing, it cannot directly verify whether the model has truly recovered the task-relevant features that were discarded during aggregation.
Questions for authors
- Table 14 shows that RawSource accounts for most of the performance gain. How can the authors isolate the specific contribution of null-space projection from the auxiliary branch and adaptive gating?
- Since NSR introduces additional parameters and computation, how does it compare with baselines under matched total computational budgets? Can actual inference latency and peak memory usage be clarified?
- How stable are the reported improvements across random seeds, particularly the small gains over RawSource? Would the conclusions hold with larger backbones or more capable decoders?
- Can the authors provide stronger evidence that NSR recovers task-relevant information rather than benefiting from additional member-wise processing? How well do the label-based diagnostics support this mechanism?
6 comments
Q3. Experimental Stability and Stronger Visual Models
Regarding the evaluation protocol, the paper reports semantic-segmentation validation mIoU from the final checkpoint of a single training run; heterophilic node classification uses ten official splits, and ZINC uses ten random seeds. To further examine the Full NSR–RawSource differences, we extended the input-source ablations to ADE20K. Full NSR exceeds RawSource by 1.42, 2.04, and 3.29 mIoU points at ToMe 2.5×, 3.4×, and 3.8×, respectively (Table C4). Full NSR outperforms RawSource at all four ADE20K compression settings, showing consistent positive improvements across the evaluated compression strengths and providing further evidence for the stability of this residual-source benefit.
We also evaluated a larger DeiT-Base/16 backbone with an FCN segmentation head on VOC. NSR improves mIoU in all 12 compression configurations across ToMe, PiToMe, and MPM, including an increase from 31.18 to 51.39 for ToMe 3.4× and from 28.43 to 48.67 at 3.8× (Table C5). These results further demonstrate the value of explicit null-space residuals across ADE20K compression settings and the effectiveness of the complete NSR framework with a larger backbone and stronger segmentation head.
Q4. Evidence for Task-Relevant Information and the Label-Based Diagnostic
Your question concerns a central motivation of NSR: whether member differences invisible to the current aggregation operator remain relevant to downstream tasks. We examine this from three complementary perspectives: a controlled task with explicit semantic requirements, a label-based null-space diagnostic, and input-source ablations.
First, Tree-NeighborsMatch: the sibling-swap construction demonstrates that, under identical local aggregation weights, swapping values associated with different keys can leave the aggregated output unchanged while changing the correct answer. This gives a concrete example of a conflict between aggregation-induced indistinguishability and the key–value associations required by the task. Second, the label-based diagnostic in Appendix E: it measures the projection of task-label differences onto the current aggregation null space. Figures 4 and 6 show that, as diagnostic scores increase, original GNN accuracy generally decreases while NSR remains relatively stable, widening the accuracy gap. The diagnostic is used only for post-training analysis, not for training or model selection. Third, RawSource versus Full NSR: comparing predictive performance within the same compensation architecture shows the additional value of explicit null-space separation, with consistent improvements on Roman-empire and ADE20K (Table C4; Tables 14–15 in the paper).
Taken together, Tree-NeighborsMatch provides a concrete example of task-relevant local indistinguishability; the label diagnostic reveals the relationship between task-label structure and model performance; and RawSource ablations demonstrate the additional value of explicitly processing null-space information in actual prediction tasks. These complementary findings support NSR's central motivation: non-injective aggregation may eliminate member differences still required by downstream tasks, and explicitly extracting and using these differences under task supervision can improve predictive performance.
Additional Clarifications on Weaknesses
(1) Member-wise Encoding and Adaptive Gating
Null-space projection constructs from the current operator. The encoder transforms the residuals into learnable representations, and member-wise gating produces based on each member and its context, adaptively modulating their contributions. For residuals satisfying , a shared linear transformation still cancels under the original weights:
In the visual implementation, member-wise RMSNorm and context-dependent sigmoid gates make the processed codes different from a single shared linear transformation, enabling effective complementary information. The RawSource and Member-wise MLP ablations further demonstrate the value of information-source choice and the complete member-wise processing mechanism. Projection, encoding, and gating have distinct but interconnected roles: projection defines the information source, encoding transforms its representation, and gating learns how to use it according to member context.
(2) Tree-NeighborsMatch Member-Level Controls
Following your suggestion, we added RawSource and Post-aggregation-only controls on Tree-NeighborsMatch under the task construction and training-fit evaluation settings of the original Alon & Yahav (ICLR 2021) paper. At , RawSource achieves 89%, 78%, and 85% maximum training accuracy for GCN, GIN, and GraphSAGE, respectively, whereas Full NSR reaches 100% for all three (Table C7). These results further distinguish post-aggregation processing, processing of original member representations, and explicit null-space residual processing in long-range matching.
(3) ZINC Inputs and Cross-Task Validation
Our three graph settings—heterophilic node classification, Tree-NeighborsMatch, and ZINC regression—all use GCN, GraphSAGE, and GIN as graph-aggregation backbones. To maintain consistent aggregation mechanisms and model designs across graph tasks, the ZINC experiments use GNNs based on node features and graph connectivity without introducing additional bond-type-specific message passing. Specifically, ZINC uses atom types and molecular connectivity to examine whether the same NSR aggregation-compensation mechanism extends from node classification and controlled matching to graph-level continuous regression. Table 3 in the paper reports test MAE over ten random seeds, with improvements from NSR for all three backbones, further demonstrating the applicability of the same compensation mechanism across graph prediction tasks.
In response to your questions, we will consolidate the Member-wise MLP controls and ADE20K input-source and integration-pathway ablations in the revised appendix.
We sincerely thank you again for your insightful comments on our methodology, experimental design, and computational fairness. Your emphasis on experimental stability was especially valuable: it prompted us to extend the input-source ablations to ADE20K, where we observed consistent improvements across compression settings, allowing us to examine the value of explicit null-space residuals more thoroughly. Your careful analysis has helped us clarify the contribution of the complete NSR framework and better connect its theoretical motivation with the experimental results.
We sincerely thank you for your careful review and insightful comments. We especially appreciate your detailed analysis of the specific contribution of null-space projection, the roles of the auxiliary pathway and adaptive gating, computational fairness, and evidence for task-relevant information.
NSR uses the null space of the currently realized non-injective operator to address a potential mismatch between operator-induced input indistinguishability and distinctions required by downstream tasks. It is a complete framework combining null-space residual extraction, member-wise encoding and adaptive gating, and task-relevant information integration.
Regarding your observation that RawSource already accounts for most of the performance improvement, we would particularly like to clarify its relationship to Full NSR. RawSource uses original member representations, which already contain null-space components. Thus, their comparison does not test whether null-space information exists; it evaluates the additional value of explicitly separating operator-invisible components and processing them within the same compensation architecture. On ADE20K, Full NSR exceeds RawSource by 2.04 and 3.29 mIoU points at 3.4× and 3.8× compression, respectively.
In response to your questions, we added Member-wise MLP controls, ADE20K input-source ablations, comparisons at nearly matched total GFLOPs, a larger visual model, and measured inference-efficiency results. Complete results are consolidated in the shared Official Comment. We address your questions individually below.
Q1. Separating the Contributions of Projection, the Auxiliary Pathway, and Adaptive Gating
We examine these factors using three controls. Post-aggregation-only processes only the aggregated representation; Member-wise MLP applies a pre-aggregation Linear–GELU–Linear transformation without null-space projection or NSR gating; and RawSource retains NSR's full encoding, gating, and integration architecture, replacing only the null-space residuals with original member representations.
For the currently realized linear operator, X=UY+Z and AZ=0. Therefore, the original representations used by RawSource already contain null-space information. RawSource's improvement reflects the value of the complete member-wise compensation pathway, while its comparison with Full NSR isolates the additional value of explicitly separating operator-invisible components within the same architecture. On Roman-empire, Full NSR exceeds RawSource by 3.69, 2.14, and 1.56 percentage points for GCN, GraphSAGE, and GIN, respectively; it also exceeds nearly parameter-matched MLP controls by 7.53, 4.36, and 4.83 points, with only 260 parameters difference per pair (Table C1; Table 15 in the paper). On ADE20K at ToMe 3.4×/3.8×, Full NSR exceeds RawSource by 2.04/3.29 mIoU points and MLP by 3.52/4.34 points (Table C4).
These comparisons distinguish the benefit of the complete framework over ordinary member-wise processing from the additional contribution of explicit null-space separation. Projection defines the complementary information source; encoding and gating learn how to process it; and task-relevant integration enables it to affect downstream predictions.
Q2. Matched Computational Budgets and Actual Costs
Thank you for your careful attention to computational fairness. We added nearly parameter-matched controls, comparisons at similar total GFLOPs, and measurements of inference latency, throughput, and peak memory. The Member-wise MLP parameter comparisons appear in Table C1. At nearly identical total GFLOPs on ADE20K, ToMe-NSR-3.8× achieves 33.74 mIoU at 3.082 GFLOPs, compared with 17.67 mIoU at 3.085 GFLOPs for original ToMe-3.4×, a gain of 16.07 points. The corresponding PiToMe comparison is 34.17 versus 31.39 mIoU, a gain of 2.78 points (Table C9). Moreover, ToMe-NSR-3.4× reaches 34.11 mIoU at 3.443 GFLOPs, surpassing original ToMe-2.5× at 31.44 mIoU and 4.202 GFLOPs.
Measured inference results are given in Table C6. With DeiT-Base + FCN on VOC, adding NSR to ToMe 3.4× improves mIoU from 31.18 to 51.39, while inference latency rises from 36.86 to 41.06 ms/batch and peak memory from 1339.94 to 1346.35 MiB. These results directly show the predictive advantage at comparable theoretical compute budgets alongside the associated measured resource costs.