- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In my view, the main contribution of this paper is to establish a new state of the art method, DiGSA, a more efficient and differentiable attention framework for large-scale face clustering, primarily improving upon Diff-Cluster, with fewer parameters, better performance, and faster inference. Although the proposed design is sufficiently clever and technically well-motivated, I believe its broader impact on the field is relatively limited. The authors provide enough evidence for readers to gain a reasonable understanding of the effectiveness of the proposed method. However, the empirical presentation is relatively conservative, with limited additional analysis or visualization to further demonstrate why and how the individual components are effective.
From my personal perspective, I would have preferred a more systematic discussion and evaluation of the proposed design. For example, beyond the current ablations, it would be helpful to provide a more thorough sensitivity analysis of key design choices, clearer qualitative visualizations showing how the proposed components affect the learned graph or attention structure, and a more explicit discussion of failure cases and the conditions under which each component is expected to be beneficial. Such analyses would make the empirical contribution more convincing and provide deeper insights beyond the reported performance gains.
Critical strengths
One clear strength is that the methodological motivation is coherent. Each design choice directly targets a specific limitation of Diff-Cluster: hard Top-K masking is replaced with differentiable sparsity, while sequence-oriented positional encoding is replaced with graph-aware structural encoding.
Another positive aspect is that the paper improves accuracy, efficiency, and model size simultaneously. DiGSA achieves slightly better results than Diff-Cluster across all five MS1M scales while using fewer parameters and less computation.
The paper also includes reasonable component-level ablations for DASA, GSDA, GraphPE, and the learnable temperature, which helps verify that the proposed components contribute positively to the final performance.
In addition, the appendix contains several useful supplementary analyses, including run-to-run stability, the roles of the primary and secondary GSDA maps, controlled similarity perturbation, and cross-dataset / cross-architecture transfer.
Critical weaknesses
The main concern is not the empirical performance, as the proposed method achieves state-of-the-art results while substantially reducing computational cost and model complexity. In fact, obtaining comparable or better accuracy with fewer parameters, lower FLOPs, and higher throughput is one of the strongest aspects of this work.
My main concern instead lies in the degree of conceptual novelty and the depth of the analysis. The proposed framework is carefully designed, but several of its key ingredients are built upon established techniques: DASA relies on entmax-based sparse normalization with a learnable temperature, while GraphPE uses degree statistics and random-walk positional encodings. GSDA provides a more task-specific contribution, but the overall framework can still be viewed as a well-engineered integration and adaptation of existing ideas to improve Diff-Cluster.
Therefore, while the empirical contribution is strong, I find the methodological contribution relatively incremental. The paper demonstrates that the proposed design is effective, but provides relatively limited insight into why these components are particularly suitable for face clustering beyond the specific baseline being improved.
I would also have preferred a more systematic analysis of the proposed components. For example, the paper could further investigate the learned sparsity patterns and per-head temperatures in DASA, the sensitivity to KNN graph density or subgraph size, and the conditions under which GSDA provides the largest benefit. Such analyses would help establish broader insights beyond the reported efficiency and accuracy improvements.
Questions for authors
DiGSA is largely built upon Diff-Cluster, which itself adopts the differential attention mechanism from Differential Transformer. One aspect I am particularly interested in is the system-level efficiency and deployment implications of retaining differential attention as the core design.
My understanding is that differential attention requires computing two attention distributions and subsequently subtracting them. While efficient implementations such as FlashAttention are possible for Differential Transformer, this computation does not necessarily map as directly to the standard single-attention SDPA execution path, and its practical efficiency may depend substantially on the underlying kernel implementation and hardware backend. Therefore, parameter count, FLOPs, and even isolated model latency may not fully characterize the actual deployment efficiency. In other words, lower FLOPs or lower measured latency does not necessarily imply proportionally better end-to-end deployment efficiency.
This is particularly relevant here because the paper emphasizes efficiency improvements over Diff-Cluster through reductions in parameters, FLOPs, latency, and increased throughput, while the reported measurements mainly characterize the relation model itself.
I would therefore like to better understand the authors' principled rationale for building DiGSA around differential attention. In particular:
- Is differential attention fundamentally necessary for face clustering, or could the proposed graph-structured sparsity and graph-aware biases be incorporated into a conventional single-attention formulation with potentially simpler deployment?
- What practical trade-offs does differential attention introduce in terms of memory traffic, kernel fusion, peak memory usage, and compatibility with commonly used optimized attention backends?
- Have the authors considered or benchmarked end-to-end deployment efficiency across different hardware (it is optional if the rebuttal period is limited for authors) or attention backends, rather than relying primarily on parameters, FLOPs, and model-level latency?
- If my understanding regarding the implementation or deployment limitations of differential attention is inaccurate, I would appreciate clarification on how these issues are handled in the current implementation.
This is probably the question I am most interested in regarding the paper. A convincing explanation of why differential attention is essential here, together with evidence that its system-level costs are properly addressed, would substantially strengthen my view of the work, and I would consider increasing my score. Otherwise, I would likely maintain my current rating and would not recommend acceptance.
3 comments
- What practical trade-offs does differential attention introduce in terms of memory traffic, kernel fusion, peak memory usage, and compatibility with commonly used optimized attention backends?
Thank you for this question, your observation on deployment details is very accurate. Compatibility with existing optimized attention backends is indeed problematic for our method: FlashAttention supports neither the adjacency-guided biases ( and ) in GSDA nor entmax (α=1.5); AdaSplash supports entmax (α=1.5) but not the adjacency-guided biases in GSDA. Since neither FlashAttention nor AdaSplash supports the adjacency-guided biases, we first attempted to solve this problem with PyTorch's FlexAttention: FlexAttention allows modifying every attention score, and the adjacency-guided biases are exactly additive correction terms on each score, so we injected the bias into FlexAttention's callback in the form of "score + coefficient × bias". Since all 16 attention heads share the same adjacency matrix and differ only in their per-head coefficients, we pass "one matrix + 16 coefficients" instead of replicating the matrix 16 times, thereby avoiding extra memory overhead. However, when addressing the support for entmax (α=1.5), we found that FlexAttention does not support it either. To thoroughly resolve both problems, we wrote a dedicated Triton kernel from scratch, which fuses bias injection, entmax normalization, and the differential subtraction into a single pass, avoiding explicit materialization of intermediate attention maps and thus reducing peak memory usage and memory traffic.
The experimental results are summarized in the table below (batch size = 32, NVIDIA A6000)
Kernel Method bs=32 Latency bs=32 Throughput PyTorch Diff-Cluster (Top-k softmax, diff attention) 26.01 ms 1,230/s PyTorch Our DiGSA (entmax α=1.5, w/ bias, diff attention) 12.80 ms 2,500/s FlashAttention Our DiGSA (softmax, w/o bias, diff attention) 3.84 ms 8,328/s AdaSplash Our DiGSA (entmax α=1.5, w/o bias, diff attention) 5.21 ms 6,137/s FlexAttention Our DiGSA (softmax, w/ bias, diff attention) 4.65 ms 6,884/s Our Triton kernel Our DiGSA (entmax α=1.5, w/o bias, diff attention) 3.97 ms 8,068/s Our Triton kernel Our DiGSA (entmax α=1.5, w/ bias, diff attention) 4.15 ms 7,714/s Our Triton kernel Our DiGSA (entmax α=1.5, w/ bias, single attention) 3.94 ms 8,119/s As shown in the table: Diff-Cluster suffers from a high inference latency of 26.01 ms due to its MoE and Top-K operations, whereas the PyTorch implementation of our DiGSA takes only 12.80 ms, a 2.03× speedup; our Triton kernel further reduces the latency to 4.15 ms, a 3.08× speedup over the PyTorch baseline (12.80 ms), raising the throughput from 2,500/s to 7,714/s; this latency is comparable to FlashAttention (3.84 ms), which supports neither bias nor entmax, and is faster than AdaSplash (5.21 ms) and FlexAttention (4.65 ms). In summary, the system-level overhead introduced by differential attention can be effectively eliminated through kernel-level optimization, and our method achieves high practical deployment efficiency.
- Have the authors considered or benchmarked end-to-end deployment efficiency across different hardware (it is optional if the rebuttal period is limited for authors) or attention backends, rather than relying primarily on parameters, FLOPs, and model-level latency?
Thank you for your suggestion. Since DiGSA adopts exactly the same KNN graph construction method and clustering algorithm as Diff-Cluster, with the only difference lying in the node-relation prediction network, we focus our comparison on the inference speed of the relation prediction network across different hardware. It is worth noting that face clustering is a fundamental application in mobile photo albums (e.g., the face grouping feature on Samsung smartphones), so on-device deployment efficiency is of direct practical value. Following your suggestion, we not only benchmarked the inference latency of different kernels but also conducted on-device tests on a Samsung Galaxy Z Fold 8:
Platform Method Avg. Latency Peak Memory Samsung Z Fold 8 Diff-Cluster (diff attention) 216.82 ms 93.99 MB Samsung Z Fold 8 Our DiGSA (diff attention) 5.74 ms 86.91 MB Samsung Z Fold 8 Our DiGSA (single attention) 5.50 ms 84.23 MB As shown, on the mobile device Diff Attention (5.74 ms) and Single Attention (5.50 ms) have nearly identical inference times, whereas Diff-Cluster requires 216.82 ms, while our method achieves a 37.8× speedup with lower peak memory (86.91 MB vs. 93.99 MB). This further confirms that our DiGSAdifferential attenti introduces almost no additional overhead in real-world deployment.
- If my understanding regarding the implementation or deployment limitations of differential attention is inaccurate, I would appreciate clarification on how these issues are handled in the current implementation.
Thank you for your patience and your careful attention to implementation details. We hope our Responses 1–3 have clarified how these implementation and deployment limitations are handled in our current implementation. In brief: (i) the incompatibility with existing attention backends has been fully resolved by our dedicated Triton kernel; (ii) after optimization, differential attention runs at nearly the same latency as single attention. If you have any further questions or concerns, please feel free to let us know, because your suggestions are highly valuable to us.
- This is probably the question I am most interested in regarding the paper. A convincing explanation of why differential attention is essential here, together with evidence that its system-level costs are properly addressed, would substantially strengthen my view of the work, and I would consider increasing my score. Otherwise, I would likely maintain my current rating and would not recommend acceptance.
Thank you for focusing on this core question, which is indeed central to our work. Regarding "why differential attention is essential": as shown in Response 1, for both Diff-Cluster and our DiGSA, differential attention consistently and significantly outperforms single attention, demonstrating that the differential mechanism, which subtracts the secondary attention map from the primary one to cancel cross-community noise, is the fundamental source of the performance gains. Regarding "evidence that its system-level costs are properly addressed": as shown in Responses 2 and 3, with our dedicated Triton kernel, the inference latency of Diff Attention is nearly identical to that of Single Attention (4.15 ms vs. 3.94 ms on the A6000; 5.74 ms vs. 5.50 ms on the Z Fold 8), reaching a speed comparable to FlashAttention while consuming less memory. In summary, differential attention delivers consistent performance gains at almost no additional system cost, and the two goals are achieved simultaneously. We hope the above experimental evidence and explanations can address your concerns, and we would be grateful if you would take them into account in your evaluation. Thank you again for your valuable comments!
We sincerely thank you for your recognition of our work and for your valuable time and constructive comments. We fully agree with you that the practical efficiency of a model may depend heavily on the underlying kernel implementation and hardware backend, and that the number of parameters and FLOPs may not fully reflect real-world deployment efficiency. Following your suggestions, we have conducted additional experiments to answer your questions and address your concerns. We hope our responses will resolve your concerns.