- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TaskHC organizes each object query's persistent residual streams into separate classification and localization groups, so that task-specific cross-attention and prediction read from and write back to their designated group while shared self-attention and feed-forward modules reconstruct a joint state from the full stream bank. This design makes the sources and destinations of task information explicit across decoder layers, with cross-group residual mixing at shared modules integrating the complementary information accumulated by both tasks. Architectural ablations on DEIM-D-FINE-L show that group-specific reconstruction and writeback plus full-bank residual mixing improve detection, while extending specialization to self-attention or FFN hurts accuracy, and sampling/probe analyses confirm that each branch acquires task-appropriate visual evidence. Matched evaluations on COCO with DEIM-D-FINE and DEIM-RT-DETRv2 demonstrate consistent gains, e.g., 54.45 to 54.68 AP on DEIM-D-FINE-L for only 0.53 ms additional latency on an NVIDIA T4 with TensorRT FP16.
Critical strengths
- TaskHC introduces a clean, well-motivated architectural mechanism that generalizes a widely-used technique (Hyper-Connections) in a principled way.
- The paper provides direct, mechanistic evidence of learned functional specialization through sampling and frozen-state probe analyses
Critical weaknesses
TaskHC's logic is: traditional decoders use a single hidden state, classification and localization have different information needs, therefore they should be stored separately. But this reasoning skips a crucial step: what information does the traditional single stream actually lose?
In fact, in standard DETR-style architectures, classification and localization share the same query hidden state, but each has its own prediction head. The hidden state itself is a task-agnostic intermediate representation, and the prediction heads are responsible for extracting task-specific information. This design works precisely because the intermediate representation can simultaneously carry the information needed by both tasks, there is no need to separate them at the representation level.
TaskHC's ablation (Table 2) shows that when self-attention or FFN are also task-specialized, performance actually drops. This itself is a signal: task separation is not "the more, the better"; shared intermediate representations have value. But the paper does not squarely address this tension, instead treating "separating cross-attention and prediction is enough" as an established conclusion.
More critically, the paper does not compare the information capacity of "grouped streams vs. ungrouped streams." For instance: 4 streams have a total of 4C dimensions; after grouping, each group has 2C dimensions. If ungrouped, the working state reconstructed from 4 mixed streams is also C-dimensional. From an information-theoretic standpoint, grouping does not increase total capacity, it only changes the read/write scope. So does the improvement come from "information being stored separately" or from "a simpler read/write path" (fewer streams participating in reconstruction fewer noise sources)? The paper does not distinguish between these two explanations.
Questions for authors
- TaskHC's improvement on DEIM-D-FINE-L is +0.23 AP (54.45 → 54.68), which is indeed very small. The paper does provide multi-seed results in Appendix D.1. Mean ± std: Baseline 54.36 ± 0.17, TaskHC 54.54 ± 0.13, ΔAP = +0.18 ± 0.19. The key problem: the standard deviation of ΔAP (0.19) exceeds the mean (0.18). This means that on seeds 1 and 3, TaskHC shows essentially no improvement (+0.03 and +0.00), and one could argue the gain is entirely drowned out by noise. Two out of five seeds show basically no effect, which makes it hard to call this a "stable" improvement. The paper reports the "mean improvement of +0.18" but does not perform any statistical significance test (e.g., paired t-test), nor does it discuss the fact that the inter-seed variance far exceeds the effect size.
- The paper's approach is to stipulate that the two groups and are "responsible for" classification and localization respectively, then show through ablation that "this allocation beats full-bank access." But this only demonstrates that this organization is effective, it does not demonstrate that the model actually stores task-specific information in these streams. The authors may argue that the paper has offered two pieces of supporting evidence: 1) Sampling analysis (Figure 3): The classification branch's cross-attention samples more from object interiors, while the localization branch emphasizes peripheral regions. 2) Frozen probes (Figure 4): Classification probes perform better on CLS inputs (+1.39 pp), localization probes perform better on LOC inputs (+3.87 pp at IoU≥0.5). But these share a common problem: they show behavioral differences, not information-content differences.
- Sampling differences can be explained as "classification and localization modules have different computational needs," not necessarily "different information is stored in the streams." Probe differences can be explained as "the CLS/LOC prediction heads produce different outputs", which is almost tautological; if the two heads produced identical outputs, that would be the surprising result.
- To genuinely prove that "classification streams store classification-specific information," one would need intervention experiments: e.g., freezing or replacing the streams in and checking whether classification performance specifically degrades; or using linear probes to decode classification/localization information from each group separately and checking for cross-decoding (i.e., whether information is decoupled). The paper does not do this kind of experiment.
- If TaskHC's mechanism is effective, why is it validated only on the specific combination of classification + localization? The paper's conclusion says it "may also be applicable to settings with multiple interdependent tasks," but this is speculation without experimental support. Tasks are defined by humans; the model may spontaneously learn multiple information representations, is hard separation even necessary? Recent work on how mHC is actually used in practice found that in DeepSeek-V4-Flash, typical attention/FFN modules effectively use only about 2 streams, with the dominant stream shifting across depth, and in later layers, residual mixing nearly degenerates to identity, with streams essentially passing through independently. This suggests that models selectively use streams on their own, without needing human-specified assignment of which stream handles what.
- In other words, TaskHC's "functional grouping" may be an overly strong prior. If the model is capable of learning "which stream should store what" on its own, then artificially splitting streams into two groups and restricting read/write scope may actually limit the model's flexibility. The paper's ablation shows full-bank access is 0.23 AP worse than group-specific, but this gap is small and not validated across tasks, making it hard to rule out hyperparameter tuning or architecture-specific coincidence.
- The ablation in Table 2 is conducted on only one model (DEIM-D-FINE-L), with only four configurations (Prediction+CA, +FFN, +SA, +SA+FFN), and no cross-architecture validation (this ablation is not done on DEIM-RT-DETRv2). This is insufficient to support a general conclusion like "module role determines access scope."
4 comments
Stream use and intervention evidence
We agree that sampling and prediction-input probe differences describe how task branches acquire and process evidence, rather than directly identifying the information stored in persistent streams. We therefore added stream freezing and read-source swapping. The following combines TaskHC results from Table E.2 with the newly completed Full/Full control in trained DEIM–D-FINE-L models:
| Intervention | TaskHC Cls. (%) | TaskHC Loc (IoU ≥ 0.75, %) | Full/Full Cls. (%) | Full/Full Loc (IoU ≥ 0.75, %) |
|---|---|---|---|---|
| No intervention | 83.83 | 78.55 | 84.05 | 78.46 |
| Freeze streams 1–2 | 1.19 | 69.01 | 27.97 | 47.92 |
| Freeze streams 3–4 | 50.92 | 25.90 | 83.31 | 72.91 |
| Swap read sources | 9.86 | 0.15 | 69.05 | 35.08 |
In TaskHC, streams 1–2 form CLS and 3–4 form LOC. Freezing holds the selected streams at initialization across all six layers. Swapping exchanges the task read groups in TaskHC and the two stream halves at Full/Full readers, while preserving write destinations. TaskHC shows task-aligned sensitivity: freezing CLS most strongly harms classification, while freezing LOC most strongly harms localization. Full/Full is more sensitive to streams 1–2 than 3–4. This tests how the trained detector uses the streams; Appendix E.3 gives the detailed TaskHC analysis.
Thank you for recommending deeper stream analyses. Following these studies, we added experiments and visualizations of, e.g., effective stream counts, winner consistency, inter-stream representations, and residual mixing. Our six-layer detection decoder is much shallower than the large language models discussed in the review, with different computational and task demands. Visualizations show information transfer and interaction between streams. Mean load-effective counts are 3.27 for reading and 2.36 for writing; see Appendix F for details.
Detection gains and reliability
COCO 2017 contains approximately 118,000 training images and 5,000 validation images. It remains challenging because detectors must recognize and precisely localize objects across clutter, occlusion, and large scale variation, including small objects with little visual evidence. Its AP evaluation spans 80 categories and IoU thresholds from 0.50 to 0.95. Improving strong real-time baselines adds a further constraint: gains must be obtained within limited model and inference budgets. Our DEIM–D-FINE and DEIM–RT-DETRv2 baselines already incorporate substantial architectural and training improvements, making further accuracy gains demanding.
Related studies also reflect the challenge of further improving strong real-time detection baselines. RT-DETRv3: Real-Time End-to-End Object Detection with Hierarchical Dense Positive Supervision improves RT-DETRv2 by 0.2/0.3 AP with ResNet-34/101. In RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models, 36-epoch experiments on RT-DETRv2-L show +0.2 AP from DSI and a further +0.3 from GAM; on D-FINE-L, these increments are +0.1 and +0.2. These findings reflect the challenge of obtaining a large gain from a single methodological improvement on an already strong baseline. We assess the reliability of TaskHC's gains through our own repeated experiments.
Repeated training also requires substantial computation: in our experimental setup, a training run takes approximately 2.5 days on a single GPU. We added 13 standard TaskHC trainings, expanding 15 comparisons to 28 across ten configurations. Equal-configuration analysis gives +0.1622 AP for the original set (95% CI: 0.0792 to 0.2451; p = 0.001664) and +0.1876 for the expanded set (95% CI: 0.1188 to 0.2564; p = 0.000165). Both support positive average gains (Appendix D.1; Tables D.8, D.9, and D.10). Additional datasets and architectures support generalization (Table 2 and Table 3).
We also repeated the main ablations on DEIM–D-FINE-L with seed 1 and DEIM–RT-DETR-L, covering state management, module topology, and read/write scopes (Appendix D.2). These results support functional grouping and task-specific CA/prediction with joint SA/FFN across initialization and detector framework. The seed-1 repetition also supports full-bank residual mixing at shared modules.
Our work focuses on state management for real-time DETR-based object detection, using Hyper-Connections to organize and update persistent representations for classification and localization. The broader statement in our conclusion was intended as a research outlook: task-structured state management may offer a useful approach when interacting tasks have distinct information needs. We apologize that the wording did not clearly distinguish this prospect from our demonstrated scope. We do not suggest that all tasks require explicit grouping; its value for other task combinations remains to be investigated.
Re: Official Comment by Authors
Thank you so much for the authors' thorough experiments and the substantial updates to the paper. I truly appreciate the care and effort that went into the rebuttal and the clarification of the design motivation. I fully recognize the value the authors see in this design, and I want to emphasize that my hesitation is not due to any lack of effort or seriousness on the authors' part.
Unfortunately, and with real regret, I find myself unable to give a positive score at this stage. Since the theoretical contribution of this work is relatively limited, a clear and substantial empirical improvement would have been important for me in assessing its value. My main concern remains that the current design is more complex than the original implementation, while the gains on the main metrics are rather modest. This makes it genuinely difficult for me to confirm the necessity of the added complexity from an empirical standpoint. Given the large volume of submissions at ICLR, I am afraid the current contribution is not yet enough for me to justify a positive score, and I say this with sincere reluctance.
I want to thank the authors once again for their careful response and hard work, and I genuinely hope this work can be further strengthened in future revisions.
Thank you for your thoughtful follow-up and for recognizing the additional experiments and clarifications. We understand that your remaining concern is whether the empirical benefit justifies the added complexity. We would like to address this with training-trajectory evidence, a breakdown of the added structure, and the engineering optimizations we made to further reduce inference latency in TensorRT.
1. How the AP gain evolves during training
In the paper, we focused on AP after the full training schedule. Given your concern about the modest final gains, we examined whether that final gap reflects the benefit of TaskHC over the course of training. The table compares COCO validation AP at matched epochs on the smallest model in each DEIM detector family, averaging the gain within each interval.
| Training epochs | D-FINE-N ΔAP | RT-DETRv2-R18 ΔAP |
|---|---|---|
| 1–10 | +1.43 | +0.87 |
| 11–20 | +1.16 | +0.49 |
| 21–30 | +0.93 | +0.23 |
| 31–40 | +0.56 | +0.40 |
| 1–50 | +0.87 | +0.48 |
| Full training | +0.24 | +0.37 |
On both detectors, TaskHC takes an early lead in COCO validation AP. Over the first 50 epochs, D-FINE-N and RT-DETRv2-R18 gain an average of 0.87 and 0.48 AP over their respective baselines. The gap narrows late in training, leaving gains of 0.24 and 0.37 AP after the full schedule. This sustained lead also translates into fewer epochs to reach the same accuracy: with TaskHC, RT-DETRv2-R18 first reaches 48 AP ten epochs earlier (87 vs. 97).
2. What TaskHC adds to the detector
TaskHC makes two straightforward changes to the decoder. It adds a second copy of the existing cross-attention (CA) module, giving classification and localization branches of equal width with independent weights. It also adds HC controllers to manage how the branches read and update the residual streams. The additional CA branch accounts for most of the added parameters. On DEIM–D-FINE-L, the parameter count increases from 30.77M to 33.26M. Of the 2.49M added parameters, 0.47M belong to the controllers, including prediction readout (18.8% of the increase and 1.4% of the full model). Control and routing use 0.32 GFLOPs, or 20.7% of the additional 1.55 GFLOPs and 0.35% of the model’s 92.69 GFLOPs.
The two CA branches also give future detectors a way to adapt evidence acquisition to each task. In this study, we give the branches equal capacity so that classification and localization have the same sampling budget. Within the same state-management framework, future work could vary branch width or the number and placement of sampling points independently, and study which allocation best serves recognition and box refinement under a fixed decoder budget.
3. TensorRT inference graph optimization
Our previous TensorRT implementation already sampled encoder features jointly for the two CA branches. We further optimized the TensorRT inference graph by batching projections across the branches, fusing the controller and stream-update operations, and fusing the prediction readout. These optimizations preserve the trained weights, parameter count, and theoretical FLOPs of each TaskHC model. Compared with our previous implementation, these changes reduce latency for DEIM–D-FINE-L + TaskHC from 9.00 to 8.15 ms (9.4%) and for DEIM–RT-DETR-L + TaskHC from 10.19 to 8.83 ms (13.4%). We used the same weights for each comparison and measured all implementations on an NVIDIA T4 with TensorRT FP16 and batch size 1. Both optimized models surpass their native baselines in inference speed: D-FINE-L reaches 122.6 FPS versus 118.2 FPS, and RT-DETR-L reaches 113.3 FPS versus 104.5 FPS. Across both detector families, optimized TaskHC exceeds native baseline throughput in nine of ten configurations. Some CA optimizations also reduce baseline latency. See Appendix C for the deployment analysis.
We appreciate your close attention to the cost of the added structure, which prompted us to revisit the implementation and further reduce inference latency.
Thank you for raising two important questions: why group streams explicitly, and why specialize only some modules.
1. Why explicit grouping?
We agree that the shared intermediate query representation in standard DETR can carry information needed by both classification and localization. This does not mean that leaving the model to discover how to organize and use that information is the most effective approach for detection.
For example, PLE (Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations, RecSys 2020) explicitly separates task-specific and shared experts while retaining learned gating. Its experiments show that this structural prior can reduce negative transfer and improve performance over adaptive sharing without explicit expert roles. mHC-HSI: Clustering-Guided Hyper-Connection Mamba for Hyperspectral Image Classification assigns physically meaningful spectral groups to residual streams while learning their interactions.
TaskHC also retains adaptive stream use: within each functional group, the model learns how to combine and update its streams and what they represent. Shared operations allow information exchange across groups.
Our grouping inherits a detector-specific prior from DESTR (DESTR: Object Detection With Split Transformer) and SD-DETR (Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection). These methods extend the original single intermediate representation into two explicitly maintained state variables for classification and localization, and report detection improvements. TaskHC adopts this established decomposition to investigate a different state-management mechanism. Revised Table 4(a) adapts the prior explicit-state scheme to the same detector and compares it with HC-gated state management and full TaskHC. Keeping the task decomposition fixed makes this a comparison of how task states are maintained and coupled under the same functional prior.
2. Controlled evidence
We test this directly. Full bank / Full bank removes the explicit task partition from read/write access: both task-specific CA branches retain their own controllers, but can reconstruct their working states from all four streams and write back to all four. Thus, the model can learn how each task uses the full bank. Task group / Task group instead constrains each task to its assigned two streams. Both retain four C-dimensional streams, two CA branches, C-dimensional working states, and joint SA/FFN. The grouped design performs better in the original comparison, a second random seed, and another detector framework (Table 5(a), Table D.6, and Table D.3):
At fixed stream capacity, task grouping improves AP by 0.23, 0.38, and 0.32 across these settings. These results support the inherited functional prior over learning unrestricted full-bank access in the evaluated detection settings.
3. Why selective sharing?
The original manuscript discusses these roles in Section 4.3, “Placement of task-specific computation” (p. 7, below Table 2). We extend DESTR/SD-DETR's task-specific CA/FFN and shared-SA topology within TaskHC:
These roles motivate task-specific CA/heads and C-dimensional joint SA/FFN. Joint processing also avoids widened attention and duplicated FFNs. Original and repeated ablations support this division in accuracy and resource cost; broader task separation is not consistently beneficial.