acceptodds
Under review as a conference paper at ICLR 2027

TaskHC: Function-Structured Persistent States with Operation-Dependent Access

10 views

Abstract

Classification and localization require different visual evidence, yet standard detection transformers accumulate and integrate information for both tasks in a single shared state per query. We introduce TaskHC, which extends Hyper-Connections by organizing persistent residual streams into functional groups. Task-specific working states are reconstructed from designated groups, and task updates are written back to them; shared modules reconstruct a joint working state from the full stream bank and mix residual information across groups. Architectural ablations show that group-specific reconstruction and writeback improve detection over access to the full stream bank with the task-specific modules held fixed, and that shared modules benefit from cross-group mixing. Sampling analyses and frozen-state probes reveal specialization in evidence acquisition and task-information accessibility. On COCO, TaskHC improves matched DEIM–D-FINE and DEIM–RT-DETRv2 baselines. For DEIM–D-FINE-L, AP increases from 54.45 to 54.68 with 0.53 ms additional latency on an NVIDIA T4 using TensorRT FP16.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
detection transformers, object detection, representation learning, hyper-connections, persistent residual streams, task specialization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_na5tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TaskHC organizes each object query's persistent residual streams into separate classification and localization groups, so that task-specific cross-attention and prediction read from and write back to their designated group while shared self-attention and feed-forward modules reconstruct a joint state from the full stream bank. This design makes the sources and destinations of task information explicit across decoder layers, with cross-group residual mixing at shared modules integrating the complementary information accumulated by both tasks. Architectural ablations on DEIM-D-FINE-L show that group-specific reconstruction and writeback plus full-bank residual mixing improve detection, while extending specialization to self-attention or FFN hurts accuracy, and sampling/probe analyses confirm that each branch acquires task-appropriate visual evidence. Matched evaluations on COCO with DEIM-D-FINE and DEIM-RT-DETRv2 demonstrate consistent gains, e.g., 54.45 to 54.68 AP on DEIM-D-FINE-L for only 0.53 ms additional latency on an NVIDIA T4 with TensorRT FP16.

Critical strengths
  • TaskHC introduces a clean, well-motivated architectural mechanism that generalizes a widely-used technique (Hyper-Connections) in a principled way.
  • The paper provides direct, mechanistic evidence of learned functional specialization through sampling and frozen-state probe analyses
Critical weaknesses

TaskHC's logic is: traditional decoders use a single hidden state, classification and localization have different information needs, therefore they should be stored separately. But this reasoning skips a crucial step: what information does the traditional single stream actually lose?

In fact, in standard DETR-style architectures, classification and localization share the same query hidden state, but each has its own prediction head. The hidden state itself is a task-agnostic intermediate representation, and the prediction heads are responsible for extracting task-specific information. This design works precisely because the intermediate representation can simultaneously carry the information needed by both tasks, there is no need to separate them at the representation level.

TaskHC's ablation (Table 2) shows that when self-attention or FFN are also task-specialized, performance actually drops. This itself is a signal: task separation is not "the more, the better"; shared intermediate representations have value. But the paper does not squarely address this tension, instead treating "separating cross-attention and prediction is enough" as an established conclusion.

More critically, the paper does not compare the information capacity of "grouped streams vs. ungrouped streams." For instance: 4 streams have a total of 4C dimensions; after grouping, each group has 2C dimensions. If ungrouped, the working state reconstructed from 4 mixed streams is also C-dimensional. From an information-theoretic standpoint, grouping does not increase total capacity, it only changes the read/write scope. So does the improvement come from "information being stored separately" or from "a simpler read/write path" (fewer streams participating in reconstruction →\rightarrow fewer noise sources)? The paper does not distinguish between these two explanations.

Questions for authors
  • TaskHC's improvement on DEIM-D-FINE-L is +0.23 AP (54.45 → 54.68), which is indeed very small. The paper does provide multi-seed results in Appendix D.1. Mean ± std: Baseline 54.36 ± 0.17, TaskHC 54.54 ± 0.13, ΔAP = +0.18 ± 0.19. The key problem: the standard deviation of ΔAP (0.19) exceeds the mean (0.18). This means that on seeds 1 and 3, TaskHC shows essentially no improvement (+0.03 and +0.00), and one could argue the gain is entirely drowned out by noise. Two out of five seeds show basically no effect, which makes it hard to call this a "stable" improvement. The paper reports the "mean improvement of +0.18" but does not perform any statistical significance test (e.g., paired t-test), nor does it discuss the fact that the inter-seed variance far exceeds the effect size.
  • The paper's approach is to stipulate that the two groups Gcls\mathcal{G}_{\mathrm{cls}} and Gloc\mathcal{G}_{\mathrm{loc}} are "responsible for" classification and localization respectively, then show through ablation that "this allocation beats full-bank access." But this only demonstrates that this organization is effective, it does not demonstrate that the model actually stores task-specific information in these streams. The authors may argue that the paper has offered two pieces of supporting evidence: 1) Sampling analysis (Figure 3): The classification branch's cross-attention samples more from object interiors, while the localization branch emphasizes peripheral regions. 2) Frozen probes (Figure 4): Classification probes perform better on CLS inputs (+1.39 pp), localization probes perform better on LOC inputs (+3.87 pp at IoU≥0.5). But these share a common problem: they show behavioral differences, not information-content differences.
  • Sampling differences can be explained as "classification and localization modules have different computational needs," not necessarily "different information is stored in the streams." Probe differences can be explained as "the CLS/LOC prediction heads produce different outputs", which is almost tautological; if the two heads produced identical outputs, that would be the surprising result.
  • To genuinely prove that "classification streams store classification-specific information," one would need intervention experiments: e.g., freezing or replacing the streams in Gcls\mathcal{G}_{\mathrm{cls}} and checking whether classification performance specifically degrades; or using linear probes to decode classification/localization information from each group separately and checking for cross-decoding (i.e., whether information is decoupled). The paper does not do this kind of experiment.
  • If TaskHC's mechanism is effective, why is it validated only on the specific combination of classification + localization? The paper's conclusion says it "may also be applicable to settings with multiple interdependent tasks," but this is speculation without experimental support. Tasks are defined by humans; the model may spontaneously learn multiple information representations, is hard separation even necessary? Recent work on how mHC is actually used in practice found that in DeepSeek-V4-Flash, typical attention/FFN modules effectively use only about 2 streams, with the dominant stream shifting across depth, and in later layers, residual mixing nearly degenerates to identity, with streams essentially passing through independently. This suggests that models selectively use streams on their own, without needing human-specified assignment of which stream handles what.
  • In other words, TaskHC's "functional grouping" may be an overly strong prior. If the model is capable of learning "which stream should store what" on its own, then artificially splitting streams into two groups and restricting read/write scope may actually limit the model's flexibility. The paper's ablation shows full-bank access is 0.23 AP worse than group-specific, but this gap is small and not validated across tasks, making it hard to rule out hyperparameter tuning or architecture-specific coincidence.
  • The ablation in Table 2 is conducted on only one model (DEIM-D-FINE-L), with only four configurations (Prediction+CA, +FFN, +SA, +SA+FFN), and no cross-architecture validation (this ablation is not done on DEIM-RT-DETRv2). This is insufficient to support a general conclusion like "module role determines access scope."
4 comments

Thank you for raising two important questions: why group streams explicitly, and why specialize only some modules.

1. Why explicit grouping?

We agree that the shared intermediate query representation in standard DETR can carry information needed by both classification and localization. This does not mean that leaving the model to discover how to organize and use that information is the most effective approach for detection.

For example, PLE (Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations, RecSys 2020) explicitly separates task-specific and shared experts while retaining learned gating. Its experiments show that this structural prior can reduce negative transfer and improve performance over adaptive sharing without explicit expert roles. mHC-HSI: Clustering-Guided Hyper-Connection Mamba for Hyperspectral Image Classification assigns physically meaningful spectral groups to residual streams while learning their interactions.

TaskHC also retains adaptive stream use: within each functional group, the model learns how to combine and update its streams and what they represent. Shared operations allow information exchange across groups.

Our grouping inherits a detector-specific prior from DESTR (DESTR: Object Detection With Split Transformer) and SD-DETR (Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection). These methods extend the original single intermediate representation into two explicitly maintained state variables for classification and localization, and report detection improvements. TaskHC adopts this established decomposition to investigate a different state-management mechanism. Revised Table 4(a) adapts the prior explicit-state scheme to the same detector and compares it with HC-gated state management and full TaskHC. Keeping the task decomposition fixed makes this a comparison of how task states are maintained and coupled under the same functional prior.

2. Controlled evidence

We test this directly. Full bank / Full bank removes the explicit task partition from read/write access: both task-specific CA branches retain their own controllers, but can reconstruct their working states from all four streams and write back to all four. Thus, the model can learn how each task uses the full bank. Task group / Task group instead constrains each task to its assigned two streams. Both retain four C-dimensional streams, two CA branches, C-dimensional working states, and joint SA/FFN. The grouped design performs better in the original comparison, a second random seed, and another detector framework (Table 5(a), Table D.6, and Table D.3):

SettingTask group / Task group APFull bank / Full bank APΔAP
DEIM–D-FINE-L, seed 054.6854.45+0.23
DEIM–D-FINE-L, seed 154.7054.32+0.38
DEIM–RT-DETR-L54.5054.18+0.32

At fixed stream capacity, task grouping improves AP by 0.23, 0.38, and 0.32 across these settings. These results support the inherited functional prior over learning unrestricted full-bank access in the evaluated detection settings.

3. Why selective sharing?

The original manuscript discusses these roles in Section 4.3, “Placement of task-specific computation” (p. 7, below Table 2). We extend DESTR/SD-DETR's task-specific CA/FFN and shared-SA topology within TaskHC:

  • CA selects visual evidence from encoder features. Classification benefits from discriminative appearance, whereas localization needs spatial extent and boundaries; separate branches can learn different sampling locations and weights.
  • Prediction heads convert this evidence into category scores and box estimates. Each reads from the group updated by its CA branch, linking task-specific acquisition, accumulation, and prediction.
  • SA exchanges information among object queries. Joint states let these interactions use both semantic and spatial cues about the objects, rather than processing each cue separately.
  • FFN applies a nonlinear channel transformation to each query. Joint processing can combine the accumulated semantic and geometric evidence within an object representation; splitting it limits this stage's cross-task integration.

These roles motivate task-specific CA/heads and C-dimensional joint SA/FFN. Joint processing also avoids widened attention and duplicated FFNs. Original and repeated ablations support this division in accuracy and resource cost; broader task separation is not consistently beneficial.

Stream use and intervention evidence

We agree that sampling and prediction-input probe differences describe how task branches acquire and process evidence, rather than directly identifying the information stored in persistent streams. We therefore added stream freezing and read-source swapping. The following combines TaskHC results from Table E.2 with the newly completed Full/Full control in trained DEIM–D-FINE-L models:

InterventionTaskHC Cls. (%)TaskHC Loc (IoU ≥ 0.75, %)Full/Full Cls. (%)Full/Full Loc (IoU ≥ 0.75, %)
No intervention83.8378.5584.0578.46
Freeze streams 1–21.1969.0127.9747.92
Freeze streams 3–450.9225.9083.3172.91
Swap read sources9.860.1569.0535.08

In TaskHC, streams 1–2 form CLS and 3–4 form LOC. Freezing holds the selected streams at initialization across all six layers. Swapping exchanges the task read groups in TaskHC and the two stream halves at Full/Full readers, while preserving write destinations. TaskHC shows task-aligned sensitivity: freezing CLS most strongly harms classification, while freezing LOC most strongly harms localization. Full/Full is more sensitive to streams 1–2 than 3–4. This tests how the trained detector uses the streams; Appendix E.3 gives the detailed TaskHC analysis.

Thank you for recommending deeper stream analyses. Following these studies, we added experiments and visualizations of, e.g., effective stream counts, winner consistency, inter-stream representations, and residual mixing. Our six-layer detection decoder is much shallower than the large language models discussed in the review, with different computational and task demands. Visualizations show information transfer and interaction between streams. Mean load-effective counts are 3.27 for reading and 2.36 for writing; see Appendix F for details.

Detection gains and reliability

COCO 2017 contains approximately 118,000 training images and 5,000 validation images. It remains challenging because detectors must recognize and precisely localize objects across clutter, occlusion, and large scale variation, including small objects with little visual evidence. Its AP evaluation spans 80 categories and IoU thresholds from 0.50 to 0.95. Improving strong real-time baselines adds a further constraint: gains must be obtained within limited model and inference budgets. Our DEIM–D-FINE and DEIM–RT-DETRv2 baselines already incorporate substantial architectural and training improvements, making further accuracy gains demanding.

Related studies also reflect the challenge of further improving strong real-time detection baselines. RT-DETRv3: Real-Time End-to-End Object Detection with Hierarchical Dense Positive Supervision improves RT-DETRv2 by 0.2/0.3 AP with ResNet-34/101. In RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models, 36-epoch experiments on RT-DETRv2-L show +0.2 AP from DSI and a further +0.3 from GAM; on D-FINE-L, these increments are +0.1 and +0.2. These findings reflect the challenge of obtaining a large gain from a single methodological improvement on an already strong baseline. We assess the reliability of TaskHC's gains through our own repeated experiments.

Repeated training also requires substantial computation: in our experimental setup, a training run takes approximately 2.5 days on a single GPU. We added 13 standard TaskHC trainings, expanding 15 comparisons to 28 across ten configurations. Equal-configuration analysis gives +0.1622 AP for the original set (95% CI: 0.0792 to 0.2451; p = 0.001664) and +0.1876 for the expanded set (95% CI: 0.1188 to 0.2564; p = 0.000165). Both support positive average gains (Appendix D.1; Tables D.8, D.9, and D.10). Additional datasets and architectures support generalization (Table 2 and Table 3).

We also repeated the main ablations on DEIM–D-FINE-L with seed 1 and DEIM–RT-DETR-L, covering state management, module topology, and read/write scopes (Appendix D.2). These results support functional grouping and task-specific CA/prediction with joint SA/FFN across initialization and detector framework. The seed-1 repetition also supports full-bank residual mixing at shared modules.

Our work focuses on state management for real-time DETR-based object detection, using Hyper-Connections to organize and update persistent representations for classification and localization. The broader statement in our conclusion was intended as a research outlook: task-structured state management may offer a useful approach when interacting tasks have distinct information needs. We apologize that the wording did not clearly distinguish this prospect from our demonstrated scope. We do not suggest that all tasks require explicit grouping; its value for other task combinations remains to be investigated.

Reviewer_na5tOpenReview ↗

Re: Official Comment by Authors

Thank you so much for the authors' thorough experiments and the substantial updates to the paper. I truly appreciate the care and effort that went into the rebuttal and the clarification of the design motivation. I fully recognize the value the authors see in this design, and I want to emphasize that my hesitation is not due to any lack of effort or seriousness on the authors' part.

Unfortunately, and with real regret, I find myself unable to give a positive score at this stage. Since the theoretical contribution of this work is relatively limited, a clear and substantial empirical improvement would have been important for me in assessing its value. My main concern remains that the current design is more complex than the original implementation, while the gains on the main metrics are rather modest. This makes it genuinely difficult for me to confirm the necessity of the added complexity from an empirical standpoint. Given the large volume of submissions at ICLR, I am afraid the current contribution is not yet enough for me to justify a positive score, and I say this with sincere reluctance.

I want to thank the authors once again for their careful response and hard work, and I genuinely hope this work can be further strengthened in future revisions.

Thank you for your thoughtful follow-up and for recognizing the additional experiments and clarifications. We understand that your remaining concern is whether the empirical benefit justifies the added complexity. We would like to address this with training-trajectory evidence, a breakdown of the added structure, and the engineering optimizations we made to further reduce inference latency in TensorRT.

1. How the AP gain evolves during training

In the paper, we focused on AP after the full training schedule. Given your concern about the modest final gains, we examined whether that final gap reflects the benefit of TaskHC over the course of training. The table compares COCO validation AP at matched epochs on the smallest model in each DEIM detector family, averaging the gain within each interval.

Training epochsD-FINE-N ΔAPRT-DETRv2-R18 ΔAP
1–10+1.43+0.87
11–20+1.16+0.49
21–30+0.93+0.23
31–40+0.56+0.40
1–50+0.87+0.48
Full training+0.24+0.37

On both detectors, TaskHC takes an early lead in COCO validation AP. Over the first 50 epochs, D-FINE-N and RT-DETRv2-R18 gain an average of 0.87 and 0.48 AP over their respective baselines. The gap narrows late in training, leaving gains of 0.24 and 0.37 AP after the full schedule. This sustained lead also translates into fewer epochs to reach the same accuracy: with TaskHC, RT-DETRv2-R18 first reaches 48 AP ten epochs earlier (87 vs. 97).

2. What TaskHC adds to the detector

TaskHC makes two straightforward changes to the decoder. It adds a second copy of the existing cross-attention (CA) module, giving classification and localization branches of equal width with independent weights. It also adds HC controllers to manage how the branches read and update the residual streams. The additional CA branch accounts for most of the added parameters. On DEIM–D-FINE-L, the parameter count increases from 30.77M to 33.26M. Of the 2.49M added parameters, 0.47M belong to the controllers, including prediction readout (18.8% of the increase and 1.4% of the full model). Control and routing use 0.32 GFLOPs, or 20.7% of the additional 1.55 GFLOPs and 0.35% of the model’s 92.69 GFLOPs.

The two CA branches also give future detectors a way to adapt evidence acquisition to each task. In this study, we give the branches equal capacity so that classification and localization have the same sampling budget. Within the same state-management framework, future work could vary branch width or the number and placement of sampling points independently, and study which allocation best serves recognition and box refinement under a fixed decoder budget.

3. TensorRT inference graph optimization

Our previous TensorRT implementation already sampled encoder features jointly for the two CA branches. We further optimized the TensorRT inference graph by batching projections across the branches, fusing the controller and stream-update operations, and fusing the prediction readout. These optimizations preserve the trained weights, parameter count, and theoretical FLOPs of each TaskHC model. Compared with our previous implementation, these changes reduce latency for DEIM–D-FINE-L + TaskHC from 9.00 to 8.15 ms (9.4%) and for DEIM–RT-DETR-L + TaskHC from 10.19 to 8.83 ms (13.4%). We used the same weights for each comparison and measured all implementations on an NVIDIA T4 with TensorRT FP16 and batch size 1. Both optimized models surpass their native baselines in inference speed: D-FINE-L reaches 122.6 FPS versus 118.2 FPS, and RT-DETR-L reaches 113.3 FPS versus 104.5 FPS. Across both detector families, optimized TaskHC exceeds native baseline throughput in nine of ten configurations. Some CA optimizations also reduce baseline latency. See Appendix C for the deployment analysis.

We appreciate your close attention to the cost of the added structure, which prompted us to revisit the implementation and further reduce inference latency.

Reviewer_vsnyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how a detection Transformer should organize persistent query states when classification and localization require different information. TaskHC divides each query's residual state bank into classification and localization groups: task-specific modules read from and write back to their designated group, while shared self-attention and feed-forward modules read and mix the full bank. The method is built on the mHC-lite connection parameterization and is evaluated by replacing the decoder in DEIM–D-FINE and DEIM–RT-DETRv2 without changing their native detection objectives. On COCO, TaskHC gives small but generally positive AP gains, and the paper further studies the design with access, mixing, stream-count, repeated-training, and frozen-state analyses. These results support functional grouping as a promising architectural choice, although the small gains and limited repeated ablations leave the strength of the mechanism claims uncertain.

Critical strengths
  1. The paper isolates concrete architectural design choices rather than presenting only an end-to-end improvement. The ablations compare functional grouping, module-specific access, full-bank access, and cross-group mixing under closely related task modules, which makes the claimed contribution easier to evaluate.
  2. The empirical evaluation and engineering disclosure are strong. The paper reports repeated training, deployment cost, probe protocols, and unfavorable results, and it also shows that the number of persistent states is not universally necessary for matching the reported accuracy. Overall, the experiments provide a useful and relatively transparent assessment of the proposed design.
Critical weaknesses
  1. The gains are small, and the core structural conclusions do not receive the same repeated-training validation as the main model comparison. Across five seeds, the mean AP improvement is 0.18 with a paired standard deviation of 0.19. This supports a modest improvement trend but is not enough to treat every 0.1–0.3 AP ablation difference as a reliable ranking.
  2. The probe results show specialization in the final representations, but do not isolate why persistent functional grouping produces it. Table E.1 reports a 1.39-point advantage in classification macro accuracy for the classification input and a 3.87-point advantage in [email protected] for the localization input. However, the two probe inputs come from different task-specific computation paths, and the localization input includes D-FINE's cross-layer state accumulation. These differences could themselves produce different probe performance. The paper does not apply the same probe analysis to the full-bank control in Table 3a. Thus, the current result cannot determine whether the observed accessibility difference is mainly caused by persistent grouping rather than by the specialized branches or native prediction paths. The performance ablations in Table 3 provide partial independent support, so this does not invalidate the method; it limits the mechanism claim about how task information is formed and preserved.
Questions for authors
  1. Statistical reliability of the ablations: Tables 3a and 3b appear to report single-run comparisons, but the reported AP differences are small. Can the authors provide repeated runs or another measure of uncertainty for the grouping/full-bank and Full/IHC comparisons, so that it is possible to determine whether these differences are statistically meaningful rather than run-to-run variation?
  2. Do the same classification/localization probe differences appear for the full-bank control while keeping the task branches unchanged? If the states are read before the native heads and localization cross-layer accumulation, do the conclusions remain? This would help separate the contribution of branch specialization from that of persistent grouping.
  3. When Table 3a changes the read/write support, are the controller input scope, parameter count, and task-specific cross-attention residual transport held fixed? Could you please list the exact configuration of each variant so that the comparison can be interpreted as changing the preservation and access constraints rather than several factors at once?
1 comment

We thank you for the detailed questions on ablations, probes, and controller design, which helped us strengthen our controls and clarify the results.

Q1. Reproducibility of grouping and mixing results

Following your suggestion, we added DEIM–D-FINE-L seed-1 ablations and a DEIM–RT-DETR-L study. All three settings support functional grouping:

Read / writeDEIM–D-FINE-L seed 0 APDEIM–D-FINE-L seed 1 APDEIM–RT-DETR-L AP
Task group / Task group54.6854.7054.50
Full bank / Task group54.5154.4554.22
Task group / Full bank54.3254.2854.03
Full bank / Full bank54.4554.3254.18

Results appear in Table 5(a), Table D.6, and Table D.3. Shared-module mixing was repeated with seed 1:

Residual mixingSeed 0 APSeed 1 AP
Identity (IHC)54.5854.22
Intra-group mixing54.4454.51
Full-bank mixing54.6854.70

Full-bank mixing remains best across both seeds (Table 5(b), Table D.7), supporting the final design choice. Further repeated experiments and supplementary results are provided in Appendix D.

Q2. Probe interpretation and direct stream interventions

Thank you for highlighting this distinction. The Full/Full control also exhibits task-aligned probe differences at the detection-head inputs: the localization-head input is better suited to training a new localization probe, while the classification-head input favors a new classification probe. These probes evaluate representations already processed for the respective heads, rather than the persistent streams themselves. The same trend in Full/Full therefore shows that the original probe results alone cannot establish the contribution of persistent functional grouping.

To examine how the detector uses persistent streams, we compare freezing and read-source swapping in trained DEIM–D-FINE-L models. The table combines TaskHC results from Table E.2 with newly completed Full bank / Full bank results; Appendix F provides complementary visual analyses of routing, representations, and residual transport:

InterventionTaskHC Cls. (%)TaskHC Loc. IoU ≥ 0.75 (%)Full/Full Cls. (%)Full/Full Loc. IoU ≥ 0.75 (%)
No intervention83.8378.5584.0578.46
Freeze streams 1–21.1969.0127.9747.92
Freeze streams 3–450.9225.9083.3172.91
Swap read sources9.860.1569.0535.08

Freezing retains the selected streams at initialization through all six layers. Swapping exchanges task read groups in TaskHC and permutes the two stream halves at full-bank readers in the control; writes retain their original destinations. TaskHC shows stronger task-aligned sensitivity: freezing CLS primarily damages classification, while freezing LOC strongly damages localization. Full/Full is much more sensitive to the first half than the second. These interventions complement the probes by testing the learned use of persistent streams. The TaskHC intervention analysis is given in Appendix E.3, with visual analyses in Appendix F.

Q3. Controller scope and residual-mixing controls

We keep the task topology fixed across the read/write ablations: two task-specific CA modules and separate classification and localization heads. Each branch retains its own controller; CA uses read/write gates, while prediction heads use read-only gates. The controller receives the streams within the specified access scope and generates one gating coefficient per accessed stream, following HC’s stream-wise gating formulation. Task access therefore conditions the gate on the corresponding two streams and produces two coefficients, whereas Full access uses all four streams and produces four coefficients. Read and write scopes are configured separately, as summarized below. This aligns the information used to generate each gate with the states it manages; under Task access, the gate does not directly depend on the other task group’s states.

Read / writeRead-gate inputWrite-gate input
Task / Task2 streams2 streams
Full / Task4 streams2 streams
Task / Full2 streams4 streams
Full / Full4 streams4 streams

Across these variants, CA residual mixing (COMB) remains within the fixed stream pairs (1, 2) and (3, 4). Keeping this residual-transport structure unchanged allows us to examine read/write access without simultaneously changing how existing stream states are mixed. The visualization and statistics in Appendix F further show that the learned CA mixing matrices are close to identity, indicating that this path mainly preserves individual stream states, with limited cross-stream mixing.

Reviewer_b6oyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TaskHC, a structured state-management framework for object-detection Transformers, aiming to mitigate representation conflicts between classification and localization. Building on Hyper-Connections (HC), TaskHC divides persistent residual states into classification and localization groups and introduces operation-dependent state access. Task-specific modules primarily access and update their corresponding state groups, while shared Self-Attention and FFN modules access the full state bank to facilitate cross-group information exchange. The authors evaluate TaskHC on COCO val2017 using multiple configurations of DEIM-D-FINE and DEIM-RT-DETRv2, with architectural ablations, attention-sampling analyses, and frozen-feature probes to examine the proposed mechanism. Experimental results show modest AP improvements in some configurations, accompanied by increased inference latency.

Critical strengths
  1. The problem is meaningful, and the proposed mechanism is reasonably motivated. This paper addresses the representational differences between classification and localization in object-detection Transformers and proposes a structured state-management mechanism based on Hyper-Connections (HC). TaskHC organizes persistent states into classification and localization groups and introduces operation-dependent access strategies to balance task-specific representation and cross-task information exchange.

  2. The evaluation includes both performance assessment and mechanistic analysis. The paper evaluates multiple configurations of DEIM-D-FINE and DEIM-RT-DETRv2 on COCO val2017. Architectural ablations, attention-sampling analyses, and frozen-feature probes are used to investigate the proposed mechanism. The paper also reports parameter counts, FLOPs, and inference latency to assess computational overhead.

  3. Functional grouping and operation-dependent state access offer an interesting architectural perspective. TaskHC adopts different state-access and update strategies for task-specific and shared modules, aiming to balance task specificity with information sharing. This provides an interesting direction for organizing and managing internal states in object-detection networks.

Critical weaknesses
  1. The novelty of the method and the independent contribution of its core mechanism require clearer delineation. The novelty of the method and the independent contribution of its core mechanism require clearer delineation. TaskHC's functional grouping and operation-dependent state access are related to existing HC-based state-management methods and decoupled detection architectures. Controlled comparisons that account for parameter count, state-stream configuration, and computational budget are needed to isolate the contribution of functional grouping.

  2. The performance gains are modest, and their statistical reliability and accuracy–efficiency trade-off require further validation. The performance gains are modest, and their statistical reliability and accuracy–efficiency trade-off remain insufficiently established. According to the existing review record, DEIM-D-FINE-L improves AP from 54.45 to 54.68, while parameter count increases from 30.77M to 33.26M and inference latency rises from 8.37 ms to 8.90 ms. The recorded five-seed results indicate a mean paired AP gain of approximately +0.18, with a 95% confidence interval of approximately [-0.06, 0.42]. If these statistics are accurate, the interval includes zero, leaving the robustness of the gains uncertain. Further repeated-run experiments and uncertainty analyses are needed to establish whether the improvements justify the additional overhead.

  3. The experimental scope is limited, and the generalizability and practical competitiveness of the method remain insufficiently established. The experimental scope is limited, and the generality and practical competitiveness of the method remain insufficiently established. The evaluation primarily focuses on COCO val2017 and DEIM-based detectors, leaving its applicability to other architectures and data distributions uncertain. Additional cross-architecture or cross-dataset evaluations, together with fair comparisons against related state-management methods and representative alternatives, would strengthen the claims regarding generality and practical value.

Questions for authors
  1. Independent Contribution of Functional Grouping. How does functional grouping contribute to the performance gains of TaskHC beyond existing HC-based methods and task-specific interaction designs? Controlled comparisons with variants that remove functional grouping, while keeping the parameter count, state-stream configuration, and computational budget as comparable as possible, would help isolate its specific contribution.

  2. Statistical Reliability of the Reported Gains. To what extent are the reported AP gains statistically robust across different random seeds? Reporting per-seed results and confidence intervals for the main configurations and key ablation studies would help establish whether the improvements are consistent and sufficient to justify the additional parameters and inference latency.

  3. Generalization Across Architectures and Datasets. How well does TaskHC generalize to detection architectures and datasets beyond the current evaluation setting? Since the experiments primarily focus on COCO and DEIM-based detectors, additional evidence or a clearer discussion of the method's applicability and limitations would strengthen the generalization claims.

  4. Accuracy–Efficiency Trade-off in Practice. Under what deployment scenarios do the accuracy gains of TaskHC justify its additional computational overhead? A comprehensive comparison of AP, parameter count, FLOPs, and inference latency under consistent evaluation conditions would help clarify its practical value.

2 comments

We thank you for the careful review and constructive questions about TaskHC’s contribution, reliability, and practical value. We have added controlled comparisons and repeated experiments to address these points, as detailed below.

Q1. Independent contribution of functional grouping

Thank you for suggesting a comparison under matched state capacity and similar computational budgets. To test functional grouping, we compare TaskHC with Full bank / Full bank (original Table 3(a), revised Table 5(a)). In the full-bank access scheme, the four streams are not explicitly assigned to functional groups: each task-specific module reconstructs its working state from all four streams and writes its update back to all four. TaskHC instead assigns two streams to each task and reconstructs and updates states within the corresponding group. The stream count and width, two CA branches, C-dimensional working states, and joint SA/FFN topology remain unchanged, with comparable parameter and computational budgets.

The table combines the original DEIM–D-FINE-L comparison with its second-seed repetition and DEIM–RT-DETR-L ablation (Table D.6 and Table D.3). Resource columns refer to DEIM–D-FINE-L.

Read / writeParams (M)GFLOPsDEIM–D-FINE-L seed 0 APDEIM–D-FINE-L seed 1 APDEIM–RT-DETR-L AP
Task group / Task group33.2692.6954.6854.7054.50
Full bank / Full bank33.3792.7554.4554.3254.18

Functional grouping improves AP by 0.23, 0.38, and 0.32 across the original experiment, the second-seed repetition, and the second detector framework. The consistent direction supports its contribution under comparable state capacity and computational budgets.

Q2. Statistical reliability of detection gains

We agree that the modest gains call for repeated training and uncertainty analysis. We added 13 standard-TaskHC trainings. Using equal configuration weights across the same ten configurations gives:

ComparisonsMean ΔAP95% CITwo-sided p
Original 15+0.1622(0.0792, 0.2451)0.001664
Expanded 28+0.1876(0.1188, 0.2564)0.000165

Both the original and expanded experiments support a positive configuration-averaged gain. Appendix D.1 gives the analysis; Tables D.8, D.9, and D.10 report individual results. Repeating the main ablations on DEIM–D-FINE-L seed 1 and DEIM–RT-DETR-L also supports the design choices (Appendix D.2).

Q3. Generalization across datasets and architectures

Following your suggestion to broaden the evaluation, Table 2 adds D-FINE-S results on CrowdHuman (one category), LVIS (1,203 long-tailed categories), and V3Det (13,204 categories), with AP gains of 0.52, 0.29, and 0.33. The gains across these markedly different category scales and distributions support cross-dataset generalization.

TaskHC also improves DAB-DETR and DINO by 0.9 and 0.3 AP (Table 3), supporting its applicability across detector architectures beyond the DEIM family.

AuthorsOpenReview ↗

Analysis of Deployment Costs and Potential Benefits

We thank you for raising this practical question. TaskHC is a state-organization mechanism that can be transferred across detectors. On the evaluated real-time architectures, it provides modest accuracy gains at moderate parameter, computation, and inference costs while retaining real-time capability. The table summarizes Table 1 under NVIDIA T4/TensorRT FP16. Arrows indicate baseline → TaskHC; parentheses give percentage changes for resources and FPS, and absolute changes for AP.

ConfigurationParams (M)GFLOPsFPSAP
DEIM–D-FINE-N3.75 → 4.18 (+11.5%)7.16 → 7.43 (+3.8%)396.8 → 380.2 (-4.2%)42.80 → 43.04 (+0.24)
DEIM–D-FINE-S10.24 → 11.48 (+12.1%)25.17 → 25.94 (+3.1%)268.1 → 247.5 (-7.7%)48.69 → 48.94 (+0.25)
DEIM–D-FINE-M19.25 → 20.91 (+8.6%)56.68 → 57.71 (+1.8%)184.8 → 158.2 (-14.4%)52.60 → 52.74 (+0.14)
DEIM–D-FINE-L30.77 → 33.26 (+8.1%)91.14 → 92.69 (+1.7%)119.5 → 112.4 (-6.0%)54.45 → 54.68 (+0.23)
DEIM–D-FINE-X61.70 → 64.19 (+4.0%)202.67 → 204.21 (+0.8%)71.7 → 68.1 (-5.0%)56.24 → 56.33 (+0.09)
DEIM–RT-DETRv2-R1819.98 → 20.83 (+4.3%)60.21 → 63.93 (+6.2%)206.6 → 184.8 (-10.5%)49.06 → 49.43 (+0.37)
DEIM–RT-DETRv2-R3431.24 → 32.37 (+3.6%)92.26 → 97.22 (+5.4%)150.2 → 135.7 (-9.6%)51.10 → 51.14 (+0.04)
DEIM–RT-DETRv2-R50-M32.99 → 33.85 (+2.6%)99.53 → 103.25 (+3.7%)140.3 → 129.9 (-7.4%)53.09 → 53.05 (-0.04)
DEIM–RT-DETRv2-R50-L42.14 → 43.78 (+3.9%)133.49 → 136.52 (+2.3%)104.5 → 97.8 (-6.5%)54.24 → 54.45 (+0.21)
DEIM–RT-DETRv2-R10174.87 → 76.58 (+2.3%)251.43 → 258.88 (+3.0%)68.5 → 64.9 (-5.2%)55.41 → 55.56 (+0.15)

All ten TaskHC configurations retain over 60 FPS under these conditions, supporting their use in real-time detection deployments. The additional cost is most relevant to applications that prioritize precise localization and reliable category recognition and can accommodate a moderate reduction in inference speed. On CrowdHuman, AP75 improves by 0.84 versus 0.22 for AP50, supporting more precise localization. On LVIS, rare-category AP rises from 21.66 to 23.64, compared with a 0.29 overall AP gain, consistent with improved category discrimination under sparse supervision (Table 2). These results support the accuracy-oriented use of the available deployment budget.

The gains are relatively pronounced in smaller configurations: DEIM–D-FINE-N/S improve by 0.24/0.25 AP with 0.43M/1.24M additional parameters, and DEIM–RT-DETRv2-R18 improves by 0.37 AP with 0.85M additional parameters. With limited model capacity, a shared representation may have greater difficulty accommodating the different evidence needed for category recognition and precise localization. Task-specific acquisition and read/write scopes allow these signals to accumulate and be updated separately, potentially reducing competition between the two tasks while preserving information integration through shared modules. For deployments that require compact detectors but place additional value on detection accuracy, TaskHC therefore offers an option to improve accuracy with moderate additional resources while retaining real-time inference.

As a future extension, the dedicated classification pathway could provide an interface for semantic alignment with CLIP or other vision-language models, while retaining task-specific localization processing. This offers a direction for investigating richer category semantics and open-vocabulary recognition within the same state-management framework.

Comments on the paper

2 comments

We thank all reviewers for their time and thoughtful feedback. Your questions helped us strengthen the experimental evidence and clarify the contribution of TaskHC. We summarize the new results and the corresponding revisions below.

1. Repeated training and statistical reliability. We added 13 standard-TaskHC experiments under the original ten DEIM–D-FINE and DEIM–RT-DETRv2 configurations, expanding the original 15 baseline–TaskHC comparisons to 28. We analyze both sets using the same equal weighting across configurations:

ComparisonsMean ΔAP95% CITwo-sided p
Original 15+0.1622(0.0792, 0.2451)0.001664
Expanded 28+0.1876(0.1188, 0.2564)0.000165

Both the original and expanded experiments support a positive average effect of TaskHC. Statistical details are in Appendix D.1; individual results are in Tables D.8, D.9, and D.10.

2. Generalization across datasets and architectures. With D-FINE-S, we added single-class pedestrian detection on CrowdHuman, long-tailed detection on LVIS (1,203 categories), and large-vocabulary detection on V3Det (13,204 categories). The following summarizes Table 2; parentheses show AP gains.

ModelCrowdHuman APLVIS APV3Det AP
D-FINE-S54.6430.7731.68
D-FINE-S + TaskHC55.16 (+0.52)31.06 (+0.29)32.01 (+0.33)

TaskHC also improves DAB-DETR and DINO by 0.9 and 0.3 AP (Table 3). Together, these results support generalization across category distributions and detector architectures.

Table 3. Generalization across detector architectures on COCO val2017.

ModelAP
DAB-DETR44.2
+ TaskHC45.1 (+0.9)
DINO48.1
+ TaskHC48.4 (+0.3)

3. Design validation across seeds and frameworks. We repeat the main ablation experiments on DEIM–D-FINE-L with a second seed and on DEIM–RT-DETR-L (Appendix D). These repetitions support functional grouping and the final topology of task-specific CA/prediction with joint SA/FFN. The second-seed results also support full-bank residual mixing at shared modules.

4. Comparison with explicit task-state management. We separate original Table 2 into the state-management and topology comparisons in revised Tables 4(a) and 4(b). DESTR: Object Detection With Split Transformer (He and Todorovic, 2022) and Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection (Zhang et al., 2023) motivate the explicit-state controls in the same DEIM–D-FINE-L detector. These controls maintain classification/localization states and concatenate them for joint processing. HC read/write gates instead reconstruct a C-dimensional joint state, reducing parameters and computation while improving AP from 54.24 to 54.41. Full TaskHC adds multiple streams per task and learned residual mixing, reaching 54.68 AP (Table 4(a)):

ConfigurationParams (M)GFLOPsAP
Baseline30.7791.1454.45
SD-DETR-style (Split FFN)40.6897.0854.24
SD-DETR-style (Joint FFN)40.6797.0854.24
SD + HC gates32.8492.3954.41
TaskHC (4 streams)33.2692.6954.68

5. Stream interventions and visual analysis. We appreciate the reviewers’ observation that the original probes examine prediction inputs after task-specific processing. We agree that these probes reflect how the model processes and uses task information, but do not sufficiently establish differences in the information stored in the persistent streams themselves. We added stream-freezing and read-source-swapping interventions (Table E.2) and visual analyses of read/write routing, inter-stream representations, and residual mixing (Appendix F).

Table E.2 freezes each group's streams at initialization or swaps the groups being read in the same trained TaskHC model on COCO val2017:

InterventionCls. accuracy (%)Loc. IoU ≥ 0.75 (%)
No intervention83.8378.55
Freeze CLS group1.1969.01
Freeze LOC group50.9225.90
Swap group reads9.860.15

Freezing each group more strongly impairs its corresponding task, while swapping read sources disrupts both tasks. These results support task-specific functional roles of the persistent stream groups.

AuthorsOpenReview ↗

Engineering optimizations to further reduce TaskHC inference latency in TensorRT

We thank the reviewers for raising the question of inference latency in real-time detection. Our previous TensorRT implementation already sampled encoder features jointly for the two cross-attention branches. We further optimized the TensorRT inference graph in two ways. Batched computation executes matching projections in the classification and localization branches in a single batched operation, with each branch retaining its own weights. Operator fusion combines successive operations within the controller, residual-stream update, and prediction readout, so fewer intermediate tensors need to be written to GPU memory and read back by the next kernel. These optimizations preserve the trained weights, parameter count, and theoretical FLOPs of each TaskHC model. Across all ten configurations, the updated TensorRT implementation reduces TaskHC inference latency by 5.0% to 13.4% relative to the previous implementation. With these optimizations, nine of the ten TaskHC configurations also run faster than their native baselines without TaskHC.

For a fair comparison, we benchmark the native baselines and both TaskHC implementations in the same environment using the same timing protocol (NVIDIA T4, TensorRT FP16, batch size 1). The remeasured latencies and corresponding FPS for the native baselines and the previous TaskHC implementation may differ slightly from those reported in the paper. The last column gives the latency reduction relative to the previous TensorRT implementation.

DEIM modelBaseline msBaseline FPSPrevious TaskHC msPrevious FPSUpdated TaskHC msUpdated FPSLatency reduction
D-FINE-N2.103475.52.613382.72.327429.811.0%
D-FINE-S3.354298.13.671272.43.303302.810.0%
D-FINE-M5.503181.75.922168.95.435184.08.2%
D-FINE-L8.463118.29.005111.18.154122.69.4%
D-FINE-X13.41574.514.09371.013.38674.75.0%
RT-R184.701212.75.079196.94.471223.712.0%
RT-R346.546152.87.058141.76.192161.512.3%
RT-R50-M*7.059141.77.362135.86.853145.96.9%
RT-R50-L9.569104.510.18998.18.825113.313.4%
RT-R10114.22970.314.83867.413.63273.48.1%

Starting from the previous TensorRT implementation, each row adds one optimization to the preceding row. We report end-to-end latency and the savings relative to the preceding row and to the starting implementation (ms).

Optimization addedD-FINE-L latencyStep savingTotal savingRT-R50-L latencyStep savingTotal saving
Previous TaskHC implementation (remeasured)9.005—0.00010.189—0.000
+ Joint controller coefficient computation and fused writeback8.8060.1980.1989.2150.9740.974
+ Joint execution of dual-branch attention, projections and writeback8.4770.3300.5289.1280.0871.061
+ Query and sampling layout changes8.499-0.0220.5069.1070.0211.082
+ Fused prediction readout8.3590.1400.6459.0400.0681.149
+ Four-stream controller projection and normalization optimizations8.1540.2050.8508.8250.2141.364

Together, these changes reduce latency by 9.4% on D-FINE-L and 13.4% on RT-R50-L relative to the previous TensorRT implementation. Some of these optimizations can also reduce inference latency for the native baseline detectors. We have updated Appendix C to describe these TensorRT inference graph optimizations and analyze how end-to-end latency changes at each step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.