acceptodds
Under review as a conference paper at ICLR 2027

SpecHyper-Track: Relation-Specific Spectral Hypergraph Reasoning for Online Multi-Object Tracking

Abstract

Relational reasoning methods often assume that all interactions share identical propagation behaviors, while real-world systems contain relations with different semantic and spectral characteristics. Online multi-object tracking (MOT) provides a challenging instance of this problem, where cooperative motion patterns and identity-conflicting interactions coexist. We introduce SpecHyper-Track, a relation-specific spectral hypergraph framework with stage-aware frequency routing for identity-preserving association. The framework constructs semantic hypergraphs with homogeneous and heterogeneous hyperedges to model cooperative and competitive relations, respectively, enabling relation-dependent spectral propagation. Cooperative relations are enhanced through low-frequency smoothing, whereas competitive relations preserve identity-sensitive variations through high-frequency responses. Leveraging tight‑framelet decomposition, we derive complementary frequency‑aware representations adaptively dispatched based on trajectory status: high-frequency features improve discrimination for established trajectories, while fused low- and high-frequency features provide stable context for newly initialized trajectories. The proposed framework couples semantic relational modeling, spectral representation learning, and state-aware association in a unified manner. Evaluations on MOT16, MOT17 and DanceTrack yield substantial improvements on association‑oriented metrics including HOTA, IDF1 and AssA in crowded interactive scenarios. Code is available at https://anonymous.4open.science/r/SpecHyper-Track-1869/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hypergraph representation learning, spectral graph learning, high-order relational reasoning, adaptive frequency routing, multi-object tracking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5DqHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpecHyper-Track adds a training-free module to an online tracker (TrackTrack) that refines appearance features with two hypergraphs (1) A cooperative hyperedges over motion-consistent groups, smoothed by a low-pass residual, and competitive hyperedges over nearby look-alike targets, sharpened by a high-pass residual. (2) A framelet decomposition then gives high-frequency features to established tracks and fused features to new tracks. Experiments cover MOT16, MOT17 and DanceTrack (Tables 4–6), with ablations on MOT17 validation (Tables 1–3).

Critical strengths

(1) SpecHyper-Track reduces identity switches from 1487 to 1101, on MOT17 (Table 2). With the same detector, ReID model, and tracking settings as their reproduced TrackTrack baseline, HOTA also improves modestly from 65.6 to 66.9 on MOT17 and from 62.4 to 64.2 on DanceTrack (Tables 5–6), without additional training

(2) Table 1 shows why the residual design matters. Keeping the original features and applying a bounded residual update achieves 71.1 IDF1, compared with 48.7 Table 1.

Critical weaknesses

(1) The benefit of using high-frequency features for established tracks and fused features for new tracks is unclear. In Table 2, the residual update adds 2.9 HOTA points, while the framelet split and routing add another 0.4 and reduce identity switches from 1376 to 1189. These results do not isolate the effect of assigning different features to the two stages. Since this is a main contribution, the authors should compare against using the same features in both stages and swapping the assignments.

(2) The reimplemented baseline is weaker than the published one. The authors' re-run of TrackTrack gets 65.6 HOTA on MOT17 and 62.4 on DanceTrack (Tables 5–6), but the published results are 67.1 and 66.5 (Shim et al., CVPR 2025, Tables 1 and 3), both above SpecHyper-Track (66.9 and 64.2). The gap is not explained, so it is unclear whether the module helps a well-tuned tracker. It also means the MOT17 claim in Sec. 3.2, of beating the strongest competitor, does not hold.

Questions for authors

Requested clarification+evidence:

(1) Please asses or elaborate contribution of routing properly. Why use high-frequency features for established tracks and fused features for new tracks? Please compare this assignment with using high-frequency features in both stages, fused features in both stages, and swapping the assignments. These comparisons would help isolate and better understand contribution of each component.

(2) Please clarify the difference in official TrackTrack (Shim et al., CVPR 2025) and its reimplementation, what causes the gap to the published results? Also does the method still works i.e. gives similar gains against release and config, at least on MOT17 or DanceTrack validation?

5 comments

Q1. Motivation and Benefits of Track-State-Aware Feature Routing

Established tracks have accumulated appearance observations, so we use high-frequency-enhanced features (XhX_h) to distinguish easily confused candidates. For unconfirmed tracks with shorter appearance histories, we use fused features (XfX_f), which retain these discriminative cues and add within-group context. To isolate the effect of routing, we varied only the feature assignment, keeping feature construction, conditional activation, and post-processing fixed.

Table R1. Routing ablation on the MOT17 ablation split

First stage → Second stageHOTAIDF1MOTAAssADetA
High-frequency-enhanced → High-frequency-enhanced68.984.979.672.565.9
Fused → Fused69.285.279.672.565.9
High-frequency-enhanced → Fused (ours)69.385.779.873.266.2
Fused → High-frequency-enhanced69.185.179.772.966.0

The proposed routing improves IDF1 by 0.5-0.8 points and AssA by 0.3-0.7 points across the three alternatives.

Q2. Published TrackTrack results and matched-protocol comparison

We reproduced TrackTrack using the authors' released code and configuration parameters. Within each dataset, we ran our TrackTrack reproduction and SpecHyper-Track on the same machine using identical detection inputs, ReID features, and evaluation settings.

Table R2. HOTA comparison of published TrackTrack, reproduced TrackTrack, and SpecHyper-Track on the MOT17 and DanceTrack test sets.

Test setTrackTrack HOTA (published)TrackTrack HOTA (reproduced)SpecHyper-Track HOTAGain over reproduced baseline
MOT1767.165.666.9+1.3
DanceTrack66.562.464.2+1.8

Using the same relation construction and frequency enhancement parameters on both test sets, SpecHyper-Track improves HOTA over our matched-protocol TrackTrack reproduction on both MOT17 and DanceTrack and achieves a HOTA score close to the published TrackTrack result on MOT17. It also improves IDF1 and AssA by 0.8 and 1.2 points on MOT17 and by 1.3 and 2.1 points on DanceTrack, respectively. We will revise Sec. 3.2 to report the published, reproduced, and augmented results side by side.

The difference appears marginal, is it even consistent across multiple seeds?

Thank you for the follow-up. Both trackers use the same fixed detections generated by a publicly released detector and the same ReID features. With these inputs fixed, the tracking stage, including our enhancement module, has no seed-dependent steps, so the reported results do not depend on the choice of random seed under this evaluation setup.

To complement the HOTA results in Table R4, we report IDF1 and AssA gains over our TrackTrack reproduction under the same evaluation protocol.

Test setIDF1 gain (points)AssA gain (points)
MOT17+0.8+1.2
DanceTrack+1.3+2.1
On MOT20, where dense crowds and frequent occlusions make identity association challenging, SpecHyper-Track improves all reported metrics over our TrackTrack reproduction. With the same enhancement parameters across datasets, it also improves HOTA on all three test sets.
Reviewer_HeCvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free module that refines appearance features inside a two-stage tracking-by-detection pipeline built on TrackTrack. At each frame, detections, previous-frame context and Kalman predictions are grouped by motion and spatial compatibility. Each group forms a homogeneous hyperedge, and appearance-confusable pairs from different groups form heterogeneous hyperedges.

The method applies a bounded low-pass correction on the homogeneous hypergraph and a high-pass correction on the heterogeneous one. It then uses Haar tight framelets to produce low- and high-frequency features. High-frequency features are used to match existing tracks, and fused features to match new tracks.

The reported test results are 66.5, 66.9 and 64.2 HOTA on MOT16, MOT17 and DanceTrack. The paper's reproduction of TrackTrack reaches 65.6 on MOT17 and 62.4 on DanceTrack.

Critical strengths

The most convincing evidence is the comparison with TrackTrack, reproduced under the same detector, ReID model and protocol:

  • MOT17 test: +1.3 HOTA, +0.8 IDF1 and +1.2 AssA.
  • DanceTrack: +1.8 HOTA, +1.3 IDF1 and +2.1 AssA. DanceTrack is the benchmark where appearance ambiguity is the main difficulty.

A training-free module that adds this much to a strong recent tracker is of practical interest.

The ablations in Tables 1 and 3 also make a useful negative point. Injecting relational information too aggressively degrades identity information; with global feature replacement, IDF1 falls to 48.7. Bounded residual corrections work best.

Critical weaknesses

The main concern is that the ablations do not isolate the two ideas the paper is built on.

The relation-specific claim needs evidence that low-pass on cooperative relations plus high-pass on competitive relations is better than the alternatives. Those alternatives are each branch alone, the same filter for both, and the two filters swapped. Table 3 only varies alpha and beta together, and none of these variants appears in the paper.

For the framelet stage and routing, Table 2 shows only a small gain over relation enhancement alone: 68.9 to 69.3 HOTA, 84.9 to 85.7 IDF1, and 72.5 to 73.1 AssA. The two framelet-only variants score at or below the base tracker. Rows (f) and (g) are said to differ only in GPU implementation details and conditional application, yet ID switches drop from 1189 to 1101. This difference needs an explanation.

All of these results come from single runs on a MOT17 validation split whose construction is not described. More than twenty hyperparameters are set by hand, presumably on that same split. This makes it difficult to judge whether a 0.4 HOTA gain is meaningful. MOT16 and MOT17 also share their test videos, so the two benchmarks do not count as independent confirmation.

A second concern is that efficiency and the claimed mechanism are not measured. This is an online tracker, the limitations section acknowledges the extra cost, and the full model only runs the hypergraph in crowded scenes. Even so, no FPS or per-frame latency is reported for any variant.

The affinity heatmaps in Figs. 8-9 show inter-target affinity falling from about 0.44 to 0.06 after refinement. However, they mask the same-identity entries and use selected frames, so they do not show whether the gap between true and false matches grows. Uniformly lowering every affinity would produce the same picture.

Questions for authors
  1. In the Table 3 setting, could results be reported with only the homogeneous branch, only the heterogeneous branch, and with the two filters swapped?
  2. Are rows (c)-(g) of Table 2 cumulative? What exactly changes between (f) and (g)?
  3. How is the MOT17 validation split built? Were the hyperparameters in Appendix A.4 tuned on it, and were any changed for DanceTrack?
  4. What is the FPS of the base tracker and of the full model on MOT17 and DanceTrack, on the hardware listed in the appendix?
  5. In Figs. 8-9, which features are compared, and how do same-identity affinities across frames change? Averages over whole sequences would be more convincing than selected frames.
  6. Could the original TrackTrack paper's numbers be listed next to the reproduction?
  7. Some entries in Tables 1 and 3 coincide although HOTA differs. Please confirm they are correct:
    • Table 1(c) and the alpha=beta=1 row of Table 3 both report IDF1 48.7 and MOTA 54.6.
    • Table 1(a) and the alpha=beta=0 row both report IDF1 67.8 and MOTA 62.1.
4 comments

Q1. Single-branch and filter-assignment ablations

We evaluated six filter assignments on the full MOT17 training split used for ablation. We varied only which branches were active and which filters they used, keeping the active branches' residual weights and all other tracking and evaluation settings fixed.

Table R1. Ablation of filter assignments to homogeneous and heterogeneous relations on the full MOT17 training split used for ablation.

Filter assignmentHOTAIDF1MOTAAssADetA
Homogeneous low-pass only68.784.479.772.065.7
Heterogeneous high-pass only68.984.879.572.665.8
Low-pass for both branches69.085.079.672.766.0
High-pass for both branches68.984.779.872.466.0
Homogeneous low-pass + heterogeneous high-pass (ours)69.385.779.873.266.2
Homogeneous high-pass + heterogeneous low-pass68.984.779.872.366.2

Our assignment achieves the highest HOTA, IDF1, and AssA among the six configurations. It improves IDF1 by 1.3 points over the homogeneous branch alone and 0.9 points over the heterogeneous branch alone. Compared with swapping the filters, it improves IDF1 and AssA by 1.0 and 0.9 points, respectively, at the same reported DetA (66.2).

Q2. Component configurations and identity switches in Table 2 of the original manuscript

Rows (c)-(g) are only partly cumulative. Rows (c) and (d) compare two feature representations, while rows (e) and (f) introduce stage-specific routing incrementally.

Table R2. Configurations of rows (c)-(g) in Table 2 of the original manuscript

Row in Table 2Configuration
(c)Uses framelet features in the analysis domain.
(d)Uses framelet features reconstructed in the node domain.
(e)Uses high-frequency-enhanced features (XhX_h) in the first association stage.
(f)Additionally uses fused features (XfX_f) in the second stage.
(g)Retains the routing in (f), uses an optimized GPU implementation, and applies Gaussian trajectory interpolation (interval = 30, τ=12\tau=12).

Rows (f) and (g) use the same activation rule and thresholds for hypergraph enhancement. At the reported precision, both rows achieve 69.3 HOTA and 85.7 IDF1. IDSW decreases from 1,189 in row (f) to 1,101 in row (g). We will clarify the configuration and post-processing used in each row of Table 2 of the revised manuscript.

Q3. MOT17 split, hyperparameter selection, and consistency

The "MOT17 validation set" in the original manuscript comprises all annotated frames from the seven official training sequences (MOT17-02, -04, -05, -09, -10, -11, and -13). We used these sequences both for ablations and to select the main hyperparameters of the enhancement module. We will relabel this evaluation as "Ablation on the official MOT17 training split." On DanceTrack, we kept the relation construction and frequency enhancement parameters fixed and used the dataset-specific post-processing (AFLink for DanceTrack and Gaussian interpolation for MOT).

On the MOT17 ablation split, the dual-relation update improves HOTA from 66.0 to 68.9. Adding framelet filtering and routing raises HOTA to 69.3 and IDF1 from 84.9 to 85.7, while reducing IDSW from 1,376 to 1,189 before interpolation. These gains are measured in aggregate across the seven MOT17 training sequences. MOT16 and MOT17 share test videos. On the separate DanceTrack test set, the full method improves HOTA by 1.8 points and AssA by 2.1 points over our matched TrackTrack reproduction.

Q4. Measured tracking throughput

On the RTX 3090, full SpecHyper-Track achieves 69.3 FPS on the 2,412-frame MOT17 timing subset and 123.8 FPS on DanceTrack. We compute FPS from the total wall-clock time of the tracking loop. On the denser MOT20, TrackTrack and full SpecHyper-Track achieve 14.1 and 9.7 FPS, respectively.

Q5. Figures 8-9 and Cross-Frame Affinity

Figures 8 and 9 compare the pre-refinement features (XbX_b) with XhX_h for the first association stage and with XfX_f for the second. They show four selected frames from MOT17-12 and DanceTrack0003. With the diagonal masked, mean off-diagonal affinity decreases in each example. We will clarify the scope of this observation in the captions and Appendix A.7.

For the cross-frame analysis, we used ground-truth identities to identify same-ID track-detection pairs and measured their mean similarity before and after refinement.

Table R3. Mean cross-frame similarity of same-ID track-detection pairs before and after refinement

Association stageMean same-ID similaritySame-ID pairs
First stage0.8053 → 0.857250,341
Second stage0.7724 → 0.8147347

Q6. Published and Reproduced TrackTrack Results

Table R4 compares the published TrackTrack results (Shim et al., CVPR 2025), our reproduction using the authors' released code and configuration parameters, and SpecHyper-Track. Within each dataset, we ran our TrackTrack reproduction and SpecHyper-Track on the same machine using identical detections, ReID features, and evaluation settings. We kept the relation construction and frequency enhancement parameters fixed across the MOT17, DanceTrack, and MOT20 test sets.

Table R4. HOTA comparison of published TrackTrack results, our matched-protocol reproduction, and SpecHyper-Track.

Test setTrackTrack (published)TrackTrack (reproduced)SpecHyper-TrackGain over reproduced baseline
MOT1767.165.666.9+1.3
DanceTrack66.562.464.2+1.8
MOT2065.765.265.3+0.1

Using the same relation construction and frequency enhancement parameters across all three datasets, SpecHyper-Track improves HOTA over our matched-protocol TrackTrack reproduction on each dataset and achieves HOTA scores close to the published TrackTrack results on MOT17 and MOT20.

Q7. Metric consistency between Tables 1 and 3 of the original manuscript

We rechecked the original evaluation outputs and confirmed that the IDF1 and MOTA values in each pair coincide when rounded to one decimal place.

In the first pair, Table 1(c) replaces the original appearance features with relation features, whereas the (\alpha=\beta=1) setting in Table 3 retains them and adds a (\tanh)-bounded dual-relation residual. Both report 48.7 IDF1 and 54.6 MOTA, but their AssA scores are 56.8 and 46.9, and their DetA scores are 68.3 and 63.1, respectively.

In the second pair, Table 1(a) disables motion grouping and relation updates, while the (\alpha=\beta=0) setting in Table 3 sets the dual-relation residual to zero. Both report 67.8 IDF1 and 62.1 MOTA, with HOTA scores of 60.9 and 61.3, respectively. HOTA combines detection and association accuracy at each localization threshold before averaging across thresholds, so equal reported IDF1 and MOTA can coexist with different HOTA scores. We will clarify the settings used in both tables in the revision.

Thanks for the detailed response and the new experiments. Both points the rating depended on, the filter-assignment ablation and the runtime numbers, are now covered, which is appreciated.

The response does raise one new point. According to the answers on Table 2 and the split, the reported results use Gaussian trajectory interpolation on MOT17 and AFLink on DanceTrack. Neither step is mentioned in the paper, which describes the method as online and compares it against online results. Both steps rely on information from later frames, so results that include them are not strictly online.

The paper also describes the difference between rows (f) and (g) as conditional application of the hypergraph, while the response says both rows use the same activation rule. The drop in ID switches from 1,189 to 1,101 therefore appears to come from interpolation rather than from the method. Two things would resolve this point:

  • confirming that the TrackTrack reproduction used exactly the same post-processing;
  • reporting the test results without post-processing alongside the current ones.

One smaller item from the original review is still open: the repeated IDF1 and MOTA values across rows of Tables 1 and 3 that have different HOTA should be checked in the revision.

Overall, the response addresses the two points the rating depended on. The rating will move to weak accept once it is confirmed that the TrackTrack reproduction used the same post-processing. If it did not, a matched comparison with and without post-processing would be needed before the test-set gains can be attributed to the method.

Thank you for pointing out this important issue. We carefully examined the evaluation pipelines of the reproduced TrackTrack baseline and SpecHyper-Track and confirmed that both methods employ exactly the same post-processing procedures and parameter settings.

Specifically, for MOT17, both methods apply Gaussian trajectory interpolation with interval=30 and tau=12. For DanceTrack, both methods apply AFLink with thrT=(0,20), thrS=100, and thrP=0.05. Furthermore, both methods are evaluated using the post-processed tracking results in our original experiments.

To further address the reviewer's concern about the potential influence of post-processing, we conducted additional comparative experiments on the MOT17 validation set with and without post-processing. The results are presented in Table R5.

Table R5. Comparison with and without post-processing on the MOT17 validation set.

MethodPost-processingHOTA↑IDF1↑MOTA↑AssA↑DetA↑
TrackTrackNo67.884.578.171.764.4
OursNo68.885.379.473.065.9
TrackTrackYes69.185.479.773.166.0
Ours (Full Model)Yes69.385.779.873.266.2

As shown in Table R5, without post-processing, SpecHyper-Track consistently outperforms the reproduced TrackTrack baseline across all five metrics, with improvements of 1.0 HOTA, 0.8 IDF1, 1.3 MOTA, 1.3 AssA, and 1.5 DetA. Under identical post-processing configurations, our full model also maintains improvements across all five metrics, with gains of 0.2, 0.3, 0.1, 0.1, and 0.2, respectively. These results demonstrate that the performance improvements of our proposed method are not attributable to differences in post-processing configurations.

Reviewer_HAMQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies identity errors in online multi-object tracking. SpecHyper-Track groups nearby objects using motion and position, then builds two hypergraphs: one for motion-consistent groups and one for potentially confusing objects across groups. It smooths appearance features within the first relation and sharpens differences within the second. Framelet filters produce further feature refinements, with different features used for established and newly initialized tracks. The module requires no additional training and is added to an existing tracking pipeline. Experiments on MOT16, MOT17, and DanceTrack report better tracking and identity association.

Critical strengths
  • The problem is technical-sound. Similar-looking people, occlusion, and crowded motion make identity matching difficult. Using surrounding objects as context while preserving differences between competing identities is a sensible direction.
  • The module requires no additional training and preserves the original appearance features through residual updates. This makes it potentially useful as an addition to existing trackers. The paper also acknowledges failures caused by incorrect grouping, noise, and dense scenes.
  • The controlled results are encouraging. In Table 2, the full method improves MOT17 validation HOTA from 66.0 to 69.3 and IDF1 from 81.1 to 85.7, while reducing identity switches from 1,487 to 1,101.
Critical weaknesses
  • The paper's demonstrated improvement is narrower than the broad framing suggests. Against the reproduced TrackTrack baseline, test HOTA improves by 1.3 points on MOT17 and 1.8 points on DanceTrack. These are useful gains, but their practical importance depends on the added runtime and consistency across sequences. The current evidence supports an improvement to an existing tracker; it does not yet establish a broadly better approach to relational reasoning.

  • Novelty appears mainly incremental on common practices, especially in relation filtering and frequency construction [1, 2, 3]. The potentially new part is the particular combination of motion-based relation types and feature selection by track status. The paper needs stronger evidence that this combination adds something beyond simpler versions of these existing ideas.

  • The homogeneous branch may be much simpler than its spectral description suggests. If each node belongs to one motion group, the construction gives one homogeneous hyperedge per group. Under Eq. 12, propagation within such a group is just averaging: the hyperedge weight from Eq. 6 cancels out. Under the exact framelet assumptions, the low-frequency residual then reduces to a fixed multiple of ordinary group smoothing. This does not simplify away the heterogeneous branch, but it raises a concrete question about what the homogeneous framelet stage contributes. The paper should clarify whether groups overlap and compare against this simpler operation.

  • The link between the relation labels and identity preservation remains a heuristic. Different people can move together, so smoothing within a motion group can mix identities. A heterogeneous hyperedge also averages all its members before sharpening; it does not explicitly distinguish every same-identity pair from every competing pair. Appendix A.5 acknowledges related failures, but there is no quantitative analysis of how often the constructed relations are correct. The spectral arguments describe smoothing and sharpening under stated assumptions; they do not establish that the changed features preserve the right identities. The selected affinity plots show reduced similarity between different targets, without measuring whether correct track-to-detection matches remain strong.

  • The need for the extra framelet and routing stages is not clearly isolated. In Table 2, the simpler dual-relation update already achieves 68.9 HOTA and 84.9 IDF1, compared with 69.3 and 85.7 for the full method. The full method does reduce identity switches further, from 1,376 to 1,101, so the extra components may be useful. However, the table lacks comparisons using the same features at both stages, reversing the feature choices, or using simpler filters with retuned strengths. Row (g) also adds conditional activation alongside implementation changes, so its improvement cannot be attributed solely to frequency routing.

  • Efficiency is not demonstrated. Appendix A.3 gives a useful complexity analysis, including O(N^2 d) relation discovery and sparse polynomial filtering. However, the paper reports no measured frames per second, per-frame overhead, or peak memory relative to the host tracker. Limiting hyperedge size does not by itself limit the total number of hyperedges. Appendix A.4 also enables the module only in selected crowded frames, which makes measurements on those active frames especially important. Requiring no extra training is an advantage, but the accuracy-versus-runtime tradeoff remains unknown.

[1] Liu, Yixin, et al. "Beyond Smoothing: Unsupervised Graph Representation Learning with Edge Heterophily Discriminating." AAAI 2023.

[2] Bo, Deyu, et al. "Beyond Low-Frequency Information in Graph Convolutional Networks." AAAI 2021.

[3] Zheng, Xuebin, et al. "How Framelets Enhance Graph Neural Networks." ICML 2021.

Questions for authors
  • What is the measured cost of the improvement? Please compare the base tracker, dual-relation update, and full model on the same hardware, reporting total tracking speed, added module time, and peak memory. How do these change with scene density, and what is the overhead on frames where enhancement is active?

  • Do motion groups overlap? If they form a partition, Eq. 12 gives a within-group averaging matrix, Theta_h = 11^T / |G|, and L_h^2 = L_h. With exact Haar filters, the residual F_l - X_b is therefore a constant times -L_h X_b. What does the homogeneous framelet stage add beyond changing the amount of smoothing? Does a direct implementation of this simpler operation give the same results?

  • Can the authors isolate the benefit of track-status routing? Please compare high-frequency-enhanced features at both stages, fused features at both stages, the proposed routing, and reversed routing, keeping conditional activation and other settings fixed. A comparison against tuned first-order filters would also help establish whether framelets are necessary.

  • How often do the relation rules connect observations of the same identity versus different identities? Please measure changes in both correct and incorrect track-to-detection similarities, including cases where different people move together and where multiple nodes represent the same person.

  • How much of the gain requires hypergraph construction? Eq. 12 ultimately produces a pairwise propagation matrix. Can a simpler weighted graph, using comparable motion and appearance cues, achieve similar accuracy at lower cost?

  • Please clarify the details needed to reproduce the method: how are velocities assigned to current detections before matching, what is the spatial scale sigma in Eq. 2, how are the framelet scales and Chebyshev intervals chosen, and which exact MOT17 frames form the validation split?

  • Reference GLMBFairMOT [Van Ma et al. (2024)] does not appear to have an publicly available publication in any form.

5 comments

Q1. Runtime and Computational Overhead

Thank you for suggesting that we quantify the runtime overhead of the relation enhancement module. We agree that online multi-object trackers must balance association accuracy and computational cost, especially in crowded scenes. To quantify this overhead, we evaluated the base tracker (TrackTrack), a variant with only dual-relation updates, and the full SpecHyper-Track. Table 1 reports the tracking throughput and computational overhead on the MOT20 test set.

We compute FPS from the total wall-clock time of the tracking loop and report the mean frame time from per-frame timings. FLOPs are estimated separately for the enhancement module.

Table 1. Runtime and computational overhead on the MOT20 test set.

ConfigurationTracking FPSMean frame time (ms)Peak memory (MB)Enhancement cost (GFLOPs/frame)
Base tracker (TrackTrack)14.156.8--
+ Dual-relation updates11.572.43694.2
Full SpecHyper-Track9.789.036928.4

Relative to the base tracker, the added module time is 15.6 ms/frame for dual-relation updates and 32.2 ms/frame for the full model, with a peak memory overhead of 369 MB. We consider this runtime and memory overhead acceptable for practical deployment. The base tracker follows the official TrackTrack code; since it does not contain the relation module, the corresponding entries are marked as "-".

Appendix A.3 analyzes the sources of this overhead. Relation discovery compares candidate node pairs with a worst-case complexity of O(N^2 d); in denser scenes, more candidate pairs are screened and the spectral filtering processes more node-hyperedge incidences. Table 2 reports the estimated enhancement cost by detection count. On MOT20, ~92% of the 4,479 frames contain at least 101 detections, where the cost averages 30.66 GFLOPs per frame; frames with 21-50 detections average only 1.64 GFLOPs. Since nearly all MOT20 frames activate enhancement on dense scenes, the 89.0 ms mean frame time approximates the overhead on frames where enhancement is active, whereas sparse frames cost close to the base tracker. On the sparser sets, the mean cost drops sharply: 1.31 GFLOPs/frame on MOT17 (69.3 FPS over the 2,412 timing frames) and 0.27 GFLOPs/frame on the DanceTrack validation set (123.8 FPS).

Table 2. Estimated enhancement cost of the full model by detection count.

DatasetDetections/frameFramesMean cost (GFLOPs/frame)
MOT17All24121.31
MOT170-2013180.35
MOT1721-501561.55
MOT1751-1009382.63
DanceTrackAll255080.27
DanceTrack0-20240060.25
DanceTrack21-5015020.71
MOT20All447928.41
MOT2021-501591.64
MOT2051-1001943.95
MOT20>=101412630.66

Q2. Simplifying the Homogeneous Low-Frequency Branch

Thank you for the insightful observation. We confirm that motion groups do not overlap: each node is assigned to at most one motion group, so the groups form a partition, and groups with fewer than two nodes are masked out of the low-frequency update. For each updated group GG, the hyperedge weight cancels during normalization, giving

Θh=11⊤∣G∣,Lh=I−Θh,Lh2=Lh.\Theta_h = \frac{\mathbf{1}\mathbf{1}^{\top}}{|G|}, \qquad L_h = I - \Theta_h, \qquad L_h^2 = L_h.

Hence LhL_h has only eigenvalues 0 and 1, and under exact Haar filters the reconstruction in Eq. (16) and its residual reduce to

Fl=ΘhXb,Fl−Xb=−LhXb,F_l = \Theta_h X_b, \qquad F_l - X_b = -L_h X_b,

i.e., a constant times −LhXb-L_h X_b, exactly as you derived. Relative to direct averaging, its contribution reduces to (i) the effective smoothing amount realized by the approximate filter response and (ii) a small accuracy gain, which we quantify below.

To answer "does the simpler operation give the same results", we implemented it directly: the low-frequency reconstruction in Eq. (16) was replaced with plain within-group feature averaging, while keeping the mask, element-wise tanh⁡\tanh, residual weight γl=0.15\gamma_l = 0.15, and per-node L2L_2 normalization in Eq. (18).

Table 3. Ablation of the homogeneous low-frequency branch on MOT17.

ConfigurationHOTAIDF1MOTAAssADetA
Homogeneous update disabled68.784.479.872.066.1
Within-group averaging69.185.179.772.966.0
Full Framelet (ours)69.385.779.873.266.2

Direct averaging already improves IDF1 and AssA by 0.7 and 0.9 points over disabling the update, confirming the value of within-group context from motion-consistent nodes. The full Framelet adds a further +0.2 HOTA, +0.6 IDF1, and +0.3 AssA over direct averaging, so the two implementations do not give identical results.

The reason is our finite-order Chebyshev approximation. Let qq denote the overall low-frequency reconstruction response in Eq. (16). Within each updated group, the realized residual is

Fl−Xb=[q(0)−1] ΘhXb+[q(1)−1] LhXb,F_l - X_b = [q(0) - 1]\,\Theta_h X_b + [q(1) - 1]\,L_h X_b,

of which direct averaging corresponds exactly to q(0)=1q(0) = 1 and q(1)=0q(1) = 0. Any deviation of the approximated response from these two values changes the residual entering Eq. (18) even at the same residual weight, and the difference propagates through Eq. (19) to the fused features used in the second association stage. This explains the small performance gap without attributing it to extra modeling power.

For budget-limited settings, direct averaging is cheaper (Table 4).

Table 4. Tracking throughput of homogeneous low-frequency configurations.

ConfigurationMOT17 FPSMOT20 FPS
Within-group averaging81.010.2
Update disabled75.19.8
Full Framelet69.39.7

In summary, the partition structure makes the homogeneous stage equivalent in form to group smoothing; direct averaging recovers most of the gain, while the Framelet stage retains a small but consistent accuracy edge at higher cost. We will add the above derivation, ablation, and clarification to the revision.

Q3. Separate Contributions of Stage-Specific Routing and Framelet Filtering

Thank you for suggesting this separation. On the MOT17 validation set, we isolated each factor by varying only one at a time, keeping feature construction, conditional activation, and post-processing fixed throughout.

Stage-specific routing. The high-frequency-enhanced feature XhX_h in Eq. (18) keeps the base appearance representation and adds a high-frequency residual from heterogeneous relations; the fused features in Eq. (19) additionally carry low-frequency context from homogeneous relations. The first association stage handles established tracks, where we use XhX_h to separate easily confused candidates; the second stage handles unconfirmed tracks with shorter histories, where fused features provide both discriminative cues and within-group context.

Table 5. Ablation of stage-specific feature routing on MOT17.

First stage → Second stageHOTAIDF1MOTAAssADetA
High-frequency-enhanced → High-frequency-enhanced68.984.979.672.565.9
Fused → Fused69.285.279.672.565.9
High-frequency-enhanced → Fused (ours)69.385.779.873.266.2
Fused → High-frequency-enhanced69.185.179.772.966.0

Our routing improves over fused→fused by +0.5 IDF1 and +0.3 AssA, and over the reverse assignment by +0.6 IDF1 and +0.3 AssA, with the gains concentrated in identity association.

Framelet versus first-order filtering. A first-order filter weights frequency components linearly, whereas the multilevel Framelet in Eqs. (14)-(17) cascades low-pass responses and aggregates high-pass components across scales, yielding a different frequency weighting. Fixing the routing, we tuned four first-order strength settings (Table 6); the untuned default scores 69.0 HOTA and the best tuned variant reaches 69.1.

Table 6. First-order filtering vs. full Framelet on MOT17.

Filtering configurationHOTAIDF1MOTAAssADetA
First-order (low_gamma=0.075, high_gamma=0.40)69.085.179.772.766.0
First-order (low_gamma=0.30, high_gamma=0.40)69.185.179.772.765.9
First-order (low_gamma=0.15, high_gamma=0.40)69.085.079.772.666.0
First-order (low_gamma=0.15, high_gamma=0.20)69.084.979.772.666.0
Full Framelet (ours, low_gamma=0.15, high_gamma=0.40)69.385.779.873.266.2

Full Framelet improves over the best first-order variant by +0.2 HOTA, +0.6 IDF1, and +0.5 AssA, again with the larger gains in identity association.

Filter assignment by relation type. We further verified the low-pass-on-homogeneous / high-pass-on-heterogeneous assignment against single-filter and swapped alternatives (Table 7).

Table 7. Filter assignment ablation on MOT17.

Filter assignmentHOTAIDF1MOTAAssADetA
Homogeneous low-pass only68.784.479.772.065.7
Heterogeneous high-pass only68.984.879.572.665.8
Low-pass for both relation types69.085.079.672.766.0
High-pass for both relation types68.984.779.872.466.0
Homogeneous low-pass + heterogeneous high-pass (ours)69.385.779.873.266.2
Swapped (homogeneous high-pass + heterogeneous low-pass)68.984.779.872.366.2

Our assignment outperforms low-pass-on-both by +0.7 IDF1 and +0.5 AssA, and matches the swapped assignment on DetA (66.2) while improving IDF1 by 1.0 and AssA by 0.9.

In summary, all three controlled ablations support the proposed routing, filtering, and assignment design, and we will include them in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.