acceptodds
Under review as a conference paper at ICLR 2027

Multiple Object Tracking via Pixel-level Association

Abstract

Existing tracking-by-query methods for Multi-Object Tracking rely heavily on the quality of the track query, which is typically generated based on detections. However, detections may contain background and parts of neighboring objects, contaminating the track query and reducing its consistency throughout the tracking propagation. This problem becomes particularly severe under long-term or severe occlusion. To address these issues, we propose PATrack, a pixel-level association framework that constructs the track query from high-quality object masks. Specifically, PATrack suppresses the effects of background features by constructing the track query using objects' masks and reduces contamination of the track query from neighboring objects through the calibration of the objects' masks. It further roughly estimates the occluded regions of object masks and suppresses unreliable trajectory generation for heavily occluded objects, thereby improving robustness to long-term and severe occlusion. PATrack achieves state-of-the-art performance on DanceTrack and SportsMOT, while delivering significant performance on MOT17, demonstrating its extremely high reliability and superiority.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Object Tracking, Tracking-by-Query

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wUW7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PATrack combines SAM3 mask propagation with YOLOX detections for multi-object tracking. It introduces holistic object estimation, mask calibration and occlusion-aware memory updates. The main experiments evaluate the method on DanceTrack, SportsMOT and MOT17, with ablations of its three components.

Critical strengths
  1. The design addresses a clearly identified problem. Occlusion can affect both current localization and the reliability of features written into memory. Detection-assisted estimation, mask calibration and selective memory updates address different parts of this problem, giving the framework a coherent motivation.
  2. The results are competitive against dedicated trackers. PATrack achieves 80.1 HOTA on DanceTrack, compared with 75.8 for SAM2MOT, and 82.9 on SportsMOT, compared with 81.0 for GTA. These comparisons provide meaningful evidence of its competitiveness.
Critical weaknesses
  1. The contribution of additional detections is not sufficiently separated from that of the association design. Adding HOE raises HOTA from 73.38 to 77.60, accounting for a substantial part of the full improvement. This step introduces both detector information and its processing pipeline. The ablation supports the system’s effectiveness, but does not isolate how much comes from the additional detections versus the proposed association mechanism. Comparisons with the original SAM3 should therefore be interpreted carefully.
  2. The claims about occlusion handling and long-term identity preservation need more direct quantitative support. The main evidence consists of aggregate tracking metrics and qualitative examples. Results grouped by occlusion severity or duration, together with identity recovery statistics, would more directly establish whether mask calibration and memory filtering address the stated failure modes.
  3. Some conclusions are stronger than the observed differences. In Table 5, the proposed association function achieves 77.60 HOTA against 77.55 for HMIoU, while obtaining lower MOTA. This supports a modest improvement on selected metrics, but does not establish substantial superiority. The contribution of the association function itself consequently appears limited.
Questions for authors
  1. Could a SAM3 baseline using the same YOLOX detections, output rules and a conventional association method help separate the contribution of detector information from the proposed association design?
  2. Are there results grouped by occlusion severity or duration that directly support the effects of PMC and OAI on identity preservation?
  3. How should the small HOTA gain and lower MOTA relative to HMIoU in Table 5 be interpreted? Should the corresponding conclusion be narrowed?
Reviewer_oTBaOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Query-based multi-object tracking typically uses bounding-box features to initialize and update track queries. However, these boxes may include background or other objects. Under severe occlusion or object interactions, contaminated query features can lead to identity switches and tracking failures. This paper proposes PATrack, a SAM3-based pixel-level multi-object tracking framework that uses cleaner and more stable pixel-level features for association across frames. It shifts query feature construction from bounding boxes to pixel masks and combines occlusion estimation with reliability filtering to improve identity consistency in long-term tracking.

Critical strengths

The combination requires no additional training and delivers substantial empirical gains on three benchmarks. It offers practical value for adapting an existing strong segmentation tracker.

Critical weaknesses

The experimental setup conflates the benefits of the core mechanisms with those of the detector. The improvement from SAM3 to +HOE already accounts for a substantial portion of the overall gains, while HOE simultaneously introduces YOLOX and full bounding-box information. On MOT17, the SAM3 baseline achieves a MOTA of only 19.2, compared with 74.5 for PATrack. This particularly calls for disentangling gains in output box/detection quality from gains in identity association.

Questions for authors

Could you add a control that strictly matches the auxiliary detection information: SAM3 with the same YOLOX detector and conventional association/bounding-box processing, but without PMC/OAI or the proposed overall estimation rules? Please also report DetA, AssA, and results grouped by occlusion level separately to identify the main sources of improvement.

Reviewer_SAkjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PATrack augments pretrained SAM3 tracking with a YOLOX detector and rule-based prediction refinement, without additional training. SAM3 produces current-frame masks from historical memories. Holistic Object Estimation matches track predictions to YOLOX boxes as estimates of full-object extent; Pixel Mask Calibration filters unreliable tracks and crops selected masks; and Occlusion-Aware Interaction uses overlap and area ratios to control track states and memory updates. These additions are intended to reduce the effects of unreliable masks under occlusion. Experiments on DanceTrack, SportsMOT, and MOT17 report improvements over the SAM3 baseline, supported by component ablations, selected threshold studies, qualitative examples, and an inference-time breakdown.

Critical strengths

I acknowledge the following positive aspects of the work, but I do not consider them substantial enough to support acceptance.

1. The component ablations support the effectiveness of the refinements within the evaluated system. On DanceTrack-val, Table 4 reports 77.60 HOTA / 75.55 AssA with HOE and 79.75 / 77.19 with both PMC and OAI. Adding PMC or OAI individually also improves HOTA to 78.72 or 79.14. These results provide evidence that the refinements contribute beyond the auxiliary detector and HOE, and constitute useful empirical support for this implementation.

2. The improvements require no additional training and add modest computation relative to SAM3. The method reuses pretrained SAM3 and YOLOX weights. Appendix A.3 reports 400.21 ms/frame for the complete system, including the auxiliary detector, versus 366.25 ms/frame for SAM3 on an RTX A6000—an increase of approximately 9.3%. This provides a practical way to improve an existing SAM3-based tracking pipeline without a further training stage.

Critical weaknesses

The paper overstates its contribution by framing rule-based refinements of SAM3 as a new "pixel-level association" mechanism. Its conceptual contribution is limited, and its technical contribution is insufficiently validated. Its system-level results also do not establish a compelling advantage in accuracy, speed, or their trade-off that would justify acceptance on practical grounds alone.

1. The paper overstates its association contribution by presenting rule-based refinements as "pixel-level association". The abstract presents PATrack as "a pixel-level association framework that constructs the track query from high-quality object masks" to address contamination in detection-based track queries. The introduction further describes this as a "novel framework," identifies initializing and updating queries with features that exclude background and neighboring objects as its "core idea" (lines 53–57), and lists the pixel-level association framework as its first main contribution (lines 87–88). Together, these statements frame mask-based association as the paper's central methodological advance.

The core association of a historical track with its current-frame target is still performed by SAM3's existing memory-conditioned tracking process. Section 3.1 states that SAM3 combines historical memories with current-frame features to produce a mask prediction for each existing track (lines 173–177). PATrack applies rules around these predictions: Equation (1) filters new-track candidates using mask overlap; Equation (2) matches propagated track boxes to YOLOX detections to obtain auxiliary spatial extents; PMC flags unreliable tracks and crops masks; and OAI controls track states and suppresses unreliable memory updates. These additions provide auxiliary matching and reliability filtering or refinement, while the core temporal association remains with SAM3. Moreover, the method does not specify an explicit pixel-level identity-association operation. Identity continuity is implicit in SAM3's interaction between historical representations and current image features. Pixel-level mask outputs and the object-pair mask IoU in Equation (1) do not establish that association itself is performed at pixel granularity.

The incremental gains in Table 4 support the usefulness of these refinements, but do not substantiate a new association mechanism. The abstract and introduction nevertheless present "pixel-level association" as the central methodological advance. This framing overstates both the association contribution and the basis for describing it as pixel-level. The method would be more accurately described as detector-guided mask refinement and memory/track management built on SAM3's existing tracking process.

I suggest that the authors exercise greater caution when packaging refinements of an existing tracking system as new methodological concepts, particularly during AI-assisted polishing, and ensure that such framing does not exaggerate the actual contribution.

2. The conceptual insight and demonstrated methodological value are limited. The conceptual premise is already well established in tracking-by-detection. FineTrack explicitly identifies contamination from background and other objects in box- and center-based appearance features, and uses part masks to reduce this interference (Sections 1 and 3.3). BoT-SORT restricts appearance-state updates to high-confidence detections because crowds, occlusion, and blur degrade features (Section 3.3), while Deep OC-SORT uses confidence-adaptive updates to downweight or reject corrupted appearance embeddings (Section 3.3). These works already establish the problem of unreliable identity evidence and the principle of selectively updating historical representations. In segmentation-based tracking, SAM2MOT provides the closer precedent of excluding corrupted current-frame memories through cross-object reasoning. PATrack uses different geometric criteria, but does not establish a new failure mechanism or substantially deepen the understanding of when and why such reliability controls succeed. Its ablations demonstrate useful gains within the evaluated system, without establishing the conceptual advance suggested by its framing.

The relevant technical question is whether the proposed rules address these shared problems more effectively than existing solutions. Table 4 establishes incremental benefits from PMC and OAI over HOE, while Tables 1–3 provide complete-system comparisons. These are useful results, but they do not isolate the relative merit of the proposed reliability rules. For example, on DanceTrack-test, SAM3 already achieves 76.6 HOTA, compared with 47.7 for ByteTrack and 55.1 for OC-SORT, before PATrack raises the score to 80.1. A substantial baseline advantage therefore already exists independently of the proposed rules; the full gap cannot be attributed to their technical contribution. Table 5 compares alternative matching costs, but does not replace a comparison against relevant correction or memory-selection strategies. An appropriate controlled comparison in a common tracking setup, or within the same SAM3-plus-detector pipeline, is needed to establish whether PATrack's rules handle the shared contamination and occlusion problems more effectively than representative existing alternatives.

Generality is a separate concern. All three benchmarks use the SAM3–YOLOX pairing. Reusing most thresholds across datasets is useful evidence of cross-scene applicability, but it does not demonstrate that the rules remain effective with other trackers, detectors, or model versions. The method assumes that matched detector boxes provide useful constraints on SAM3 masks, and uses fixed geometric ratios to judge reliability. Experiments across multiple tracker–detector model pairs, including changes to each component, would help determine whether the rules capture a transferable principle or mainly compensate for the particular error patterns of SAM3 and YOLOX. For each pair, the comparison should measure the gain from adding the same rules under a matched output protocol, first with the original thresholds and, if retuned, with a comparable tuning budget. Without evidence beyond this model pairing, the results establish a working SAM3–YOLOX configuration but do not establish the proposed rules as a broadly applicable solution to the shared tracking problem.

3. The paper does not establish a compelling advantage in absolute accuracy, speed, or the accuracy–speed trade-off. With a limited conceptual contribution and an insufficiently validated technical contribution, practical performance becomes the remaining potential basis for acceptance. Such a contribution could come from substantially higher accuracy, substantially faster inference, or a better accuracy–speed trade-off than existing systems. The paper does not establish a sufficiently strong advantage along any of these three dimensions.

In terms of accuracy, PATrack reports strong DanceTrack results, but its performance does not establish a clear overall lead. On SportsMOT-test, NOOUGAT's offline version reports 85.6 HOTA versus PATrack's 82.9; on MOT17-test, SPAM, Table 2 reports 67.5 versus 64.7 using an offline model fine-tuned on target-dataset ground truth. These different settings limit direct comparison, but the results do not support an unrestricted claim of superior accuracy. In terms of speed, the complete PATrack system runs at approximately 2.5 FPS on an RTX A6000, without demonstrating a throughput advantage over competing trackers. Finally, its small additional overhead relative to SAM3 does not establish a better accuracy–speed trade-off against other complete systems; that comparison is missing. The reported gains over SAM3 are useful, but they do not establish a practical advantage strong enough to support acceptance on system performance alone.

Questions for authors
  1. Comparative technical effectiveness. How do the proposed rules compare with representative existing solutions to appearance contamination and unreliable historical updates under a controlled setup? This could be evaluated in a common tracking setting or within the same SAM3-plus-detector pipeline, using alternatives such as confidence-based update filtering or an appropriately adapted SAM2MOT rule. Please specify the models, training, output-box policy, retained SAM3 procedures, and tuning budget. Tables 4–5 provide useful component and matching-cost comparisons; the missing evidence is whether the proposed reliability rules improve on relevant existing alternatives. Please also complement Appendix A.2's qualitative examples with a quantitative analysis using the available box and identity annotations: report rule-trigger counts and changes in localization, false positives/negatives, and identity errors on shared validation clips with and without the rules. These statistics would identify which tracking errors the rules correct, quantify any new errors they introduce, and assess whether they improve identity continuity in addition to detection and localization. State the sample size and selection procedure. Any direct claims about pixel contamination or memory-update correctness should be supported by suitable annotations or an audited subset.

  2. Generality across model pairs. Do the same rules remain effective beyond SAM3–YOLOX? Please consider multiple compatible tracker–detector pairs, covering changes to the tracker and detector across the tested configurations. Within each pair, compare performance with and without the rules under the same output protocol. Report results using the original thresholds and, if retuned, a comparable tuning budget. This would help distinguish transferable improvements from compensation for the particular error patterns of SAM3 and YOLOX.

  3. Practical advantage of the complete system. Does PATrack offer substantially higher accuracy, higher throughput, or a better accuracy–speed trade-off than representative existing trackers? Please provide a comparison under comparable evaluation and computing conditions, reporting end-to-end timing that includes SAM3, YOLOX, and the proposed rules. Specify differences in training data, hardware, and input resolution, and clarify whether the evaluated outputs use future frames, delayed confirmation, or offline post-processing. The relevant evidence is an advantage over alternative complete systems, beyond the small additional overhead relative to SAM3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.