- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new method in Language-guided multi-camera tracking. The method predicts future target observations in terms of camera, arrival time, and entry region, and recursively updates a persistent belief using posterior-weighted candidate and null-conditioned states.
Critical strengths
- The problem is clearly characterized. By distinguishing between "waiting" (null), camera unavailability, and the actual absence of the target, and by using a censored likelihood to incorporate the evidence inherent in "no observation" (Eq. 9), the probabilistic semantics are formulated with greater rigor.
Critical weaknesses
- The improvement attributable to the proposed structured posterior update is relatively modest. While removing the event prior or recursive feedback causes substantial performance degradation, the structured ReWorld update improves HOTA by only 0.50 points over a comparably sized generic learned updater on MTMMC.
- Limited novelty. The concepts of posterior-weighted candidate hypotheses, a null/missed-detection hypothesis, and recursive state prediction are long established in PDA/JPDA, IPDA, MHT, and Gaussian-sum filtering. The authors cite this literature, and Appendix T frames the difference as an application to language-specified identities and camera-conditioned events rather than as a new inference mechanism. Mechanistically, the new element is the learned projection that reduces the posterior mixture (Eq. 4) to a fixed-size state.
Questions for authors
The abstract states that the model “preserves uncertainty about alternative trajectories,” and Sec. 3.4 states that “later evidence can resolve an earlier ambiguity.” Yet Sec. 3.4 and Appendix T confirm that the mixture is collapsed at every step “without retaining a tree of past decisions.” The retained belief is a per-step marginal: a categorical route distribution, unimodal velocity moments, and a learned 256-d summary. It is not a distribution over association histories, and there is no retrodiction, so later evidence cannot re-weight an earlier assignment. Please state precisely which uncertainty is preserved. Please also clarify whether the appearance memory is updated with posterior-weighted candidate features or only with committed matches.
4 comments
Response to Reviewer 6DY1 (2/2)
W2. Limited novelty relative to PDA/JPDA, IPDA, MHT and Gaussian-sum filtering.
We agree with your summary: the recursion is classical, and inside the update the new element is the learned projection. We rewrote the contribution statement to say this. Sec. 2 and App. T now discuss IPDA and Gaussian-sum filtering, and note that our network-presence probability (Eq. 8) plays the role of IPDA's target-existence probability.
The claim we keep is narrower than our original text suggested: for this problem the collapse is worth learning.
In kinematic PDA the state lives in a Euclidean space and moment matching is the natural summary. Here the quantity being predicted is a camera-network event: which camera next, when, and where in the frame, with waits whose meaning depends on which streams are available (Eq. 9). A moment-matched collapse still exists, and it is our fixed-moment control. What is not known in advance is which statistics of an ambiguous return matter for the forecast two handoffs later. So we let later forecast and association errors decide, through rollouts of up to four handoffs (Sec. 3.5). Two results suggest this is the right call.
First, the learned collapse is not interchangeable with the classical one. With the same predictor, scorer and data, the fixed-moment collapse retains 89.08% of identities after three handoffs. Ours retains 93.83%.
Second, it recovers most of what keeping hypotheses would give, at a fixed cost. We built a depth-3 MHT on the same features, candidates, priors, null definition and causal history, and evaluated it on the same 1,200 chains (Table 39). Core time and memory are measured at 19 cameras, 64 active identities and 412 gated pairs, and include MHT expansion, scoring, pruning and history maintenance.
| Method | IR@3 | Core time (ms) | Core memory (GB) |
|---|---|---|---|
| MHT, 4 branches | 91.17% (1,094) | 31.6 | 4.8 |
| MHT, 8 branches | 93.17% (1,118) | 47.2 | 6.1 |
| MHT, 16 branches | 94.58% (1,135) | 73.9 | 8.7 |
| ReWorld (one collapsed belief) | 93.83% (1,126) | 25.4 | 3.3 |
ReWorld lands between the 8- and the 16-branch tracker, at about half the time and memory of the former. The 16-branch tracker is 0.75 points higher at 2.9x the time and 2.6x the memory, and the paper reports that as it is.
W1. The structured update gains only 0.50 HOTA over the generic updater.
We agree that 0.50 is modest as an aggregate, and the revised abstract no longer leads with it. Two things compress the aggregate. All update variants share the same frozen detections and local tracklets, so the detection half of HOTA = √(DetA · AssA) barely moves; at ReWorld's DetA of 44.37 (Table 1), 0.50 HOTA is about one point of AssA. And the two updates behave alike on unambiguous returns, which are the majority.
To see where the structure matters, we split the same 1,200 chains by a candidate-score margin whose cutoff was fixed on validation data (Table 32):
| Chains | Generic updater, IR@3 | ReWorld, IR@3 | Gain |
|---|---|---|---|
| High ambiguity (360) | 76.67% (276) | 86.67% (312) | +10.00 |
| Low ambiguity (840) | 95.83% (805) | 96.90% (814) | +1.07 |
| All (1,200) | 90.08% (1,081) | 93.83% (1,126) | +3.75 |
The ambiguous chains account for 36 of the 45 chains gained. Paired by chain, ReWorld fixes 59 chains that the generic updater loses and breaks 14 that it keeps (Table 33; exact McNemar test treating chains as independent, p < 1e-6). The next-camera forecast improves on the same chains, from 87.17% to 89.83% top-1, and median arrival-time error falls from 0.80 to 0.72 s (Table 34).
The original matched comparison points the same way: +0.50, +0.47 and +0.46 HOTA on three disjoint held-out scenes with five seeds each (Table 28). The extra structure costs 0.02 ms and 0.02M parameters per update (Table 27).
Presentation. We took the score of 2 seriously and reread the paper as a newcomer would. Two things made it hard to follow, and both are fixed. Secs. 3.3–3.4 now state side by side what enters the commitment-gated appearance memory and what enters the per-step belief. And because different analyses use different event groups and two configurations, App. L.1 now has one table that maps every result table to its population, configuration and purpose.
Thank you for a critique that made both the claims and the positioning against classical filtering sharper. If some part is still unconvincing, please tell us which; we can run a further check before the discussion closes. If your concerns are resolved, we would be grateful if you would reconsider your rating.
Additional follow-up on the posterior update
Thank you for your careful review and for raising specific questions about the contribution and interpretation of the posterior update. Your comments helped us clarify what uncertainty the state retains and how the update relates to PDA-style inference. We have also reported additional controlled experiments on the learned posterior summary in part 2 of our recent follow-up response to Reviewer wZm6. These compare update structures and single-handoff versus multi-handoff supervision under matched conditions, with ambiguity-stratified and paired analyses. On a separate, fixed set of 1,289 three-handoff chains, ReWorld achieves 94.18% IR@3, compared with 90.07% for the capacity-matched generic updater and 87.82% for the fixed-moment PDA-style control under multi-handoff supervision.
We would be grateful for your thoughts on whether our response, the revised manuscript, and these additional results address your concerns. If you find the clarifications and evidence persuasive, we hope you might consider a more favorable assessment. We welcome any remaining questions and would be glad to discuss them. If the paper is accepted, we will address all issues raised by you and the other reviewers and complete the corresponding revisions in the camera-ready version. This will include documenting the additional experiments and refining the explanation of the learned state and its limitations. Thank you for the time and care you have devoted to helping us improve the paper.
Thank you for your reconsideration
Thank you for reconsidering your assessment and for the care you have devoted to reviewing our work. Your comments helped us state more precisely which uncertainty the recurrent belief retains, distinguish the belief update from the appearance-memory update, and clarify the relationship to classical probabilistic data association. They also motivated more focused comparisons of the learned posterior update. We sincerely appreciate your constructive critique, which has improved the clarity and rigor of the paper.
Response to Reviewer 6DY1 (1/2)
Thank you for this review. Your reading of Sec. 3.4 and Appendix T is correct, and it caught two sentences of ours that claimed more than the method does. We answer your question here and the two weaknesses in part 2. New results are in App. V and X of the revised PDF (blue).
Q (first part). Which uncertainty is preserved?
Eq. (4) is collapsed at every update, with no tree of past assignments. What is carried is a single-step forward summary: posterior-weighted probabilities over cameras and blind regions, velocity moments, a network-presence probability, and a learned 256-d vector. It is not a distribution over association histories. There is no retrodiction, and an emitted assignment is never revised.
Our phrases "preserves uncertainty about alternative trajectories" (Abstract) and "later evidence can resolve an earlier ambiguity" (Sec. 3.4) claimed more than this. Both are replaced:
What the state does keep is the mass of the unselected alternatives, inside the prior for the next observation. Your question made us test whether that mass matters. On 1,200 fixed three-handoff chains, we intervened on the trained model's feedback while holding the current decision and observation features fixed (Table 31). Failed chains stay in the denominator.
Dropping the non-top mass costs 6.00 points of three-handoff retention. Redistributing it costs 8.25. Later decisions therefore depend on where the unselected mass sits, and not only on the candidate that was selected. A test-time intervention also takes the update outside its training distribution, so we read these two rows together with the separately trained controls below them, which point the same way.
These chains are a new fixed set, so absolute values differ from the original Table 30, where the same full-versus-generic comparison gives 95.79% vs 93.14%.
Q (second part). Is appearance memory updated with posterior-weighted features or only with committed matches?
Only with committed matches. The appearance memory is written after a match passes the commitment rule (threshold, margin, consecutive confirmations), using that candidate's descriptor. During a wait it is frozen: neither an unconfirmed candidate nor a posterior-weighted average of candidates is written into it.
The recurrent belief follows a different rule. It is updated with posterior-weighted candidate and null states whenever new evidence arrives, including when the output is "wait". Temporary null is an inference hypothesis and wait is a decision, so a wait can still carry nonzero candidate mass. With no new observation, elapsed time and the absence interval advance the state once; cached candidate evidence is not fused a second time.
The separation is deliberate. A soft update of the appearance memory would let distractor features drift into the identity anchor during a wait. So an unconfirmed candidate can change where and when we expect the target next, but never what we match against.
The revised text now says this in one sentence: "The global appearance template is maintained separately and updated only after a match satisfies the commitment rule; it remains unchanged during waiting." Secs. 3.3–3.4 set the two write rules side by side.