acceptodds
Under review as a conference paper at ICLR 2027

ReWorld-Track: A Recursive Event World Model for Language-Guided Multi-Camera Tracking

Abstract

Language-guided multi-camera tracking must preserve a target identity across unobserved gaps, where similar candidates and uncertain returns can make early associations unreliable. A wrong match can corrupt the history used to predict later observations and propagate identity errors across subsequent camera handoffs. We propose ReWorld-Track, a recursive event world model that carries association uncertainty into future predictions. Candidate matches and continued waiting define alternative target states, whose posterior probabilities are used to update a persistent recurrent belief. This representation preserves uncertainty about alternative trajectories through successive observations. This belief predicts the next camera, arrival time, and entry region, while appearance and language evidence guide association. By training across successive handoffs, the model learns to retain uncertainty that remains useful for later predictions and identity decisions. ReWorld-Track achieves HOTA scores of 65.19 on CityFlowV2 and 45.36 on MTMMC, with improved identity continuity across repeated handoffs. On MTMMC, its structured posterior update gains 0.50 HOTA points over a similarly sized generic updater and 0.94 points over fixed-moment soft association, raising next-camera accuracy from 86.03% to 87.41% and reducing median arrival-time error from 0.78 s to 0.71 s for subsequent target returns.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Language-guided multi-camera tracking, Recursive event world model, Posterior state update, Association uncertainty, Identity continuity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6DY1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new method in Language-guided multi-camera tracking. The method predicts future target observations in terms of camera, arrival time, and entry region, and recursively updates a persistent belief using posterior-weighted candidate and null-conditioned states.

Critical strengths
  1. The problem is clearly characterized. By distinguishing between "waiting" (null), camera unavailability, and the actual absence of the target, and by using a censored likelihood to incorporate the evidence inherent in "no observation" (Eq. 9), the probabilistic semantics are formulated with greater rigor.
Critical weaknesses
  1. The improvement attributable to the proposed structured posterior update is relatively modest. While removing the event prior or recursive feedback causes substantial performance degradation, the structured ReWorld update improves HOTA by only 0.50 points over a comparably sized generic learned updater on MTMMC.
  2. Limited novelty. The concepts of posterior-weighted candidate hypotheses, a null/missed-detection hypothesis, and recursive state prediction are long established in PDA/JPDA, IPDA, MHT, and Gaussian-sum filtering. The authors cite this literature, and Appendix T frames the difference as an application to language-specified identities and camera-conditioned events rather than as a new inference mechanism. Mechanistically, the new element is the learned projection that reduces the posterior mixture (Eq. 4) to a fixed-size state.
Questions for authors

The abstract states that the model “preserves uncertainty about alternative trajectories,” and Sec. 3.4 states that “later evidence can resolve an earlier ambiguity.” Yet Sec. 3.4 and Appendix T confirm that the mixture is collapsed at every step “without retaining a tree of past decisions.” The retained belief is a per-step marginal: a categorical route distribution, unimodal velocity moments, and a learned 256-d summary. It is not a distribution over association histories, and there is no retrodiction, so later evidence cannot re-weight an earlier assignment. Please state precisely which uncertainty is preserved. Please also clarify whether the appearance memory is updated with posterior-weighted candidate features or only with committed matches.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6DY1 (1/2)

Thank you for this review. Your reading of Sec. 3.4 and Appendix T is correct, and it caught two sentences of ours that claimed more than the method does. We answer your question here and the two weaknesses in part 2. New results are in App. V and X of the revised PDF (blue).

Q (first part). Which uncertainty is preserved?

Eq. (4) is collapsed at every update, with no tree of past assignments. What is carried is a single-step forward summary: posterior-weighted probabilities over cameras and blind regions, velocity moments, a network-presence probability, and a learned 256-d vector. It is not a distribution over association histories. There is no retrodiction, and an emitted assignment is never revised.

Our phrases "preserves uncertainty about alternative trajectories" (Abstract) and "later evidence can resolve an earlier ambiguity" (Sec. 3.4) claimed more than this. Both are replaced:

  • Abstract: "The update carries uncertainty about the current association into forecasts of subsequent observations."
  • Sec. 3.4: "At each observation event, the posterior over candidate-conditioned states and the temporary-null state is compressed into a recurrent belief, allowing uncertainty in the current association to influence subsequent event predictions. The belief does not retain separately recoverable association histories."

What the state does keep is the mass of the unselected alternatives, inside the prior for the next observation. Your question made us test whether that mass matters. On 1,200 fixed three-handoff chains, we intervened on the trained model's feedback while holding the current decision and observation features fixed (Table 31). Failed chains stay in the denominator.

Feedback passed to the next forecastHow obtainedIR@1IR@2IR@3
Full posterior (ReWorld)trained and evaluated as is98.4295.7593.83
MAP-only: all mass moved to the top hypothesistest-time intervention97.7593.1787.83
Non-MAP mass redistributedtest-time intervention97.5891.8385.58
Fixed-moment (PDA-style) updatetrained separately97.9293.8389.08
Generic learned updatetrained separately98.1794.3390.08

Dropping the non-top mass costs 6.00 points of three-handoff retention. Redistributing it costs 8.25. Later decisions therefore depend on where the unselected mass sits, and not only on the candidate that was selected. A test-time intervention also takes the update outside its training distribution, so we read these two rows together with the separately trained controls below them, which point the same way.

These chains are a new fixed set, so absolute values differ from the original Table 30, where the same full-versus-generic comparison gives 95.79% vs 93.14%.

Q (second part). Is appearance memory updated with posterior-weighted features or only with committed matches?

Only with committed matches. The appearance memory is written after a match passes the commitment rule (threshold, margin, consecutive confirmations), using that candidate's descriptor. During a wait it is frozen: neither an unconfirmed candidate nor a posterior-weighted average of candidates is written into it.

The recurrent belief follows a different rule. It is updated with posterior-weighted candidate and null states whenever new evidence arrives, including when the output is "wait". Temporary null is an inference hypothesis and wait is a decision, so a wait can still carry nonzero candidate mass. With no new observation, elapsed time and the absence interval advance the state once; cached candidate evidence is not fused a second time.

The separation is deliberate. A soft update of the appearance memory would let distractor features drift into the identity anchor during a wait. So an unconfirmed candidate can change where and when we expect the target next, but never what we match against.

The revised text now says this in one sentence: "The global appearance template is maintained separately and updated only after a match satisfies the commitment rule; it remains unchanged during waiting." Secs. 3.3–3.4 set the two write rules side by side.

AuthorsOpenReview ↗

Response to Reviewer 6DY1 (2/2)

W2. Limited novelty relative to PDA/JPDA, IPDA, MHT and Gaussian-sum filtering.

We agree with your summary: the recursion is classical, and inside the update the new element is the learned projection. We rewrote the contribution statement to say this. Sec. 2 and App. T now discuss IPDA and Gaussian-sum filtering, and note that our network-presence probability (Eq. 8) plays the role of IPDA's target-existence probability.

The claim we keep is narrower than our original text suggested: for this problem the collapse is worth learning.

In kinematic PDA the state lives in a Euclidean space and moment matching is the natural summary. Here the quantity being predicted is a camera-network event: which camera next, when, and where in the frame, with waits whose meaning depends on which streams are available (Eq. 9). A moment-matched collapse still exists, and it is our fixed-moment control. What is not known in advance is which statistics of an ambiguous return matter for the forecast two handoffs later. So we let later forecast and association errors decide, through rollouts of up to four handoffs (Sec. 3.5). Two results suggest this is the right call.

First, the learned collapse is not interchangeable with the classical one. With the same predictor, scorer and data, the fixed-moment collapse retains 89.08% of identities after three handoffs. Ours retains 93.83%.

Second, it recovers most of what keeping hypotheses would give, at a fixed cost. We built a depth-3 MHT on the same features, candidates, priors, null definition and causal history, and evaluated it on the same 1,200 chains (Table 39). Core time and memory are measured at 19 cameras, 64 active identities and 412 gated pairs, and include MHT expansion, scoring, pruning and history maintenance.

MethodIR@3Core time (ms)Core memory (GB)
MHT, 4 branches91.17% (1,094)31.64.8
MHT, 8 branches93.17% (1,118)47.26.1
MHT, 16 branches94.58% (1,135)73.98.7
ReWorld (one collapsed belief)93.83% (1,126)25.43.3

ReWorld lands between the 8- and the 16-branch tracker, at about half the time and memory of the former. The 16-branch tracker is 0.75 points higher at 2.9x the time and 2.6x the memory, and the paper reports that as it is.

W1. The structured update gains only 0.50 HOTA over the generic updater.

We agree that 0.50 is modest as an aggregate, and the revised abstract no longer leads with it. Two things compress the aggregate. All update variants share the same frozen detections and local tracklets, so the detection half of HOTA = √(DetA · AssA) barely moves; at ReWorld's DetA of 44.37 (Table 1), 0.50 HOTA is about one point of AssA. And the two updates behave alike on unambiguous returns, which are the majority.

To see where the structure matters, we split the same 1,200 chains by a candidate-score margin whose cutoff was fixed on validation data (Table 32):

ChainsGeneric updater, IR@3ReWorld, IR@3Gain
High ambiguity (360)76.67% (276)86.67% (312)+10.00
Low ambiguity (840)95.83% (805)96.90% (814)+1.07
All (1,200)90.08% (1,081)93.83% (1,126)+3.75

The ambiguous chains account for 36 of the 45 chains gained. Paired by chain, ReWorld fixes 59 chains that the generic updater loses and breaks 14 that it keeps (Table 33; exact McNemar test treating chains as independent, p < 1e-6). The next-camera forecast improves on the same chains, from 87.17% to 89.83% top-1, and median arrival-time error falls from 0.80 to 0.72 s (Table 34).

The original matched comparison points the same way: +0.50, +0.47 and +0.46 HOTA on three disjoint held-out scenes with five seeds each (Table 28). The extra structure costs 0.02 ms and 0.02M parameters per update (Table 27).

Presentation. We took the score of 2 seriously and reread the paper as a newcomer would. Two things made it hard to follow, and both are fixed. Secs. 3.3–3.4 now state side by side what enters the commitment-gated appearance memory and what enters the per-step belief. And because different analyses use different event groups and two configurations, App. L.1 now has one table that maps every result table to its population, configuration and purpose.

Thank you for a critique that made both the claims and the positioning against classical filtering sharper. If some part is still unconvincing, please tell us which; we can run a further check before the discussion closes. If your concerns are resolved, we would be grateful if you would reconsider your rating.

AuthorsOpenReview ↗

Additional follow-up on the posterior update

Thank you for your careful review and for raising specific questions about the contribution and interpretation of the posterior update. Your comments helped us clarify what uncertainty the state retains and how the update relates to PDA-style inference. We have also reported additional controlled experiments on the learned posterior summary in part 2 of our recent follow-up response to Reviewer wZm6. These compare update structures and single-handoff versus multi-handoff supervision under matched conditions, with ambiguity-stratified and paired analyses. On a separate, fixed set of 1,289 three-handoff chains, ReWorld achieves 94.18% IR@3, compared with 90.07% for the capacity-matched generic updater and 87.82% for the fixed-moment PDA-style control under multi-handoff supervision.

We would be grateful for your thoughts on whether our response, the revised manuscript, and these additional results address your concerns. If you find the clarifications and evidence persuasive, we hope you might consider a more favorable assessment. We welcome any remaining questions and would be glad to discuss them. If the paper is accepted, we will address all issues raised by you and the other reviewers and complete the corresponding revisions in the camera-ready version. This will include documenting the additional experiments and refining the explanation of the learned state and its limitations. Thank you for the time and care you have devoted to helping us improve the paper.

AuthorsOpenReview ↗

Thank you for your reconsideration

Thank you for reconsidering your assessment and for the care you have devoted to reviewing our work. Your comments helped us state more precisely which uncertainty the recurrent belief retains, distinguish the belief update from the appearance-memory update, and clarify the relationship to classical probabilistic data association. They also motivated more focused comparisons of the learned posterior update. We sincerely appreciate your constructive critique, which has improved the clarity and rigor of the paper.

Reviewer_kinPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReWorld-Track, a recursive event world model for language-guided multi-camera tracking. The method models the target's next observation as a structured event (camera, arrival time, entry region), uses a recursive belief state to carry candidate and temporary-null evidence across successive camera handoffs, and updates the predictive state through a closed loop of "prediction–association–waiting–feedback." Experiments are conducted on CityFlowV2 vehicles and MTMMC pedestrians, including component ablations, missing-observation tests, repeated handoffs, language perturbations, cross-domain transfer, and latency analysis.

Critical strengths
  1. Clear motivation and a closed-loop method. The paper targets the core difficulty of ambiguous identity continuation in multi-camera tracking and feeds the posterior of uncertain association back into the next event prediction. Prediction, association, and waiting are connected through a shared belief, making the design logically coherent and novel.

  2. Rigorous experimental controls. The predictor-matched control (Table 17), confidence intervals over five paired training seeds (Table 18), false-match diagnostics at matched recovery coverage (Figure 4(b)), and controlled language perturbation analysis (Tables 3/7/22) reasonably isolate the contribution of each module, making the conclusions more credible than overall metrics alone.

  3. Detailed analysis of the waiting mechanism and language evidence. Temporary null is separated from network presence, and Table 19 and Figure 4(b) show the role of waiting in reducing premature matches. The gains of language under visual ambiguity and the risks when descriptions are misleading are also quantified, and the analysis is relatively honest.

Critical weaknesses
  1. Strong dependence on a fixed camera graph and language query quality. The method assumes that the camera graph G and installation features are available at test time; Table 25 shows that HA drops from 85.1 to 74.6 under unseen topology. On the language side, misleading descriptions can make performance fall below the no-query baseline (MTMMC 73.08 vs 78.39). Robustness to missing/erroneous graphs and inaccurate queries in real deployment is insufficiently discussed.

  2. Insufficient analysis of the sensitivity and transferability of hyperparameters across datasets and scenes. The method involves key hyperparameters such as the acceptance threshold, the margin over the second-best candidate, and the number of consecutive confirmations. However, the paper does not systematically report whether these hyperparameters need different settings for CityFlowV2 versus MTMMC, for different camera densities, or for different scenes, nor does it analyze the stability of hyperparameters selected on the source domain when transferred to the target domain.

Questions for authors
  1. How does the method perform when the camera graph is unavailable, partially incorrect, or dynamically changing? Can the authors provide robustness experiments on missing/erroneous camera graphs and inaccurate language queries?

  2. How sensitive are the acceptance threshold, margin, and confirmation count to dataset and scene changes, and how stable are source-domain selected hyperparameters when transferred to the target domain?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kinP (1/2)

Thank you for the review, and for reading the controls closely enough to single out the predictor-matched comparison and the matched-coverage diagnostic. Both of your questions are about deployment, and both led to new experiments (App. W, blue in the revised PDF). Q1 is answered here and Q2 in part 2.

W1 / Q1(a). What happens when the camera graph is unavailable, partly wrong, or changing?

Short answer: a wrong graph costs accuracy, no graph at all costs 4 points, and in every corrupted condition posterior feedback helps more than it does with the correct graph.

Graph given at test timeCoverageOne-way HAFull HAFull − One-wayFull FM
Installation graph98.1082.4084.10+1.705.40
10% of edges removed91.4076.5080.80+4.305.95
20% of edges removed84.5069.9075.10+5.207.35
10% spurious edges added98.1080.4083.60+3.205.88
20% spurious edges added98.1077.9081.70+3.806.78
No graph (all cameras)99.5076.2080.10+3.907.90

All rows use the same 1,000 MTMMC handoffs and 4,000 absence decisions, with no retraining (Table 35). Coverage is the share of handoffs whose true candidate survives gating; handoffs that lose it stay in the HA denominator.

Three findings matter for deployment.

  • Missing edges hurt most, and the loss comes from the gate. With 20% of edges removed, the true candidate survives gating in 84.50% of handoffs instead of 98.10%, and HA falls by 9.00 points. Spurious edges cost little: 2.40 points at 20%.
  • Past a certain point, no graph is safer than an incomplete one. With 10% of edges missing the graph still helps (80.80%, against 80.10% with no graph, and fewer false matches). With 20% missing it no longer does: treating every camera as a possible successor is 5.00 points better, at 7.90% false matches against 7.35%.
  • Feedback compensates when the prior degrades. Its gain over one-way prediction is 3.20 to 5.20 points in the corrupted conditions, against 1.70 with the correct graph. The same holds on directed routes never seen in training: 77.25% HA, 4.00 points above one-way prediction (Table 37).

We did not test edges that change within a sequence; each graph was fixed per sequence. The related case we do cover is a camera going offline mid-sequence. Availability enters the update online, with no retraining: on CityFlowV2, HA under an 8.8 s mask is 87.83%, against 67.17% for reactive memory on the same 600 events (Table 20).

W1 / Q1(b). Inaccurate language queries.

You are right that a misleading description can push accuracy below the no-query model. The table below shows how accuracy degrades as the query gets worse. Each column uses the same events in every row (Table 7). Entries are HA for a hard attribute gate / our full scorer, with the full scorer's difference from no query in parentheses.

Query conditionCityFlowV2 (637 events; no query 81.63)MTMMC (509 events; no query 78.39)
Natural description85.40 / 87.76 (+6.12)80.16 / 84.28 (+5.89)
Paraphrase84.14 / 86.66 (+5.02)81.73 / 83.69 (+5.30)
Weak ("a pedestrian")79.43 / 83.05 (+1.41)77.21 / 79.37 (+0.98)
Names an attribute hidden in the current view74.10 / 84.46 (+2.83)68.96 / 81.14 (+2.75)
Misleading (an attribute replaced by a contradictory one)65.31 / 75.67 (−5.97)64.05 / 73.08 (−5.30)

In both domains the full scorer stays above the no-query model when the description is reworded, vague, or names something the current view hides. In that last case a hard gate already falls 7.54 and 9.43 points below no query. The full scorer falls below only at the adversarial end, where an attribute is replaced by a wrong one, and even there it loses about a third of what a hard gate loses (5.30 vs 14.34 points on MTMMC).

Misleading descriptions remain an open limitation, and the Conclusion says so.

AuthorsOpenReview ↗

Response to Reviewer kinP (2/2)

W2 / Q2. Do the commitment hyperparameters need different settings across datasets and scenes, and do source-selected settings transfer?

To your first question: the two datasets do not need different settings. The threshold, margin and confirmation count are selected on each dataset's own validation data, with the objective, the grid and the tie-breaking rule fixed before any test evaluation. The two selections come out the same: threshold 0.80, margin 0.15 and two confirmations on both CityFlowV2 and MTMMC.

Three further observations bear on how stable they are.

They are always applied to a scene they were not chosen on. The split is scene-disjoint: the validation scene and the test scene differ (App. L.1). Every main result therefore already uses settings selected on one installation and applied to another.

The rule acts on a calibrated posterior, not on a raw similarity. ECE is 2.8% for the joint belief, against 5.0% for the temperature-scaled scorer and 8.7% for the raw similarity score (Table 23), and the null mass absorbs what the event forecast leaves unassigned. A threshold of 0.80 then means a risk level rather than a dataset-specific score scale, which is why we expect it to carry over.

The response to each setting is smooth. On MTMMC, HOTA stays between 45.07 and 45.36 for thresholds from 0.65 to 0.90, and within 1.1 points over the whole sweep from 0.50 to 0.95 (Table 19). The new sweep varies margin and confirmation count on the 1,000-handoff group used in part 1, with the checkpoint and threshold fixed (Table 36):

MarginConfirmationsHA (%)FM (%)Delay (s)
0.10285.406.480.94
0.15 (operating point)284.105.401.06
0.20282.304.531.21
0.15186.707.700.78
0.15381.503.901.34

The settings move the tracker along one trade-off between recovering more returns and accepting fewer distractors. From the loosest to the strictest row, HA goes from 86.70% to 81.50%, FM from 7.70% to 3.90%, and mean delay from 0.78 to 1.34 s. Nothing breaks at either end.

Transfer across domains. Table 9 moves source-selected checkpoints, calibration and commitment settings between vehicles and pedestrians with no target tuning. ReWorld reaches 74.68% HA and 5.14% FM from vehicles to pedestrians, and 77.31% HA and 4.45% FM in reverse. That is 4.61 and 5.22 HA points above appearance-plus-time association under the same rule, with lower FM in both directions (7.29% and 6.63% for the baseline).

We did not test other camera densities or frame rates. A deployment that differs strongly in either should re-select these three settings on a short validation clip of its own, and App. W now says so.

Thank you again. If the graph, query or sensitivity results leave a case you care about uncovered, tell us and we will try to add it before the discussion closes. If they answer your concerns, we would be grateful if you would consider updating your rating.

AuthorsOpenReview ↗

Follow-up on robustness and deployment analyses

We appreciate your thoughtful suggestions on the method’s robustness and practical use. Your questions helped us identify deployment conditions that needed closer examination. Our response presents additional analyses of missing and erroneous camera graphs, inaccurate language queries, parameter sensitivity, and cross-domain transfer. The parameter analyses examine the acceptance threshold, candidate-score margin, and confirmation count, together with whether settings selected in one dataset transfer to the other without retuning. We also discuss the performance losses caused by graph errors and misleading descriptions, making the method’s practical limitations more explicit.

We would welcome your feedback on whether these analyses and the accompanying clarifications have sufficiently addressed the points you raised. If the additional evidence strengthens your assessment of the paper, we would be grateful if you could consider a stronger recommendation. We remain happy to answer further questions or discuss any suggestions you may have. If the paper is accepted, we will address all issues raised by you and the other reviewers and complete the corresponding revisions in the camera-ready version, including clearer experimental protocols, parameter-selection procedures, and deployment limitations. Thank you for your time and for helping improve the quality of our paper.

Reviewer_wZm6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ReWorld-Track, a recursive event world model for language-guided multi-camera tracking. The key idea is to preserve uncertainty across camera handoffs instead of committing to a single candidate at each association step. Candidate matches and a temporary-null alternative are represented as alternative target states, whose posterior probabilities are used to update a persistent recurrent belief. This belief predicts the next camera, arrival time, and entry region, while appearance, language, motion, and detector evidence guide candidate association.

The method is evaluated on CityFlowV2 and MTMMC. The reported results show improvements in HOTA, identity continuity, next-camera prediction, and arrival-time estimation. The paper also includes controlled ablations of event priors, recurrent feedback, language queries, temporary null states, and posterior update mechanisms.

Critical strengths
  1. The paper addresses an important failure mode in multi-camera tracking.
    Early commitment to an incorrect candidate can corrupt the history used for later predictions and cause identity errors across repeated handoffs. ReWorld-Track explicitly preserves candidate and null uncertainty, which is a meaningful and practically motivated design.

  2. The proposed posterior update is well aligned with the stated problem.
    Instead of using only the most likely candidate, the method propagates weighted candidate-conditioned and null-conditioned states into the next event forecast. The ablations show that the event prior, recurrent update, and temporary-null mechanism contribute to identity continuity and reduce false matches.

  3. The experimental analysis is relatively comprehensive.
    The paper evaluates complete trajectories, repeated handoffs, missing observations, camera masking, language perturbations, calibration, scene changes, topology shifts, and runtime. It also reports results over multiple training seeds, which strengthens the empirical analysis.

Critical weaknesses
  1. The method appears to rely substantially on fixed scene and camera-topology priors.
    The predictive state uses a predefined camera graph, transition structure, availability information, and route-dependent timing distributions. Although the paper includes topology-shift experiments, performance still degrades under installation changes. It is therefore unclear how broadly the proposed world-model formulation generalizes to unseen or dynamically changing camera networks.

  2. The practical complexity and computational cost are not fully characterized.
    ReWorld-Track maintains structured beliefs over candidates and null states, performs graph-based propagation, and predicts camera, time, and entry-position distributions. The reported latency is close to the 10 Hz update interval in the largest workload. More analysis is needed to determine how the method scales with the number of active identities, cameras, and candidate associations.

  3. The contribution of language modeling is somewhat secondary.
    The main performance gains appear to come from event priors, recursive belief updates, and uncertainty-aware waiting, while language mainly enters as one additional association score. The paper should clarify whether the method provides a substantial advance specifically for language-guided tracking, or whether the same framework would be effective without language input.

  4. The improvement over the generic learned updater is relatively modest.
    The structured posterior update improves HOTA by 0.50 points over a similarly sized generic updater on MTMMC. Although this gain is consistent, the paper should better discuss whether the additional modeling complexity is justified by the magnitude of the improvement.

Questions for authors
  1. Generalization to changing camera networks:
    How does ReWorld-Track behave when the camera graph, transition probabilities, or travel-time statistics are substantially different from those used during training? Can the authors provide results for graph edges or camera routes that are entirely unseen during training?

  2. Scaling behavior:
    How does runtime and memory usage scale with the number of active identities, cameras, and feasible identity–candidate pairs? In particular, can the authors report results for larger workloads than the 19-camera setting?

  3. Role of language:
    What is the exact contribution of the language branch relative to the event prior and visual appearance features? Could the authors provide a controlled comparison using the same recurrent world model with and without language input across both easy and appearance-ambiguous cases?

  4. Calibration and threshold selection:
    Are all posterior calibration procedures and commitment thresholds selected exclusively on validation data? Please clarify whether any test-set information is used when defining the null decision rule or the evaluation populations.

  5. Comparison with multi-hypothesis tracking:
    How does the proposed posterior state update differ from classical multiple-hypothesis tracking or probabilistic data association beyond the learned recurrent projection? A more direct comparison would help establish the novelty of the formulation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer wZm6 (1/2)

Thank you for the detailed review. You raised four weaknesses and five questions; we answer each under your own numbering, W1–W2 here and the rest in part 2. New results are in App. V–X of the revised PDF (blue).

W1 / Q1. Camera networks that differ from training: unseen edges, routes, and travel-time statistics.

Yes, we can report routes that are entirely unseen. We held out directed camera edges on MTMMC. Every chain that crosses a held-out edge was removed from training, validation and model selection; models were retrained and timing priors refitted on what remained. The test group has 800 handoffs and 3,200 absence decisions (Table 37). Both methods receive the installation graph, so this tests unseen routes under known connectivity.

ConditionOne-way HAFull HAFull − One-wayFull FMArrival MedAE (s)
Held-out routes73.25% (586)77.25% (618)+4.006.53%1.02
Held-out routes, timing-prior center shifted by +30%68.50% (548)73.88% (591)+5.388.00%1.27

On routes it has never seen, the full model stays 4.00 points above one-way prediction. When the travel-time prior is also off by 30%, its own accuracy falls by 3.38 points and the gap to one-way prediction widens to 5.38.

For reference, with seen topology the gap is 1.7 points (85.1% vs 83.4% in Table 25; 84.10% vs 82.40% in Table 35). These are different event groups, so absolute values are not comparable across them. The direction matches the original Table 25, though, where the gap grows from 1.7 to 2.3 and 3.8 points from seen topology to unseen cameras to unseen topology. Feedback matters more as the prior becomes less reliable.

The same holds when the graph itself is wrong. With 20% of edges removed, HA falls from 84.10% to 75.10%, and with no graph at all to 80.10%; the gain over one-way prediction grows from 1.70 to 5.20 and 3.90 points (Table 35; details in our reply to Reviewer kinP).

Two limits remain. Accuracy does fall under installation shift, and we did not test a graph that changes within a sequence. A camera going offline mid-sequence is handled online through the availability term (Table 20).

W2 / Q2. Cost and scaling beyond 19 cameras.

On one RTX 4090 the full pipeline holds 10 Hz at 19 cameras and no longer at 24. P95 end-to-end latency is 111.8 ms at 24 cameras and 148.3 ms at 32, i.e. about 9 and 7 Hz. The paper now states this limit.

CamerasActive IDsGated pairsCore mean (ms)E2E P95 (ms)Core memory (GB)
19481,18638.792.64.6
24641,60047.3111.85.6
32962,40067.8148.37.4

Where the cost comes from:

  • The tracker core is one part of it. Its mean time is 47.3 ms at 24 cameras and 67.8 ms at 32. The end-to-end figure also includes detection, local tracking and CLIP features for every stream, all on the same GPU.
  • The core scales with gated pairs, roughly linearly over this range. From 412 pairs (Table 24, Heavy) to 2,400, core time goes from 25.4 to 67.8 ms: a fixed part plus about 0.02 ms per additional pair. Pairs matter more than identities or cameras as such: in Table 24, 64 identities with 412 pairs run faster than 48 identities with 1,186 pairs (25.4 vs 38.7 ms).
  • Cost does not grow with track length. The belief has a fixed size, so time and memory are constant in the number of past handoffs. Core memory grows from 4.6 to 7.4 GB over these workloads.
  • The structured update is not the expensive part. It takes 0.41 ms against 0.39 ms for the generic updater (Table 27), and the whole core is cheaper than an 8-branch MHT on the same workload (25.4 vs 47.2 ms, Table 39).
AuthorsOpenReview ↗

Response to Reviewer wZm6 (2/2)

W3 / Q3. Is language secondary? Would the framework work without it?

We agree: language is one source of evidence, not the core contribution, and the recursive state works without it. The revised Secs. 1 and 4.5 now say plainly that the mechanism is modality-agnostic and that it is the task, not the update, that is language-guided.

Component removals on MTMMC give the order of importance (Tables 1 and 2(a), five seeds):

VariantHOTAvs full
Full ReWorld-Track45.36
without the language query44.10−1.26
without recursive feedback41.28−4.08
without the event prior40.02−5.34
Adapted CRTracker (uses the query)44.12

Without any query, ReWorld-Track (44.10 HOTA, 81.72% HA, 5.42% FM) is level with the query-guided CRTracker baseline (44.12, 81.59%, 5.67%). So the event model and the recursion carry most of the result.

Where language earns its place is in ambiguous returns. Table 22 is the controlled comparison you ask for: the same events, split by how close the strongest appearance distractor is, with only the query changed.

MTMMC subsetNo queryNatural queryGainMisleading query
Easy appearance90.192.2+2.184.9
Appearance-ambiguous63.274.1+10.957.0

A natural description adds 2.1 points when appearance already separates the target and 10.9 when it does not. Because query evidence changes the candidate weights, that disambiguation also enters the state passed to the next forecast. A misleading description costs 6.2 points on the ambiguous subset; the paper lists this as a limitation.

W4. Is +0.50 HOTA worth the extra modeling complexity?

The aggregate is small because all variants share the same detections and behave alike on unambiguous returns. On ambiguous three-handoff chains, identity retention rises from 76.67% to 86.67%, and chain by chain the structured update fixes 59 chains and breaks 14. The added cost is 0.02 ms and 0.02M parameters per update. Details: point 1 of the general response and our reply to Reviewer 6DY1 (Tables 32–34).

Q4. Are calibration and thresholds selected only on validation data?

Yes. Nothing the tracker runs with is chosen on test data.

  • The scorer is fitted on training pairs. Its temperature, the checkpoint, and the commitment threshold, margin and confirmation count are selected on validation data and frozen before testing.
  • The null decision has no tuned parameter. Its prior mass is what the event forecast leaves unassigned, η0=1−∑jηj\eta_0 = 1 - \sum_j \eta_j.
  • Test annotations are used to define which handoffs are evaluated and whether an outcome is correct. Detector misses stay in the evaluated set. Absence decisions are sampled on a model-independent 2 Hz grid.

Three analyses do use test outcomes, and they are diagnostics, not operating choices: the threshold sweep (Table 19), the easy/ambiguous grouping in Table 22, which uses the true-candidate/distractor margin, and the 96%-reacquisition comparison in Fig. 4(b). The captions now say so. The new chain grouping in App. V avoids this: it uses shared candidate-score margins and a cutoff defined on validation data.

Q5. How does the update differ from MHT or PDA, beyond the learned projection?

In the recursion itself, it does not: candidate/null weighting and collapse-every-step are PDA-class, and Sec. 2 and App. T now say so. MHT keeps separate histories and can revise them; we keep neither. Beyond the learned projection, two things differ from kinematic PDA. The state is filtered over camera-network events (next camera, arrival time, entry region) and keeps explicit route, velocity and presence statistics next to the learned vector. And the no-observation update is availability-aware (Eq. 9): an available camera that did not see the target counts against routes through it, while a masked stream gives no evidence either way.

We also compared against a depth-3 MHT on the same 1,200 chains, sharing features, candidates, priors, null definition and causal history (Table 39; 19 cameras, 64 identities, 412 gated pairs):

MethodIR@3Core time (ms)Core memory (GB)
MHT, 4 branches91.17% (1,094)31.64.8
MHT, 8 branches93.17% (1,118)47.26.1
MHT, 16 branches94.58% (1,135)73.98.7
ReWorld (one collapsed belief)93.83% (1,126)25.43.3

ReWorld lands between 8 and 16 branches at about half the time and memory of the 8-branch tracker. The 16-branch tracker is 0.75 points higher at 2.9x the time.

Thank you again; the route-transfer, scaling and MHT experiments exist because of your questions. If any answer is incomplete, we are glad to follow up. If the new results answer your concerns, we would be grateful if you would consider updating your rating.

Thanks for the detailed rebuttal and for adding the new experiments on route transfer, scaling, language, calibration, and MHT. I’ve gone through the response and the new results, and most of my original concerns have been addressed.

W1/Q1: The held-out edge experiment is helpful. On unseen routes, the full model improves HA by 4.00 points over one-way prediction, and the gap increases to 5.38 points when the timing prior is shifted. This suggests that the recurrent feedback remains useful when the event prior is inaccurate. The main limitation is that the camera graph is still available at test time and remains fixed within each sequence, so dynamic topology changes are not covered. Still, I think the new results largely address my concern about unseen routes and prior mismatch.

W2/Q2: The additional latency and memory results make the practical operating range much clearer. The main scaling factor appears to be the number of gated identity–candidate pairs rather than the length of the track history. On one RTX 4090, the full pipeline runs at about 10 Hz with 19 cameras, and drops to roughly 9 Hz and 7 Hz with 24 and 32 cameras. This is a useful characterization of the system, and I recommend stating this deployment limit clearly in the paper.

W3/Q3: The new ablations make the roles of recursion, event priors, and language much clearer. Language has only a modest effect on the overall results, but it provides a much larger gain on appearance-ambiguous cases: +10.9 points compared with +2.1 points on easy cases. The drop under misleading descriptions is also an important limitation to report. Overall, the revised framing that the recurrent update is modality-agnostic and that language mainly helps with difficult disambiguation is reasonable.

W4: The chain-level results make the relatively small overall HOTA gain easier to interpret. On ambiguous three-handoff chains, identity retention improves from 76.67% to 86.67%, with 59 chains fixed and 14 chains broken. Given the very small additional computation and parameter cost, I think this provides a reasonable justification for the structured update in the cases where uncertainty matters most.

Q4: The clarification about the train/validation/test split addresses my concern about calibration and possible test-set leakage. The reported calibration and commitment settings are selected on validation data and frozen before testing. This distinction should be kept clear in the final paper and the table captions.

Q5: I appreciate the authors’ clarification that the candidate/null update is PDA-like rather than a fundamentally new Bayesian inference scheme. In my view, the main contribution is the event-level recurrent fusion, the availability-aware handling of missing observations, and the learned fixed-size collapse. The MHT comparison is useful: in terms of IR@3, ReWorld falls between the 8- and 16-branch MHT variants while using substantially less computation and memory. I would therefore avoid describing the method as a fundamentally new general association paradigm, and instead emphasize this event-level integration and its accuracy–efficiency trade-off.

Overall, the new experiments address most of my questions and make the paper stronger. I still have some reservations about the limited novelty beyond PDA-style inference and the fact that the method has only been tested with a static camera graph known at test time. These issues keep the paper in the borderline range for me.

I will keep my rating at 3 (Weak Accept), with confidence remaining at 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC

We thank the three reviewers. They agree on what the paper is for: an early commitment to the wrong candidate corrupts the history used for later handoffs, and carrying candidate and null uncertainty forward is a well-motivated answer to it (wZm6, kinP). They also found the controls rigorous (kinP), the analysis comprehensive (wZm6), and the treatment of waiting, camera unavailability and censored non-observation probabilistically careful (6DY1).

Four themes run across the reviews. We answer them here once; the individual replies give the details. A revised PDF is uploaded, with changes in blue and new results in App. V–X (Tables 31–39).

1. Is the structured update worth it for +0.50 HOTA over a generic learned updater? (6DY1 W1, wZm6 W4)

The aggregate is small for two reasons. All update variants share the same frozen detections, so the detection half of HOTA = √(DetA · AssA) barely moves. And the two updates differ only when a return is ambiguous, which most returns are not.

We therefore reran the comparison on 1,200 fixed three-handoff chains, split by a validation-defined ambiguity margin. On the 360 ambiguous chains, three-handoff identity retention rises from 76.67% to 86.67%. On the other 840 it rises from 95.83% to 96.90%. Chain by chain, the structured update fixes 59 chains and breaks 14 (exact McNemar test, p < 1e-6). It adds 0.02 ms and 0.02M parameters per update.

2. What is new relative to PDA, JPDA, IPDA and MHT? (6DY1 W2, wZm6 Q5)

The recursion is classical, and the revised paper says so. What we add is a collapse whose content is learned from later forecast errors, for a state that predicts camera-network events (next camera, arrival time, entry region) under availability-aware censoring. Two comparisons show that this matters.

  • A PDA-style fixed-moment collapse with the same predictor retains 89.08% of identities after three handoffs. Ours retains 93.83%.
  • A depth-3 MHT sharing our features, candidates, priors and null definition reaches 93.17% with 8 branches and 94.58% with 16, at 1.9x and 2.9x our core time. One collapsed belief lands between the two, at about half the time and memory of the 8-branch tracker.

3. How much does the method depend on the camera graph, and how does it scale? (wZm6 W1–W2/Q1–Q2, kinP W1/Q1)

It uses the graph, and a wrong graph costs accuracy. We now measure how much. Handoff accuracy falls from 84.10% to 75.10% with 20% of edges missing, to 81.70% with 20% spurious edges, and to 80.10% with no graph at all. In all five corrupted conditions, posterior feedback gains more over one-way prediction than it does with the correct graph (3.20 to 5.20 points, against 1.70). On directed routes never seen in training, handoff accuracy is 77.25%, 4.00 points above one-way prediction.

On scale: core time grows roughly linearly with the number of gated identity–candidate pairs. On one GPU the full pipeline holds 10 Hz at 19 cameras and no longer at 24, and the paper now states this.

4. Language, commitment settings and test-set hygiene (wZm6 W3/Q3–Q4, kinP W1–W2)

The recursive state does not need language: without the query the model reaches 44.10 HOTA on MTMMC, level with the query-guided CRTracker baseline (44.12). Language earns its place on visually ambiguous returns (+10.9 HA points, against +2.1 on easy ones). Every checkpoint, calibration temperature and commitment setting is selected on validation data and frozen before testing. Selected independently on CityFlowV2 and MTMMC, the commitment settings come out identical.

What we corrected. Reviewer 6DY1 pointed out that the abstract and Sec. 3.4 overstated what is retained. The state is a per-step summary; there is no association history and no retrodiction. The reviewer is right, and both passages are rewritten.

Limits the paper states. Two are new in the revision: the 10 Hz limit above, and a 16-branch MHT that scores 0.75 points higher at about three times the core time. One was already in the submission: a misleading description lowers accuracy below the no-query model.

Revision summary

ChangeWherePrompted by
Scope of retained uncertainty; contribution narrowed; IPDA and Gaussian-sum filtering discussedAbstract, Secs. 2 and 3.4, App. T6DY1, wZm6
Write conditions of appearance memory vs the beliefSecs. 3.3–3.46DY1
Roadmap of evaluation populations and configurationsApp. L.16DY1
Feedback interventions, ambiguity split, paired outcomes, forecastsApp. V, Tables 31–346DY1, wZm6
Graph perturbations; margin and confirmation sweepApp. W, Tables 35–36kinP, wZm6
Held-out routes and timing shift; scaling to 24 and 32 cameras; MHT comparisonApp. X, Tables 37–39wZm6, 6DY1
AuthorsOpenReview ↗

ReWorld-Track: Summary of the Discussion and Revisions

Dear Area Chair and Reviewers,

We thank the Area Chair for guiding the discussion throughout the review. We are grateful to all three reviewers for reading the paper closely and taking the time to consider our responses. Your questions helped us explain what the tracker retains after an uncertain match and test when that information matters.

ReWorld-Track can defer a match and still use the current observation to predict the target’s next appearance. At each camera handoff, the tracker weighs the candidate matches alongside the possibility of no match. This weighting builds on classical probabilistic data association. Our contribution is a learned update that uses these weights to form a fixed-size predictive state. During training, errors in later forecasts and identity decisions teach the update which information to retain. The state predicts the next camera, arrival time, and entry region.

Table 32 shows where this update helps most. On 360 ambiguous MTMMC chains, ReWorld-Track preserved the target’s identity through all three handoffs on 312 chains (86.67%), compared with 276 (76.67%) for a similarly sized generic updater. On the other 840 chains, the gain was 1.07 percentage points. Of the 45 net additional successful chains, 36 came from the ambiguous subset. This result complements the 0.50-point HOTA gain on complete trajectories by showing where the two updaters differ most.

Reviewer 6DY1, thank you for highlighting the distinction between an offline camera and an available camera that does not see the target. Your question about which uncertainty survives a handoff led us to revise the abstract and Section 3.4. We now explain how the current posterior informs later forecasts and when a confirmed match updates appearance memory. We also added comparisons with fixed-moment and multi-hypothesis trackers. We appreciate the time you took to reassess our work and thank you for raising your recommendation.

Reviewer kinP, you highlighted two useful controls in the paper. One compares update rules with the same predictor. The other measures false matches at equal recovery coverage. Your questions about deployment led us to quantify the effects of missing camera links, incorrect links, and misleading descriptions. We also examined the sensitivity of the commitment settings and their transfer across datasets. Thank you for these suggestions and for confirming that our responses addressed your concerns.

Reviewer wZm6, you identified the failure that motivated this work: one wrong match can affect several later handoffs. Thank you for examining the additional results on unseen routes, scaling, language, calibration, and tracking baselines. Your follow-up led to two further experiments, reported in Appendices Y and Z. One compared update rules under single- and multi-handoff supervision. The other tested route openings and closures during continuous tracking and measured the effect of delays in receiving an updated camera graph. We appreciate the time you devoted to both rounds of discussion.

The revised manuscript is available on OpenReview. Blue text marks revisions prompted by the initial reviews. Red text marks revisions prompted by the follow-up with Reviewer wZm6. If accepted, we will fully integrate these revisions into the camera-ready paper. We will give the ambiguous-handoff results greater prominence in the main text and explain the state update and the camera-graph assumptions alongside them.

We thank the reviewers for the questions that led to these experiments. We are grateful to the Area Chair for guiding this exchange and for the time and attention you have given our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.