- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies persistent event verification under intermittent UAV revisits, e.g., whether a parking or roadway region remains occupied across passes. It introduces a benchmark of 11 sites that organizes observations around fixed semantic map regions and distinguishes confirmed absence from missing or unreliable evidence. The proposed U²-Revisit fuses visual, telemetry, and map features through quality-aware masked fusion, and maintains separate spatial and instance memories with time decay. Under a seven-fold LOSO protocol, it achieves an Event AP of 0.9594.
Critical strengths
- The task is practically relevant, and explicitly separating confirmed absence from missing or unreliable observations is a reasonable design for intermittent sensing.
- The benchmark uses a site-disjoint LOSO protocol and reports results over multiple seeds.
Critical weaknesses
- Insufficient and unconvincing experiments. All baselines are self-constructed variants; no published methods for temporal or irregularly sampled sequence modeling are compared, including those discussed in the related work. Moreover, Unordered Set, which ignores revisit order and time, outperforms U²-Revisit on AUROC, F1, and BAcc, which contradicts the claimed benefit of ordered temporal modeling. The AP gains over the baselines and in the ablations are within the standard deviations over three seeds, so the improvements are not significant.
- Core claim is not validated and key results are missing. Distinguishing missing evidence from confirmed absence is presented as the key challenge, but no experiment evaluates it, e.g., robustness on sequences with many unavailable observations. The external-site evaluation on R10/R11, the auxiliary task results, and a description of the "Statistics" baseline (whose AUROC of 0.42 is below chance) are not provided.
- Unclear writing and task definition. In particular, the second paragraph of the introduction abruptly describes a "lightweight proposal stage" before the task, the challenges, or the model have been introduced, making it unclear what is being proposed and why. More generally, the introduction lists several difficulties without explaining how each is addressed, and the term "persistent event" is not defined until Section 3.3, where it is still unclear what condition makes an occupancy "persistent."
Questions for authors
- How does U²-Revisit compare with published methods for irregular or missing-data sequence modeling? Why does Unordered Set outperform U²-Revisit on AUROC, F1, and BAcc?
- Are the update mask and quality descriptors computed automatically, or derived from the annotated evidence states?
- What is the "Statistics" baseline, and why is its AUROC below 0.5?
4 comments
Response to Reviewer ygiJ (2)
CW2: Missing-Evidence Robustness
Concern: The paper emphasizes distinguishing unavailable evidence from confirmed absence but originally lacks direct experimental validation.
Response: We have added a controlled synthetic missing-evidence experiment using frozen trained models.
At selected revisits, dynamic evidence and associated quality/availability inputs are masked. We compare two strategies under identical missing positions and original time intervals:
- Skip Update: Preserve elapsed-time decay without incorporating unavailable observations into recurrent memory.
- Forced Update: Perform recurrent updates at the same masked positions despite the absence of new dynamic evidence.
Additional results (Appendix D.8):
| Missingness | Skip Update AP | Forced Update AP | AP |
|---|---|---|---|
| 10% | 0.9561 | 0.9522 | +0.0039 |
| 30% | 0.9537 | 0.9336 | +0.0200 |
| 50% | 0.9489 | 0.8926 | +0.0563 |
The advantage of skipping updates increases with missingness, reaching +0.0563 Event AP at 50% missingness.
Both strategies retain elapsed-time decay, so this experiment evaluates the recurrent update policy rather than isolating the effect of time decay. Complementarily, the ablation in Table 3 shows that removing time decay decreases mean Event AP from 0.9594 to 0.9521.
We also evaluate 95 sequences containing naturally occurring detector-missing observations, identified from automatic high-recall records. U$^2$-Revisit achieves Event AP 0.7700, compared with 0.7563 for Unordered Set.
These experiments provide complementary evidence for selective state updates, elapsed-time modeling, and performance under incomplete observations.
Manuscript Revision: Sections 5.2–5.4, Table 3, and Appendix D.8, which now includes a quantitative table and an Event AP trend figure.
Q2: Quality Descriptors and Update-Mask Sources
Question: Are the update mask and quality descriptors computed automatically or derived from annotated evidence states?
Response: Both the quality descriptors and temporal update masks are generated through automatic preprocessing, without using manually annotated persistent-event or vehicle-identity labels as model inputs.
The five quality descriptors capture detection confidence, visual-feature availability, image-to-map registration reliability, GPS-field availability, and image sharpness estimated using Laplacian variance. These signals support quality-aware feature fusion.
The temporal update mask is derived from automatically generated observation-state information and controls whether a revisit triggers a recurrent state update. When an update is suppressed, elapsed-time memory decay remains active.
Importantly, manual event and identity annotations are used only as supervision targets, rather than as inputs to the quality-aware fusion or temporal update mechanism. We have also clarified the provenance of the automatically generated quality features and update masks in the implementation details.
Manuscript Revision: Sections 3.3 and 5.2–5.3.
Response to Reviewer ygiJ (3)
CW2: External-Site Evaluation and Auxiliary Tasks
Concern: The external-site evaluation and auxiliary-task results are insufficiently documented.
Response: Appendix D.5 reports external-site evaluation. Models trained on the seven development sites are evaluated on held-out R10 and R11, comprising 211 labeled candidates.
| Method | Event AP | AUROC |
|---|---|---|
| Unordered Set | 0.963 | 0.789 |
| Single Memory | 0.972 | 0.812 |
| U$^2$-Revisit | 0.973 | 0.817 |
U$^2$-Revisit achieves the highest pooled Event AP and AUROC among the original baselines on these sites.
We evaluated the frozen auxiliary heads at positions with valid supervision. The occupancy and revisit-level persistence heads achieve AP values of 0.9948 and 0.9173, respectively.
Identity-related supervision is considerably sparser: the sample-level identity evaluation contains only three replacement-positive sequences, limiting conclusions about replacement recognition. We report the auxiliary results separately from the primary candidate-level Event AP.
Q3: Statistics Baseline and AUROC Below 0.5
Question: What is the Statistics baseline, and why does it achieve AUROC below 0.5?
Response: Statistics is a supervised logistic-regression baseline using nine handcrafted sequence-level features summarizing vehicle detections, detection continuity, empty and unavailable observations, and revisit intervals.
Features are standardized with StandardScaler, followed by class-balanced logistic regression trained independently on the six training sites of each LOSO fold.
Our additional analysis identifies cross-site score misranking as the main explanation for its low pooled AUROC.
| Evaluation | AUROC |
|---|---|
| Within-site pairs | 0.9639 |
| Cross-site pairs | 0.3061 |
| Site-macro | 0.8917 |
| Pooled | 0.4222 |
The classifier ranks examples relatively well within individual sites, but its scores are poorly aligned across sites. In particular, high-scoring negative candidates from R1 contribute substantially to incorrect cross-site rankings.
Thus, the pooled AUROC below 0.5 does not indicate reversed class labels; it reflects poor cross-site score comparability for this statistical baseline.
Manuscript Revision: We clarified the baseline definition and its pooled-AUROC interpretation in the experimental analysis.
CW3: Task Definition and Presentation
Concern: The introduction presents the proposal stage before defining the task, and the meaning of persistent events remains insufficiently clear.
Response: We have clarified the operational meaning of persistent occupancy in our region-centered setting.
A persistent event refers to an occupancy episode supported by temporally separated observations of the same fixed semantic region, rather than an isolated vehicle detection within a single revisit. The essential distinction is that persistence is assessed from evidence accumulated across revisits, with spatial correspondence established through the reference map.
For example, when a region is observed as occupied during multiple revisits, an intervening unavailable observation does not automatically invalidate the persistence hypothesis. In contrast, a confirmed-empty observation provides evidence of an occupancy interruption and must be distinguished from missing evidence.
Importantly, persistent occupancy is a region-level property, not a vehicle-identity claim. Different vehicles may occupy the same region across revisits, and the UAV cannot directly observe every transition during the intervening gaps. Accordingly, our labels represent evidence-supported persistent occupancy rather than guaranteed uninterrupted occupancy throughout unobserved intervals.
Operationally, rule-based proposals identify sequences with repeated vehicle observations, within-revisit stationary evidence, and spatial consistency. These criteria select candidates for manual review, not positive event labels. Final persistence labels are assigned through review of the available multi-revisit evidence, while vehicle-identity annotations are handled separately.
We have revised the task presentation to distinguish persistent occupancy, confirmed absence, unavailable evidence, and vehicle-identity continuity.
Manuscript Revision: Introduction, Section 3.3, and Appendices A.4–A.5.
Closing Remarks
We thank the reviewer again for the constructive suggestions. The revised manuscript now includes four published baseline comparisons, a parameter-matched architecture comparison, paired statistical analyses, controlled missing-evidence experiments, and clearer task and implementation descriptions.
We believe these additions substantially strengthen the empirical evaluation and clarify the contributions of U$^2$-Revisit. We welcome further discussion and would appreciate the reviewer's reconsideration in light of the additional evidence.
We hope that our responses have addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarifications and engage in further discussion. We sincerely appreciate your time and valuable feedback.
Response to Reviewer ygiJ (1)
We sincerely thank the reviewer for the detailed and constructive feedback, particularly the concerns regarding experimental validity, missing-evidence robustness, and task clarity. We have substantially expanded our evaluation and revised the manuscript to address these concerns.
CW1 & Q1: Published Baselines and Temporal Modeling
Concern: The original baselines are self-constructed, and the benefits of explicit temporal modeling are not sufficiently established.
Response: We have added four published methods for irregular or incomplete sequence modeling. All methods are evaluated on the same seven-site LOSO benchmark with 544 labeled candidates and three training seeds.
U²-Revisit achieves higher mean Event AP than all four task-adapted published methods. These methods use candidate-level supervision, whereas U²-Revisit additionally uses auxiliary multitask supervision; the comparison therefore evaluates complete task-adapted systems rather than isolating architectural differences.
Regarding Internal Baselines and Capacity Control
We report the strongest internal baselines separately, because they test different questions from the published-method comparison above.
Unordered Set. Multi-revisit aggregation itself provides substantial improvements over single-revisit prediction. U²-Revisit improves the primary Event AP metric by 0.0132 (0.9594 vs. 0.9462), while Unordered Set achieves slightly higher AUROC, F1, and balanced accuracy. We report this as a competitive comparison rather than a claim of superiority on every metric.
Parameter-matched GRU. To control for model capacity, we evaluated a single-memory GRU with 1,003,676 parameters, compared with 1,004,175 for U²-Revisit (a difference below 0.05%). The matched GRU obtains 0.9596 Event AP versus 0.9594 for U²-Revisit, while U²-Revisit achieves higher AUROC (0.9453 vs. 0.9440), F1 (0.9269 vs. 0.9175), and BAcc (0.9013 vs. 0.8846). This shows that simple aggregation is a strong capacity-matched baseline; we therefore position the dual-memory design as a task-motivated inductive bias rather than the source of the main Event-AP gain.
Paired bootstrap. The mean AP difference between U²-Revisit and Unordered Set is +0.0132, with a candidate-level 95% CI of [-0.0013, 0.0264] and a site-cluster 95% CI of [-0.0056, 0.0460]. The interval includes zero, which is why we frame the AP comparison as competitive evidence rather than a statistical superiority claim. The stronger evidence for the proposed design comes from the controlled missing-evidence experiments in CW2.
Manuscript Revision: Section 6.3, Table 4, and Appendices D.6–D.7.