acceptodds
Under review as a conference paper at ICLR 2027

EventFieldNet: Structured Event Scoring for Video Moment Retrieval

Abstract

Highly overlapping candidates pose a ranking challenge in video moment retrieval: truncated, overextended, and well-aligned intervals can share query-relevant content yet differ in event coverage and excess context. We propose EventFieldNet, which makes semantic and temporal ranking criteria explicit through three complementary fields that augment learned base span scores. Evidence aggregates learned clip-level relevance across the entire candidate interval; Support assesses event persistence within the interval relative to local context derived from the model's own predictions; and Transition separately evaluates directional entry and exit at the two boundaries. The fields receive role-specific supervision; in particular, Evidence can be constrained by reciprocal counterfactual video–query pairing built from moment annotations instead of by clip-level saliency GT, with the same retrieval accuracy. With the same InternVideo2 features as the strongest baseline and no additional pretraining, EventFieldNet improves average mAP on QVHighlights by 1.7 points, with larger gains at stricter IoU thresholds.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Moment Retrieval, Temporal Grounding, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aCyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the difficulty of ranking highly overlapping candidates in MR, where truncated, overextended, and well-aligned intervals share similar content but differ in temporal extent. The proposed EventFieldNet augments a base span score with three explicitly supervised scoring fields: Evidence for interval-wide semantic relevance, Support for event coverage relative to temporal context, and Transition for directional boundary alignment, each trained with counterfactual contrasts built from moment annotations. Experiments on three datasets show improvements, mainly at strict IoU thresholds.

Critical strengths

The experimental section covers a wide range of analyses, including retrained ablations, field-removal studies across IoU thresholds and moment lengths, and a comparison between counterfactual and saliency-based supervision.

Critical weaknesses
  1. The design of Support is overly complex relative to its contribution. It stacks several hand-crafted components, and its context correction relies on references selected from the model’s own provisional predictions, so errors in these predictions can propagate into the correction. However, removing Support at inference causes only a marginal drop, and the need to manually attenuate its ranking gradient suggests that its intended coverage role is not stably learned.

  2. The evaluation scope is narrow, and the results only partially support the claimed advantage. The main paper evaluates only on QVHighlights, where the improvements over the strongest baselines are inconsistent across metrics, and the additional results on other datasets are mixed. Moreover, the paper includes no MLLM-based baselines or long-video benchmarks, and does not report computational cost, which is a concern given that the method scores all grid candidates and may scale poorly to longer videos.

  3. The method relies on many hand-crafted designs and hyperparameters without corresponding ablations, including those in the Support construction, the ranking loss, and the score fusion. As a result, it is difficult to judge which design choices are essential and how robust the method is to these settings.

Questions for authors

Since the base path is a newly designed architecture rather than an existing detector, could the authors integrate the three fields into an established detector such as SG-DETR? This would help separate the contribution of the proposed fields from that of the base architecture, and would also support the claim that the approach generalizes to other temporal localization models.

2 comments
AuthorsOpenReview ↗

Response to Main Concerns

We thank the reviewer for the careful assessment and for recognizing the breadth of our analyses. Your suggestions helped us explain Support more clearly and examine computational cost, cross-dataset performance, and parameter stability. The revision addresses these three concerns below; all references use its updated numbering.

W1: Support's complexity comes from assessing event coverage using both internal relationships and external context; its components serve distinct roles.

Highly overlapping candidates can share semantic content yet cover different parts of an event. Evidence scores clip-level relevance, and Transition scores directional boundary alignment. Support additionally needs to detect omitted event content, which requires relating the candidate to surrounding content as well as examining its interior.

S₀ captures interactions and persistence between adjacent clips, while reference regions and outer rings provide coverage and contextual information. Up to eight references accommodate multiple plausible locations; the candidate–reference IoU threshold of 0.3 excludes insufficiently overlapping references from the correction. Coverage counterfactuals require truncations to lower the score and coverage-preserving expansions to remain neutral. The ranking-gradient coefficient of 0.1 balances aggregate ranking and coverage supervision without changing the forward score or attenuating Support's own auxiliary loss. Sections 3.2–3.4 and Appendix A.2.2 explain these roles.

Table 5(a) tests reference correction and gradient attenuation separately: removing either reduces validation average mAP from 57.46 to 56.60/56.75. Table 18 tests the learned behavior: under InternVideo2/CLIP+SlowFast, Support's truncation-ranking accuracy increases from 31%/31% without its loss to 55%/52%, and its expansion absolute score gap decreases from 0.12/0.11 to 0.04/0.04. These results address both component utility and the coverage objective; the smaller fixed-weight inference-removal effect does not by itself characterize Support's role during training.

Reference errors are examined directly. Table 16 reports positive average Support contributions of 1.50 ± 0.16 / 1.10 ± 0.20 mAP points in the low-quality-reference group. Recomputing references after E/T removal adds only 0.025–0.069 points to the frozen-reference drops in Table 17. This supports the average contribution conclusions while retaining reference quality as a limitation.

W2: The revised presentation makes cross-dataset competitiveness clearer while explicitly reporting computational cost and long-video limitations.

Table 2 now brings the original Charades-STA and TACoS results into the main text. In the supplementary matched-split Charades comparison, EventFieldNet and SG-DETR have similar R1, while average mAP is 47.64 versus 42.95 (Table 8); each method retains its native recipe. Table 10 adds published MLLM comparisons, with their different inputs and fine-tuning or joint-training settings identified.

The current implementation incurs additional cost. Table 20 reports batch-1 inference latency of 25.30 ms for EventFieldNet versus 15.50 ms for SG-DETR on the same H20 GPU with FP32/no-TF32, starting from pre-extracted features and including post-processing.

Dense grid enumeration limits scaling because candidate counts grow quadratically: 2,850 candidates at 75 clips and 720,600 at 1,200 clips, as explained in Appendix F.2. Table 21 reports 17.03 GiB peak allocated memory at 1,200 clips, with 127.87 ms for the forward pass and 371.73 ms for post-processing. This is a scaling measurement, not a long-video accuracy benchmark. Appendix F.2 discusses sparse candidates and hierarchical processing as future directions, and the conclusion acknowledges the efficiency limitation.

W3: Mean performance is stable across the tested loss weights, Support gradient coefficients, and counterfactual margins.

We jointly scale the three field-loss weights by ×0.5/×2, change Support's ranking-gradient coefficient to 0.03/0.3, and scale counterfactual margins by ×0.5/×2. Across these six settings, test average mAP changes by −0.230 to +0.031 points relative to the same three-seed default, whose mean is 55.679. Table 5(b) summarizes these results; Table 19 reports validation/test metrics and paired changes, and Figure 5 shows variability.

Appendix E also reports learned-scalar, MLP, and alternative Transition-weight fusion results of 55.66/55.56/55.65, versus 55.68 for the same default. Together with the component ablations, these results distinguish the roles of the designs from sensitivity to their numerical settings. The stability conclusion is limited to the tested ranges and does not assume arbitrary settings or parameter interactions are harmless.

AuthorsOpenReview ↗

Response to Specific Questions

All section references below use the revised manuscript's numbering.

Q1: The current fields are trained jointly with EventFieldNet's candidate-scoring mechanism; direct integration into SG-DETR does not reproduce their gains.

EventFieldNet jointly ranks dense, highly overlapping candidates, with the field scores, base score, and Support reference correction learned within the same pipeline. SG-DETR uses decoder-generated candidates, different confidence semantics, and its native objectives. Directly adding the scores therefore need not provide suitable calibration; candidate geometry and optimization coupling are other possible factors.

Following your suggestion, we retain SG-DETR's original losses, attach E/S/T to its encoded clip features, and add their scores to the decoded confidence. Appendix F.3 reports test average mAP of 54.11 for native SG-DETR versus 50.68 after integration. This result does not support a direct plug-in gain. The proposed explanations remain factors to investigate, rather than established causes of the decrease.

Appendix F.3 reports this negative result, and Section 5 discusses the next step: a more efficient, loosely coupled field-scoring mechanism adapted to the candidate representations, confidence scales, and training objectives of established temporal localization frameworks.

Reviewer_qWK2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes EventFieldNet, a method for video moment retrieval that augments a base span score with three separate fields to model semantic relevance (Evidence), event coverage and contextual support (Support), and boundary alignment (Transition). It introduces role-specific counterfactual supervision for these fields to improve the ranking of highly overlapping candidate intervals. On QVHighlights, using the same InternVideo2 features without additional pretraining, the method improves average mAP from 54.10 to 55.84 over the baseline. Ablation studies indicate that all three fields contribute, while counterfactual Evidence supervision achieves retrieval performance comparable to saliency-ground-truth supervision.

Critical strengths

1.The paper addresses a clearly defined and practically meaningful problem of ranking highly overlapping candidate intervals, including truncation, overextension, and one-sided boundary errors. It also introduces role-specific counterfactual supervision for the three scoring fields, providing targeted learning objectives for different ranking criteria.

2.The experimental evaluation is relatively rigorous, including multi-seed experiments, architectural and supervision loss ablations, and analyses across different IoU thresholds and moment lengths. These experiments provide evidence for the method's effectiveness and the complementary contributions of its components, strengthening the credibility of the results.

Critical weaknesses

1.The paper mainly integrates existing ideas in semantic matching, event completeness assessment, and boundary modeling into three separate scoring fields with corresponding counterfactual supervision. However, these modeling strategies and counterfactual supervision techniques have been extensively explored in prior work. The overall approach limits its novelty.

2.The main experiments and ablation analyses are concentrated on QVHighlights. Although additional results on other datasets are provided in the appendix, the improvements are inconsistent, leaving the generalization ability insufficiently demonstrated. Moreover,some metrics show negligible gains or even underperform existing methods.These relatively modest improvements call for stronger experimental evidence to establish the method's effectiveness and advantages.

Questions for authors

1.Compared with existing methods such as “ DE²TR: Dual Evidence Detection Transformer for Video Temporal Grounding,” European Conference on Computer Vision (ECCV),2026, what are the key novel aspects of the three scoring fields and their counterfactual supervision? Is there direct experimental evidence demonstrating their advantages over simpler score fusion or unified auxiliary supervision strategies?

2.The main experiments focus on QVHighlights, while the improvements on Charades-STA and TACoS are inconsistent. Could the authors explain these differences and provide stronger cross-dataset evidence to support the generalization ability of the proposed method?

4 comments
AuthorsOpenReview ↗

Response to Main Concerns

Thank you for your careful review and constructive suggestions. Your comments helped us clarify the design contribution and strengthen the cross-dataset evidence. In trying to save space, we placed the cross-dataset results in a compact appendix table and did not adequately explain their comparison settings or ranking performance. The revision brings the key results into Section 4.2 and Table 2, with supplementary comparisons in Tables 8–9. References below use the revised manuscript's numbering.

W1: Our contribution is an explicit field-based scoring mechanism coupled with error-specific counterfactual supervision.

Highly overlapping candidates can share similar semantic content while differing substantially in event coverage and boundary accuracy. EventFieldNet makes these distinctions explicit in the final candidate score: Evidence assesses interval-wide semantic relevance, Support assesses coverage relative to temporal context, and Transition assesses directional boundary alignment. Sections 2–3 clarify this coupling of separate scoring criteria and training targets, including the distinction from semantic and boundary evidence within moment queries.

The counterfactual objectives specify how each field should distinguish its corresponding errors: query/video swaps for semantic mismatches, truncation and coverage-preserving expansion for coverage, and endpoint shifts and reversals for boundary alignment. Table 18 directly tests these behaviors. Under InternVideo2/CLIP+SlowFast, Evidence AUC improves from 0.75/0.70 to 0.82/0.76, and Support's truncation-ranking accuracy improves from 31%/31% to 55%/52%. Transition's diagnostic gains are smaller, but removing its score reduces InternVideo2 validation average mAP by 1.51 points in Table 14. Together with the unified-head comparison in Table 5(a), these experiments make the proposed division of scoring roles testable.

W2: EventFieldNet is competitive on Charades-STA and TACoS, with its ranking advantage clearer in the supplementary mAP comparison.

Without additional task-level pretraining, our submitted model achieves Charades [email protected]/0.7 of 70.59/48.80, close to SG-DETR's 70.20/49.50. On TACoS, it achieves 45.81/30.63, exceeding SG-DETR's 44.70/29.90 on both metrics. Table 2 now presents these results in the main text and distinguishes feature and training settings. SG-DETR+PT uses additional task pretraining, SDST uses side-tuning, and FlashVTG's Charades result uses InternVideo2-6B features. We also retain the cases where another method has better R1 under comparable settings.

The fields target candidate ordering, whose benefits need not appear equally in Top-1 recall. In Table 8's additional Charades comparison, both methods use the same 11,203/1,201 training–holdout split and selection rule, while retaining their native training recipes. EventFieldNet and SG-DETR obtain similar [email protected]/0.7—68.54/49.01 versus 68.52/48.71—but average mAP is 47.64 versus 42.95, and [email protected] is 48.39 versus 42.86. Table 9 further shows MR-only gains of 1.49/2.08 average mAP points over removing the three field losses on Charades/TACoS. These results provide cross-dataset evidence for candidate ranking and field supervision without highlight annotations.

AuthorsOpenReview ↗

Response to Specific Questions

All table and section references below use the revised manuscript's numbering.

Q1: What distinguishes the fields from existing methods, and is there evidence beyond simple fusion or unified supervision?

The distinction is the explicit correspondence between candidate error types, final scoring terms, and counterfactual training targets. As clarified in Section 2, DE²TR separates semantic and boundary evidence within moment queries. EventFieldNet instead exposes interval-wide relevance, contextual coverage, and directional boundary alignment as separate candidate scores, each constrained by contrasts specifying its intended response. Sections 3.2–3.3 describe both the scoring mechanisms and their paired objectives.

Table 5(a) compares this decomposition with a unified width-192 head U trained using all three counterfactual families. The MR heads have 369,217 parameters for U versus 371,270 for E/S/T, a difference of 0.553%, with the architecture fixed before training. EventFieldNet and U achieve validation average mAP of 57.46/57.24 and test average mAP of 55.83/55.69. The improvement is modest. Appendix D.2 also specifies the preserved HD input and its gradient paths, so the parameter comparison is explicitly limited to the MR heads.

Appendix E separately tests score fusion: learned scalar weights, MLP fusion, and a Transition weight of 1 yield test average mAP of 55.66/55.56/55.65, versus 55.68 for the same three-seed default. More elaborate fusion therefore does not improve the mean result in these tests. The direct diagnostics in Table 18 provide complementary evidence about the learned roles: stronger improvements for Evidence and Support, and smaller positive improvements for Transition, whose retrieval contribution is also shown in Tables 14–15.

Q2: Why are the cross-dataset improvements mixed, and what stronger evidence supports generalization?

As discussed in W2, our compact R1-only appendix table did not fully present the method's cross-dataset ranking performance. Table 2 retains the submission results: Charades [email protected]/0.7 is 70.59/48.80 versus SG-DETR's 70.20/49.50, while TACoS is 45.81/30.63 versus 44.70/29.90. These support competitive localization without highlight annotations, with mixed Charades R1 rather than improvement on every metric.

The supplementary comparison in Table 8 directly examines ranked-list retrieval under the same Charades training–holdout split and selection rule. R1 remains close, while EventFieldNet improves average mAP from 42.95 to 47.64 and [email protected] from 42.86 to 48.39. Each method retains its native training recipe. Table 9 additionally compares EventFieldNet with removal of the three field losses: 47.64 versus 46.15 on Charades, and 35.50 versus 33.42 on TACoS, with HD disabled throughout.

These results are consistent with the method's focus on ordering overlapping candidates: better ranking across a prediction list need not produce the same gain in its first prediction. They strengthen the cross-dataset evidence without attributing all differences between datasets to a single cause. Section 4.2 now discusses this distinction, and Appendices B.2 and C.2 specify the protocols and separate the supplementary runs from the submission checkpoints.

Thank you for your detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. However, several important issues remain.

1. Limited novelty and insufficient justification for the proposed design.

The authors emphasize that the main contribution is the explicit coupling of candidate-scoring mechanisms with error-specific counterfactual supervision. However, the distinction from existing approaches remains insufficiently convincing.

More importantly, compared with a parameter-matched unified head trained using the same counterfactual samples, the proposed three-field design improves validation/test average mAP by only 0.22/0.14 points, which the authors also acknowledge as modest. While Support improves truncation-ranking accuracy over the base scorer by 3/2 percentage points under the two feature settings, its relatively complex design involves multiple components whose individual contributions are not fully justified. The diagnostic improvements of Transition are also limited.

Furthermore, since the main contribution is a scoring and supervision framework rather than a new detector, its applicability to existing architectures is important. However, integrating the proposed fields into SG-DETR actually decreases average mAP from 54.11 to 50.68, raising further concerns about its general applicability.

2. Insufficient analysis of performance differences and competitiveness.

The additional cross-dataset experiments provide useful evidence for the method's effectiveness and generalization, which I appreciate.

However, the explanation for the modest improvements and weaker performance on certain metrics remains rather general. The authors argue that their method primarily improves ranked-list quality (mAP) rather than Top-1 localization (R1). While reasonable, this does not fully explain the observed performance differences. Further analysis of dataset characteristics or representative failure cases would make this argument more convincing.

Additionally, the authors attribute the superior performance of some existing methods to additional pretraining (SG-DETR+PT), data augmentation (DE²TR+BAS), or different input features. However, without sufficiently controlled comparisons, these explanations remain speculative, and the relative advantages of the proposed method are still unclear.

Overall, while the additional experiments strengthen the evaluation, my main concerns regarding novelty, the necessity of the complex design, and the limited performance advantages remain unresolved. I therefore maintain my original rating.

Thank you for considering our additional experiments and explaining why you are maintaining your original rating. We would like to clarify two points:

  1. What the unified-head comparison measures. The parameter-matched unified head uses the same counterfactual examples designed for Evidence, Support, and Transition. It therefore retains the training signal from the three-field design. The 0.14-point gain in test mAP measures what explicit field-wise scoring adds on top of those shared examples. The design we propose pairs each scoring field with counterfactual examples that target its corresponding error.

  2. What the SG-DETR experiment tests. We included this experiment to test direct integration into another detector. The drop in mAP shows that this integration did not work in the tested setting. Our paper evaluates the proposed scoring and supervision design in EventFieldNet; it does not claim that the fields can be plugged into other detectors without adaptation.

Reviewer_3fVGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EventFieldNet addresses the ranking of overlapping temporal candidates in video moment retrieval. It adds three query-conditioned fields to a learned base span score: Evidence averages bounded clip relevance across an interval; Support combines adjacent-clip interactions with a context correction based on predicted reference intervals; and Transition scores directional entry and exit separately. Moment annotations generate field-specific training contrasts through video/query swaps, interval truncations and expansions, and endpoint shifts/reversals. The final scores rank grid-aligned candidates before temporal NMS. The main experiments use the same InternVideo2 feature release as SG-DETR and report a 1.74-point improvement in QVHighlights test average mAP, with additional experiments using CLIP+SlowFast and on Charades-STA and TACoS.

Critical strengths
  1. Using the exact same released InternVideo2 features as SG-DETR with no task pretraining, EventFieldNet improves QVHighlights test average mAP from 54.1 to 55.84 and [email protected] from 55.8 to 57.51 (Table 1, p. 8), with every single one of its five random seeds (55.05–56.26 in Table 7, p. 23) beating SG-DETR's published 54.1. The paper also clearly separates feature/pretraining regimes and reports the full landscape of published results in Table 9 rather than cherry-picking baselines.

  2. The ablations cleanly separate retraining without field losses (Table 2, p. 9, where dropping all three field losses lowers validation average mAP from 57.46 to 53.13) from removing individual field scores at inference (Tables 4–5, pp. 21–22, where removing Transition costs 4.49 mAP points at IoU=0.95 vs. 0.25 at IoU=0.5). Section 4.4 also explicitly notes that replacing L_E with a saliency loss on QVHighlights still leaves L_HD active, while Appendix B.5 and Table 6 test moment-only training without L_HD on Charades-STA and TACoS.

Critical weaknesses
  1. About Direct Field Diagnostics and Cascaded Error in Support. While EventFieldNet decomposes the ranking score into CC, EE, SS, and TT (Eq. 1), the paper only evaluates their utility indirectly via final retrieval mAP after loss removal (Table 2) or score subtraction (Tables 4–5), without reporting any direct field-level metrics to verify that each field actually measures its intended physical quantity (e.g., clip-level relevance AUC for ete_t, boundary recall/precision for TsT_s and TeT_e, or truncation-ranking accuracy for SS). Moreover, computing the Support correction ΔS(I)\Delta S(I) introduces an explicit cascaded dependency (Eqs. 4, 15–18): reference intervals RIR_I and outer rings OIO_I are selected via greedy NMS on the provisional score F0=C+E+S0+12(Ts+Te)F_0 = C + E + S_0 + \frac{1}{2}(T_s + T_e), and Transition itself depends on upstream adapter cues UsU_s and UeU_e (Eq. 5). When CC, EE, or TT makes early errors in F0F_0, corrupted reference intervals can misguide ΔS\Delta S and compound ranking failures—an effect masked in Tables 4–5 because Support references are kept frozen rather than recomputed when removing EE or TT (Appendix B.4). Please report direct diagnostic metrics for each field, measure the quality (IoU/recall) of F0F_0-selected reference intervals, and evaluate how reference errors propagate through ΔS\Delta S (including oracle-reference controls and recomputed references in Table 4) to better demonstrate the soundness and robustness of your proposed method.

  2. Across Tables 1, 6, and 9, EventFieldNet underperforms several existing methods on key metrics, especially strict Top-1 accuracy ([email protected]), and Section 4.2 only briefly notes that mAP rewards the full ranked list while R1 checks the top prediction, without thoroughly discussing the method's core competitiveness.

Specifically, with CLIP+SlowFast on QVHighlights (Table 1), EventFieldNet trails DualGround on 5 of 7 metrics (including [email protected]: 67.74 vs. 68.20, [email protected]: 50.45 vs. 51.72, and HD mAP: 40.20 vs. 41.15) as well as DE2TR [R2] (51.39 [email protected]) and Sim-DETR (50.91 [email protected]), while removing EE even hurts accuracy at IoU≥0.80\text{IoU} \ge 0.80 (Table 5). Even with InternVideo2 features, EventFieldNet's [email protected] and HD mAP on QVHighlights (56.57 / 43.41) remain slightly below DualGround (56.94 / 44.02) and SG-DETR (56.6 / 43.8); on Charades-STA (Table 6), its [email protected] (48.80) falls behind all listed InternVideo2 baselines—SDST* (52.6), FlashVTG (49.87), and SG-DETR (49.5)—and Table 9 shows still larger gaps to 2D boundary modeling (A2FA+F2SGD at 63.4 Avg mAP) and augmented/pretrained variants (DE2TR+BAS, CVA, SG-DETR+PT).

Please clarify the core practical advantage of EventFieldNet when Top-1 localization ([email protected]) lags behind these methods, and include matched-split/matched-feature comparisons with boundary-scoring baselines such as BAM-DETR [R1] and DE2TR [R2] (as well as field ablations without LHDL_{\text{HD}} on Charades-STA/TACoS or an additional benchmark like ActivityNet Captions [R3]).

References:

[R1] Pilhyeon Lee and Hyeran Byun. BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos. ECCV 2024. https://arxiv.org/abs/2312.00083

[R2] Yifan Zhang et al. DE2TR: Dual Evidence Detection Transformer for Video Temporal Grounding. ECCV 2026.

[R3] Ranjay Krishna et al. Dense-Captioning Events in Videos (ActivityNet Captions). ICCV 2017. https://arxiv.org/abs/1705.00754

Questions for authors
  1. In Table 2, what is the validation average mAP if Support uses only the local compatibility score S0 (without reference-interval correction), or if the 0.1x ranking-gradient attenuation on Support is set to 1.0?

  2. Can you provide field-loss or field-removal ablations on Charades-STA or TACoS (where L_HD is absent), and explain why removing Evidence slightly improves CLIP+SlowFast mAP and R1 at IoU >= 0.80 in Table 5?

  3. How does SG-DETR perform on Charades-STA when trained on the same 11,203-query split and selected on the 1,201-query validation holdout, and what are the parameter count, training time, and per-query inference latency of EventFieldNet compared with the base-only model and SG-DETR?

3 comments
AuthorsOpenReview ↗

Response to Main Concerns

We thank the reviewer for recognizing our feature-controlled comparisons and the distinction between retraining ablations and inference-time field removal. Your suggestions helped us directly examine field behavior and reference dependencies, and clarify the method's practical advantage. References below use the revised manuscript's numbering.

W1: The added diagnostics support the intended field behaviors, and accounting for reference changes preserves the average field-contribution conclusions. Transition also shows positive, though modest, diagnostic gains.

For the concern about whether the fields learn their intended behaviors, we compare EventFieldNet with variants removing only the corresponding loss while retaining the scoring structure. Under InternVideo2/CLIP+SlowFast, Evidence's clip-relevance AUC increases from 0.75/0.70 to 0.82/0.76. Support's truncation-ranking accuracy increases from 31%/31% to 55%/52%, exceeding the base score's 52%/50%. Following the expansion-neutral objective in Section 3.3, the mean absolute score gap between original and expanded intervals decreases from 0.12/0.11 to 0.04/0.04. Transition shows smaller positive boundary-diagnostic gains. These measurements are reported in Table 18 and Appendix D.5.

For the concern that inaccurate references may misguide Support, Table 16 reports reference-set GT recall at IoU ≥ 0.5 of 87.69%/84.35%. Among queries whose top-1 reference has IoU < 0.5, EventFieldNet outperforms Support-score removal by 1.50 ± 0.16 / 1.10 ± 0.20 mAP points. Oracle references with reconstructed outer rings further improve overall mAP by 0.96/0.86 points. Reference quality remains a limitation, but Support provides a positive average contribution within the low-quality-reference group.

For the concern that frozen references conceal cascaded effects, we recompute the provisional scores, references, Support, and final ranking after removing E/T. Appendix D.4 reports reference-set changes for 30%–40% of queries, while Table 17 shows an additional mAP decrease of only 0.025–0.069 points relative to frozen references. The average E/T removal effects therefore persist when reference changes are included. This supports the contribution analysis under these interventions, rather than robustness to arbitrary upstream errors.

W2: EventFieldNet improves ranked-list retrieval under the standard feature and training conditions, while maintaining competitive Top-1 localization.

In Table 1, EventFieldNet achieves the highest average mAP among the reported methods in both feature groups without additional task pretraining or augmentation. With InternVideo2, average mAP and [email protected] are 55.84/57.51, compared with SG-DETR's published 54.10/55.80; [email protected] is nearly identical at 56.57 versus 56.60. With CLIP+SlowFast, average mAP is 49.38, above DualGround, BAM-DETR, and DE²TR at 49.02/45.36/48.74. Our [email protected] is 50.45 versus the standard group's highest 51.72. The advantage is stronger ranked-list retrieval, not leadership on every metric.

This pattern is consistent with the design: highly overlapping candidates can share relevant content while differing in coverage and boundary alignment. Separate Evidence, Support, and Transition scores provide information for ordering these candidates beyond an aggregate span score. Average precision reflects the ranked list, whereas R1 evaluates only its first prediction. The larger gain at [email protected] is consistent with better ranking of well-aligned intervals; the field-specific evidence comes from the ablations and diagnostics, rather than from the metric difference alone.

Tables 1 and 7 retain the stronger published results obtained under different settings: SG-DETR+PT uses InterVid-MR task pretraining, DE²TR+BAS uses augmentation, and A²FA+F²SGD uses different visual inputs. These comparisons do not replace matched retraining of BAM-DETR and DE²TR, which we have not completed. Our reproduced Charades comparison and MR-only field-loss ablations are reported separately in Tables 8–9 and discussed below.

AuthorsOpenReview ↗

Response to Specific Questions

All table and section references below use the revised manuscript's numbering.

Q1: Do reference correction and ranking-gradient attenuation improve Support?

Both improve validation performance in the design ablations of Table 5(a). EventFieldNet achieves 57.46 average mAP, compared with 56.60 using S₀ alone and 56.75 with the ranking-gradient coefficient set to 1.0. The decreases of 0.86/0.71 points support the two design choices. These experiments differ from Table 3's loss-removal ablations: they specifically test reference correction and gradient allocation. Appendix D.2 explains the configurations.

Q2: Does field supervision help without highlight training, and why can removing Evidence help at strict IoU under CLIP+SlowFast?

Table 2 retains the submission's MR-only Charades-STA and TACoS results. The supplementary ablations in Table 9 directly test field supervision without HD: EventFieldNet achieves 47.64/35.50 average mAP, compared with 46.15/33.42 after removing the three field losses. The gains are 1.49/2.08 points. Both variants omit HD and retain the scoring fields; only the field losses are removed. These supplementary runs are separate from the submission checkpoints, with their protocols specified in Appendix B.2 and the comparison explained in Appendix C.2.

Evidence's content relevance need not resolve small boundary differences between highly overlapping intervals. It can therefore favor a slightly misaligned interval with similar relevant content. This is a possible explanation for Table 15: removing Evidence improves CLIP+SlowFast AP by 0.39/0.33 points at IoU 0.90/0.95, while lowering overall average mAP by 0.13 points. Table 18's improved clip-relevance AUC does not contradict this result: clip discrimination and precise interval ranking measure different properties. Appendix D.5 discusses this limitation without treating the explanation as a verified causal mechanism.

Q3: What happens under the same Charades split and selection rule, and what is the computational cost?

Table 8 reports SG-DETR retrained on the same 11,203-query training split and selected using the same 1,201-query holdout and selection rule. Its test [email protected]/[email protected] is 68.52/48.71, versus EventFieldNet's 68.54/49.01. Average mAP is 42.95 versus 47.64, and [email protected] is 42.86 versus 48.39. SG-DETR retains its native training recipe and 100-epoch budget; the split, features, and selection rule are matched, rather than the training budget.

Table 20 and Appendix F.1 report cost on the same H20 GPU in FP32/no-TF32, starting from pre-extracted features. EventFieldNet / w/o E/S/T / SG-DETR have 22.99M/22.99M/13.47M registered parameters and batch-1 inference latency of 25.30/13.64/15.50 ms, including post-processing. The ablation retains bypassed modules in its registered parameter count, so equal counts do not imply equal active computation. In a separate timing comparison with batch 64 and 24 epochs, training-stage time is 3,312/3,228/2,511 seconds. This timing protocol is distinct from SG-DETR's native training recipe.

Thanks for the additional experiments. The direct field diagnostics and the reference-recomputation ablation address my questions about field behavior and cascading effects. The Support ablations and the moment-only results on Charades-STA and TACoS also clarify the contribution of the design. On Charades, the comparison with matched features, data split, and checkpoint-selection rule shows a meaningful average mAP gain over SG-DETR (47.64 vs. 42.95).

My remaining concern is the trade-off between ranking gains and inference cost. Top-1 results are nearly tied on Charades and mixed on QVHighlights. In the reported timing from pre-extracted features, EventFieldNet takes 25.30 ms per query, compared with 15.50 ms for SG-DETR and 13.64 ms for the base-only ablation. Matched retraining of BAM-DETR and DE²TR remains open. Given these limitations, I am keeping my rating at 2 (weak rejection).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for their careful assessment and constructive suggestions. Your feedback has helped us clarify EventFieldNet's contribution, improve its presentation, and strengthen the supporting analyses. Building on your suggestions, we have submitted a revised manuscript, with changes highlighted in blue for ease of review. All table and section references in our responses refer to this revised manuscript.

Our contribution is the explicit coupling of candidate-scoring mechanisms with role-specific counterfactual supervision. EventFieldNet assigns interval-wide semantic relevance, contextual coverage, and directional boundary alignment to separate fields that jointly rank each candidate. Query/video swaps, coverage removal, coverage-preserving expansion, and boundary perturbations specify how each score should respond to its corresponding error. Sections 2–3 clarify this distinction from aggregate quality scoring and from semantic or boundary modeling within a detector.

Our original presentation understated the cross-dataset evidence and contributed to concerns about generalization. To save space, we placed these results in a compact appendix table containing only R1 metrics. The revision brings the Charades-STA and TACoS results into the main text in Table 2 and Section 4.2. The supplementary ranking comparison and MR-only field-loss ablations appear in Tables 8–9. We distinguish published results from reproduced comparisons and identify differences in features, additional training, and checkpoint selection. This makes the method's candidate-ranking advantages and its mixed Top-1 results clearer.

The added experiments directly address the main design and applicability questions. Table 5 examines Support's components and a parameter-matched unified head; Tables 16–18 examine reference quality, reference dependencies, and field behavior. Table 19 and Appendix E report sensitivity and score-fusion comparisons. Tables 20–21 report computational cost and long-sequence scaling, while Appendix F.3 reports the unsuccessful direct integration into SG-DETR. These results clarify both the strengths of the current design and the adaptation it still requires.

Responding to the reviews has helped us make the contribution and scope of our work more precise. We welcome further discussion and would be happy to address any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.