- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CueRator searches for symbolic decision rules that adapt frozen multimodal encoders to audio-visual event perception. An offline loop comprises Plan, Design, Oracle, and Policy Agents. Candidate formulations produce modality-specific thresholds, while a neural policy predicts formulation parameters for each video. Candidates receive feedback from per-video oracle parameter search and validation performance after policy training. At inference, only the policy and discovered formulation are required. Experiments cover OV-AVEBench, two additional audio-visual tasks, cross-encoder transfer, and several search ablations.
Critical strengths
(1) The separation between a symbolic formulation and video-specific parameter prediction is a useful design. It restricts adaptation to an inspectable decision scaffold while preserving input-dependent behavior. The explicit formulation in Appendix F makes the resulting predictor substantially easier to inspect than an unrestricted learned prediction head.
(2) The paper includes meaningful controls and substantial supplementary analysis. In particular, AV2A-Policy in Table 1 tests whether adding the same policy architecture to an existing rule accounts for the improvement. The search-session statistics, constraint ablations, and transfer experiments also provide useful evidence beyond a single selected result.
Critical weaknesses
(1) The generalization claims are stronger than what the supervision and selection protocols establish.
Section 4.1 states that validation contains both seen and unseen categories. Sections 3.3 and 3.6 describe repeated use of validation metrics and diagnostic reports to revise formulations and parameter ranges, while Algorithm 1 selects the final formulation–policy pair by validation performance. Consequently, the categories called “unseen” are excluded from policy-training examples but are represented in the labeled feedback that shapes the decision rule itself.
Held-out test videos support sample-level evaluation. Establishing that symbolic structure enables generalization to categories unobserved during discovery additionally requires categories excluded from the entire formulation-search process. The current experiments do not separate this stronger form of category generalization from adaptation through repeated target-category validation feedback. This distinction is particularly consequential because Section 4.2 attributes the improved unseen performance and reduced seen–unseen gap to the discovered formulation.
The comparisons also involve unequal validation access. Appendix B tunes global AV2A parameters on a 200-video subset, whereas CueRator repeatedly uses validation feedback for candidate design, checkpoint selection, and final selection. Following a baseline’s original tuning protocol is reasonable, but it does not isolate the benefit of the formulation from the benefit of broader adaptive validation access. AV2A-Policy is a valuable control, although it does not receive an analogous search over its rule structure and parameter ranges.
The adjacent-task experiments introduce further supervision differences. Appendix C fits the OV-AVVP pseudo-label formulation by gradient descent using segment-level validation labels, in addition to fitting a confidence filter. Table 3 compares this system against a published training-free AV2A result. This demonstrates the performance of the complete adapted pipeline, but does not establish how much improvement comes from the proposed discovery method versus the additional learned use of temporal annotations.
These issues affect the interpretation of the paper’s central generalization claim and the comparability of its supporting results.
(2) The ablations do not isolate the distinctive contribution of report-guided multi-agent search.
In Section 4.3, the score-only baseline simultaneously replaces the role-decomposed system with a single agent and removes the diagnostic reports. Its degradation therefore combines changes in information content, reasoning organization, and potentially inference expenditure. Table 2 establishes that the complete system outperforms this restricted alternative, but does not establish whether multiple specialized agents are necessary or whether one agent receiving the same reports could achieve comparable results.
Likewise, removing the Oracle or Policy Agent removes substantive diagnostic information along with the corresponding role. These experiments support the usefulness of the complete feedback package, but provide weaker evidence for the particular agent decomposition. Appendix D reports the full system’s computational cost without a corresponding cost breakdown for these alternatives. Equal iteration counts alone do not ensure comparable LLM inference or analysis budgets.
A related issue concerns the interpretation of “learnability.” Sections 3.2 and 4.3 compare oracle performance on training videos with policy performance on validation videos. This gap combines parameter predictability with distribution differences, policy optimization, and generalization error. The oracle itself is estimated through finite Sobol search. Consequently, this gap is not an isolated measurement of whether a formulation’s optimal parameters are learnable from embeddings, although it is repeatedly used to motivate and interpret the search.
The manuscript also lacks a comparison against a single agent with identical diagnostic access or a conventional symbolic-search method using the same formulation constraints and policy evaluation budget. Given that the principal methodological contribution is the discovery procedure, rather than merely the final threshold rule, these missing controls leave its incremental value insufficiently established.
Questions for authors
(1) Which labeled validation information about unseen categories is available to each agent, including aggregate metrics, per-category analyses, and per-sample diagnostics? Does the unseen-category improvement persist when all formulation discovery, range revision, and checkpoint selection use only seen-category validation data?
(2) How sensitive are the comparisons in Table 1 to validation access? What happens when AV2A and other decision-rule baselines receive the same validation pool and a comparable tuning budget, or when CueRator is restricted to the same 200-video validation subset?
(3) Does a single agent receiving the same oracle results, policy diagnostics, reports, and memory achieve comparable performance under matched LLM inference and policy-training budgets? Please clarify whether the search ablations in Table 2 match these budgets or only iteration counts.
(4) Can oracle and policy performance be evaluated on the same held-out videos, with oracle labels used strictly for post hoc diagnosis? How much of the reported oracle–policy gap remains after controlling for the different data splits and finite oracle-search budget?
(5) For OV-AVVP, what portion of the improvement in Table 3 comes from fitting the pseudo-label formulation with segment-level validation annotations and from the confidence filter? Does the advantage persist when the baseline receives equivalent annotation access and auxiliary calibration?
(6) Appendix G retains task-specific auxiliary formulations discovered on the target task when evaluating cross-task transfer. How much of the transferred system’s performance depends on these auxiliary discoveries? Please distinguish transfer of the threshold formulation from transfer of the complete decision pipeline.