acceptodds
Withdrawn from ICLR 2027

CueRator: Agentic Search for Symbolic Rules to Adapt Frozen Multimodal Encoders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents have been used to search over symbolic structures such as programs and equations. We propose CueRator, an agentic framework for policy-aware decision-rule discovery, which adapts frozen contrastive multimodal encoders by searching for the decision rule that converts their cross-modal similarities into predictions. We validate it on open-vocabulary audio-visual event perception, where existing methods involve a trade-off between adaptivity and generalization to unseen categories: trained modules adapt at the cost of generalization, and fixed rules the reverse. The framework pairs a symbolic formulation for generalization with a lightweight policy that predicts its parameters per video for adaptivity. A report-guided multi-agent loop discovers the formulation offline, evaluating each candidate on its expressive ceiling and on whether a trained policy can realize it. On OV-AVEBench, CueRator raises the total average from 57.8 to 60.2 and unseen-category performance from 55.8 to 59.9 over the best existing method, reducing the seen-unseen gap from 7.1 to 1.2. Ablations attribute the gains to both the formulation and the policy and show that both feedback signals are necessary for effective search. CueRator also improves over the respective baselines on two further audio-visual event perception tasks, and the discovered rule remains competitive across encoders with only the policy retrained.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
symbolic rule discovery, LLM-guided program search, agentic search, open-vocabulary learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vueAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CueRator searches for symbolic decision rules that adapt frozen multimodal encoders to audio-visual event perception. An offline loop comprises Plan, Design, Oracle, and Policy Agents. Candidate formulations produce modality-specific thresholds, while a neural policy predicts formulation parameters for each video. Candidates receive feedback from per-video oracle parameter search and validation performance after policy training. At inference, only the policy and discovered formulation are required. Experiments cover OV-AVEBench, two additional audio-visual tasks, cross-encoder transfer, and several search ablations.

Critical strengths

(1) The separation between a symbolic formulation and video-specific parameter prediction is a useful design. It restricts adaptation to an inspectable decision scaffold while preserving input-dependent behavior. The explicit formulation in Appendix F makes the resulting predictor substantially easier to inspect than an unrestricted learned prediction head.

(2) The paper includes meaningful controls and substantial supplementary analysis. In particular, AV2A-Policy in Table 1 tests whether adding the same policy architecture to an existing rule accounts for the improvement. The search-session statistics, constraint ablations, and transfer experiments also provide useful evidence beyond a single selected result.

Critical weaknesses

(1) The generalization claims are stronger than what the supervision and selection protocols establish.

Section 4.1 states that validation contains both seen and unseen categories. Sections 3.3 and 3.6 describe repeated use of validation metrics and diagnostic reports to revise formulations and parameter ranges, while Algorithm 1 selects the final formulation–policy pair by validation performance. Consequently, the categories called “unseen” are excluded from policy-training examples but are represented in the labeled feedback that shapes the decision rule itself.

Held-out test videos support sample-level evaluation. Establishing that symbolic structure enables generalization to categories unobserved during discovery additionally requires categories excluded from the entire formulation-search process. The current experiments do not separate this stronger form of category generalization from adaptation through repeated target-category validation feedback. This distinction is particularly consequential because Section 4.2 attributes the improved unseen performance and reduced seen–unseen gap to the discovered formulation.

The comparisons also involve unequal validation access. Appendix B tunes global AV2A parameters on a 200-video subset, whereas CueRator repeatedly uses validation feedback for candidate design, checkpoint selection, and final selection. Following a baseline’s original tuning protocol is reasonable, but it does not isolate the benefit of the formulation from the benefit of broader adaptive validation access. AV2A-Policy is a valuable control, although it does not receive an analogous search over its rule structure and parameter ranges.

The adjacent-task experiments introduce further supervision differences. Appendix C fits the OV-AVVP pseudo-label formulation by gradient descent using segment-level validation labels, in addition to fitting a confidence filter. Table 3 compares this system against a published training-free AV2A result. This demonstrates the performance of the complete adapted pipeline, but does not establish how much improvement comes from the proposed discovery method versus the additional learned use of temporal annotations.

These issues affect the interpretation of the paper’s central generalization claim and the comparability of its supporting results.

(2) The ablations do not isolate the distinctive contribution of report-guided multi-agent search.

In Section 4.3, the score-only baseline simultaneously replaces the role-decomposed system with a single agent and removes the diagnostic reports. Its degradation therefore combines changes in information content, reasoning organization, and potentially inference expenditure. Table 2 establishes that the complete system outperforms this restricted alternative, but does not establish whether multiple specialized agents are necessary or whether one agent receiving the same reports could achieve comparable results.

Likewise, removing the Oracle or Policy Agent removes substantive diagnostic information along with the corresponding role. These experiments support the usefulness of the complete feedback package, but provide weaker evidence for the particular agent decomposition. Appendix D reports the full system’s computational cost without a corresponding cost breakdown for these alternatives. Equal iteration counts alone do not ensure comparable LLM inference or analysis budgets.

A related issue concerns the interpretation of “learnability.” Sections 3.2 and 4.3 compare oracle performance on training videos with policy performance on validation videos. This gap combines parameter predictability with distribution differences, policy optimization, and generalization error. The oracle itself is estimated through finite Sobol search. Consequently, this gap is not an isolated measurement of whether a formulation’s optimal parameters are learnable from embeddings, although it is repeatedly used to motivate and interpret the search.

The manuscript also lacks a comparison against a single agent with identical diagnostic access or a conventional symbolic-search method using the same formulation constraints and policy evaluation budget. Given that the principal methodological contribution is the discovery procedure, rather than merely the final threshold rule, these missing controls leave its incremental value insufficiently established.

Questions for authors

(1) Which labeled validation information about unseen categories is available to each agent, including aggregate metrics, per-category analyses, and per-sample diagnostics? Does the unseen-category improvement persist when all formulation discovery, range revision, and checkpoint selection use only seen-category validation data?

(2) How sensitive are the comparisons in Table 1 to validation access? What happens when AV2A and other decision-rule baselines receive the same validation pool and a comparable tuning budget, or when CueRator is restricted to the same 200-video validation subset?

(3) Does a single agent receiving the same oracle results, policy diagnostics, reports, and memory achieve comparable performance under matched LLM inference and policy-training budgets? Please clarify whether the search ablations in Table 2 match these budgets or only iteration counts.

(4) Can oracle and policy performance be evaluated on the same held-out videos, with oracle labels used strictly for post hoc diagnosis? How much of the reported oracle–policy gap remains after controlling for the different data splits and finite oracle-search budget?

(5) For OV-AVVP, what portion of the improvement in Table 3 comes from fitting the pseudo-label formulation with segment-level validation annotations and from the confidence filter? Does the advantage persist when the baseline receives equivalent annotation access and auxiliary calibration?

(6) Appendix G retains task-specific auxiliary formulations discovered on the target task when evaluating cross-task transfer. How much of the transferred system’s performance depends on these auxiliary discoveries? Please distinguish transfer of the threshold formulation from transfer of the complete decision pipeline.

Reviewer_UhunOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

CueRator is a report‑guided multi‑agent search framework for adapting fully frozen contrastive multimodal encoders, targeting open‑vocabulary audio‑visual event localization (OV‑AVEL). CueRator improves overall performance and substantially narrows the performance gap between seen and unseen categories.

Critical strengths
  1. Resolves the classic adaptability‑generalization trade‑off. The approach does not fine‑tune large multimodal encoders. It combines generalization from symbolic formulations and per‑video adaptability from the policy network, reducing the seen‑unseen performance gap from 7.1 to 1.2 and achieving state‑of‑the‑art results for open‑vocabulary audio‑visual event localization.
  2. Dual‑signal multi‑agent search paradigm. It leverages two complementary feedback signals: the oracle expressive ceiling and policy‑driven learnability. This filters out theoretically powerful formulations whose parameters cannot be practically learned by the policy network, outperforming conventional LLM‑guided symbolic search that relies merely on scalar evaluation scores.
  3. Produces interpretable closed‑form symbolic formulations instead of black‑box modules. The discovered rule can be reused across encoders. Only lightweight policy retraining is required for encoder replacement, avoiding costly full‑cycle search and lowering adaptation overhead.
  4. Efficient inference. No LLM calls are needed during inference. Only forward passes of the small‑size policy network plus deterministic threshold computation are performed, making the method applicable to multiple related audio‑visual tasks.
  5. Comprehensive ablation studies verify the necessity of each component. Experiments validate the importance of the four‑agent loop, oracle‑policy dual feedback, and per‑video dynamic parameters. Controlled comparisons disentangle performance gains contributed respectively by the discovered formulation and the policy network.
Critical weaknesses
  1. Restricted observation scope for agents. Agents only have access to aggregated statistical reports, metrics and ablation outputs, and “cannot inspect raw audio‑video samples directly”. The search tends to prioritize failure patterns captured by statistics, while fine‑grained sample‑level defects may remain undetected.
  2. Constraints on symbolic formulation interfaces. The Design Agent is bounded by strict AST (Abstract Syntax Tree) constraints, prohibiting loops and conditional branches. Current symbolic formulas are built purely upon “cross‑modal similarity features”. The existing interface lacks support for more sophisticated multimodal interaction logic.
  3. Limited cross‑task transferability. The fixed discovered formulation F∗F^* transfers well to structurally similar tasks such as OV‑DAVEL. Performance degrades when directly applied to tasks with different output structures or supervision paradigms (e.g., weakly‑supervised OV‑AVVP), where full agent‑driven rediscovery is still required.
  4. High computational cost for offline search. A complete search session takes around 9.8 hours on a single RTX 3090. The overhead comes from LLM invocation, oracle grid‑search, and repeated policy training. Although inference is fast, “the offline rule‑discovery phase is computationally expensive”.
  5. Heavy dependency on the quality of frozen backbone encoders. The whole pipeline builds on similarity outputs from pre‑trained encoders. Noise and defects originating from encoders propagate directly to final predictions, as illustrated by typical failure cases in the paper.
  6. Sensitivity to the underlying LLM backbone. Different large‑model backbones yield varied performance when serving as agents. Though valid formulations can always be generated, noticeable fluctuations are observed in downstream test metrics.
Questions for authors

as above

Reviewer_JPqQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents an agentic framework for discovering symbolic decision rules to adapt frozen multimodal encoders for open-vocabulary audio-visual event localization. A report-guided multi-agent loop searches for candidate formulations, while an Oracle Agent estimates their expressive capacity and a Policy Agent evaluates whether their parameters can be learned by a lightweight network. On OV-AVEBench, CueRator improves the total average score from 57.8 to 60.2 and the unseen-category score from 55.8 to 59.9 over the strongest reported baseline, substantially reducing the seen–unseen performance gap.

Critical strengths

None

Critical weaknesses

The abstract and introduction are difficult to follow, with insufficient logical connections between the key concepts and the proposed method. The paper introduces “symbolic structures” in the context of LLM-guided search, but does not clearly explain how this concept connects to contrastive multimodal encoders and the specific problem of OV-AVEL. The subsequent reference to “such a rule” is also ambiguous, and the notion of a “symbolic rule” is not sufficiently defined before it becomes central to the proposed framework. More importantly, the claim that “the rule provides generalization, and the per-input parameters provide adaptivity” is asserted rather than adequately motivated: it remains unclear why a symbolic formulation should improve generalization to unseen categories, or why predicting its parameters per input resolves the limitations of existing methods. Consequently, the transition from these observations to “policy-aware decision-rule discovery” is not convincing. The authors should substantially rewrite the abstract and introduction to clearly define the decision rule and its role in OV-AVEL, explain the motivation for jointly searching the rule structure and learning its input-dependent parameters, and establish how this design addresses a specific limitation of existing approaches. In its current form, the paper does not provide a sufficiently clear and coherent account of its central idea or methodological contribution.

Questions for authors
  1. What exactly constitutes a “symbolic rule” in CueRator, and how does it operate on the similarity scores produced by frozen contrastive multimodal encoders to solve OV-AVEL? Please clarify the relationship between symbolic structures, decision rules, and the proposed framework.
  2. The paper claims that symbolic rules provide generalization while input-dependent parameter prediction provides adaptivity. What theoretical or empirical evidence supports this claim? In particular, how does the proposed decomposition improve generalization to unseen categories compared with existing dynamic-threshold methods?
  3. How does the proposed policy-aware decision-rule discovery differ from conventional rule search or validation-score-based candidate selection? What evidence demonstrates that the multi-agent feedback and the joint consideration of oracle performance and policy learnability are necessary for the reported improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.