acceptodds
Under review as a conference paper at ICLR 2027

Learning Evidence-Adaptive Composition for Compositional Generalization

Abstract

Modern vision systems increasingly combine pretrained models that provide complementary yet uncertain evidence about the same instance. However, existing approaches combine such evidence using fixed-parameter fusion functions or predefined composition rules, even when the appropriate interaction depends on the surrounding evidence. This limitation becomes more pronounced in multi-source settings, where an intermediate composition can change how subsequent evidence should be interpreted. We therefore ask whether a model should learn not only what evidence to combine, but also how that evidence should be composed. We introduce AdaCompose, a general framework for evidence-adaptive composition in which the complete evidence states being composed determine the parameters of the composition operator itself. The generated operator is applied to local evidence representations to produce pairwise relations and a new evidence state, which can then determine the operator for subsequent compositions. AdaCompose incorporates within-source permutation consistency, prediction-score gating, controlled low-rank operator adaptation, bounded relations, and a common state interface that enables recursive composition. We evaluate AdaCompose on video anomaly detection, compositional zero-shot recognition, and visual question answering, covering context-dependent recognition, unseen semantic combinations, and multimodal reasoning. Across these tasks, these evaluations assess generalisation to unseen attribute-object combinations in CZSL and the integration of heterogeneous evidence in VAD and multimodal question answering. These results demonstrate the potential of making the composition operator itself dependent on the evidence being composed, providing a general mechanism for composing heterogeneous evidence beyond fixed composition rules.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
compositional generalization, compositional learning, multimodal learning, neural operators, representation learning, vision-language learning, heterogeneous evidence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_goUhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AdaCompose, a framework for combining evidence from different sources. The main idea is that the way evidence is combined should depend on the evidence itself, rather than relying on a fixed composition rule. To achieve this, the authors use the evidence from two sources to generate the parameters of a composition operator, which is then used to model relationships between individual hypotheses. The resulting state can also be used in subsequent compositions, allowing the framework to handle multiple evidence sources recursively. The paper also discusses several structural properties of the proposed method, including permutation consistency, bounded global states, and controlled low-rank adaptation. The authors evaluate AdaCompose on four tasks: synthetic relation learning, compositional zero-shot learning, video anomaly detection, and multimodal question answering. The results show clear improvements on context-dependent synthetic tasks, competitive performance on some real-world benchmarks, and promising results on ScienceQA.

Critical strengths

One strength of this paper is its idea of making the composition operator depend on the evidence being combined, rather than using a fixed fusion function. The resulting evidence state can also be reused in later composition steps. I think this is an interesting way to handle multiple evidence sources, and the framework is flexible enough to be applied to different tasks.

Another strength is the controlled relation-learning experiments. The authors explicitly consider settings where the relationship between two hypotheses depends on the surrounding evidence, and also test how predictions change when the context varies while the queried pair remains fixed. AdaCompose performs better than the compared methods across all three contextual relation-learning settings. I find these experiments particularly valuable because they directly evaluate the context-dependent behavior that motivates the proposed approach.

Critical weaknesses

My main concern is that the experiments do not sufficiently isolate the contributions of the proposed mechanisms. The paper emphasizes evidence-conditioned operator generation and recursive composition, but it is unclear how much of the improvement comes from these components rather than contextual representations or learned fusion. In the synthetic experiments, some baselines do not have access to the same contextual information, while the ablations on real-world tasks do not fully separate the effects of dynamic operators and recursion. This makes it difficult to determine whether the central design choices are necessary. Another concern is the fairness and reliability of the experimental comparisons. For ScienceQA, the simple aggregation baselines do not use the same semantic features or training setup as AdaCompose, making it hard to attribute the improvements to the proposed composition mechanism. For VAD, the evaluation protocol differs from that used for the reported baselines, so the results are not directly comparable. In addition, most downstream experiments are conducted with a single random seed. These issues weaken the empirical evidence for the effectiveness and generality of the proposed approach.

Questions for authors
  1. The main idea of this paper is to generate composition operators based on the input evidence. However, it is still unclear whether this is more effective than simply feeding the same contextual information into a fixed-parameter relation network. Have the authors compared these two approaches under the same input and training settings? If not, could the authors clarify which existing results specifically support the benefit of dynamic operator generation?
  2. How much does recursive composition actually contribute to the final performance? In Table 7, the standalone recursive branch performs relatively poorly, while the full model also combines direct-pair and marginal scores. Is there an ablation that removes recursive composition while keeping the other components unchanged? This would help clarify whether recursion is an important part of the observed improvements.
  3. For video anomaly detection, the paper mentions that the evaluation protocol differs from that of the published baselines. Were any baseline methods evaluated using the same frame alignment and metrics as AdaCompose? If not, how should we interpret the performance comparisons in Table 4?
3 comments

We thank the reviewer for the thoughtful comments. We address the concerns and questions below.

W1 / Q1. We completed a context-matched fixed-parameter relation baseline. It retains the evidence encoder and context generator, but concatenates the context code to each fixed-weight relation-network layer; AdaCompose instead uses that code for low-rank weight modulation. Both models receive the same contextual inputs and training settings, with approximately matched parameter counts.

AdaCompose has lower error on Weierstrass and Hardy Z, while the fixed-parameter network achieves lower error on Julia (Table R1). This isolates a task-dependent benefit of dynamic parameterization, rather than establishing that dynamic weights are necessary whenever contextual information is available.

Table R1. Context-matched synthetic relation learning (nRMSE ↓, mean ± std).

MethodWeierstrassJuliaHardy Z
DeepONet (reported)0.4921 ± 0.00430.2362 ± 0.00800.9690 ± 0.0048
Fixed context-conditioned network0.2599 ± 0.00330.1513 ± 0.01440.7205 ± 0.0195
AdaCompose (reported)0.2447 ± 0.00690.1837 ± 0.00760.5506 ± 0.0458

R1 uses three seeds and population standard deviation, following the original reporting convention. DeepONet and AdaCompose retain their reported results. The fixed network is newly trained under the matched protocol.

W2 / Q2. We completed two retrained ablations under the reported dataset-specific staged training schedules. Each intervention starts at the first stage, and subsequent stages are initialized from the corresponding ablation checkpoint, with validation-based selection:

  1. Without recursion: Retain the first attribute–object relation field, direct-pair branch, and marginal branch. Remove the composed evidence state and the second operator relating it to pair evidence. The fusion gate receives primitive summaries and the first operator code instead of recursive inputs.

  2. Non-recursive replacement: Add a fixed-weight MLP receiving attribute, object, and pair local states, their summaries, and the first operator code. It approximately matches the original core's capacity, but constructs neither an intermediate evidence state nor a second operator. It uses the same non-recursive gate inputs as the first ablation.

On C-GQA, removing recursion lowers all four metrics, and AdaCompose also exceeds the capacity-matched replacement in the closed world. However, the replacement is better in the open world, and removing recursion slightly improves MIT-States. These retrained comparisons support a setting-dependent contribution of recursion.

Table R2. Retrained recursion ablation on CZSL (HM / AUC %, ↑).

MethodMIT-States ClosedMIT-States OpenC-GQA ClosedC-GQA Open
Without recursion34.57 / 17.7617.25 / 5.4423.24 / 7.826.85 / 1.31
Non-recursive replacement34.42 / 17.5616.77 / 5.3224.18 / 8.428.21 / 1.66
AdaCompose34.34 / 17.5817.10 / 5.3824.67 / 8.577.30 / 1.43

The two ablations are retrained. The AdaCompose row retains the reported checkpoints. All use the reported dataset-specific training schedules and closed/open-world evaluation protocols.

Depth transfer: A separate completed probabilistic-parity experiment trains and selects checkpoints at depths 1 to 3 and tests unseen depths 4 to 5. Here, depth d denotes d binary compositions of d+1 Bernoulli sources. Both models reuse a recursive cell; only the relation-field parameterization differs. AdaCompose achieves RMSE 0.1042 ± 0.0899 / 0.1413 ± 0.1106 at depths 4/5, versus 0.2695 ± 0.1491 / 0.2903 ± 0.1556 for fixed-weight composition (three seeds, sample SD). It improves in every paired seed. Because seen-depth fitting also differs, this establishes transfer on the controlled task.

W3. We completed a trained context-conditioned fixed-parameter fusion baseline using identical CLIP-derived evidence and training settings. It retains AdaCompose's evidence encoder, context generator, recursive composition, and answer readout. Only dynamic low-rank weight modulation is replaced by concatenating context to fixed-weight relation layers, with approximately matched capacity. Both models are trained independently; the baseline is not fine-tuned from an AdaCompose checkpoint.

Both trained models substantially outperform Mean/Product, supporting the value of supervised contextual fusion. Across all three completed seeds, their Overall accuracy is similar, so this comparison does not establish an overall advantage for dynamic weights.

Table R3. Trained semantic fusion on ScienceQA (accuracy %, ↑).

MethodIMGNOOverall
Mean44.8247.5348.27
Product44.9247.5348.29
Trained fixed-parameter fusion70.286 ± 0.69661.440 ± 0.90766.423 ± 0.330
AdaCompose70.550 ± 1.51960.093 ± 1.44966.195 ± 1.385

The trained models share contextual inputs, approximately matched parameter counts, training budgets, and validation-based checkpoint selection. Values are mean ± sample SD over all three completed seeds. The limited number of seeds prevents strong conclusions about the statistical reliability of the observed differences.

W4 / Q3. The discrepancy in Appendix F.3 concerns evaluation positions, not test membership: all 240 test-video IDs match the official Protocol II list, and validation videos come exclusively from the official training set. Our original evaluation covers 146,015 decoder-aligned positions; the released labels contain 143,920 positions, using 16 per I3D time step. Tail handling and effective feature lengths explain the count difference.

We use the released ground-truth array and test order to evaluate the same frozen predictions using the released per-video lengths. ROC-AUC/AP change by only +0.080/+0.094 percentage points (rebuttal Table R4). This indicates limited sensitivity to this specific prefix mapping; it does not establish frame-level equivalence or matching baseline subgroup definitions.

We also distinguish average precision from trapezoidal PR-AUC, which the released RTFM/MGFN evaluators use. A strict cross-method ranking requires predictions evaluated at verified common positions with the same metric implementation. Rebuttal Table R4 provides a within-model sensitivity check.

Table R4. AdaCompose under original and released-label evaluation (%, ↑).

Evaluation settingPositionsPositive positionsROC-AUCAPPR-AUC
Original decoder-aligned146,01533,73982.73955.46855.456
Released labels, per-video prefix mapping143,92033,16082.81955.56155.549

The second row retains the first N frozen frame scores per video, with N = released I3D length × 16, and uses the released test order and ground-truth array. No retraining, checkpoint reselection, or score normalization is performed. This assumes correspondence of the retained frame indices. AP is average precision; PR-AUC is trapezoidal integration, as used by the released RTFM/MGFN evaluators.

W5. Table R5 reports three seeds for the VAD ablations under the same training and validation-selection rules. Full AdaCompose has higher mean AUC and AP than each ablation.

The original full-versus-type-only result improves AUC by 0.279 points but lowers AP by 1.917 points. Across three seeds, the paired differences are +0.956 ± 1.607 AUC and +1.515 ± 2.973 AP points (sample SD).

Table R5. Multi-seed VAD mechanism ablations (%, ↑).

VariantExisting AUC / APAUC (mean ± std)AP (mean ± std)
Full AdaCompose82.739 / 55.46881.574 ± 1.36556.614 ± 1.053
Type-only operator codes82.460 / 57.38580.618 ± 1.69655.099 ± 2.022
Context-aware static factors81.953 / 56.58479.119 ± 2.64752.140 ± 3.870
Without interaction feedback81.570 / 54.81080.405 ± 1.14654.889 ± 1.674

Mean ± sample SD over three seeds under the same training and validation-selection rules.

We have incorporated the corresponding clarifications and new experimental results into the revised manuscript.

Reviewer_goUhOpenReview ↗

Response

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.

However, the new results also show that the benefits of dynamic operator generation and recursive composition are not consistent across tasks. In particular, the fixed-parameter baseline achieves comparable performance on ScienceQA, and recursion does not consistently improve the CZSL results. These findings leave me unconvinced about the advantages of the proposed mechanisms. Therefore, I will keep my original score.

Reviewer_6mdGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AdaCompose, a framework for combining heterogeneous evidence through dynamically generated composition operators. Source summaries condition low-rank adaptations of a relation network, whose outputs form new evidence states for recursive composition. The authors establish structural properties and evaluate the framework on synthetic relation learning, compositional zero-shot recognition, video anomaly detection, and ScienceQA, demonstrating contextual adaptation and applicability across multiple tasks with a relatively small trainable task-specific parameter budget.

Critical strengths
  1. Coherent formulation of evidence-dependent composition. AdaCompose integrates evidence-conditioned operators and recursive state updates through a common interface, allowing intermediate results to influence subsequent composition.

  2. Broad evaluation with encouraging contextual-learning results. Experiments cover synthetic relation learning and three downstream tasks, showing promising contextual adaptation with compact task-specific modules.

Critical weaknesses
  1. Insufficient validation of dynamic operators. Several baselines differ in contextual inputs or training, while the VAD ablation shows a small ROC-AUC gain but lower AP. Matched comparisons are needed to establish the benefit of operator adaptation. 
  2. Unclear contribution of recursion. CZSL results combine recursive, direct-pair, and marginal scores without retrained ablations isolating recursion. Generalization to greater composition depth is also untested. 
  3. Evaluation reliability concerns. VAD comparisons use different frame alignments and evaluation protocols. The acknowledged inspection of test results during development further weakens confidence in the generalization claims.
Questions for authors
  1. How is the composition order determined for each task, and what guides this choice given that the operator is neither commutative nor associative?

  2. How do the task-specific implementations differ from the generic composition cell, and which theoretical guarantees remain applicable?

  3. How should the opposing ROC-AUC and AP trends in the operator ablation be interpreted when assessing the benefit of evidence-conditioned composition?

4 comments

We thank the reviewer for the constructive feedback. We address each of the reviewer's points below.

W1. We conducted two controlled comparisons.

  1. Matched relation network: We compare AdaCompose against a fixed-parameter relation network that receives the same local states, global summaries, and type information, with matched parameter budgets and training settings. The baseline retains the same evidence-encoder and context-generator architectures but concatenates the learned context code to each fixed-weight MLP layer, whereas AdaCompose uses that code to modulate the relation field's weights through low-rank updates.

  2. Trained semantic fusion: We evaluate a context-conditioned fixed weight relation field ablation of AdaCompose on ScienceQA, retaining its evidence-encoder and context-generator architectures, recursive composition, and answer readout. Only the dynamic low-rank weight modulation is replaced by concatenating the context code to each fixed-weight relation-network layer, with approximately matched parameter counts. All trainable modules are trained independently rather than fine-tuned from an AdaCompose checkpoint. Thus, both models retain context-dependent reasoning, and this comparison examines the parameterization of the relation field rather than the overall contribution of the AdaCompose architecture. Across all three seeds, Overall accuracy is similar between the models (Table R2).

VAD multi-seed results (Table R6 in W3): Using the paired runs underlying Table R6, full AdaCompose improves over the type-only ablation by 0.956 ± 1.607 ROC-AUC and 1.515 ± 2.973 AP percentage points (paired mean ± sample SD; Table R6). The AP decrease observed in the original run does not persist across seeds. Because type-only codes also remove contextual input, this ablation complements rather than replaces the context-matched comparisons above.

These comparisons separate dynamic weight parameterization from access to context. AdaCompose improves on Weierstrass and Hardy Z but underperforms the fixed network on Julia (Table R1). Both trained ScienceQA models substantially outperform untrained aggregation, supporting learned contextual fusion. The gains from dynamic modulation vary across subsets (Table R2). Together with the uncertain VAD gains, these results support task-dependent benefits of operator adaptation.

Table R1. Context-matched synthetic relation learning (nRMSE ↓, mean ± std).

MethodWeierstrassJuliaHardy Z
DeepONet0.4921 ± 0.00430.2362 ± 0.00800.9690 ± 0.0048
Fixed context-conditioned network0.2599 ± 0.00330.1513 ± 0.01440.7205 ± 0.0195
AdaCompose0.2447 ± 0.00690.1837 ± 0.00760.5506 ± 0.0458

R1 uses three seeds and population standard deviation, following the original reporting convention. DeepONet and AdaCompose retain their reported results. The fixed network is newly trained under the matched protocol.

Table R2. Trained semantic fusion on ScienceQA (accuracy %, ↑).

MethodIMGNOOverall
Mean44.8247.5348.27
Product44.9247.5348.29
Trained fixed-parameter fusion70.286 ± 0.69661.440 ± 0.90766.423 ± 0.330
AdaCompose70.550 ± 1.51960.093 ± 1.44966.195 ± 1.385

The trained models share contextual inputs, matched parameter counts, training budgets, and validation-based checkpoint selection. Values are mean ± sample SD over all three completed seeds.

W2. We completed two retrained ablations using the reported dataset-specific training configurations and staged training schedules. Each ablation is introduced at the first training stage, and subsequent stages are initialized from the corresponding ablation checkpoint, with validation-based checkpoint selection throughout:

  1. Without recursion: Retain the first attribute–object dynamic relation field, but remove the composed evidence state and the second operator generation that relates it to pair evidence. Retain the direct-pair and marginal branches, and replace recursive inputs to the fusion gate with primitive summaries and the first-stage operator code. Retrain the resulting model under the original staged protocol.

  2. Non-recursive replacement: Replace the removed recursive path with a fixed-weight MLP that directly receives attribute, object, and pair local states, their global summaries, and the first-stage operator code. This branch neither constructs an intermediate evidence state nor generates a second operator code. Its capacity approximately matches the original core, and it uses the same non-recursive fusion-gate inputs as the first ablation.

Table R3 shows a dataset- and evaluation-dependent contribution. On C-GQA, removing recursion lowers all four metrics; AdaCompose also exceeds the capacity-matched replacement in closed-world HM/AUC (24.67/8.57 versus 24.18/8.42). However, the replacement performs better in the open world (8.21/1.66 versus 7.30/1.43). On MIT-States, removing recursion gives small improvements. Thus, recursion contributes to the current C-GQA closed-world model, but is not uniformly superior to non-recursive contextual fusion.

The recursive branch receives mean closed-world mixture weights of 47.34% on C-GQA and 1.12% on MIT-States. This is consistent with different branch usage, but is not a causal importance measure: branch scales and joint training also matter.

Depth transfer: We now evaluate a controlled probabilistic-parity task: predict the probability that an odd number of independent Bernoulli events occur. Depth d denotes d binary compositions of d+1 sources. Training and checkpoint selection use depths 1 to 3 only. Depths 4 to 5 are held out. Both models recursively reuse their composition cell. The baseline replaces dynamic field weights with fixed weights receiving the same context. Thus, Table R4 tests dynamic parameterization in depth transfer, complementing the recursion ablation in Table R3. AdaCompose achieves lower error at both unseen depths in every paired seed, and outperforms a constant-0.5 predictor (RMSE 0.3678/0.3456 at depths 4/5). However, seen-depth fitting also differs, with one fixed-weight seed near the constant baseline. These results demonstrate depth transfer on this task.

Table R3. Retrained recursion ablation on CZSL (HM / AUC %, ↑).

MethodMIT-States ClosedMIT-States OpenC-GQA ClosedC-GQA Open
Without recursion34.57 / 17.7617.25 / 5.4423.24 / 7.826.85 / 1.31
Non-recursive replacement34.42 / 17.5616.77 / 5.3224.18 / 8.428.21 / 1.66
AdaCompose34.34 / 17.5817.10 / 5.3824.67 / 8.577.30 / 1.43

The AdaCompose row retains the reported checkpoints. All use the reported dataset-specific training schedules and closed/open-world evaluation protocols.

Table R4. Controlled depth transfer on probabilistic parity (RMSE ↓).

MethodSeen depth 3Unseen depth 4Unseen depth 5
Fixed-parameter sequential composition0.2138 ± 0.15910.2695 ± 0.14910.2903 ± 0.1556
AdaCompose0.0501 ± 0.04560.1042 ± 0.08990.1413 ± 0.1106

Mean ± sample standard deviation over all three seeds. Models use identical paired training data, 12,000 updates with batch size 64, and approximately matched parameter counts (32,517/32,457 for AdaCompose/fixed). Checkpoints minimize mean validation MSE over depths 1 to 3. Unseen depths are evaluated only after selection is locked.

W3. VAD evaluation: The discrepancy in Appendix F.3 concerns evaluation positions, not dataset membership: all 240 test-video IDs match the official Protocol II list, and validation videos come exclusively from the official training set. Our original evaluator covered the full decoder-aligned sequences; the released evaluator uses 16 positions per I3D time step. Tail handling and effective feature lengths account for the difference. Using the released per-video lengths, we reconstructed the official labels exactly. Re-evaluating the same frozen predictions under the mapping described below changes ROC-AUC/AP by only +0.080/+0.094 percentage points (Table R5), indicating limited sensitivity to this adjustment. This does not establish identical decoded-frame correspondence or baseline subgroup definitions; published baseline scores remain reference comparisons.

Evaluation protocol: Test data were not used for training or model selection; checkpoints were selected using validation performance. The test-result inspection disclosed in Appendix F.5 occurred during exploratory development and did not guide training or model selection.

Table R6 reports all three seeds for the VAD mechanism ablations. Full AdaCompose has higher mean ROC-AUC and AP than each ablation; the paired differences and their variability qualify the single-run interpretation in Q3.

Table R5. AdaCompose under original and released-label evaluation (%, ↑).

Evaluation settingPositionsPositive positionsROC-AUCAPPR-AUC
Original decoder-aligned146,01533,73982.73955.46855.456
Released labels, per-video prefix mapping143,92033,16082.81955.56155.549

The second row retains the first N frozen frame scores per video, with N = released I3D length × 16, and uses the released test order and ground-truth array. No retraining, checkpoint reselection, or score normalization is performed. This assumes correspondence of the retained frame indices; decoded-image identity has not been independently verified. AP is average precision; PR-AUC is trapezoidal integration, as used by the released RTFM/MGFN evaluators.

Table R6. Multi-seed VAD mechanism ablations (%, ↑).

VariantExisting AUC / APAUC (mean ± std)AP (mean ± std)
Full AdaCompose82.739 / 55.46881.574 ± 1.36556.614 ± 1.053
Type-only operator codes82.460 / 57.38580.618 ± 1.69655.099 ± 2.022
Context-aware static factors81.953 / 56.58479.119 ± 2.64752.140 ± 3.870
Without interaction feedback81.570 / 54.81080.405 ± 1.14654.889 ± 1.674

Mean ± sample SD over three seeds, using the same training and validation-selection rules and the original 146,015-position timeline.


Q1. The composition order is determined by the task-specific organization of evidence, rather than being learned as an optimal permutation. AdaCompose is explicitly ordered and does not assume commutativity or associativity.

  • VAD: Action and object evidence are composed first, followed by scene evidence. This organizes local action–object interactions before incorporating broader scene context.

  • ScienceQA: Question and textual-hint evidence are composed first, followed by visual evidence, allowing the intermediate textual state to condition the subsequent visual interaction.

  • CZSL: Attribute and object evidence are composed for each aligned candidate pair; the resulting evidence state is then related to pair evidence: (Attribute, Object) → Pair. Direct-pair and marginal scoring branches are fused with this recursive branch.

  • Synthetic relation learning (R1): Only pairwise relation learning is evaluated. The separate depth-transfer task in R4 recursively composes Bernoulli sources in a fixed sequential order.

These orders reflect task-specific modeling choices, not theoretically optimal composition structures. We will clarify the exact composition trees and distinguish the selected order from any claim of order invariance. Alternative-order evaluations would further quantify the sensitivity of the mechanism.

Table R7. Task-specific composition structure.

TaskComposition orderRole
SyntheticPairwise onlyContext-dependent relation learning
VAD(Action, Object) → SceneLocal interaction followed by scene conditioning
ScienceQA(Question, Text) → ImageTextual composition followed by visual integration
CZSL(Attribute, Object) → PairCandidate-aligned composition followed by pair-evidence interaction

CZSL uses aligned candidates rather than a full Cartesian product. No order-optimality claim is made.

Q2. Section 2.2 defines a generic Cartesian composition cell. The downstream models adapt its interaction structure and prediction readout to their task-specific evidence spaces:

TaskImplementation differences and theoretical scope
VADUses evidence-weighted concept factors, low-rank interactions, and temporal fusion. The generic-cell proofs do not directly establish guarantees for the complete temporal predictor.
CZSLUses aligned attribute–object candidates rather than Cartesian interactions, and combines recursive, direct-pair, and marginal scores. Properties of the recursive module do not automatically extend to the fused predictor.
ScienceQAUses answer-conditioned evidence and masked averaging of final relation logits. Symmetric averaging alone does not establish permutation consistency of the complete model; evidence construction and interaction processing must also satisfy the required conditions.

The theoretical guarantees apply only where their assumptions are preserved. The algebraic rank bound remains applicable to weight updates that retain the specified low-rank parameterization. Global-state boundedness additionally requires convex updates and bounded relation summaries; fixed-query sensitivity requires bounded, fixed pair inputs and the assumed Lipschitz conditions. Neither result automatically guarantees robustness of the complete downstream predictor.

Permutation consistency concerns reordering elements within an evidence set, not exchanging evidence sources or changing the composition tree. It therefore does not imply commutativity or associativity.

We have clarified these boundaries in Appendices D and E, distinguishing guarantees proved for the generic cell from properties retained by individual modules and empirical results for the task-specific models.

Q3. Compared with type-only operator codes, full AdaCompose improves ROC-AUC from 82.460% to 82.739% (+0.279 points), while AP decreases from 57.385% to 55.468% (−1.917 points).

These metrics evaluate different aspects of detection. ROC-AUC measures positive–negative ranking across thresholds, whereas AP summarizes precision–recall performance and is particularly sensitive to the ranking of positive instances in imbalanced evaluations.

The three-seed results qualify this interpretation: full AdaCompose improves mean ROC-AUC and AP over type-only codes, with paired differences of +0.956 ± 1.607 and +1.515 ± 2.973 percentage points, respectively. Type-only codes also remove contextual input to the generator; the context-matched baselines in W1 more directly isolate dynamic weight parameterization.


We have incorporated the corresponding clarifications and new experimental results into the revised manuscript.

Reviewer_95G8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AdaCompose proposes an evidence-adaptive framework for combining heterogeneous predictions from pretrained models. Instead of applying a fixed relation function, it generates composition-specific parameters from summaries of the current evidence. Each source contains local hypothesis states, a permutation-invariant global summary, and a type embedding. A hypernetwork produces bounded coefficients that modulate a shared relation network through low-rank updates. The generated network scores hypothesis pairs and constructs a new evidence state. This state retains the input interface, allowing it to participate in subsequent compositions with newly generated operators. The framework aims to support context-sensitive evidence integration across compositional recognition, anomaly detection, and multimodal question answering.

Critical strengths
  1. The motivation is clear, the same hypothesis pair can require different interpretations depending on the surrounding evidence. Adapting the composition rule is a reasonable approach to this problem.
  2. The shared composition interface and low-rank parameterization give the framework a coherent structure. The permutation and boundedness analyses also make several architectural properties explicit.
  3. The evaluation covers controlled relation learning and several downstream applications.
Critical weaknesses
  1. The contextual intervention changes both local representations and operator codes, while ScienceQA compares a trained semantic head against untrained probability fusion. These comparisons do not establish that generating weights is better than using a fixed context-conditioned relation network.
  2. In Eqs. 6–9, newly generated relation information reaches the next operator only through the mean relation score. With input summaries fixed, different relation patterns sharing this mean become indistinguishable to operator generation, which seems restrictive for the proposed recursive mechanism.
  3. Recursive composition is central to the proposal, but its empirical contribution remains unclear. There is no unseen-depth evaluation, and the CZSL fusion diagnostics do not separate recursion from the direct-pair and marginal branches.
  4. Some cross-method comparisons use different evaluation pipelines. VAD results differ in frame alignment and metric implementation, while CZSL timing does not measure the exact pipeline used for the reported accuracy, limiting the conclusions about relative performance and efficiency.
  5. Downstream results use a single seed, and Appendix F.5 acknowledges exploratory inspection of test results. This weakens confidence in small mechanism gains, such as the 0.28-point VAD ROC-AUC improvement over type-only codes, which also comes with lower AP.
Questions for authors

See weakness.

4 comments

We thank the reviewer for the constructive comments. We address the concerns and questions below.

W1. We completed two context-matched comparisons:

  1. Synthetic relation learning: The fixed network retains the evidence encoder and context generator, but concatenates the context code to each fixed-weight relation-network layer. AdaCompose instead uses this code for low-rank weight modulation. Both models use the same inputs and training settings, with approximately matched parameter counts.

  2. ScienceQA: The fixed fusion model retains the same contextual evidence, recursive composition, and answer readout. Only the relation field's dynamic weight modulation is replaced by context concatenation; both models are trained independently under the same optimization and validation-selection rules. This comparison isolates the effect of relation-field parameterization rather than evaluating the overall contribution of AdaCompose.

VAD multi-seed results (Table R8 in W5): Across the three paired runs reported in Table R8, full AdaCompose improves over the type-only ablation by 0.956 ± 1.607 ROC-AUC and 1.515 ± 2.973 AP percentage points (paired mean ± sample SD). The AP decrease observed in the original run does not persist across seeds.

These comparisons separate dynamic weight parameterization from access to context. AdaCompose improves on Weierstrass and Hardy Z but underperforms the fixed network on Julia (Table R1). Across all three ScienceQA seeds, Overall accuracy is similar between the trained models (Table R2). Both trained models substantially outperform untrained aggregation, supporting learned contextual fusion. The VAD gains remain uncertain across seeds (Table R8). These results support task-dependent benefits of operator adaptation.

Table R1. Context-matched synthetic relation learning (nRMSE ↓, mean ± std).

MethodWeierstrassJuliaHardy Z
DeepONet0.4921 ± 0.00430.2362 ± 0.00800.9690 ± 0.0048
Fixed context-conditioned network0.2599 ± 0.00330.1513 ± 0.01440.7205 ± 0.0195
AdaCompose0.2447 ± 0.00690.1837 ± 0.00760.5506 ± 0.0458

R1 uses three seeds and population standard deviation, following the original reporting convention. DeepONet and AdaCompose retain their reported results. The fixed network is newly trained under the matched protocol.

Table R2. Trained semantic fusion on ScienceQA (accuracy %, ↑).

MethodIMGNOOverall
Mean44.8247.5348.27
Product44.9247.5348.29
Trained fixed-parameter fusion70.286 ± 0.69661.440 ± 0.90766.423 ± 0.330
AdaCompose70.550 ± 1.51960.093 ± 1.44966.195 ± 1.385

The trained models share contextual inputs, approximately matched parameter counts, training budgets, and validation-based checkpoint selection. Values are mean ± sample SD over all three completed seeds.

W2. As defined in Section 2.2, the composed evidence state contains two complementary information pathways:

  1. Local interaction pathway: Each pair produces a local state hijXY=rijXY⋅ψpair([hiX,hjY,rijXY])h_{ij}^{XY} = r_{ij}^{XY} \cdot \psi_{\text{pair}}([h_i^X, h_j^Y, r_{ij}^{XY}]). These representations form HXYH_{XY} and participate directly in subsequent pairwise interactions.
  2. Global conditioning pathway: The mean relation score contributes to zXYz_{XY}, which determines the next operator together with the new source summary and type embeddings.

We agree that different relation patterns with identical means, input summaries, types, and subsequent source produce the same next operator code. Nevertheless, their local pairwise states can remain different, allowing subsequent relation evaluation to distinguish them.

Thus, the bottleneck restricts operator conditioning, not all subsequent computation. Local states retain pair-specific representations, but their learned projection and relation weighting are not guaranteed to be injective; neither pathway ensures lossless propagation. This explanation applies to the generic Cartesian composition cell. CZSL uses aligned candidate pairs rather than retaining a full Cartesian set.

Table R3. Information pathways in recursive composition.

ComponentInformation retainedRole in subsequent composition
Local states HXYH_{XY}Individual relation-conditioned pair representationsInputs to subsequent pairwise relation evaluation
Global summary zXYz_{XY}Input summaries and aggregated relation informationConditions the next operator code
Type embedding τXY\tau_{XY}Composition-type informationIdentifies the composition type for operator generation

This clarification follows the existing formulation in Eqs. 6–10; no additional empirical claim is made.

W3. We completed two retrained ablations under the reported dataset-specific staged training schedules. Each intervention starts at the first stage, with subsequent stages initialized from the corresponding ablation checkpoint and selected using validation data:

  1. Without recursion: Keep the first attribute-object relation field and the direct-pair/marginal branches. Remove the intermediate composed state and the second operator relating it to pair evidence; replace the fusion gate's recursive inputs with primitive summaries and the first operator code.

  2. Non-recursive replacement: Add a fixed-weight MLP receiving attribute, object, and pair local states, their summaries, and the first operator code. It constructs no intermediate evidence state or second operator and approximately matches the full core's capacity; its fusion gate uses the same non-recursive inputs as the first ablation.

On C-GQA, removing recursion lowers all four metrics, and AdaCompose also exceeds the capacity-matched replacement in the closed world. However, the replacement performs better in the open world, while removing recursion slightly improves MIT-States. These results support a setting-dependent contribution.

Depth transfer: Table R5 reports prediction errors on the probabilistic-parity task. Depth d means d binary compositions of d+1 sources. Training and validation use depths 1 to 3, depths 4 to 5 are held out. Both models recursively reuse their cell, differing in dynamic versus context-conditioned fixed field weights. AdaCompose has lower unseen-depth error in every paired seed and beats a constant-0.5 predictor (RMSE 0.3678/0.3456 at depths 4/5). Seen-depth fitting also differs, with one fixed-weight seed near the constant baseline. This demonstrates transfer on the controlled task.

Table R4. Retrained recursion ablation on CZSL (HM / AUC %, ↑).

MethodMIT-States ClosedMIT-States OpenC-GQA ClosedC-GQA Open
Without recursion34.57 / 17.7617.25 / 5.4423.24 / 7.826.85 / 1.31
Non-recursive replacement34.42 / 17.5616.77 / 5.3224.18 / 8.428.21 / 1.66
AdaCompose (reported)34.34 / 17.5817.10 / 5.3824.67 / 8.577.30 / 1.43

The two ablations are retrained. The AdaCompose row retains the reported checkpoints. All use the reported dataset-specific training schedules and closed/open-world evaluation protocols.

Table R5. Controlled depth transfer on probabilistic parity (RMSE ↓).

MethodSeen depth 3Unseen depth 4Unseen depth 5
Fixed-parameter sequential composition0.2138 ± 0.15910.2695 ± 0.14910.2903 ± 0.1556
AdaCompose0.0501 ± 0.04560.1042 ± 0.08990.1413 ± 0.1106

Mean ± sample standard deviation over all three seeds. Models use identical paired training data, 12,000 updates with batch size 64, and approximately matched parameter counts (32,517/32,457 for AdaCompose/fixed). Checkpoints minimize mean validation MSE over depths 1 to 3; unseen depths are evaluated only after selection is locked.

W4. VAD: The discrepancy concerns evaluation positions, not test membership: all 240 test-video IDs match the official Protocol II list, and validation videos come only from the official training set. Our original evaluation covers decoder-aligned sequences; the released labels use 16 positions per I3D time step. Tail handling and effective feature lengths explain the count difference. Re-evaluating the same frozen predictions with the explicit mapping in Table R6 changes ROC-AUC/AP by only +0.080/+0.094 percentage points. This quantifies sensitivity to the released labels. Published RTFM/MGFN/UR-DMU/EGO scores therefore remain reference comparisons, not matched rankings.

CZSL: The earlier benchmark omitted evidence reconstruction, so its latency could not be paired with the reported accuracy. Table R7 now includes this step and measures AdaCompose from preprocessed image input to prediction using the reported checkpoint. We remove the cross-method efficiency comparison because we could not locate publicly downloadable C-GQA-trained DFSP/Troika checkpoints. The revised measurement establishes AdaCompose's inference cost within the stated scope. Full-test accuracy verification remains pending.

Table R6. AdaCompose under original and released-label evaluation (%, ↑).

Evaluation settingPositionsPositive positionsROC-AUCAPPR-AUC
Original decoder-aligned146,01533,73982.73955.46855.456
Released labels, per-video prefix mapping143,92033,16082.81955.56155.549

The second row retains the first N frozen frame scores per video, with N = released I3D length × 16, and uses the released test order and ground-truth array. No retraining, checkpoint reselection, or score normalization is performed. This assumes correspondence of the retained frame indices; decoded-image identity has not been verified. AP is average precision; PR-AUC is trapezoidal integration, as used by the released RTFM/MGFN evaluators.

Table R7. AdaCompose inference timing on C-GQA (RTX 4090, batch size 1).

MethodLatency (ms/image ↓)
AdaCompose9.924 ± 0.031

Timed path: image encoding → primitive similarities → evidence reconstruction → candidate scoring → prediction. Mean ± SD across 1,280 measurements (256 test images × 5 passes), after 50 warm-up iterations; 6,515 closed-world candidates; FP32 image encoding and BF16 task-head autocast. Excludes I/O, preprocessing, host-to-device transfer, and one-time text/projection setup. Predictions match the archived-evidence path on 254/256 images

W5. Table R8 reports three seeds under the same training and validation-selection rules, using the original 146,015-position VAD timeline.

The original full-versus-type-only comparison improves ROC-AUC by 0.279 points but lowers AP by 1.917 points. Across three seeds, the paired full-minus-type-only differences are instead +0.956 ± 1.607 ROC-AUC and +1.515 ± 2.973 AP points (sample SD). Full AdaCompose has higher mean AUC/AP than every ablation.

Evaluation protocol: Test data were not used for training or model selection; checkpoints were selected using validation performance. Appendix F.5 refers to viewing test results during exploratory development, which is distinct from using them to train or select models. The additional three-seed experiments quantify training variability.

Table R8. Multi-seed VAD mechanism ablations (%, ↑).

VariantExisting AUC / APAUC (mean ± std)AP (mean ± std)
Full AdaCompose82.739 / 55.46881.574 ± 1.36556.614 ± 1.053
Type-only operator codes82.460 / 57.38580.618 ± 1.69655.099 ± 2.022
Context-aware static factors81.953 / 56.58479.119 ± 2.64752.140 ± 3.870
Without interaction feedback81.570 / 54.81080.405 ± 1.14654.889 ± 1.674

Mean ± sample SD over three seeds, using the same training and validation-selection rules.

We have incorporated the corresponding clarifications and new experimental results into the revised manuscript.

OK, address most of my concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and revision

Dear AC,

We sincerely thank you and all three reviewers (95G8, 6mdG, and goUh) for their valuable feedback, which has helped us strengthen our work. We have carefully addressed the major concerns through additional experiments, theoretical clarifications, and revisions to the manuscript.

We are encouraged that the reviewers recognized the following strengths:

  1. Novel and well-motivated formulation (95G8, 6mdG, goUh):

    • The interesting idea of making composition operators dependent on the evidence being combined.
    • A coherent framework where intermediate evidence can influence subsequent composition rules.
  2. Principled and reusable architecture (95G8, 6mdG):

    • A shared composition interface with controlled low-rank operator adaptation.
    • Explicit theoretical characterization of permutation consistency, boundedness, and other structural properties.
  3. Broad evaluation and contextual learning (95G8, 6mdG, goUh):

    • Controlled relation-learning experiments that directly examine context-dependent behavior.
    • Applications across three downstream tasks, with promising performance and compact task-specific modules.

Summary of rebuttal and evidence

For your convenience, we summarize the main concerns and our responses below:

Key concernOur response
Effectiveness of dynamic operators (95G8, 6mdG, goUh)We introduced context-matched fixed-parameter baselines with comparable capacity and identical training protocols. AdaCompose achieves lower errors on two of three contextual synthetic fields. Additional trained ScienceQA comparisons show similar overall accuracy, clarifying the task-dependent benefits of dynamic operator adaptation.
Contribution of recursive composition (95G8, 6mdG, goUh)We conducted two retrained CZSL ablations, including a capacity-matched non-recursive replacement. Recursion improves C-GQA closed-world performance. We further introduced unseen-depth experiments, where AdaCompose achieves lower errors than fixed-parameter sequential composition at depths 4–5.
Information bottleneck and theoretical scope (95G8, 6mdG)We clarified how local relation-conditioned states preserve pairwise information for subsequent interactions, while global summaries determine the next operator. We explicitly discuss the mean-aggregation limitation and distinguish generic theoretical guarantees from task-specific implementations.
Evaluation fairness and reliability (95G8, 6mdG, goUh)We re-evaluated VAD using released MSAD labels and frame-length mapping, obtaining consistent ROC-AUC/AP (82.819/55.561 versus 82.739/55.468). We also added three-seed VAD ablations, showing higher mean ROC-AUC and AP for the full model, while reporting variability and limitations of cross-method comparisons.
Efficiency and reproducibility (95G8)We updated the CZSL inference benchmark to include evidence reconstruction, reporting 9.924 ± 0.031 ms/image under the specified setup. We also clarified implementation details, evaluation protocols, and the scope of efficiency comparisons.

Rebuttal outcome

  • Reviewer 95G8 acknowledged that our responses had addressed their concerns.
  • We have incorporated the additional experiments and clarifications into the revised manuscript, providing stronger evidence for the framework while explicitly identifying where the benefits of dynamic adaptation and recursion depend on the evaluation setting.

We sincerely appreciate the AC's time and consideration of our revised work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.