- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes AdaCompose, a framework for combining evidence from different sources. The main idea is that the way evidence is combined should depend on the evidence itself, rather than relying on a fixed composition rule. To achieve this, the authors use the evidence from two sources to generate the parameters of a composition operator, which is then used to model relationships between individual hypotheses. The resulting state can also be used in subsequent compositions, allowing the framework to handle multiple evidence sources recursively. The paper also discusses several structural properties of the proposed method, including permutation consistency, bounded global states, and controlled low-rank adaptation. The authors evaluate AdaCompose on four tasks: synthetic relation learning, compositional zero-shot learning, video anomaly detection, and multimodal question answering. The results show clear improvements on context-dependent synthetic tasks, competitive performance on some real-world benchmarks, and promising results on ScienceQA.
Critical strengths
One strength of this paper is its idea of making the composition operator depend on the evidence being combined, rather than using a fixed fusion function. The resulting evidence state can also be reused in later composition steps. I think this is an interesting way to handle multiple evidence sources, and the framework is flexible enough to be applied to different tasks.
Another strength is the controlled relation-learning experiments. The authors explicitly consider settings where the relationship between two hypotheses depends on the surrounding evidence, and also test how predictions change when the context varies while the queried pair remains fixed. AdaCompose performs better than the compared methods across all three contextual relation-learning settings. I find these experiments particularly valuable because they directly evaluate the context-dependent behavior that motivates the proposed approach.
Critical weaknesses
My main concern is that the experiments do not sufficiently isolate the contributions of the proposed mechanisms. The paper emphasizes evidence-conditioned operator generation and recursive composition, but it is unclear how much of the improvement comes from these components rather than contextual representations or learned fusion. In the synthetic experiments, some baselines do not have access to the same contextual information, while the ablations on real-world tasks do not fully separate the effects of dynamic operators and recursion. This makes it difficult to determine whether the central design choices are necessary. Another concern is the fairness and reliability of the experimental comparisons. For ScienceQA, the simple aggregation baselines do not use the same semantic features or training setup as AdaCompose, making it hard to attribute the improvements to the proposed composition mechanism. For VAD, the evaluation protocol differs from that used for the reported baselines, so the results are not directly comparable. In addition, most downstream experiments are conducted with a single random seed. These issues weaken the empirical evidence for the effectiveness and generality of the proposed approach.
Questions for authors
- The main idea of this paper is to generate composition operators based on the input evidence. However, it is still unclear whether this is more effective than simply feeding the same contextual information into a fixed-parameter relation network. Have the authors compared these two approaches under the same input and training settings? If not, could the authors clarify which existing results specifically support the benefit of dynamic operator generation?
- How much does recursive composition actually contribute to the final performance? In Table 7, the standalone recursive branch performs relatively poorly, while the full model also combines direct-pair and marginal scores. Is there an ablation that removes recursive composition while keeping the other components unchanged? This would help clarify whether recursion is an important part of the observed improvements.
- For video anomaly detection, the paper mentions that the evaluation protocol differs from that of the published baselines. Were any baseline methods evaluated using the same frame alignment and metrics as AdaCompose? If not, how should we interpret the performance comparisons in Table 4?
3 comments
W3. We completed a trained context-conditioned fixed-parameter fusion baseline using identical CLIP-derived evidence and training settings. It retains AdaCompose's evidence encoder, context generator, recursive composition, and answer readout. Only dynamic low-rank weight modulation is replaced by concatenating context to fixed-weight relation layers, with approximately matched capacity. Both models are trained independently; the baseline is not fine-tuned from an AdaCompose checkpoint.
Both trained models substantially outperform Mean/Product, supporting the value of supervised contextual fusion. Across all three completed seeds, their Overall accuracy is similar, so this comparison does not establish an overall advantage for dynamic weights.
Table R3. Trained semantic fusion on ScienceQA (accuracy %, ↑).
| Method | IMG | NO | Overall |
|---|---|---|---|
| Mean | 44.82 | 47.53 | 48.27 |
| Product | 44.92 | 47.53 | 48.29 |
| Trained fixed-parameter fusion | 70.286 ± 0.696 | 61.440 ± 0.907 | 66.423 ± 0.330 |
| AdaCompose | 70.550 ± 1.519 | 60.093 ± 1.449 | 66.195 ± 1.385 |
The trained models share contextual inputs, approximately matched parameter counts, training budgets, and validation-based checkpoint selection. Values are mean ± sample SD over all three completed seeds. The limited number of seeds prevents strong conclusions about the statistical reliability of the observed differences.
W4 / Q3. The discrepancy in Appendix F.3 concerns evaluation positions, not test membership: all 240 test-video IDs match the official Protocol II list, and validation videos come exclusively from the official training set. Our original evaluation covers 146,015 decoder-aligned positions; the released labels contain 143,920 positions, using 16 per I3D time step. Tail handling and effective feature lengths explain the count difference.
We use the released ground-truth array and test order to evaluate the same frozen predictions using the released per-video lengths. ROC-AUC/AP change by only +0.080/+0.094 percentage points (rebuttal Table R4). This indicates limited sensitivity to this specific prefix mapping; it does not establish frame-level equivalence or matching baseline subgroup definitions.
We also distinguish average precision from trapezoidal PR-AUC, which the released RTFM/MGFN evaluators use. A strict cross-method ranking requires predictions evaluated at verified common positions with the same metric implementation. Rebuttal Table R4 provides a within-model sensitivity check.
Table R4. AdaCompose under original and released-label evaluation (%, ↑).
| Evaluation setting | Positions | Positive positions | ROC-AUC | AP | PR-AUC |
|---|---|---|---|---|---|
| Original decoder-aligned | 146,015 | 33,739 | 82.739 | 55.468 | 55.456 |
| Released labels, per-video prefix mapping | 143,920 | 33,160 | 82.819 | 55.561 | 55.549 |
The second row retains the first N frozen frame scores per video, with N = released I3D length × 16, and uses the released test order and ground-truth array. No retraining, checkpoint reselection, or score normalization is performed. This assumes correspondence of the retained frame indices. AP is average precision; PR-AUC is trapezoidal integration, as used by the released RTFM/MGFN evaluators.
W5. Table R5 reports three seeds for the VAD ablations under the same training and validation-selection rules. Full AdaCompose has higher mean AUC and AP than each ablation.
The original full-versus-type-only result improves AUC by 0.279 points but lowers AP by 1.917 points. Across three seeds, the paired differences are +0.956 ± 1.607 AUC and +1.515 ± 2.973 AP points (sample SD).
Table R5. Multi-seed VAD mechanism ablations (%, ↑).
| Variant | Existing AUC / AP | AUC (mean ± std) | AP (mean ± std) |
|---|---|---|---|
| Full AdaCompose | 82.739 / 55.468 | 81.574 ± 1.365 | 56.614 ± 1.053 |
| Type-only operator codes | 82.460 / 57.385 | 80.618 ± 1.696 | 55.099 ± 2.022 |
| Context-aware static factors | 81.953 / 56.584 | 79.119 ± 2.647 | 52.140 ± 3.870 |
| Without interaction feedback | 81.570 / 54.810 | 80.405 ± 1.146 | 54.889 ± 1.674 |
Mean ± sample SD over three seeds under the same training and validation-selection rules.
We have incorporated the corresponding clarifications and new experimental results into the revised manuscript.
Response
Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.
However, the new results also show that the benefits of dynamic operator generation and recursive composition are not consistent across tasks. In particular, the fixed-parameter baseline achieves comparable performance on ScienceQA, and recursion does not consistently improve the CZSL results. These findings leave me unconvinced about the advantages of the proposed mechanisms. Therefore, I will keep my original score.
We thank the reviewer for the thoughtful comments. We address the concerns and questions below.
W1 / Q1. We completed a context-matched fixed-parameter relation baseline. It retains the evidence encoder and context generator, but concatenates the context code to each fixed-weight relation-network layer; AdaCompose instead uses that code for low-rank weight modulation. Both models receive the same contextual inputs and training settings, with approximately matched parameter counts.
AdaCompose has lower error on Weierstrass and Hardy Z, while the fixed-parameter network achieves lower error on Julia (Table R1). This isolates a task-dependent benefit of dynamic parameterization, rather than establishing that dynamic weights are necessary whenever contextual information is available.
Table R1. Context-matched synthetic relation learning (nRMSE ↓, mean ± std).
R1 uses three seeds and population standard deviation, following the original reporting convention. DeepONet and AdaCompose retain their reported results. The fixed network is newly trained under the matched protocol.
W2 / Q2. We completed two retrained ablations under the reported dataset-specific staged training schedules. Each intervention starts at the first stage, and subsequent stages are initialized from the corresponding ablation checkpoint, with validation-based selection:
Without recursion: Retain the first attribute–object relation field, direct-pair branch, and marginal branch. Remove the composed evidence state and the second operator relating it to pair evidence. The fusion gate receives primitive summaries and the first operator code instead of recursive inputs.
Non-recursive replacement: Add a fixed-weight MLP receiving attribute, object, and pair local states, their summaries, and the first operator code. It approximately matches the original core's capacity, but constructs neither an intermediate evidence state nor a second operator. It uses the same non-recursive gate inputs as the first ablation.
On C-GQA, removing recursion lowers all four metrics, and AdaCompose also exceeds the capacity-matched replacement in the closed world. However, the replacement is better in the open world, and removing recursion slightly improves MIT-States. These retrained comparisons support a setting-dependent contribution of recursion.
Table R2. Retrained recursion ablation on CZSL (HM / AUC %, ↑).
The two ablations are retrained. The AdaCompose row retains the reported checkpoints. All use the reported dataset-specific training schedules and closed/open-world evaluation protocols.
Depth transfer: A separate completed probabilistic-parity experiment trains and selects checkpoints at depths 1 to 3 and tests unseen depths 4 to 5. Here, depth d denotes d binary compositions of d+1 Bernoulli sources. Both models reuse a recursive cell; only the relation-field parameterization differs. AdaCompose achieves RMSE 0.1042 ± 0.0899 / 0.1413 ± 0.1106 at depths 4/5, versus 0.2695 ± 0.1491 / 0.2903 ± 0.1556 for fixed-weight composition (three seeds, sample SD). It improves in every paired seed. Because seen-depth fitting also differs, this establishes transfer on the controlled task.