- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies the decision of whether to acquire a third modality or make a prediction or abstain when only two modality scores are initially available. It introduces a synthetic benchmark with different score-corruption mechanisms and a specified utility function, and evaluates several heuristic and learned policies for acquisition and abstention. The paper also reports experiments on real multimodal datasets. The intended contribution is to study this decision problem in a controlled setting, but I found it difficult to understand what general insights the benchmark establishes and how the results translate to practical multimodal systems.
Critical strengths
The paper addresses a relevant problem: acquiring an additional modality can have a cost, so a multimodal system should ideally determine when the extra information is worth obtaining rather than always using every available modality.
The controlled benchmark also offers a potential way to compare acquisition and abstention policies under known conditions. However, the value of this contribution depends on whether the chosen formulation produces generalizable insights, which I do not think the current paper establishes sufficiently.
Critical weaknesses
- The motivation, formulation, and experimental design are difficult to follow and insufficiently justified.
I found it difficult to understand what a practitioner or researcher should take away from the proposed benchmark. Although the paper is motivated by conflicting modalities in multimodal systems, the central setup uses synthetic scalar scores and prescribed corruption mechanisms. The paper does not clearly establish why this abstraction captures an important aspect of the motivating problem, what specific scientific question it enables the authors to answer, or what general conclusions can be drawn from the resulting experiments.
The utility function is a particularly important example. The rewards for correct and incorrect predictions and the costs of acquisition and abstention directly determine the preferred behavior of each policy. However, the rationale for the chosen relative values is not sufficiently established, nor is it clear how robust the main findings are to alternative choices. The treatment of ambiguity-related abstention further makes the results dependent on the benchmark's definitions of different corruption types and their associated rewards.
Similar questions arise regarding the policy construction and comparisons. The threshold-based policy relies on hand-designed rules, while the learned selector uses a particular model and training target without establishing a convincing advantage over simpler alternatives. Moreover, the threshold policy and always-query selective policy make the same terminal decisions under the described rule, so the threshold policy's utility advantage primarily comes from saving acquisition costs rather than changing decisions through selective acquisition.
Taken together, these choices make it difficult to distinguish a general scientific contribution from an analysis whose outcomes are substantially determined by the constructed benchmark. This is a fundamental concern about the significance of the work, rather than something that could be resolved through clearer writing alone.
- The real-data experiments are difficult to assess and do not establish a convincing practical benefit.
The paper reports experiments on real multimodal datasets, but the experimental setup, evaluation protocols, and significance of the results are not explained clearly enough for me to understand how these experiments validate the central claims. The different datasets and representations also make it difficult to determine how the experiments relate to one another and to the synthetic benchmark.
More importantly, the reported results do not convincingly demonstrate a consistent, meaningful benefit from the proposed acquisition policies. Several results appear small or uncertain, and the experiments do not establish a robust advantage over simpler alternatives across realistic settings. Thus, the real-data experiments do not provide the evidence needed to bridge the gap between the synthetic formulation and the practical motivation.
Questions for authors
- What general scientific insight does the proposed synthetic benchmark establish about multimodal acquisition and abstention, beyond comparing policies under the specified corruption mechanisms and utility function? Why are the chosen reward and cost values appropriate, and how sensitive are the main conclusions to alternative utility functions and treatments of abstention?
- Could the authors clarify the protocols and main conclusions of the real-data experiments, including the representations, baselines, acquisition-budget comparisons, and uncertainty around the reported results? What evidence do these experiments provide that the proposed approach yields a meaningful practical benefit beyond the synthetic setting?
3 comments
Thank you for the detailed follow-up and for acknowledging the scope clarification and the reporting of negative results. We understand that your concerns about motivation, contribution, comparative evidence, and presentation remain, and that you will maintain your rating. We appreciate the careful assessment.
Scope of the evidence
The matched-action equality follows from the hand-designed terminal rule; we use it to isolate request savings, not as an algorithmic gain. The submitted matched-budget results also show learned value selection beating matched random at 10% and 25%, while pair uncertainty has higher utility at every tested budget. We therefore do not present learned value selection as the leading policy.
The supported result is narrower: request cost, value-based score selection, and the abstention target affect different parts of this controlled decision problem, and alternative abstention conventions change the utility ordering. The result is conditional on the specified score generator and reward rule. The media checks do not establish practical gains: the CMU-MOSEI video-group interval spans zero and falls below the registered 0.005 margin; CREMA-D effects change sign across encoder pairs with intervals spanning zero; AMI uses two teams. The current evidence supports a diagnostic decomposition, not generalization or a practical acquisition advantage.
Thank you for the detailed response and for clarifying the scope and limitations of the experiments. I appreciate the distinction between acquisition savings and improvements in final decision quality, as well as the acknowledgement that the real-data experiments do not establish consistent practical gains.
However, my main concerns remain unresolved. While the responses clarifies what the synthetic protocol measures, I am still not convinced that the experimental setup is sufficiently motivated to establish a meaningful contribution. The matched-action result appears to follow directly from the hand-designed decision rule, and the matched-budget experiments do not demonstrate an advantage over the simpler pair-uncertainty heuristic.
I also remain concerned about the presentation. I found it difficult to identify the paper's central scientific claim and understand how the different experimental settings collectively support it. The additional explanations help clarify the intended scope, but do not fully address these concerns.
I recognize the value of carefully reporting negative and conditional findings. However, I am not convinced that the current results establish sufficiently novel or generalizable insights beyond the specific constructed protocol. For these reasons, I will maintain my current overall rating.