acceptodds
Under review as a conference paper at ICLR 2027

Controlled Acquisition and Abstention in Three-Channel Score Conflicts

Abstract

When audio, video, and text disagree, prediction accuracy alone does not reveal whether another source is worth acquiring or whether the system should withhold an answer. We separate these decisions in a controlled benchmark: a policy sees two signed scores, can request a third at a cost, and may abstain. The primary evaluation uses a mechanism-specific reward: abstention is correct only for one designated ambiguity mechanism and is scored as incorrect on mixed corruption. We call accuracy under this target targeted decision accuracy. Matched controls show that the threshold policy makes the same final decisions as an always-request policy while issuing fewer requests. Its advantage over always-answer fusion depends on rewarding abstention for one specified ambiguity pattern. Under the mechanism-conditioned target, the threshold policy reaches 0.789±0.006 targeted decision accuracy and 0.481±0.014 utility on the partially held-out synthetic split across 83 seeds. A zero-request majority reference that receives all three scores reaches 0.626±0.008 and 0.252±0.016; these results use different information budgets. A matched-budget test uses one terminal rule that allows the requested score to change the action. The train-only value selector improves utility over no-query and matched random at 10% and 25% budgets, but pair uncertainty has higher utility at every budget and the learned selector reduces utility at 50% and 63.7%. At those higher budgets, the learned selector has slightly higher non-ambiguous binary accuracy, so the utility result is not an unqualified accuracy ranking. Under the all-abstentions-error target, Threshold utility falls below majority, reversing their ordering. At the central temporal setting, full-trace controls match the neural models, while position perturbations separate them. On voice and face emotion clips from held-out actors, eight-frame fusion has opposite signed accuracy differences across two frozen encoder pairs; both actor intervals include zero. Matched-request routing gains are small and uncertain. Together, these results distinguish full-modality accuracy from pre-request selection value and show that the latter depends on the request budget and the observed-pair ranking.

Primary area:
datasets and benchmarks
Keywords:
multimodal conflict, active feature acquisition, abstention, selective prediction, matched-budget evaluation, audio-video-text scores

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XikWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper studies the decision of whether to acquire a third modality or make a prediction or abstain when only two modality scores are initially available. It introduces a synthetic benchmark with different score-corruption mechanisms and a specified utility function, and evaluates several heuristic and learned policies for acquisition and abstention. The paper also reports experiments on real multimodal datasets. The intended contribution is to study this decision problem in a controlled setting, but I found it difficult to understand what general insights the benchmark establishes and how the results translate to practical multimodal systems.

Critical strengths

The paper addresses a relevant problem: acquiring an additional modality can have a cost, so a multimodal system should ideally determine when the extra information is worth obtaining rather than always using every available modality.

The controlled benchmark also offers a potential way to compare acquisition and abstention policies under known conditions. However, the value of this contribution depends on whether the chosen formulation produces generalizable insights, which I do not think the current paper establishes sufficiently.

Critical weaknesses
  1. The motivation, formulation, and experimental design are difficult to follow and insufficiently justified.

I found it difficult to understand what a practitioner or researcher should take away from the proposed benchmark. Although the paper is motivated by conflicting modalities in multimodal systems, the central setup uses synthetic scalar scores and prescribed corruption mechanisms. The paper does not clearly establish why this abstraction captures an important aspect of the motivating problem, what specific scientific question it enables the authors to answer, or what general conclusions can be drawn from the resulting experiments.

The utility function is a particularly important example. The rewards for correct and incorrect predictions and the costs of acquisition and abstention directly determine the preferred behavior of each policy. However, the rationale for the chosen relative values is not sufficiently established, nor is it clear how robust the main findings are to alternative choices. The treatment of ambiguity-related abstention further makes the results dependent on the benchmark's definitions of different corruption types and their associated rewards.

Similar questions arise regarding the policy construction and comparisons. The threshold-based policy relies on hand-designed rules, while the learned selector uses a particular model and training target without establishing a convincing advantage over simpler alternatives. Moreover, the threshold policy and always-query selective policy make the same terminal decisions under the described rule, so the threshold policy's utility advantage primarily comes from saving acquisition costs rather than changing decisions through selective acquisition.

Taken together, these choices make it difficult to distinguish a general scientific contribution from an analysis whose outcomes are substantially determined by the constructed benchmark. This is a fundamental concern about the significance of the work, rather than something that could be resolved through clearer writing alone.

  1. The real-data experiments are difficult to assess and do not establish a convincing practical benefit.

The paper reports experiments on real multimodal datasets, but the experimental setup, evaluation protocols, and significance of the results are not explained clearly enough for me to understand how these experiments validate the central claims. The different datasets and representations also make it difficult to determine how the experiments relate to one another and to the synthetic benchmark.

More importantly, the reported results do not convincingly demonstrate a consistent, meaningful benefit from the proposed acquisition policies. Several results appear small or uncertain, and the experiments do not establish a robust advantage over simpler alternatives across realistic settings. Thus, the real-data experiments do not provide the evidence needed to bridge the gap between the synthetic formulation and the practical motivation.

Questions for authors
  1. What general scientific insight does the proposed synthetic benchmark establish about multimodal acquisition and abstention, beyond comparing policies under the specified corruption mechanisms and utility function? Why are the chosen reward and cost values appropriate, and how sensitive are the main conclusions to alternative utility functions and treatments of abstention?
  2. Could the authors clarify the protocols and main conclusions of the real-data experiments, including the representations, baselines, acquisition-budget comparisons, and uncertainty around the reported results? What evidence do these experiments provide that the proposed approach yields a meaningful practical benefit beyond the synthetic setting?
3 comments

Thank you for the detailed response and for clarifying the scope and limitations of the experiments. I appreciate the distinction between acquisition savings and improvements in final decision quality, as well as the acknowledgement that the real-data experiments do not establish consistent practical gains.

However, my main concerns remain unresolved. While the responses clarifies what the synthetic protocol measures, I am still not convinced that the experimental setup is sufficiently motivated to establish a meaningful contribution. The matched-action result appears to follow directly from the hand-designed decision rule, and the matched-budget experiments do not demonstrate an advantage over the simpler pair-uncertainty heuristic.

I also remain concerned about the presentation. I found it difficult to identify the paper's central scientific claim and understand how the different experimental settings collectively support it. The additional explanations help clarify the intended scope, but do not fully address these concerns.

I recognize the value of carefully reporting negative and conditional findings. However, I am not convinced that the current results establish sufficiently novel or generalizable insights beyond the specific constructed protocol. For these reasons, I will maintain my current overall rating.

Thank you for the detailed follow-up and for acknowledging the scope clarification and the reporting of negative results. We understand that your concerns about motivation, contribution, comparative evidence, and presentation remain, and that you will maintain your rating. We appreciate the careful assessment.

AuthorsOpenReview ↗

Scope of the evidence

The matched-action equality follows from the hand-designed terminal rule; we use it to isolate request savings, not as an algorithmic gain. The submitted matched-budget results also show learned value selection beating matched random at 10% and 25%, while pair uncertainty has higher utility at every tested budget. We therefore do not present learned value selection as the leading policy.

The supported result is narrower: request cost, value-based score selection, and the abstention target affect different parts of this controlled decision problem, and alternative abstention conventions change the utility ordering. The result is conditional on the specified score generator and reward rule. The media checks do not establish practical gains: the CMU-MOSEI video-group interval spans zero and falls below the registered 0.005 margin; CREMA-D effects change sign across encoder pairs with intervals spanning zero; AMI uses two teams. The current evidence supports a diagnostic decomposition, not generalization or a practical acquisition advantage.

Reviewer_MswAOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a synthetic decision protocol in which a binary latent label generates three signed scalar "audio/video/text" scores with Gaussian noise, one or more scores are corrupted by one of seven mechanisms (invert, swap, dropout, mixed, burst, ambiguity, clean), and a policy observes two scores, may pay 0.10 to request the third, and may abstain (cost 0.20 unless the mechanism is "ambiguity"). A hand-written threshold rule is compared with majority/median/weighted voting, two small MLPs, and a learned acquisition scorer, over 83 seeds. Under the authors' "targeted decision accuracy" the rule scores 0.789 vs 0.626 for majority; under plain binary accuracy it scores 0.721 vs 0.834. A matched-budget selector study, a temporal-trace extension, and several natural-media diagnostics are reported; in every natural-media case, the acquisition policy shows no reliable gain, and in some it is worse than a static baseline.

Critical strengths

None.

Critical weaknesses

-The writing is very hard to follow.

-The machine-learning contribution may be limited. No model, objective, or algorithm is proposed; the synthetic generator is six lines of Gaussian sampling; the policy is a two-threshold rule. The paper reads like a lab notebook, not a research paper suitable for ICLR.

Questions for authors

Q1. Why is the "mixed" mechanism charged for abstention while "ambiguity" is rewarded, given that the paper proves both are label-non-identifiable? What application motivates rewarding abstention for one and not the other?

Q2. Why are the majority/median/MLP/gated not given an abstain option and not trained with ambiguity examples, when the central comparison is about abstention?

2 comments

The central scientific result is conditional on the benchmark's score generator and reward rule. Mixed and ambiguity cases are both label-nonidentifiable, but the benchmark assigns the abstention target only to ambiguity and charges abstention on mixed corruption. When both mechanisms receive the target, Threshold utility is 0.826 ± 0.007 versus majority at 0.252 ± 0.016; when every abstention is scored as an error, Threshold utility is −0.064 ± 0.019 versus 0.252 ± 0.016. These comparisons show how the target convention changes the policy ordering; they do not establish a universal abstention rule.

The fusion references always answer with all three scores, whereas query policies start with two, so their comparison does not isolate acquisition value. The paper separately reports a matched-action test where Threshold and always-query-selective make the same terminal decisions, and a matched-budget test where the requested score can change the action. In the latter, pair uncertainty has higher utility than learned value selection at all four budgets.

The current results provide a controlled analysis of request savings, selection value, and reward semantics. The media studies use different representations and units and remain exploratory; they do not establish practical acquisition gains.

The paper studies how a policy should act when it initially observes two signed source scores and can request a third or abstain. The synthetic model makes this question testable with known corruption mechanisms and separate records of requests and terminal actions. The score is a signed scalar observation for one source, not a waveform or calibrated probability. The request cost of 0.10 and non-ambiguity abstention cost of 0.20 define the benchmark's utility rule; the study does not measure real acquisition latency, preprocessing expense, or user-facing cost.

The headline full-information references receive all three scores before acting, while query policies begin with two. The matched-action test shows Threshold and always-query-selective make the same final decisions, with Threshold paying for fewer requests. In the exact-budget test, a shared terminal rule lets the requested score change the action; learned value selection beats matched random at 10% and 25%, while pair uncertainty has higher utility at every budget.

The real-data protocols remain diagnostics. The CMU-MOSEI router margin is 0.000499 with a video-group 95% interval [−0.000439, 0.001908]. CREMA-D fusion effects change sign across encoder pairs and intervals span zero; the AMI replay uses two teams. The submission does not report implementations of ACO/AACO, GSMRL, A2MT, or SelectiveNet, so it does not establish a ranking against those methods or a practical advantage beyond the tested settings.

Reviewer_PTrAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • This paper studies acquisition and abstention in a controlled three-score setting. A policy observes two signed scores, can request a third score at a cost, and then answers or abstains. This paper evaluates a fixed Threshold policy, learned acquisition models, and several fusion references. It also includes matched-budget selector tests, temporal simulator checks, and exploratory studies on CMU-MOSEI, CREMA-D, and AMI.

  • The main findings are conditional. Threshold saves requests while preserving the final actions of 'always-query-selective '. Its advantage over majority voting depends on the mechanism-conditioned abstention target. Learned value selection improves over matched random at low budgets but has lower utility than pair uncertainty at every tested budget. The natural-media studies do not establish a reliable selective-acquisition gain.

Critical strengths
  • The separation of acquisition savings from final decision quality is useful. The matched-action identity between Threshold and always-query-selective provides an explanation of the request savings. The reward convention analysis also shows that the utility ordering can reverse. These controls help prevent acquisition savings from being confused with better prediction or generally better selective prediction.

  • The paper reports important results. The equal-information temporal controls expose the role of the simulator representation. The media studies also report encoder dependence and uncertainty.

Critical weaknesses
  • The scientific objective and the motivation for the specific formulation remain insufficiently explaned. The paper connects multimodal conflict resolution, cost-sensitive acquisition, and abstention, but does not clearly identify which central claim the experiments are intended to verify. Moreover, the abstract combines results from several protocols and tasks without clearly explaining how they support one contribution.

  • Many parts of the paper are difficult to follow. Eq. 1 appears to define homogeneous Gaussian scores with shared signal strength and noise scale, but I don't think the motivation is clearly stated. Section 3 does not adequately explain the statistical meaning of these scores, their relationship to actual modality outputs, or the rationale for this model and its parameters. A self-defined simulator does not necessarily require a citation for each equation. However, its assumptions need a stronger justification if the resulting observations are intended to inform actual multimodal systems.

  • One unresolved issue is the justification for this paper’s particular protocol: two initially available scores and one designated missing score with a fixed request cost. The paper does not clearly identify what resource the request consumes or what concrete availability process this represents.

  • The experiments do not show sufficient progress relative to the relevant acquisition and selective-prediction literature. The principal baselines are rules-based and small models defined within the paper. The experiments do not include implementations or controlled adaptations of directly relevant methods such as ACO/AACO, GSMRL, A2MT, or SelectiveNet. The MLP, Gated, and Learned acquisition baselines therefore provide limited evidence about the relationship to established approaches.

  • Table 1 also varies information availability and terminal-action rules. The full-information references receive all three scores without a request cost, while Threshold permits abstention under a target that specifically rewards it on ambiguity. The paper acknowledges these differences, but this means the headline comparison cannot establish superior acquisition.

  • The real-data studies do not resolve this gap. Using general multimodal datasets for an acquisition task is reasonable, but the availability assumptions, avoidable costs, and evaluation contract must be clearly defined. Section 5 does not provide a unified description of the datasets and tasks reported in Section 6, while essential details are included in Appendix A. These studies use different representations, targets, and statistical units. These results provide useful boundaries, but do not show practical acquisition benefits.

Questions for authors

Please refer to the points listed in Critical Weaknesses.

3 comments

The evaluated question is whether selecting a third signed source score improves a final decision under an explicit request and abstention rule. The synthetic benchmark uses scalar score abstractions, a request cost of 0.10 utility units, and a non-ambiguity abstention cost of 0.20; these values define the tested reward convention rather than a measured deployment cost. The paper reports sensitivity to alternative abstention rules and higher costs.

The full-information references receive all three scores without paying a request cost, while query policies begin with two, so Table 1 does not isolate acquisition value. The matched-action control shows that Threshold and always-query-selective make identical final decisions; Threshold's utility difference comes from fewer requests. In the matched-budget test, learned value selection improves over matched random at 10% and 25%, while pair uncertainty has higher utility at every tested budget. We have not run controlled implementations of ACO/AACO, GSMRL, A2MT, or SelectiveNet, so the current baseline suite does not establish a relative ranking against them.

The media checks are exploratory and do not establish a practical selective-acquisition gain. The CMU-MOSEI margin is 0.000499 with video-group 95% interval [−0.000439, 0.001908], below the registered 0.005 margin. CREMA-D effects change sign across encoder pairs and their intervals span zero; the AMI replay uses two teams. These protocols use different representations and units, so their results remain separate. The evidence supports separating request savings from decision quality and interpreting the synthetic and media results within their stated settings.

Reviewer_PTrAOpenReview ↗

After-rebuttal comments

Thank you for responding. It clarifies the scope and acknowledges the limitations of the comparisons. However, my main concerns about motivation, formulation, experiment, and presentation remain largely unresolved. I therefore maintain my rating of Weak rejection.

Thank you for the follow-up and for clarifying that our response did not resolve your concerns. We understand that you maintain your Weak Reject rating, and we appreciate the time you took to assess the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.