acceptodds
Under review as a conference paper at ICLR 2027

Efficient Multimodal Inference through Adaptive Acquisition and Sequential Fusion

Abstract

Multimodal systems often encode every available input, even when a subset suffices for prediction. Adaptive acquisition can reduce this cost by using predictions from incrementally fused evidence to *decide which modality to encode next and when to stop*. However, sequential fusion makes these predictions order-dependent, so decisions based on them may need to distinguish factorially many histories of the same acquired set. We introduce SeMARC, which couples a Sequential Modality Aggregator (SeMA) with an Adaptive Runtime Controller (ARC) and uses acquired evidence to select each modality *before* its encoder runs. SeMA executes only selected encoder and fusion branches, updates a fixed-size state, and predicts after each acquisition without recomputing earlier branches. We supervise every acquisition prefix under randomized modality subsets and orders to encourage consistent predictions across acquisition orders. ARC combines a set-dependent marginal-utility prior with residual fitted-Q learning to select the next available modality or stop, without inspecting unacquired inputs or retaining acquisition order. Across six multimodal classification datasets and eleven baselines, SeMARC achieves 3.2% higher macro-F1 and 61.4% lower total inference GFLOPs on average relative to each dataset's most accurate baseline. End-to-end latency falls by 44.0% across GPU and CPU and by 47.2% on Android INT8 relative to the fastest measured baseline, on average. Under varying runtime modality missingness, SeMARC still skips available modalities, matching or exceeding the best baseline macro-F1 in 21 of 24 conditions with 14.8% lower total GFLOPs on average. SeMARC thus offers a practical path toward efficient multimodal inference across heterogeneous devices.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal learning, adaptive modality acquisition, sequential fusion, efficient inference, reinforcement learning, missing-modality robustness

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_GTKwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work introduces SEMARC, an framework for efficient multimodal machine learning inference which incrementally fuses available data and selects on when to acquire additional evidence or halt processing. Empirical evaluations across multiple benchmarks demonstrate that SEMARC reduces inference GFLOPs, latency, and energy consumption while maintaining or improving predictive accuracy.

Critical strengths
  • Placing the decision to include new evidence before the feature extractors gives significant energy and latency savings
  • The order invariant training of SEMA gives an empirical justification for ARC's compact state representation thereby keeping the action-value space tractable
  • Great hardware and robustness empirical evaluation scope.
Critical weaknesses
  • Table 2 shows that pairing existing controllers with SEMA (+ Joint-AFA: .75 F1; + EAMA: .74 F1) give similar task performance to ARC (0.77 F1) with lower latency in some cases (EAMA). The accuracy edge then is mostly from SEMA not ARC and this should be noted. It would be good to see pareto frontiers for all controllers on the shared backbone if not cost prohibitive.
  • Section 3.2 seems to overstate the theoretical backing. Corrollary 3.2 assumes small error terms for the compact state, but they arent established by the prior proposition. Though this is acknowledged later in the appendix, the characterization in the main body of this section is most accurately a formal intuition rather than a guarantee.
  • The server power measurements include idle draw but android phones measurements do not. This should be consistent across both either as latency adjusted power draw or unadjusted power draw.
  • λds\lambda_{ds} and λkd\lambda_{kd} are only ablated together. All ablations are limited to IEMOCAP, with some components falling within error bounds of each other. Could use a second dataset to verify more concretely eg. CMI.

Minor Issues/Formatting:

  • ln 265, 301-302: the double struck 1 (indicator fn) is rendered incorrectly
  • ln 229: Figure 4's placement inline is distracting and contains multiple text sizes with strange placements. Making the plot larger and text formatting more consistent may help
  • figure 5 is very cluttered in its current state. referencing a table for the specific results and just showing the comparison without the magnified areas may be more clear.
  • Figure 8,9,10 has a similar issue to figure 4's text sizing and placement. Difficult to interpret due to clutter.
  • Figure 7 appears after figures 8,9,10
Questions for authors
  • How are empty or single-sample (S, y) cells in TST_S handled?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GTKw [1/3]

Thank you for taking the time to review our work and for your constructive feedback to improve the paper. We appreciate your recognition of the significant energy and latency savings enabled by acquisition decisions before feature extraction and the empirical justification that SeMA's order-robust training provides for ARC's compact state representation, keeping the action-value space tractable. We also appreciate your praise for our hardware and robustness empirical evaluation breadth. Below, we address all your questions and concerns point by point.

W1 — Table 2 shows similar F1 with SeMA + Joint-AFA (0.75), SeMA + EAMA (0.74), and ARC (0.77), with EAMA faster in some cases. Can you acknowledge that the accuracy edge is mostly from SeMA and show Pareto frontiers for all controllers on the shared backbone, if not cost prohibitive?

Thank you for highlighting this distinction. Our paper already recognizes SeMA’s strong predictive foundation. Figure 6 (Section 4.1) compares SeMA with SeMA+ARC (SeMARC) on IEMOCAP. Across 0–40% modality missingness, adding ARC reduces post-extraction GFLOPs by 34.9–50.0% and total GFLOPs by 50.5–63.9%, with a decrease of at most 0.03 absolute macro-F1 (Appendix G). As stated in Section 4.1: "This supports our premise that sample-adaptive acquisition can substantially reduce computation while largely preserving the backbone’s predictive quality.”

To make this attribution explicit, we have added the following sentence to Section 4.2 (Component replacements): “These results support our coupled design: SeMA provides the shared predictive foundation, while ARC adds adaptive acquisition and stopping to improve the accuracy–cost trade-off.”

Additionally, we have also added new controller sweeps on the shared frozen SeMA backbone (Appendix H.1, Figure 12). The key observations are:1) Higher accuracy at comparable latency. Within a 400-ms mean-latency budget, SeMARC achieves 0.76 macro-F1 at 393.4 ms, compared with 0.73 at 388.5 ms for SeMA+EAMA—an improvement of approximately 3 percentage points. 2) Higher accuracy with lower latency. SeMARC achieves 0.76 macro-F1 at 355.5 ms, exceeding EAMA’s highest-F1 setting in this sweep (0.74 at 416.5 ms) with 14.6% lower latency. Every displayed Joint-AFA point is also dominated by a SeMARC point. 3) Overall SeMARC offers a favorable accuracy–latency trade-off. EAMA offers faster settings with lower macro-F1. GMS’s highest-F1 setting in this sweep exceeds SeMARC’s by approximately 1 percentage points (0.78 versus 0.77), but requires 2.55 times the latency (1327.5 versus 520.2 ms).

Quick-reference table: best macro-F1 under a latency budget

Entries are macro-F1, with equal-weight means across six datasets. Each row selects the highest F1 among the supplied settings meeting that mean-latency budget.

Mean latency budget (ms)SeMARCSeMA + EAMASeMA + Joint-AFASeMA + GMS
≤ 350—0.67 (342.2)——
≤ 4000.76 (393.4)0.73 (388.5)0.67 (383.9)—
≤ 6000.77 (520.2)0.74 (416.5)0.74 (516.9)—
≤ 1,0000.77 (520.2)0.74 (416.5)0.74 (516.9)0.73 (772.1)
≤ 1,5000.77 (520.2)0.74 (416.5)0.76 (1459.0)0.78 (1327.5)

— means that no supplied setting meets the budget.

W2 — Does Section 3.2 overstate the theoretical backing, since Corollary 3.2 assumes small compact-state errors not established by the preceding proposition? Although acknowledged in the appendix, should the main text present this as formal intuition rather than a guarantee?

Thank you. As you note, our current appendix already clarifies that prediction-level order robustness motivates our compact-state design, while the control guarantee requires additional assumptions. Specifically, Appendix K.1 states: " These results concern average predictive distributions, not identical hidden states or AIS transition sufficiency."

To make this distinction equally explicit in the main text, we have updated the Implication paragraph in Section 3.2, following Corollary 3.2, to state: "The AIS control bound remains conditional on additional uniform reward and transition approximation assumptions beyond prediction-level order robustness."

Additionally, Appendix K.4 (Table 38) reports a new acquisition-order ablation. On IEMOCAP, appending a position-specific one-hot encoding of acquisition order to ARC’s compact state yields no consistent improvement in macro-F1 or return, supporting the omission of explicit order in this setting.

AuthorsOpenReview ↗

Response to Reviewer GTKw [2/3]

W3 — Why does server power include idle draw while Android excludes it, and can both use consistent adjusted or unadjusted accounting?

Thank you for highlighting this. CPU, GPU, and Android energy measurements all exclude idle consumption. We have updated the Table 22 caption and the measurement descriptions in Appendices D and F to make this common convention explicit across all platforms.

W4 — a) Can λds\lambda_{ds} and λkd\lambda_{kd} be ablated separately? b) Given that all ablations use IEMOCAP and some component results fall within one another's error bounds, could a second dataset, e.g. CMI, verify the effects more concretely?

Thank you. Following your suggestion, we expanded the ablations beyond IEMOCAP to CMI, EAV, and MM-Fi, and added separate ablations of prefix CE and distillation. Results follow the manuscript’s ablation protocol.

a) We remove prefix CE (λ_{ds}), distillation (λ_{kd}), or both, retaining final retained-prefix CE throughout. Each backbone is retrained and ARC is refitted.

λ_{ds}λ_{kd}IEMOCAPCMIEAVMM-Fi
1.000.500.67_{±0.02}0.56_{±0.01}0.69_{±0.02}0.95_{±0.04}
0.000.500.67_{±0.05}0.53_{±0.02}0.66_{±0.02}0.93_{±0.06}
1.000.000.67_{±0.04}0.56_{±0.01}0.69_{±0.04}0.91_{±0.11}
0.000.000.65_{±0.01}0.54_{±0.01}0.66_{±0.04}0.89_{±0.10}

Using values before display rounding, the combined loss improves mean F1 over final-prefix CE by 1.40–6.10 percentage points across the four datasets. Prefix CE benefits CMI, EAV, and MM-Fi; distillation provides its largest mean F1 benefit on MM-Fi (+4.20 points) and lowers acquisition cost at similar mean F1 on IEMOCAP and CMI.

b) Below are all non-loss variants; CMI loss results appear above. GFf\mathrm{GF}_{\mathrm f} denotes post-extraction GFLOPs and MU denotes modality utilization.

CMI variantMacro-F1 ↑GFf\mathrm{GF}_{\mathrm f} ↓MU ↓
SeMARC (reference)0.56_{±0.01}0.430.75
Fixed modality order0.55_{±0.01}0.530.91
No dropout curriculum0.53_{±0.04}0.450.78
Bottleneck tokens last0.56_{±0.01}0.450.78
Learnable aggregation0.55_{±0.01}0.410.71
Shapley + confidence exit0.56_{±0.01}0.420.74
No Q-prior0.50_{±0.05}0.220.38
No STOP (acquire all)0.57_{±0.01}0.581.00
Confidence STOP0.56_{±0.01}0.430.75
Masks-only residual input0.56_{±0.01}0.550.95
Posterior-only residual input0.55_{±0.01}0.350.61

These results support the coupled design through prediction quality and acquisition cost. The combined prefix losses and randomized-order training improve mean F1 on all four datasets. On CMI, disabling STOP gains 1.00 F1 percentage point while increasing post-extraction GFLOPs from 0.43 to 0.58 and MU from 0.75 to 1.00. Static Shapley remains competitive on CMI, and other component effects are dataset-dependent. We interpret small mean differences cautiously; fold standard deviations alone do not establish statistical significance.

Section 4.2 and Appendix I (new Tables 30–33) provide the complete results and discussion. The updated Figure 7 summarizes all four datasets using equal-weight mean relative F1 changes and observed minimum–maximum ranges.

Question for authors

Q1 — How are empty or single-sample (S,y)(S, y) cells in TST_S handled?

In our implementation, if a class has no fitting samples, its corresponding gain entries in T_{S} are left at zero. With one sample, we use that sample’s marginal gain, computed from correctness averaged over the cached orders of each subset. With multiple samples, we average their marginal gains.

In practice, our fitted-cohort is stratified and contains multiple samples from every class. As described in Equation 6 and Appendix A, each (S,y) cell uses all fitting samples of class y, with predictions cached for every ordered modality prefix. Thus, neither empty nor single-sample cells occur in the current datasets.

AuthorsOpenReview ↗

Response to Reviewer GTKw [3/3]

Minor writing and formatting comments

W5 — Can the incorrectly rendered double-struck 1 (indicator function) at lines 265 and 301–302 be corrected?

We have corrected the indicator-function rendering. Please let us know if it still does not display correctly.

W6 — Can Figure 4’s distracting inline placement near line 229 and inconsistent text sizes and positions be improved, e.g., with a larger plot and consistent formatting?

We have corrected the placement and formatting of Figure 4.

W7 — Can Figure 5’s clutter be reduced by tabulating specific results and showing the comparison without magnified regions? W8 — Can Figures 8–10 be made easier to interpret by fixing inconsistent text sizing, placement, and clutter?

We have updated Figures 5, 8, 9, and 10 to improve readability. We appreciate your suggestions and welcome any further feedback on the revised figures.

W9 — Why does Figure 7 appear after Figures 8–10?

We have corrected the figure order.

Reviewer_GTKwOpenReview ↗

Concerns addressed

Thank you for the rebuttal. I have raised my score to a 3.

Reviewer_RsLFOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SeMARC, which couples a Sequential Modality Aggregator (SeMA) with an Adaptive Runtime Controller (ARC) and uses acquired evidence to select each modality before its encoder runs. Under varying runtime modality missingness, SeMARC still skips available modalities, matching or exceeding the best baseline macro-F1 in 21 of 24 conditions with 14.8% lower total GFLOPs on average. SeMARC thus offers a practical path toward efficient multimodal inference across heterogeneous devices.

Critical strengths

The author combines random modality sorting, modality dropout training, intermediate prediction supervision, and distillation, and provides a theoretical analysis between prediction risk and order divergence, making the method design have a clearer motivation and analytical basis.

Critical weaknesses

The selection of the first modality lacks sample-level adaptability: SeMARC selects the first modality based on offline estimated overall Shapley values, rather than dynamically deciding based on information from the current sample, which may result in the first encoding not prioritizing the modality most valuable for the specific sample.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RsLF

Thank you for taking the time to review our work. We appreciate your recognition that our training components and the analysis linking prediction risk to order disagreement give SeMARC a clearer motivation and analytical basis. Below, we address your question about the initial modality selection.

W1 — Why is the first modality chosen using offline-estimated population Shapley values rather than current-sample information?

The first modality is selected from each sample's available modality set, using the offline-estimated population Shapley values described in Section 3.3. Before this acquisition, our pipeline has no content-derived predictive summary of the sample. The population-level estimate provides an initial choice without requiring a preliminary feature-extraction stage. After the first acquisition, ARC uses the sample's predictive summary and availability to adapt subsequent acquisition and stopping decisions.

Figure 9 (Section 4.2) demonstrates this availability-aware fallback: successively removing the preceding first choice causes SeMARC to select an alternative from the remaining modalities. SeMARC maintains macro-F1 close to SeMA processing all remaining inputs while leaving approximately 35.00–55.00% of available modalities unacquired. Figure 8 complements this analysis by showing that overall acquisition frequencies differ across classes. These results support availability-aware initialization and subsequent evidence-dependent acquisition.

Thank you for rebuttal. My main concerns are addressed, and I will keep my score.

Reviewer_LtYwOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SeMARC, a framework for efficient multimodal classification through adaptive modality acquisition and sequential fusion. The central idea is to selectively process modalities based on the evidence already acquired, thereby avoiding unnecessary computation from expensive modality encoders.

The framework combines two components. SeMA is a sequential modality aggregator that incrementally integrates newly acquired modalities through bottleneck attention, without recomputing previously processed branches. It is trained using randomized modality subsets and orders, together with intermediate prediction supervision and distillation, to encourage order-robust predictions. ARC is an adaptive runtime controller that uses the current prediction and acquired-modality information to decide which modality to process next or when to stop. It combines a class-conditional marginal-utility prior with residual fitted-Q learning.

The authors evaluate SeMARC on six multimodal classification datasets, comparing it against eleven baselines. They report competitive or improved classification performance with substantial reductions in inference computation, latency, and energy consumption. Additional experiments investigate missing-modality robustness, component contributions, modality acquisition behavior, and accuracy–efficiency trade-offs.

Critical strengths
  1. Well-motivated problem with clear practical significance.**

I appreciate the central idea of trading additional offline training and optimization effort for substantially more efficient inference. This is particularly relevant for multimodal systems, where processing every available modality can be computationally expensive and unnecessary for individual samples. The proposed approach directly addresses this problem by selecting modalities before running their expensive encoders, rather than selecting among representations that have already been computed.

  1. Coherent methodological design.**

The combination of SeMA and ARC is well motivated. SeMA enables incremental fusion without recomputing previous modality branches, while randomized-order training and intermediate supervision improve the reliability of predictions from partial modality subsets. ARC then uses these intermediate predictions to guide further acquisition and stopping decisions. Although the individual building blocks have precedents in the literature, I find their integration logical and technically meaningful. The components address complementary challenges and form a coherent framework.

  1. Comprehensive and convincing experimental validation.**

The evaluation is a major strength of the manuscript. Across six datasets, SeMARC achieves competitive or improved macro-F1 while substantially reducing inference computation. Importantly, the authors do not rely solely on theoretical GFLOPs. Figure 5 and the supplementary experiments demonstrate actual latency and energy savings across GPU, CPU, and Android platforms, supporting the practical benefits of the approach.

The experiments under missing-modality conditions are also valuable. In particular, comparisons with SeMA processing all available modalities demonstrate that the computational savings arise not only from missing inputs but also from deliberately skipping modalities that are available.

  1. Meaningful component analysis and exploration of practical trade-offs.**

Table 2 provides useful comparisons against alternative acquisition controllers using the same SeMA backbone, while Figure 7 examines the contributions of the main architectural and training choices. These experiments support the proposed design rather than relying exclusively on end-to-end comparisons.

I also appreciate the cost-sensitivity analysis in Figure 10 and Appendix J. The authors recognize that the optimal acquisition strategy depends on the computational cost of individual modalities. The results show that cost-aware acquisition can provide substantial additional acceleration, sometimes with little loss in classification performance. This demonstrates an understanding of the practical potential and limitations of the framework.

Critical weaknesses

I do not find any critical weakness of the paper. Yet I do have two questions for authors, which is not the limitation of current manuscript but could be potential future research directions.

Questions for authors
  1. Offline computational cost and scalability: Could the authors provide a rough estimate of the computational time and memory required to construct the ordered-prefix cache, in addition to the reported controller-fitting time? Given the factorial growth in the number of prefixes, would sampling a subset of acquisition orders be a feasible alternative when the number of modalities becomes larger?

  2. Efficiency comparison and selection timing: Could the authors clarify to what extent the large end-to-end computational improvements over selective-fusion baselines, particularly AdaMML, depend on the baseline implementations extracting modality features before selection? A brief discussion of whether these baselines could avoid part of this cost under alternative implementations would help contextualize the reported efficiency improvements.

2 comments
AuthorsOpenReview ↗

Response to Reviewer LtYw

Q1 — a) What time and memory are needed to construct the ordered-prefix cache, separately from controller fitting? b) Could sub-sampling acquisition orders address factorial growth for larger modality sets?

a) We have added cache-construction and controller-fitting profiles to Appendix A.2 (Table 6) and also reported them below for your quick reference.

DatasetCache construction (s)ARC fitting (s)Cache peak host memory (GiB)Cache GPU memory (GiB)
IEMOCAP59.909.5411.221.38
EAV11.900.567.381.21
MM-Fi10.500.769.601.10
CZU-MHAD81.2027.642.651.32
UTD-MHAD2.300.623.200.85
CMI249.6087.455.602.24

Cache construction is the mean validation-cache build time per fold. ARC fitting is the mean controller-fitting time per fit, including state preparation and holdout selection. Cache peak host memory and cache GPU memory report mean peak host-process memory and mean GPU memory, respectively, measured during complete cache construction.

b) Thank you for this question. We acknowledge the factorial growth of the offline cache in Section 3.2 and Appendix A. We have added a new preliminary IEMOCAP cache-subsampling study in Appendix A.1: retaining 20.00% of random cache entries reduces macro-F1 by 0.30 percentage points while preserving all 63 nonempty acquired sets. These findings encourage coverage-preserving subsampling, although actual cache-construction time and memory savings remain to be evaluated.

Our contribution targets per-sample latency and energy during deployment. The cache is constructed offline once for a fixed backbone and data split and reused across controller fits. Developing budgeted cache construction requires further investigation of coverage, prior estimation, and controller performance at larger modality counts. We therefore explicitly identify this extension in Section 5, Limitations and future work.

Q2 — How much do efficiency gains over selective-fusion baselines, especially AdaMML, depend on feature extraction before selection, and could alternative implementations avoid some of that cost?

Thank you for this helpful question. Much of the large end-to-end compute reduction comes from avoiding expensive feature extraction. In our evaluated baseline adaptations, selection operates on common per-modality representations (Appendix C.1), so extraction costs are incurred before downstream encoders can be skipped. Table 1 separates total and post-extraction GFLOPs:

DatasetAdaMML: total (post-extraction) GFLOPsSeMARC: total (post-extraction) GFLOPsApproximate total-GFLOPs reduction
IEMOCAP6,696.00 (0.20)3,311.00 (0.28)50.55%
MM-Fi18,098.00 (0.17)738.00 (0.07)95.92%
CMI0.51 (0.51)0.43 (0.43)15.69%
CZU-MHAD6,091.00 (0.17)6,091.00 (0.23)0.00%
EAV6,230.00 (0.17)719.00 (0.10)88.46%
UTD-MHAD7,306.00 (0.19)4,706.00 (0.19)35.59%

On IEMOCAP, SeMARC reduces total computation despite slightly higher post-extraction cost, illustrating the benefit of skipping extraction for unselected modalities. MM-Fi shows larger savings with expensive extractors, whereas CMI’s smaller reduction comes entirely from downstream processing. CZU-MHAD is an exception: SeMARC always acquires the dominant depth stream, and its total GFLOPs match AdaMML’s. Thus, the magnitude of the gains depends on both extraction cost and selection timing.

Alternative implementations using low-cost previews directly from raw inputs could reduce the front-end extraction cost and narrow the measured gap. Their accuracy–cost trade-off would require end-to-end evaluation including preview computation. SeMARC reuses the predictive summary of already-acquired evidence for subsequent acquisition and stopping decisions, without computing a separate preview of unacquired inputs.

Reviewer_LtYwOpenReview ↗

Thank you for the additional experiment results

I would like to thank the authors for providing additional experimental results, which are super helpful to the manuscript's value. As for the score, I already gave the top score in the initial review. I really like the paper.

Reviewer_D6UEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SeMARC for efficient multimodal inference. SeMARC decides which modality to process next and when to stop. It contains a sequential fusion backbone called SeMA and a controller called ARC. SeMA updates the prediction as new modalities are acquired, while ARC selects the next modality based on the current state and prediction. Since the decision is made before running the modality encoder, unused modalities can be skipped. Experiments on six multimodal classification datasets show that SeMARC achieves competitive accuracy while reducing computation and inference cost, including latency and energy consumption.

Critical strengths
  1. The paper addresses a practically relevant problem, and the design allows modality selection to happen before expensive encoders are executed, leading to real end-to-end computational savings.

  2. The experimental evaluation is comprehensive, covering six datasets and a wide range of baselines, with additional latency and energy measurements on GPU, CPU, and Android.

  3. The ablation studies are thorough and provide good support for the main design choices, including randomized-order training, modality dropout, prefix supervision, and learned stopping.

Critical weaknesses
  1. The scalability of ARC training is unclear, since it relies on caching all ordered modality prefixes and this cost grows factorially with the number of modalities. The authors should discuss how the method could be extended to settings with substantially more modalities, or whether approximate prefix sampling could be used.

  2. The comparison with adaptive acquisition baselines could be more complete, since some competing methods achieve lower latency but also lower accuracy. The authors should provide matched-latency or matched-accuracy comparisons, or full accuracy–latency Pareto curves, to better characterize the trade-off.

  3. The theoretical analysis is useful for motivating the design, but the guarantees rely on assumptions that are not verified empirically. The authors should clarify how well the learned controller state satisfies the approximate Markov assumptions in practice, or provide empirical evidence for these assumptions.

Questions for authors

See the weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer D6UE [1/2]

Thank you for taking the time to review our work and for your constructive feedback to improve the paper. We appreciate your recognition of the practical relevance of our work and the real end-to-end computational savings enabled by selecting modalities before expensive encoders. We also appreciate your positive assessment of our comprehensive evaluation across six datasets and a wide range of baselines, including GPU, CPU, and Android latency and energy measurements, and our thorough ablations supporting the main design choices. Below, we address all your questions and concerns point by point.

W1 — How can ARC training scale to substantially more modalities given that caching all ordered modality prefixes grows factorially, and could approximate prefix sampling help?

We agree that the current offline cache grows factorially with the number of modalities, as described in Section 3.2 and Appendix A in the construction of the Q-prior.

Following your suggestion, we conducted a preliminary additional study on IEMOCAP, added in Appendix A.1. We retain approximately 80%, 50%, or 20% of cached observations using two schemes: random entries, indexed by ordered prefix, example; and whole acquired sets, removing every order and example associated with selected sets. We retrain ARC with retained observations, keeping the frozen SeMA, controller settings, and acquisition price fixed. The table below (also included as Table 5 in Appendix A.1), reports performance under these reduced training budgets (MU is the mean fraction of available modalities acquired).

Retention schemeEntries retainedMacro-F1MU
Full cache100%0.670.66
Random entries80%0.670.67
Random entries50%0.650.64
Random entries20%0.670.64
Whole acquired sets80.10%0.650.57
Whole acquired sets49.90%0.660.78
Whole acquired sets19.80%0.610.69

These results suggest some redundancy in cached observations and potential for coverage-preserving subsampling. At 20% random retention, all 63 nonempty acquired sets remain represented, with a paired macro-F1 difference of −0.3 percentage points relative to the full cache [95.00% interval: −1.50, +0.80 percentage points]. Mean MU is 0.64, compared with 0.66 for the full cache. Removing whole acquired sets at approximately 20.00% retention reduces macro-F1 to 0.61 while also removing prior support. Since this study subsamples existing cached observations, actual budgeted cache construction and its potential cost savings remain to be investigated in future.

We also provide a new comprehensive profiling of cache construction and ARC fitting, reported separately in Appendix A.2 (Table 6).

Our focus in this work is efficient inference and runtime adaptation to modality availability, enabling deployment of multimodal models on resource-constrained devices. Scaling offline cache construction remains future work, and we have explicitly included this direction in Section 5, Limitations and future work as “…and developing budgeted cache construction schemes to improve offline scalability as the number of modalities grows.”

AuthorsOpenReview ↗

Response to Reviewer D6UE [2/2]

W2 — Can you provide matched-latency or matched-accuracy comparisons, or full accuracy–latency Pareto curves, given that some adaptive-acquisition baselines are faster but less accurate?

Thank you for this suggestion. Since the controllers expose different cost and budget controls, their operating points do not generally yield identical accuracy or latency. We therefore provide accuracy–latency sweeps on the shared frozen SeMA backbone in Appendix H.1, Figure 12. We vary the modality-cost weighting γ\gamma for ARC and EAMA, the per-acquisition penalty rcostr_{\mathrm{cost}} for Joint-AFA, and the modality-count budget kk for GMS.

The key observations from the newly added pareto sweeps are : 1) Higher accuracy at comparable latency. Within a 400-ms mean-latency budget, SeMARC achieves 0.76 macro-F1 at 393.4 ms, compared with 0.73 at 388.5 ms for SeMA+EAMA—an improvement of approximately 3 percentage points. 2) Higher accuracy with lower latency. SeMARC achieves 0.76 macro-F1 at 355.5 ms, exceeding EAMA’s highest-F1 setting in this sweep (0.74 at 416.5 ms) with 14.6% lower latency. Every displayed Joint-AFA point is also dominated by a SeMARC point. 3) Overall SeMARC offers a favorable accuracy–latency trade-off. EAMA offers faster settings with lower macro-F1. GMS’s highest-F1 setting in this sweep exceeds SeMARC’s by approximately 1 percentage points (0.78 versus 0.77), but requires 2.55 times the latency (1327.5 versus 520.2 ms).

Quick-reference table: best macro-F1 under a latency budget

Entries are macro-F1, with equal-weight means across six datasets. Each row selects the highest F1 among the supplied settings meeting that mean-latency budget.

Mean latency budget (ms)SeMARCSeMA + EAMASeMA + Joint-AFASeMA + GMS
≤ 350—0.67 (342.2)——
≤ 4000.76 (393.4)0.73 (388.5)0.67 (383.9)—
≤ 6000.77 (520.2)0.74 (416.5)0.74 (516.9)—
≤ 1,0000.77 (520.2)0.74 (416.5)0.74 (516.9)0.73 (772.1)
≤ 1,5000.77 (520.2)0.74 (416.5)0.76 (1459.0)0.78 (1327.5)

— means that no supplied setting meets the budget.

W3 — Could the authors clarify how well the learned controller state satisfies the approximate Markov assumptions in practice, or provide empirical evidence for these assumptions?

Thank you for this suggestion. We have added a new controlled acquisition-order experiment in Appendix K.4 (Table 38) to test whether explicitly retaining order improves ARC’s acquisition and stopping decisions. On IEMOCAP, we append a position-specific one-hot encoding of the acquisition sequence to the compact state and compare the variants across three folds and three paired fitting seeds. We hold the frozen SeMA, caches, splits, prior, reward, acquisition price, availability draws, and fitting schedule fixed, and include a wider compact controller with comparable parameter count.

Adding explicit order yields no consistent improvement in macro-F1 or return. Macro-F1 changes by −0.33, −0.23, and +0.10 percentage points under 0.00%, 20.00%, and 40.00% missingness, respectively; mean modality utilization changes by at most 0.52 percentage points.

We acknowledge that this experiment assesses one source of history dependence; the uniform reward and transition approximation conditions remain assumptions of the control bound. We have made this distinction explicit in the Implication paragraph in Section 3.2, following Corollary 3.2 as "The AIS control bound remains conditional on additional uniform reward and transition approximation assumptions beyond prediction-level order robustness."

Proposition 3.1 and Corollary 3.2 provide theoretical motivation for our design, with the corollary specifying conditions under which the compact state supports effective control. We make this conditional scope explicit, without claiming that the learned state satisfies the required reward and transition approximation assumptions. The new experiment complements this analysis with empirical evidence supporting the omission of explicit acquisition order in the evaluated setting.

Thank you for your response. Your explanations have resolved my concerns, and I am inclined to raise my score by one point.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We are very grateful for your time and insightful feedback. In this final note, we summarize the contributions acknowledged by the reviewers and the additional experiments and revisions completed during rebuttal.

Summary of Contributions:

  • Practical computational savings. Reviewers D6UE, LtYw, and GTKw recognize that selecting modalities before expensive encoders execute enables real computational, latency, and energy savings.
  • Coherent integration of sequential fusion and adaptive acquisition. Reviewer LtYw describes the integration of SeMA and ARC as logical and technically meaningful. Reviewer RsLF appreciates the training components and analysis connecting prediction risk to order disagreement.
  • Comprehensive empirical evaluation. Reviewers D6UE, LtYw, and GTKw acknowledge the evaluation breadth across six datasets and GPU, CPU, and Android platforms. Reviewer LtYw particularly highlights savings from deliberately skipping available modalities.

Additional Experiments During Rebuttal to Address Concerns:

  1. Shared-backbone controller sweeps comparing ARC, EAMA, Joint-AFA, and GMS to address D6UE and GTKw’s accuracy–latency questions (Appendix H.1, Figure 12).
  2. Expanded component ablations across IEMOCAP, CMI, EAV, and MM-Fi, including separate ablations of prefix supervision and distillation, following GTKw’s suggestions (Appendix I, Tables 30–33).
  3. Acquisition-order ablation on IEMOCAP to address D6UE and GTKw’s compact-state questions. Explicit order yields no consistent improvement; we also clarify the additional assumptions required by the control bound (Appendix K.4, Table 38; Section 3.2).
  4. Cache-construction and controller-fitting profiles across all six datasets to answer LtYw’s offline-cost question (Appendix A.2, Table 6).
  5. Preliminary cache-subsampling experiments on IEMOCAP to investigate D6UE and LtYw’s scalability questions, while retaining budgeted cache construction as future work (Appendix A.1, Table 5).

We also clarified consistent idle-subtracted energy accounting and revised the figures and notation for readability.

Following discussion, D6UE and GTKw raised their ratings, RsLF confirmed that their main concerns were addressed, and LtYw welcomed the additional experiments and reiterated their strong support. We thank the reviewers and Area Chair for the constructive discussion and suggestions that helped strengthen our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.