- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This work introduces SEMARC, an framework for efficient multimodal machine learning inference which incrementally fuses available data and selects on when to acquire additional evidence or halt processing. Empirical evaluations across multiple benchmarks demonstrate that SEMARC reduces inference GFLOPs, latency, and energy consumption while maintaining or improving predictive accuracy.
Critical strengths
- Placing the decision to include new evidence before the feature extractors gives significant energy and latency savings
- The order invariant training of SEMA gives an empirical justification for ARC's compact state representation thereby keeping the action-value space tractable
- Great hardware and robustness empirical evaluation scope.
Critical weaknesses
- Table 2 shows that pairing existing controllers with SEMA (+ Joint-AFA: .75 F1; + EAMA: .74 F1) give similar task performance to ARC (0.77 F1) with lower latency in some cases (EAMA). The accuracy edge then is mostly from SEMA not ARC and this should be noted. It would be good to see pareto frontiers for all controllers on the shared backbone if not cost prohibitive.
- Section 3.2 seems to overstate the theoretical backing. Corrollary 3.2 assumes small error terms for the compact state, but they arent established by the prior proposition. Though this is acknowledged later in the appendix, the characterization in the main body of this section is most accurately a formal intuition rather than a guarantee.
- The server power measurements include idle draw but android phones measurements do not. This should be consistent across both either as latency adjusted power draw or unadjusted power draw.
- and are only ablated together. All ablations are limited to IEMOCAP, with some components falling within error bounds of each other. Could use a second dataset to verify more concretely eg. CMI.
Minor Issues/Formatting:
- ln 265, 301-302: the double struck 1 (indicator fn) is rendered incorrectly
- ln 229: Figure 4's placement inline is distracting and contains multiple text sizes with strange placements. Making the plot larger and text formatting more consistent may help
- figure 5 is very cluttered in its current state. referencing a table for the specific results and just showing the comparison without the magnified areas may be more clear.
- Figure 8,9,10 has a similar issue to figure 4's text sizing and placement. Difficult to interpret due to clutter.
- Figure 7 appears after figures 8,9,10
Questions for authors
- How are empty or single-sample (S, y) cells in handled?
4 comments
Response to Reviewer GTKw [2/3]
W3 — Why does server power include idle draw while Android excludes it, and can both use consistent adjusted or unadjusted accounting?
Thank you for highlighting this. CPU, GPU, and Android energy measurements all exclude idle consumption. We have updated the Table 22 caption and the measurement descriptions in Appendices D and F to make this common convention explicit across all platforms.
W4 — a) Can and be ablated separately? b) Given that all ablations use IEMOCAP and some component results fall within one another's error bounds, could a second dataset, e.g. CMI, verify the effects more concretely?
Thank you. Following your suggestion, we expanded the ablations beyond IEMOCAP to CMI, EAV, and MM-Fi, and added separate ablations of prefix CE and distillation. Results follow the manuscript’s ablation protocol.
a) We remove prefix CE (λ_{ds}), distillation (λ_{kd}), or both, retaining final retained-prefix CE throughout. Each backbone is retrained and ARC is refitted.
| λ_{ds} | λ_{kd} | IEMOCAP | CMI | EAV | MM-Fi |
|---|---|---|---|---|---|
| 1.00 | 0.50 | 0.67_{±0.02} | 0.56_{±0.01} | 0.69_{±0.02} | 0.95_{±0.04} |
| 0.00 | 0.50 | 0.67_{±0.05} | 0.53_{±0.02} | 0.66_{±0.02} | 0.93_{±0.06} |
| 1.00 | 0.00 | 0.67_{±0.04} | 0.56_{±0.01} | 0.69_{±0.04} | 0.91_{±0.11} |
| 0.00 | 0.00 | 0.65_{±0.01} | 0.54_{±0.01} | 0.66_{±0.04} | 0.89_{±0.10} |
Using values before display rounding, the combined loss improves mean F1 over final-prefix CE by 1.40–6.10 percentage points across the four datasets. Prefix CE benefits CMI, EAV, and MM-Fi; distillation provides its largest mean F1 benefit on MM-Fi (+4.20 points) and lowers acquisition cost at similar mean F1 on IEMOCAP and CMI.
b) Below are all non-loss variants; CMI loss results appear above. denotes post-extraction GFLOPs and MU denotes modality utilization.
| CMI variant | Macro-F1 ↑ | ↓ | MU ↓ |
|---|---|---|---|
| SeMARC (reference) | 0.56_{±0.01} | 0.43 | 0.75 |
| Fixed modality order | 0.55_{±0.01} | 0.53 | 0.91 |
| No dropout curriculum | 0.53_{±0.04} | 0.45 | 0.78 |
| Bottleneck tokens last | 0.56_{±0.01} | 0.45 | 0.78 |
| Learnable aggregation | 0.55_{±0.01} | 0.41 | 0.71 |
| Shapley + confidence exit | 0.56_{±0.01} | 0.42 | 0.74 |
| No Q-prior | 0.50_{±0.05} | 0.22 | 0.38 |
| No STOP (acquire all) | 0.57_{±0.01} | 0.58 | 1.00 |
| Confidence STOP | 0.56_{±0.01} | 0.43 | 0.75 |
| Masks-only residual input | 0.56_{±0.01} | 0.55 | 0.95 |
| Posterior-only residual input | 0.55_{±0.01} | 0.35 | 0.61 |
These results support the coupled design through prediction quality and acquisition cost. The combined prefix losses and randomized-order training improve mean F1 on all four datasets. On CMI, disabling STOP gains 1.00 F1 percentage point while increasing post-extraction GFLOPs from 0.43 to 0.58 and MU from 0.75 to 1.00. Static Shapley remains competitive on CMI, and other component effects are dataset-dependent. We interpret small mean differences cautiously; fold standard deviations alone do not establish statistical significance.
Section 4.2 and Appendix I (new Tables 30–33) provide the complete results and discussion. The updated Figure 7 summarizes all four datasets using equal-weight mean relative F1 changes and observed minimum–maximum ranges.
Question for authors
Q1 — How are empty or single-sample cells in handled?
In our implementation, if a class has no fitting samples, its corresponding gain entries in T_{S} are left at zero. With one sample, we use that sample’s marginal gain, computed from correctness averaged over the cached orders of each subset. With multiple samples, we average their marginal gains.
In practice, our fitted-cohort is stratified and contains multiple samples from every class. As described in Equation 6 and Appendix A, each (S,y) cell uses all fitting samples of class y, with predictions cached for every ordered modality prefix. Thus, neither empty nor single-sample cells occur in the current datasets.
Response to Reviewer GTKw [3/3]
Minor writing and formatting comments
W5 — Can the incorrectly rendered double-struck 1 (indicator function) at lines 265 and 301–302 be corrected?
We have corrected the indicator-function rendering. Please let us know if it still does not display correctly.
W6 — Can Figure 4’s distracting inline placement near line 229 and inconsistent text sizes and positions be improved, e.g., with a larger plot and consistent formatting?
We have corrected the placement and formatting of Figure 4.
W7 — Can Figure 5’s clutter be reduced by tabulating specific results and showing the comparison without magnified regions? W8 — Can Figures 8–10 be made easier to interpret by fixing inconsistent text sizing, placement, and clutter?
We have updated Figures 5, 8, 9, and 10 to improve readability. We appreciate your suggestions and welcome any further feedback on the revised figures.
W9 — Why does Figure 7 appear after Figures 8–10?
We have corrected the figure order.
Concerns addressed
Thank you for the rebuttal. I have raised my score to a 3.
Response to Reviewer GTKw [1/3]
Thank you for taking the time to review our work and for your constructive feedback to improve the paper. We appreciate your recognition of the significant energy and latency savings enabled by acquisition decisions before feature extraction and the empirical justification that SeMA's order-robust training provides for ARC's compact state representation, keeping the action-value space tractable. We also appreciate your praise for our hardware and robustness empirical evaluation breadth. Below, we address all your questions and concerns point by point.
Thank you for highlighting this distinction. Our paper already recognizes SeMA’s strong predictive foundation. Figure 6 (Section 4.1) compares SeMA with SeMA+ARC (SeMARC) on IEMOCAP. Across 0–40% modality missingness, adding ARC reduces post-extraction GFLOPs by 34.9–50.0% and total GFLOPs by 50.5–63.9%, with a decrease of at most 0.03 absolute macro-F1 (Appendix G). As stated in Section 4.1: "This supports our premise that sample-adaptive acquisition can substantially reduce computation while largely preserving the backbone’s predictive quality.”
To make this attribution explicit, we have added the following sentence to Section 4.2 (Component replacements): “These results support our coupled design: SeMA provides the shared predictive foundation, while ARC adds adaptive acquisition and stopping to improve the accuracy–cost trade-off.”
Additionally, we have also added new controller sweeps on the shared frozen SeMA backbone (Appendix H.1, Figure 12). The key observations are:1) Higher accuracy at comparable latency. Within a 400-ms mean-latency budget, SeMARC achieves 0.76 macro-F1 at 393.4 ms, compared with 0.73 at 388.5 ms for SeMA+EAMA—an improvement of approximately 3 percentage points. 2) Higher accuracy with lower latency. SeMARC achieves 0.76 macro-F1 at 355.5 ms, exceeding EAMA’s highest-F1 setting in this sweep (0.74 at 416.5 ms) with 14.6% lower latency. Every displayed Joint-AFA point is also dominated by a SeMARC point. 3) Overall SeMARC offers a favorable accuracy–latency trade-off. EAMA offers faster settings with lower macro-F1. GMS’s highest-F1 setting in this sweep exceeds SeMARC’s by approximately 1 percentage points (0.78 versus 0.77), but requires 2.55 times the latency (1327.5 versus 520.2 ms).
Quick-reference table: best macro-F1 under a latency budget
Entries are macro-F1, with equal-weight means across six datasets. Each row selects the highest F1 among the supplied settings meeting that mean-latency budget.
— means that no supplied setting meets the budget.
Thank you. As you note, our current appendix already clarifies that prediction-level order robustness motivates our compact-state design, while the control guarantee requires additional assumptions. Specifically, Appendix K.1 states: " These results concern average predictive distributions, not identical hidden states or AIS transition sufficiency."
To make this distinction equally explicit in the main text, we have updated the Implication paragraph in Section 3.2, following Corollary 3.2, to state: "The AIS control bound remains conditional on additional uniform reward and transition approximation assumptions beyond prediction-level order robustness."
Additionally, Appendix K.4 (Table 38) reports a new acquisition-order ablation. On IEMOCAP, appending a position-specific one-hot encoding of acquisition order to ARC’s compact state yields no consistent improvement in macro-F1 or return, supporting the omission of explicit order in this setting.